Motivation
AtomPack datasets should be able to carry dataset-level metadata directly, without relying only on repository sidecars such as external JSON manifests.
Useful metadata includes:
- source and export provenance
- split policy
- schema/version information
- label definitions
- checksums
- citations or links
- arbitrary JSON-like metadata
Requested capability
Add a small API for storing and reading flexible dataset-level metadata.
A minimal interface could be something like:
db.set_metadata({...})
metadata = db.get_metadata()or support named metadata blobs:
db.set_metadata("split_policy", {...})
db.set_metadata("README", "...")The stored metadata should support at least JSON-like values and strings. It should be possible to read this metadata without scanning all records.
Why this matters
This would make published AtomPack datasets more self-describing and reduce reliance on repo-specific sidecar conventions. It is separate from grouped structure records, which should be handled in a different PR.
Motivation
AtomPack datasets should be able to carry dataset-level metadata directly, without relying only on repository sidecars such as external JSON manifests.
Useful metadata includes:
Requested capability
Add a small API for storing and reading flexible dataset-level metadata.
A minimal interface could be something like:
or support named metadata blobs:
The stored metadata should support at least JSON-like values and strings. It should be possible to read this metadata without scanning all records.
Why this matters
This would make published AtomPack datasets more self-describing and reduce reliance on repo-specific sidecar conventions. It is separate from grouped structure records, which should be handled in a different PR.