Add repository Save(), DeleteCache(), FetchFiles(), and embedded repository (hub.NewEmbed) support - #54
Merged
Merged
Conversation
• func (r *Repo) Save(dirPath string, linkOnly bool) error • func (r *Repo) SaveCtx(ctx context.Context, dirPath string, linkOnly bool) error • Resolves/downloads all files in the repository into the Hugging Face cache structure and copies or hard-links (os.Link) them into dirPath. • Always overwrites existing files in dirPath. • Fails with an error if r.IsLocal() is true. ### 2. DeleteCache() in delete.go • func (r *Repo) DeleteCache() error • Deletes the repository's cache directory (e.g., ~/.cache/huggingface/hub/models--...). • Documented that calling any Repo method after DeleteCache() will require files to be re-downloaded from Hugging Face Hub. • Fails with an error if r.IsLocal() is true. ### 3. Command-line Flags in main.go Added three flags to hubinfo: • -save=<target_dir>: Saves the repository files to target_dir. • -link_only: Uses hard links instead of copying files when -save is specified. • -delete_cache: Deletes the downloaded cache directory for the repository (executed after -save if both are specified).
Improved documentation.
• hub.NewEmbed(fsys fs.FS, subDir string) *Repo & Repo.WithEmbedFS(fsys, subDir): Allows backing a *hub.Repo with any fs.FS (such as embed.FS). • Repo.IsEmbed() bool: Checks if the repository is in embedded-filesystem mode (alongside IsLocal()). • Repo.Open(name) & Repo.ReadFile(name): Stream reading methods that work across all repository modes (embedded, local, and remote) directly from memory/FS without disk extraction. • Fallback for DownloadFile(s): If legacy functions requesting disk file paths are called on an embedded repository, the files are extracted to a temporary directory under os. TempDir()/go-huggingface-embed/<ID>/ and the local disk path is returned. • Detailed docstrings added to NewEmbed explaining semantics, side-effects, and stream reading usage. ### 2. Downstream Libraries Updated • tokenizers (tokenizers.go & hftokenizer.go): Updated GetConfig() and hftokenizer.New() to use repo.ReadFile() directly. • models/transformer (model.go): Updated model configuration loader to use repo.ReadFile(). • models/safetensors (header.go): Updated parseHeader to read safetensors headers directly via repo.Open(). ### 3. Documentation & Changelog • Added "Embedded Models (//go:embed)" sub-section to README.md with a complete usage example. • Updated ## Unreleased section of CHANGELOG.md describing all new features.
…pos). Added comments to avoid DownloadFile(s), and instead use Open(), ReadFile() or FetchFile(s) instead.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
This PR introduces comprehensive support for creating local and embedded copies of HuggingFace repositories, stream-reading files directly from memory or disk, and managing repository caches.
Key Additions
Repository Saving & Cache Management (
hubpackage):Repo.Save(dirPath string, linkOnly bool) error/SaveCtx: Exports/resolves all repository files to a local directory (dirPath) suitable forhub.NewLocal(), offline deployment, or containerization. Hard links (os.Link) can optionally be used withlinkOnly = true.Repo.DeleteCache() error: Deletes the downloaded cache folder (~/.cache/huggingface/hub/models--...) for a repository.Embedded Repositories (
hub.NewEmbed):hub.NewEmbed(fsys fs.FS, subDir string) *Repo/Repo.WithEmbedFS: Enables backing a repository with a Gofs.FS(e.g.//go:embed). No network calls are made in this mode.Repo.IsEmbed()/Repo.IsLocal(): Helper methods to check repository mode.Stream Reading & Pre-fetching (
hubpackage):Repo.Open(name string) (fs.File, error)/Repo.ReadFile(name string) ([]byte, error): Opens or reads repository files directly across all repository modes (embedded, local, and remote) without requiring disk file extraction in embedded mode.Repo.FetchFiles(repoFiles ...string) error/Repo.FetchFile: Pre-downloads and caches repository files upfront without returning OS disk path strings.CLI Tool Updates (
cmd/hubinfo):-localflag: treats argument as a local path usinghub.NewLocal().-save=<dir>flag: saves repository to target directory.-link_onlyflag: creates hard links instead of copying.-delete_cacheflag: deletes the repository cache after saving.Downstream Package Updates:
tokenizers,models/transformer,models/safetensors, andexamples/kalmgemma3to useRepo.Open/Repo.ReadFile/Repo.FetchFiles.Documentation & Changelog:
README.mdwith new sub-sections for Local Copies and Embedded Models.docs/CHANGELOG.md.