Skip to content

Harden LSUNClass key cache: store under dataset root and load with weights_only - #9628

Open
fjankovi wants to merge 1 commit into
pytorch:mainfrom
fjankovi:harden-lsun-key-cache
Open

Harden LSUNClass key cache: store under dataset root and load with weights_only#9628
fjankovi wants to merge 1 commit into
pytorch:mainfrom
fjankovi:harden-lsun-key-cache

Conversation

@fjankovi

Copy link
Copy Markdown

Summary

LSUNClass caches the enumerated LMDB keys in a pickle file whose path is derived from root but resolved against the current working directory, and reads it back with pickle.load:

cache_file = "_cache_" + "".join(c for c in root if c in string.ascii_letters)
if os.path.isfile(cache_file):
    self.keys = pickle.load(open(cache_file, "rb"))
else:
    ...
    pickle.dump(self.keys, open(cache_file, "wb"))

This has two issues:

  1. CWD-relative, predictable name. The cache is looked up in the process CWD, not under root, and the name is a lossy transform of root (letters only). A file dropped into the directory the user happens to run from — a shared scratch dir, a cloned example repo, a CI workspace — is loaded on the next LSUN()/LSUNClass() call. Because it's unpickled with no integrity check, that is arbitrary code execution on load (CWE-502). The user never asked to load anything; the cache is written and re-read as a side effect.
  2. Unrestricted pickle even in the benign case.

Fix

This mirrors what the sibling datasets in this package already do (ImageNet, MNIST, PhotoTour all use torch.load(..., weights_only=True)):

  • Store the cache next to the LMDB store as root/_cache_keys.pt — a trusted, per-class location, with no CWD exposure and no cross-root name collisions.
  • Load with torch.load(..., weights_only=True), so a cache file can never execute code on load.
  • Skip caching gracefully if root is read-only.

The cache only memoizes keys that are fully re-derivable from the trusted LMDB store, so pre-existing CWD caches are simply not found under the new path and are regenerated safely on first use — no migration required. torch.load round-trips the raw bytes LMDB keys, including non-UTF-8 keys.

Test plan

  • Built an LMDB store containing a non-UTF-8 key; confirmed the keys enumerate, a _cache_keys.pt is written under root, and a second construction reads it back with the non-UTF-8 key intact.
  • Confirmed a pre-existing _cache_<letters> file in the CWD is now ignored (the loader looks under root), so a poisoned CWD cache is no longer read.

…ights_only

`LSUNClass.__init__` derived its key-cache filename from the ASCII letters of
`root` and read/wrote it in the current working directory via `pickle.load`
/ `pickle.dump`:

    cache_file = "_cache_" + "".join(c for c in root if c in string.ascii_letters)
    if os.path.isfile(cache_file):
        self.keys = pickle.load(open(cache_file, "rb"))

Two problems:

- The path is resolved against the process CWD, not `root`, and the name is a
  predictable, lossy transform of `root`. Anyone able to write a file into the
  directory the user runs from (a shared scratch dir, a cloned repo, a CI
  workspace) can pre-plant `_cache_<letters>` and have it deserialized on the
  next `LSUN()` call. `pickle.load` on an attacker-controlled file is arbitrary
  code execution (CWE-502); there is no integrity check on the path.
- Even setting that aside, the cache is a plain pickle with no restriction.

Fix, matching the pattern already used by ImageNet/MNIST/PhotoTour in this
package:

- Store the cache next to the LMDB store as `root/_cache_keys.pt` (a trusted,
  per-class location; no CWD, no cross-root name collisions).
- Load with `torch.load(..., weights_only=True)` so a cache file can never
  execute code on load.
- Degrade gracefully when `root` is read-only (skip caching, re-enumerate).

The cache only memoizes keys that are fully re-derivable from the trusted LMDB
store, so old CWD caches are simply not found under the new path and are
regenerated safely on first use — no migration needed. `torch.load` round-trips
the raw `bytes` LMDB keys, including non-UTF-8 keys.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pytorch-bot

pytorch-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9628

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla

meta-cla Bot commented Aug 25, 2026

Copy link
Copy Markdown

Hi @fjankovi!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant