Add tools/synthdata: RePoE-driven synthetic userstore generation

[aurokin]

Author Note: The goal was to create generated synthetic data using RePoE for the best chance to cover all current scenarios now and going forward. Added a quirks engine to manually inject known "inactive" items

What this is

Commit generators, not data. A Python generator under tools/synthdata/ that builds arbitrarily large, deterministic userstore databases from two account-free inputs:

• RePoE (repoe_data.py) — same repoe-fork.github.io source the app fetches, plus mods.min.json for legal mod pools, tier ranges, and display templates. Cached and keyed by upstream version.txt, so game updates invalidate naturally.
• A statistics-only profile (profile.py) — tab-type mix, items-per-tab quantiles, league shares by rank, buyout density. Counts and histograms only, so a profile is committable where its source DB never is. fresh-account.json (committed) is a small new-player account shape.

Scale is a parameter: 100k items ≈ 43s; ~2M extrapolates to ≈14 min — so the webview-scale spike's 1M target is one flag.

Legacy wire idiosyncrasies

Following up on the colour-string/legacy-items point: --coverage emits three instruments —

• a Coverage tab sweeping essentially every optional field in item.h (influences, corruption states, Standard-only survivals like crucible/scourge/logbooks, PoE2 parse fields);
• a Quirks tab driven by quirks-registry.json — 28 curated entries, each with provenance and an evidence status (verified-in-data / documented / reported / community). Sources include the real 4,658-item public-stash feed embedded in the 2016-10-15 web.archive.org snapshot of the developer docs (re-verified locally), the GGG changelog, and third-party parser workarounds (PoB, exile-diary, Procurement). Covers leading-zero-stripped tab colours ("ff", "bf00"), <<set:…>> prefixes, explicit-false booleans, pre-3.1 sockets without sColour, DV sockets, flags: [], string-tuple properties, stackSizeText, literal-null arrays, unmodeled keys (talismanTier), empty/homoglyph/Remove-only tab names, and more;
• quirks.py, a scanner that reports keys in any real userstore that src/poe/types/ doesn't declare — so an old account becomes a concrete catalog delta instead of a silent parse gap. (Your 43k-item DB scans fully modeled: 0 unmodeled keys.)

One open question found along the way: item.h models prophecyText as an array of strings, the developer docs say ?string, and no corpus I could find contains a real prophecy item payload to settle it. The registry follows the app model and records the question.

Validation

• tests/tst_synthdata.cpp opens a generated DB through the app's own UserStore/repo layer and requires every stash and character payload to parse into the typed poe:: structs. Skips unless ACQ_SYNTH_DATA_DIR/ACQ_SYNTH_ACCOUNT are set, so the regular suite is unaffected.
• Dogfooded the real GUI against a 100k-item / 8,610-tab synthetic store: zero crashes, tooltips render for every quirk shape, tab-colour padding verified live, mod/name/rarity searches instant, item- and tab-level buyouts round-trip to SQLite. RSS was 639 MB after load / 865 MB after heavy use — a possibly useful native baseline for the 1M webview spike.

Notes

• Purely additive: no src/ changes.
• Design rationale, rejected alternatives, and known limits are in docs/redesign/topics/synthetic-data.md.
• Nothing derived from a real account is committed; a veteran-account profile can be added later if you want your account's shape represented.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VMh7nZndnFDht8mCFV7EAV

[gerwaric]

You're welcome to address the issues below however you feel best--and that includes deciding that they should not be addressed, or addressed with doc-only changes. -- @gerwaric

---

Thanks for putting this together. The generator/registry approach looks potentially very useful for redesign experiments, especially the decision to keep generated databases out of the repository and make scale configurable.

I reviewed this as executable tooling and test infrastructure rather than as a documentation-only change: the PR adds roughly 2,000 lines across the Python generator/scanner, a C++ test, and test build configuration. I have several questions about the validation contract, long-term maintenance, and a few claims in the documentation.

1. What is intended to exercise the generator automatically?

tst_synthdata skips unless ACQ_SYNTH_DATA_DIR and ACQ_SYNTH_ACCOUNT already point to a generated database. That means an ordinary build and CTest run only establish that the C++ test compiles; they do not run generate.py, validate a generated database, or detect changes in the RePoE input format.

As a result, regressions in areas such as these would not be caught by the checked-in suite:

• Python/runtime errors in the generator
• changes to the RePoE JSON shape
• invalid or incompatible generated SQLite schemas
• malformed coverage or quirk entries
• loss of determinism
• profile-extraction errors
• generator output that no longer parses

Could we establish at least one repeatable validation path? Some possibilities:

• a small hermetic Python test using minimal checked-in RePoE fixtures;
• a CMake/CTest target that generates a very small database before running tst_synthdata;
• a separate, explicitly manual integration target with documented commands and expected assertions;
• or CI generation using a pinned RePoE snapshot, if network and runtime costs are acceptable.

I do not think the full 100k dataset needs to run in CI, but right now the main implementation and its test can drift independently without any normal test failure.

2. Could the scope of tst_synthdata be described more narrowly or expanded?

The test currently verifies that:

• at least one stash row exists;
• all fetched stash and character rows deserialize through their repos;
• some top-level stash items exist;
• if any generated items parse as rare, at least one has explicit mods.

That is a useful parse smoke test, but it does not presently verify several guarantees discussed in the PR and design:

• that each coverage axis and quirk entry was emitted;
• that the intended value or unusual wire shape survived parsing;
• tab parent/child and folder relationships;
• stash indexes and item placement;
• buyout loading or round trips;
• unknown-key/raw-JSON preservation;
• deterministic output for identical inputs;
• requested dataset scale;
• model, search, tooltip, or worker behavior.

The design says the database is parsed through “the same path ItemsManagerWorker::ParseCachedItems uses.” More precisely, the test exercises the repository/deserialization layer also used by that path; it does not invoke ItemsManagerWorker::ParseCachedItems itself.

Could the document state that distinction? Alternatively, could the test assert the registry/coverage contract more directly—for example, by giving each probe a stable synthetic identifier and confirming that every expected probe appears and retains the intended representation?

Without named per-probe assertions, a coverage entry could silently stop being emitted or be overwritten and the test would still pass as long as the resulting tab remained parseable.

3. The GUI-testing statements appear inconsistent

The PR description says the real GUI was dogfooded against a 100k-item / 8,610-tab store, including tooltip rendering, searches, buyout round trips, tab-colour handling, and memory observations.

The design’s Known Limits section says:

Loading is proven at the datastore/parse layer; a logged-in GUI session over a synthetic store has not been exercised.

Is the intended distinction that the GUI was launched against the synthetic database, but no authenticated refresh/session was performed? If so, could the document say exactly that and record the GUI exercise described in the PR body? As written, the design makes the validation sound substantially narrower than the PR description does.

It would also be useful to distinguish measured observations from guarantees. For example, the RSS and search observations describe one manual run, whereas the checked-in test only establishes repo-level parsing.

4. The privacy wording is internally inconsistent

The design currently says:

Nothing derived from a real account is committed

and describes the two inputs as “both account-free.” It then explains that the profile is extracted from a real userstore and that fresh-account.json is committed.

The profile may be sufficiently aggregated and acceptable to commit, but it is still derived from a real account. It also contains a fairly specific combination of exact or near-exact attributes, including:

• tab count and type distribution;
• item-count quantiles by tab type;
• character count and level quantiles;
• buyout density;
• public-tab density;
• folder usage;
• stack-fill distribution.

Could this be reworded to make the actual guarantee narrower—for example, that no raw item records, IDs, names, league names, or directly identifying account data are committed?

I would also like to understand the rule for deciding that a profile is safe to commit. “Counts and histograms only” is helpful, but it does not automatically guarantee anonymity, particularly for a very small account whose histogram values may closely reproduce its exact shape. Do we want minimum bucket sizes, coarser rounding, suppression of small populations, explicit owner consent, or simply a documented project policy that these account-shape statistics are considered acceptable?

This matters more if veteran profiles from additional contributors are expected later.

5. Can the generated schema avoid duplicating UserStore’s schema and version?

generate.py contains a handwritten copy of the current characters, stashes, and buyout schemas, then sets:

PRAGMA user_version = 3

This creates a maintenance hazard. If the application schema changes, the generator can continue creating a database with the old shape while stamping it as the new/current version. UserStore will then see a current version and will not run migrations.

Could the generator create the empty database through UserStore, invoke a small application-owned schema helper, or otherwise derive the schema from the canonical datastore implementation?

If duplicating the schema is unavoidable, I think it needs an automated schema-equivalence check. For example, generate a database, create another through UserStore, and compare relevant sqlite_master and PRAGMA table_info results. Merely checking whether existing rows deserialize will not necessarily catch missing indexes, constraints, or newly added columns.

6. The determinism contract needs a more complete definition

The design says generation is deterministic per:

(seed, profile, RePoE version)

The generator docstring includes scale as well, and output also varies with:

• --items;
• --coverage;
• --now;
• --leagues;
• the current quirks registry;
• the generator implementation/revision;
• the exact contents and ordering of the downloaded RePoE files.

The upstream version string is only sufficient as a content identity if every file published under that version is immutable. The cache also fetches version.txt on every invocation, so reproducing an older run requires retaining the corresponding cached directory or otherwise obtaining the exact old inputs.

Could the documentation define reproducibility in terms of all effective inputs? It may also be helpful for the generated database or a sidecar manifest to record:

• generator revision;
• seed;
• requested item target;
• profile content hash;
• RePoE version and preferably input hashes;
• coverage enabled/disabled;
• league list;
• effective timestamp;
• quirks-registry hash.

That would make a benchmark dataset meaningfully reproducible after the repository and upstream RePoE data have moved forward.

A small test that runs generation twice and compares outputs—or compares canonicalized logical content if SQLite bytes are not stable—would enforce the claim.

7. Is --items intentionally a minimum rather than an exact size?

The main generation loop stops after a complete tab causes emitted to reach or exceed --items. Character items are then added, followed by coverage and quirk items when enabled.

The PR description’s 100k example reports 100,224 generated items, which confirms that the parameter is not an exact output count.

That behavior seems reasonable for shape-preserving scale generation, but could the CLI and design describe --items as an approximate minimum target? Otherwise benchmark sweeps may assume that --items 1000000 produces datasets directly comparable by exact cardinality.

It would also help to clarify whether reported “generated N items” includes:

• stash items;
• character equipment;
• character inventory;
• socketed items;
• coverage probes;
• quirk probes.

The C++ validator currently counts only direct items in stash payloads, so its count is not necessarily the same population as the generator’s emitted value.

8. How should the quirks registry’s provenance be maintained?

The evidence-status distinction is helpful, but some sources are not durable or independently reviewable, such as:

• “research agent dump”;
• “research-agent-verified”;
• Discord reports;
• unspecified community datasets or relayed captures.

For a registry intended to become lasting compatibility evidence, could sources use stable references wherever possible—archived URLs and dates, repository issue numbers, exact upstream files/commits, or small sanitized fixtures?

For evidence that cannot be committed or linked, it may be worth recording:

• who inspected it;
• when it was inspected;
• the exact query or observation;
• whether the source can be rechecked;
• whether the entry is merely a plausible stress case rather than a verified wire shape.

That would preserve the useful distinction between “the application should tolerate this” and “GGG is known to have emitted this.”

9. A few claims could be calibrated more precisely

The document says the coverage tab sweeps “essentially every optional field in item.h” and that the structure “matches the live API.” Given the acknowledged approximations—synthetic uniques, simplified dedicated-tab contents, independent hybrid-mod rolls, parse-only PoE2 coverage, and curated unresolved legacy shapes—those statements may be broader than necessary.

Would wording such as the following be more maintainable?

• “covers the currently selected optional fields enumerated by coverage_axes()”;
• “matches the application’s currently modeled payload shapes for the exercised structures”;
• “uses RePoE-derived mod candidates and ranges, subject to the documented generation approximations.”

That would keep the tool’s value clear without making completeness or fidelity claims that will be difficult to preserve as item.h and RePoE evolve.

Suggested disposition

I do not see a production-code regression here, since the generator is isolated from src/. I also would not block on perfect game-data realism; the documented approximations are reasonable for parse and scale testing.

The areas I think should be resolved or explicitly tracked before merge are:

1. establishing how the generator will actually be tested rather than skipped in normal runs;
2. addressing schema/version drift from the duplicated SQLite DDL;
3. reconciling the GUI-validation statements;
4. correcting and defining the privacy/anonymity claims.

The remaining questions could potentially be handled through documentation changes or clearly recorded follow-up work, depending on how this tooling is expected to be used during the redesign.

---

[aurokin]

Thanks for the thorough review — this was exactly the level of scrutiny I hoped this would get. Everything below landed as a second commit so it maps cleanly onto your numbers. Point-by-point:

1. What exercises the generator automatically?
An ordinary ctest run now generates and validates a store, offline, with no env vars:

• tools/synthdata/fixtures/ is a checked-in, deterministic slice of real RePoE data (~260 KB: a few bases per emittable item class, their spawn-legal mods, the referenced stat translations), extracted by make_fixtures.py. generate.py --repoe-dir fixtures runs fully offline from it.
• A CTest fixture (synthdata_generate, registered when a Python interpreter exists) runs selftest.py: generates a small store twice from identical inputs into the build dir, fails on any row-level difference, and hands the result to tst_synthdata. Without Python the fixture isn't registered and tst_synthdata skips with a message; CMake warns at configure time.
• So the regressions you listed — generator runtime errors, RePoE shape changes, invalid schema, malformed coverage/quirk entries, loss of determinism, output that no longer parses — now all fail a normal ctest run.

On snapshot staleness: .github/workflows/synthdata-freshness.yml is a separate low-cost weekly check — it fetches only version.txt, and only on a version change re-runs the extractor to classify the drift ("upstream bumped, subset unchanged" vs "the shapes we rely on moved"). It never fails a build: warning annotation + step summary, plus one idempotent issue while stale. (Cron only fires on the default branch, so until merge it also runs as a non-blocking pull_request check on tools/synthdata/** changes.)

2. Scope of tst_synthdata
Both suggestions taken. The generator now writes a sidecar <db>.manifest.json naming every coverage axis and registry quirk with the exact serialized bytes of the item it emitted, and the test asserts each probe's item id appears exactly once store-wide and that its emitted bytes survive verbatim in its stash payload (a single streamed pass, so multi-million-item stores stay cheap to validate). Because the assertion is byte-level, an emitted null or a deliberately deleted key that gets silently normalized is a named failure ("probe 'crlf-in-mod-text': emitted bytes not found"), not a still-parseable tab. The design doc now also states the layer distinction precisely: the test exercises the repository/deserialization layer that ParseCachedItems also uses; it does not invoke the worker itself.

3. GUI-testing statements
Your reading was right — the GUI was launched and used extensively against the synthetic store (real OAuth login, ParseCachedItems over synthetic data, ~45 min of browsing/searches/tooltips/buyout round trips), but no authenticated refresh ever touched the network. The Known Limits section now records exactly that as one manual measured run, and states that the checked-in suite asserts only repo-layer parsing, probe presence, and schema equivalence.

4. Privacy wording
Reworded to the narrow guarantee you suggested: no raw item records, item/stash ids, item or tab names, league names, or directly identifying account data are ever committed. A profile is derived from a real account, so committing one is now stated as a consent decision, not an anonymity proof: profiles are committed only with the account owner's explicit consent (fresh-account.json is my own account, committed by me), and the doc says outright that aggregation is not a mathematical anonymity guarantee for small accounts — which is why consent is the rule. Future veteran profiles follow the same terms.

5. Schema duplication
The duplication itself is hard to remove (the canonical DDL lives in C++), so I took your fallback and made it automated: tst_synthdata now creates a fresh store through UserStore itself and compares whitespace-normalized DDL for every object, PRAGMA table_info for every table, and user_version against the generated database. Any app schema change the generator misses — including a missing index or constraint, or the stamped-version-on-old-shape hazard you called out — fails the suite.

6. Determinism contract
The doc now defines the full tuple (seed, item target, profile bytes, league list, --now timestamp, RePoE input bytes, quirks-registry bytes), and the manifest's repro block records all of it with sha256 content hashes for the profile, each RePoE input file, and the registry — so a benchmark dataset stays identifiable after the repo and upstream RePoE move on. The generate-twice-and-compare test you suggested is exactly what selftest.py enforces on every ctest run (canonicalized logical content, since SQLite bytes aren't guaranteed stable).

7. --items semantics
Documented as an approximate minimum for stash items in both the CLI help and the doc. The generator now reports populations separately — generated N items (a stash, b character, c probe) — with counts defined as top-level items only (socketed sub-items excluded), and the doc notes the C++ validator counts stash-payload items, a different population from the generator's total.

8. Registry provenance
Restructured under explicit rules, documented in the registry header: source now holds durable references only (archive.org URLs with dates, issue numbers, forum thread ids, files in this repo — e.g. the 2016 corpus is now pinned to its exact snapshot URL). Evidence that can't be linked gets a structured inspected record: who, when, the exact observation, and whether it's recheckable. Entries probing tolerance with shapes we hold no captured bytes for are flagged stress_case: true — preserving your distinction between "the application should tolerate this" and "GGG is known to have emitted this".

9. Claim calibration
Adopted your wording nearly verbatim: "covers the optional fields enumerated by coverage_axes()", "matches the application's currently modeled payload shapes for the exercised structures", "uses RePoE-derived mod candidates and ranges, subject to the documented approximations". The RSS/timing observations are now tagged measured/estimated per the redesign README's lane convention.

Validation of the changes themselves: full ctest suite green (37/37, including the new fixture chain), and the 100k external lane regenerated and passed all three tst_synthdata checks via the env-var override.