Skip to content

Add reference spec framework for noise-free benchmark outputs (#924) - #941

Open
grageragarces wants to merge 4 commits into
munich-quantum-toolkit:mainfrom
grageragarces:main
Open

Add reference spec framework for noise-free benchmark outputs (#924)#941
grageragarces wants to merge 4 commits into
munich-quantum-toolkit:mainfrom
grageragarces:main

Conversation

@grageragarces

Copy link
Copy Markdown

Summary

Implements the reference specification framework proposed in #924.
Each benchmark can now expose a create_reference function that returns
a compact, JSON-serialisable description of the ideal output distribution —
no dense state vectors required.

Distribution kinds

Kind Example What is stored
sparse GHZ, BV, Grover {bitstring: prob} table
uniform W state predicate + support size (P(x) = 1/size if predicate holds)
simulate Random circuit advisory max_qubits — caller runs statevector sim
none QAOA, VQE reason string — no compact form exists

An optional objective field carries the semantic answer (hidden string,
marked states, phase, …) separately from the distribution.

Implemented benchmarks

ghz, wstate, bv, dj, grover, qpeexact, randomcircuit

Public API

from mqt.bench.benchmarks import get_reference_spec, has_reference

spec = get_reference_spec("grover", 5)
print(spec.to_dict())   # JSON-ready dict

…lkit#924)

Introduces a compact, JSON-serialisable reference specification for
MQT Bench circuits so downstream users can validate noisy experiments
against an ideal ground truth without needing a dense state vector.

New files:
- benchmarks/_reference.py  — ReferenceSpec + SparseReference /
  UniformReference / SimulateReference / NoneReference / ObjectiveSpec
- tests/test_reference.py   — 52 tests (all passing)

Framework changes:
- _registry.py: parallel _REFERENCE_REGISTRY + register_reference decorator
- __init__.py:  get_reference_spec() public API + lazy-load-aware has_reference()

Reference specs implemented for: ghz (sparse), wstate (uniform),
bv (sparse + objective), dj (sparse + objective), grover (sparse +
marked-states objective), qpeexact (sparse phase readout),
randomcircuit (simulate).

Closes munich-quantum-toolkit#924
@grageragarces

Copy link
Copy Markdown
Author

Hey @burgholzer! 👋

I've opened a draft PR implementing the reference spec framework proposed in #924.
Here's what I have added:

  • A new _reference.py module with the type hierarchy we sketched: SparseReference, UniformReference, SimulateReference, NoneReference, ObjectiveSpec, and a top-level ReferenceSpec with a to_dict() method for JSON serialisation.
  • A register_reference decorator that mirrors register_benchmark — each benchmark file can optionally define a companion create_reference(num_qubits, ...) function.
  • Public API: get_reference_spec("ghz", 5) and has_reference("grover").
  • Reference specs for 7 benchmarks as worked examples of each kind: ghz (sparse), wstate (uniform), bv, dj, qpeexact (all sparse/deterministic), grover (sparse + marked-states objective), randomcircuit (simulate).

The remaining benchmarks need someone who knows the generator internals to fill in the right answer. Some thoughts on the obvious ones:

  • QFT / QFT-entangled: is the reference the QFT of a known input state, or do these go under simulate?
  • AE / QPE-inexact: inexact phase estimation has a distribution over neighbouring bitstrings; happy to implement if you can confirm the target phase used.
  • Adder circuits: these are deterministic given fixed inputs; what operand values does the generator use?
  • QAOA / VQE: I've left these as NoneReference since the output depends on optimised parameters. Is that the right call, or would you prefer a SimulateReference so at least small instances can be validated?
  • Graph state / random circuits beyond ~30q: SimulateReference with an advisory max_qubits field seems right. Alt we could just have a NoneReference.

Any guidance on the above would let me fill in the rest.

Also very happy to adjust the schema if something doesn't work quite right!
Perhaps having the references in a subfolder might make it cleaner?

@grageragarces

Copy link
Copy Markdown
Author

will give it a day for the tests to fail before trying to fix them :)

@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

- Annotated GHZ entries dict as dict[str, float] to stop ty inferring
  LiteralString keys (invariant generic mismatch)
- Replaced # type: ignore[misc] workarounds with proper isinstance
  narrowing guards before all .entries accesses in tests
- Added test_has_reference_nonexistent_benchmark to cover the
  except ValueError branch in has_reference, closing the 2-line
  coverage gap reported by Codecov
- Replace en dashes with hyphens in ObjectiveSpec docstring (RUF002)
- Rename `dynamic` to `_dynamic` in bv.create_reference to mark it as
  intentionally unused (ARG001)
- Add missing docstrings to all public test functions (D103)
- Remove default argument from test_wstate_reference_probability (PT028)
- Replace float == comparisons with math.isclose / abs checks (RUF069)
- Add isinstance narrowing guards before .entries accesses so ty can
  narrow the ReferenceKind union (replaces # type: ignore[misc])
- Add test_has_reference_nonexistent_benchmark to cover the
  except ValueError branch in has_reference (closes Codecov gap)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant