Skip to content

Add IBM bivariate-bicycle code discovery challenge - #514

Open
ifsheldon wants to merge 1 commit into
mainfrom
codex/add-bivariate-bicycle-code-discovery
Open

Add IBM bivariate-bicycle code discovery challenge#514
ifsheldon wants to merge 1 commit into
mainfrom
codex/add-bivariate-bicycle-code-discovery

Conversation

@ifsheldon

Copy link
Copy Markdown
Contributor

Summary

Add bivariate-bicycle-code-discovery-qcode-discovery, a CPU separated_evaluator challenge that ports the pinned original CSS fitness from IBM/Qiskit's qcode-discovery campaign.

Challenge Contract

  • Target: linux-arm64-cpu
  • Ranking metric: score, descending
  • Validation: IBM stage-1 k-only screen on (6,6) and (12,6)
  • Official evaluation: the published eight-lattice suite
  • Submission limits: 2 validation, 1 official
  • Private assets: none
  • Solution interface: ordered polynomial-pair candidates in bounded candidates.json

Evaluation And Trust Boundary

The trusted evaluator reconstructs every code from JSON and never imports participant source. It preserves IBM's shortlist ordering, BP-OSD upper-bound estimates, d/sqrt(n) credibility filter, and the pinned source behavior that disables OSD-CS and exact distance on the default path. Agentics adds a three-independent-pass mean and reports its spread.

The statement explicitly warns that the score is not a distance proof, novelty result, or code-equivalence result. A later MILP campaign must use a distinct challenge handle.

Public And Private Data

All official lattices and configuration are committed public data, matching the original search setting. The challenge declares no private benchmark and requires no private ZIP overlay.

Validation Evidence

  • agentics challenge-creator check passes for this challenge and all 249 catalog entries.
  • Ten adapter contract tests pass.
  • The uv lock resolves with managed Python 3.12 on Linux ARM64.
  • The IBM seed baseline passes public validation with score 0.878290 and 2/2 lattice coverage.
  • A full three-process official smoke completed in 11m02s.
  • Official smoke mean score: 174.903494; stddev: 19.176067.
  • Vendored evaluator files and seed baseline match upstream commit 4e828d0bc74066df9484e80f751a52674af7251f byte-for-byte.
  • Apache-2.0 license and upstream attribution are included.

Creator Notes

The pinned executable adapter uses a 1,000-trial initial estimate and three 500-trial refinements, despite nearby source prose describing different counts. The challenge documents and preserves the executable behavior.

Source provenance and implementation notes are recorded in the challenge README and issue #513.

Closes #513

Port the pinned CSS bivariate-bicycle evolutionary evaluator as a CPU separated-evaluator challenge. Preserve the public lattice suite, shortlist ordering, trust-filtered BP-OSD score, and executable source behavior, while averaging three independent passes for leaderboard stability.\n\nInclude the Apache-2.0 vendored evaluator, uv-locked Python setup, hostile JSON boundary tests, and a meaningful seed-derived baseline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port IBM qcode-discovery original CSS fitness

1 participant