Add IBM bivariate-bicycle code discovery challenge - #514
Open
ifsheldon wants to merge 1 commit into
Open
Conversation
Port the pinned CSS bivariate-bicycle evolutionary evaluator as a CPU separated-evaluator challenge. Preserve the public lattice suite, shortlist ordering, trust-filtered BP-OSD score, and executable source behavior, while averaging three independent passes for leaderboard stability.\n\nInclude the Apache-2.0 vendored evaluator, uv-locked Python setup, hostile JSON boundary tests, and a meaningful seed-derived baseline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
bivariate-bicycle-code-discovery-qcode-discovery, a CPUseparated_evaluatorchallenge that ports the pinned original CSS fitness from IBM/Qiskit's qcode-discovery campaign.Challenge Contract
linux-arm64-cpuscore, descending(6,6)and(12,6)candidates.jsonEvaluation And Trust Boundary
The trusted evaluator reconstructs every code from JSON and never imports participant source. It preserves IBM's shortlist ordering, BP-OSD upper-bound estimates,
d/sqrt(n)credibility filter, and the pinned source behavior that disables OSD-CS and exact distance on the default path. Agentics adds a three-independent-pass mean and reports its spread.The statement explicitly warns that the score is not a distance proof, novelty result, or code-equivalence result. A later MILP campaign must use a distinct challenge handle.
Public And Private Data
All official lattices and configuration are committed public data, matching the original search setting. The challenge declares no private benchmark and requires no private ZIP overlay.
Validation Evidence
agentics challenge-creator checkpasses for this challenge and all 249 catalog entries.0.878290and 2/2 lattice coverage.174.903494; stddev:19.176067.4e828d0bc74066df9484e80f751a52674af7251fbyte-for-byte.Creator Notes
The pinned executable adapter uses a 1,000-trial initial estimate and three 500-trial refinements, despite nearby source prose describing different counts. The challenge documents and preserves the executable behavior.
Source provenance and implementation notes are recorded in the challenge README and issue #513.
Closes #513