Agora Scorers

Public source for Agora's official scorer runtime images.

This repo owns the public, deterministic execution substrate images for Agora's compiled scoring programs.

It owns:

• official runtime image source
• scorer-side runtime manifest helpers
• scorer regression tests
• the standalone public replay receiver CLI
• GHCR publication workflow

It does not own:

• poster authoring UX
• challenge taxonomy
• scoring method, metric, or aggregator vocabulary
• capability discovery
• the python-v1 helper SDK for compiled programs
• runtime profile selection in Agora
• worker orchestration
• proof publication
• on-chain settlement

Those remain in the main Agora repo.

Third-Party Replay Receiver

This repo also publishes the standalone receiver for Agora score proofs:

npx @moleculeagora/agora-replay --proof <cid> --format json

The receiver consumes public optimistic-private proof bundles, verifies the public proof shape and optional proof hash, and reports the privacy boundary without cloning the main Agora monorepo. Non-challenged submissions do not publish answer replay inputs; full public re-score verification is only available from challenge-by-reveal evidence after a solver voluntarily reveals a committed answer.

The output includes:

• runtime_manifest_schema_sha256
• proof_bundle_schema_sha256
• supported_program_abi_versions
• status
• reason
• replay_available
• replay_scope
• proof_hash_matches
• score_proof_facts
• mismatches

Runtime Contract

Every scorer runtime in this repo now speaks the same V2 mounted contract:

• /input/runtime-manifest.json
• /input/evaluation/<role>/<filename>
• /input/submission/<role>/<filename>
• /input/scoring_assets/<role>/<filename>
• /output/score.json

The runtime manifest declares:

• runtime_profile
• artifact_contract
• evaluation_bindings
• artifacts
• scoring_assets
• objective
• final_score_key
• policies

Official images do not own metric logic, relation templates, challenge taxonomy, or the python-v1 helper SDK. They read compiler-produced scoring assets and execute them. Variation belongs in staged scoring assets, not in image identity.

Capability enumeration belongs to the main Agora repo. Agents and verifiers should discover available methods, metrics, aggregators, and authoring shapes through GET /api/authoring/capabilities, not by reading this repo.

Official Runtime

Official images:

Container	Runtime profile id	What it does
agora-scorer-compiled	official_compiled_runtime	Executes one staged compiled program plus any staged scoring config/bundles against the mounted runtime manifest
agora-scorer-rdkit	rdkit_python_runtime	Executes the same Python-v1 mounted contract with RDKit plus the pinned OpenSOL XGBoost/RDKit-2D prediction assets available for deterministic molecule artifact checks

The runtime profile owns the deterministic child-process environment. The current official profile pins LANG, LC_ALL, PYTHONHASHSEED, SOURCE_DATE_EPOCH, and TZ; replay and publication prove that environment by its canonical sorted-key JSON SHA-256.

The image is the L5 runtime substrate. It does not branch on scoring method, metric, or aggregator names. The main Agora compiler stages one Python-v1 program per invocation. That program can implement one scoring primitive or a composition program that calls staged component logic. The image still sees one program asset and writes one /output/score.json.

agora-scorer-rdkit is a scoped dependency envelope, not a broad scientific Python image. It pins:

• base image: python:3.11.9-slim@sha256:8fb099199b9f2d70342674bd9dbccd3ed03a258f26bbd1d556822c6dfc60c317
• rdkit==2025.3.1
• numpy==2.4.4
• Pillow==12.2.0
• xgboost-cpu==3.0.5 (imports as xgboost==3.0.5)
• pandas==2.3.3
• scipy==1.17.1
• joblib==1.5.3
• transitive date/time pins required by pandas

The image also includes the public OpenSOL model Models/xgboost_rdkit_2d_clustering_model.json from sutropub/OpenSOL@89e6d30d0ce84aaf9ee2bd9c93619d3c2a4a95c4, pinned at SHA-256 bb0e4c542c8172b717239f62be3d538bf1ede214a385af055411c02f1d928da0. No apt packages, raw CCDC/CSD datasets, notebooks, docking engines, alternate OpenSOL models, DNN assets, or scoring-time package installs are added.

Repo Layout

common/                     shared scorer runtime helpers
agora-scorer-compiled/      official compiled runtime image
agora-scorer-rdkit/         official RDKit dependency-envelope runtime image
bin/                        agora-replay executable entry point
src/                        standalone replay receiver implementation
test/                       replay receiver fixtures and tests
docs/                       scorer-side extension notes
schema/                     vendored Agora main canonical runtime/proof schemas
scripts/                    local test helpers and container guards

Shared runtime helpers:

• common/runtime_manifest.py
  • V2 runtime manifest parsing
  • role-bound artifact resolution
  • scoring-asset resolution
• common/runtime_test_support.py
  • local fixture helpers for official runtime tests

Official runtime files:

• agora-scorer-compiled/entrypoint.py
  • validates the official runtime profile
  • discovers the staged program scoring asset
  • discovers the staged python_v1_runtime_sdk document asset first in PYTHONPATH
  • sets runtime-profile determinism and ABI environment variables before executing the staged program
• agora-scorer-compiled/test_score.py
  • scorer regression tests for the official compiled runtime
• agora-scorer-rdkit/Dockerfile
  • installs only the hash-locked RDKit/OpenSOL dependency envelope
  • reuses the compiled runtime entrypoint and staged Python-v1 SDK path
• agora-scorer-rdkit/requirements.txt
  • records the exact wheel hashes accepted for linux/amd64 and linux/arm64

Code-Only Policy

Official runtime images must stay public and code-only except for explicitly allowlisted public runtime assets such as the pinned OpenSOL model in agora-scorer-rdkit/OPENSOL-PROVENANCE.md. This repo must not ship:

• hidden evaluation labels
• private reference outputs
• benchmark datasets
• harness payloads
• large embedded assets outside the allowlist

Those belong in mounted evaluation artifacts or scoring assets, not in the image. The guard in scripts/check-scorer-containers.mjs enforces that rule.

Published Image

The official runtime publishes to ghcr.io/moleculeprotocol/.

Convenience tags:

docker pull ghcr.io/moleculeprotocol/agora-scorer-compiled:latest
docker pull ghcr.io/moleculeprotocol/agora-scorer-compiled:sha-<git-commit>
docker pull ghcr.io/moleculeprotocol/agora-scorer-rdkit:latest
docker pull ghcr.io/moleculeprotocol/agora-scorer-rdkit:sha-<git-commit>

Agora itself must bind the runtime profile to an immutable digest, not a floating tag.

Local Development

Run all scorer regression tests:

bash scripts/run-scorer-tests.sh

Run replay receiver tests and boundary checks:

npm ci
npm test
npm run check:replay-boundary
npm run check:release-provenance
npm run check:rdkit-image

Verify that the vendored runtime manifest and proof bundle schemas match Agora main:

AGORA_MAIN_RUNTIME_MANIFEST_SCHEMA_PATH=/path/to/Agora/packages/common/src/schemas/scorer-runtime-manifest.canonical.schema.json \
AGORA_MAIN_PROOF_BUNDLE_SCHEMA_PATH=/path/to/Agora/packages/common/src/schemas/proof-bundle.canonical.schema.json \
  npm run check:agora-main-schema-sync

When using GitHub instead of a local checkout, set AGORA_MAIN_GITHUB_TOKEN or GITHUB_TOKEN to a token with read access to moleculeprotocol/Agora.

Release notes for the npm receiver package live in RELEASING.md.

Run specific tests directly:

python3 agora-scorer-compiled/test_score.py
python3 common/test_runtime_manifest.py

Canonical Discovery

The main Agora repo owns product and scoring vocabulary. Use these public surfaces instead of copying capability lists into this repo:

• Methods, metrics, aggregators, and authoring shapes: GET /api/authoring/capabilities
• Runtime manifest schema: /.well-known/scorer-runtime-manifest.schema.json
• Proof bundle schema: packages/common/src/schemas/proof-bundle.canonical.schema.json in the main Agora repo
• Scorer result schema: /.well-known/scorer-result-schema.schema.json
• Product scoring model: docs/product/scoring-layer-invariants.md in the main Agora repo
• Pattern catalog: docs/product/scoring-pattern-catalog.md in the main Agora repo

CI And Publication

The publish workflow:

• installs the standalone replay receiver package
• runs replay receiver tests
• checks that the replay receiver does not import Agora workspace packages
• runs scorer regression tests
• rejects retired scorer vocabulary in active public-repo surfaces
• checks that the official runtime image stays code-only
• builds and runs the RDKit image smoke fixture
• verifies the vendored canonical runtime manifest and proof bundle schema hashes
• verifies that the vendored runtime manifest and proof bundle schema bytes match Agora main
• builds multi-arch images for linux/amd64 and linux/arm64
• emits max-mode BuildKit provenance for the pushed image
• publishes a GitHub/Sigstore provenance attestation for the pushed image digest
• publishes :latest and :sha-<git-commit> tags to GHCR
• emits runtime_manifest_schema_sha256, proof_bundle_schema_sha256, determinism_env_sha256, and supported_program_abi_versions in official-runtime-release.json
• emits verifier-oriented provenance metadata in official-runtime-release.json: subject name, subject digest, source repository, source ref, source commit, signer workflow, and attestation URL

The Docker build context is the repo root so the shared runtime helpers in common/ are available to the image.

Verify an official image's source provenance with the GitHub CLI:

gh attestation verify \
  oci://ghcr.io/moleculeprotocol/<image-package>@sha256:<digest> \
  --repo moleculeprotocol/agora-scorers \
  --signer-workflow moleculeprotocol/agora-scorers/.github/workflows/publish.yml \
  --source-digest <commit>

Use the digest, provenance.subject_name, provenance.source_commit, and provenance.signer_workflow fields from official-runtime-release.json. Agora main digest rotation must reject a release whose provenance does not match those fields.

Related Links

• Agora main repo
• Runtime profile registry
• Poster/scorer V2 contract
• Agora protocol
• Scoring extension guide