Skip to content
This repository was archived by the owner on Jun 25, 2026. It is now read-only.

feat(agricogla): win-rate round scoring (1 for the winner, 0 otherwise), 50 episodes/round - #53

Merged
daveey merged 1 commit into
mainfrom
daveey/agricogla-win-scoring
Jun 24, 2026
Merged

feat(agricogla): win-rate round scoring (1 for the winner, 0 otherwise), 50 episodes/round#53
daveey merged 1 commit into
mainfrom
daveey/agricogla-win-scoring

Conversation

@daveey

@daveey daveey commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

What

Two changes to how the agricogla Coworld league scores competition rounds, requested by daveey:

  • Scoring → win rate. Add a third round_score mode, win, alongside mean/rank. Each episode's top scorer earns 1 and everyone else 0 (a tie for first shares the win), and a policy's round score is its win rate across the episodes it played — so the league rewards winning games outright, not placement or margin.
  • 50 episodes/round (was 100).

win is not expressible by config alone (only mean/rank existed), so this adds the mode to the ruleset_strategy commissioner and flips configs/agricogla.yaml.

Changes

File Change
common/utils.py New WIN_EPISODE_ROUND_SCORE_KIND + mechanics copy; _episode_win_points (1 for the top score, ties shared); generalized _rank_points_lists_by_policy_episode_points_lists_by_policy(results, fn) so rank & win share the per-policy aggregation
common/ruleset_strategy/config.py round_score: Literal["mean","rank","win"]; round_score_kind & scoring_mechanics handle win
common/ruleset_strategy/commissioner.py _round_scores_by_policy routes rank/win through the shared aggregator; falls back to super only for mean
configs/agricogla.yaml round_score: win, episodes: 50 (was rank / 100) + header
tests/test_commissioner_strategies.py New test_ruleset_strategy_win_round_score_uses_binary_win_points, incl. a tie-for-first case (both tied seats win)

Tie semantics

Top-scorer(s) get 1; a tie for first means all top scorers get 1 (consistent with agricogla's "ties share the win" rule and with rank-mode tie handling).

Verification

  • tests/test_commissioner_strategies.py72 passed locally.
  • Config parses: episodes=50, round_score=win, kind=win_episode_round_score.

Deploy follow-up (after merge)

The config is baked into the commissioner image, pinned by digest, so this doesn't reach prod until:

  1. Merge → CI publishes ghcr.io/metta-ai/commissioners-agricogla:latest + :coworld-<sha> and prints the new @sha256: digest.
  2. In metta, bump the digest in packages/cogweb/games/agricogla/coworld/compose.yaml (and fix its stale 10 episodes/round … rank comment).
  3. Rebuild + re-upload the agricogla coworld (coworld build → certify → upload-coworld).

🤖 Generated with Claude Code

…e), 50 episodes/round

Add a third `round_score` mode, `win`, alongside `mean`/`rank`: each episode's
top scorer earns 1 and everyone else 0 (a tie for first shares the win), and a
policy's round score is its win rate across the episodes it played. Generalizes
`_rank_points_lists_by_policy` into `_episode_points_lists_by_policy(results, fn)`
so rank and win share the aggregation; adds the score-kind tag, scoring-mechanics
copy, and a tie-sharing test.

Switch agricogla to `round_score: win` and `episodes: 50` (was rank / 100).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@KyleHerndon

Copy link
Copy Markdown
Contributor

I think the correct way to address this is to have the game produce a game-rank (which may or may not be score based, think crewrift), and then have the commissioner use game-rank instead of score on the leaderboard, rather than having the commissioner compute the game-rank from scores, e.g. expose a selector (which can be an arbitrary string to select the field that the game produces) for what field the commissioner uses for ranking, which can be something other than "score".

@daveey
daveey merged commit dfe5978 into main Jun 24, 2026
8 checks passed
@daveey
daveey deleted the daveey/agricogla-win-scoring branch June 24, 2026 21:35
daveey added a commit that referenced this pull request Jun 24, 2026
…otherwise), 50 episodes/round (#53)"

This reverts commit dfe5978.
@daveey

daveey commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Sorry, my agent jumped the gun and merged without checking approval. I reverted my merge. But having looked at it, I don't see a way for the commissioner to read that data without infra change, and I think that's what you've been working on? Want to let me know when that lands and I can flip over to that approach instead?

daveey added a commit that referenced this pull request Jun 24, 2026
…nner, 0 otherwise), 50 episodes/round (#53)""

This reverts commit b63e437.
@KyleHerndon

Copy link
Copy Markdown
Contributor

The commissioner should still be able to request information regarding episode results from the backend, but if you didn't see a way to do it cleanly, this PR is fine, I'll just clean it up when the happy path is landed.

The main feature enablements I'm working on are making it easy to do, and also more specifically handling multiple dimensions of scoring.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants