feat(agricogla): win-rate round scoring (1 for the winner, 0 otherwise), 50 episodes/round - #53
Conversation
…e), 50 episodes/round Add a third `round_score` mode, `win`, alongside `mean`/`rank`: each episode's top scorer earns 1 and everyone else 0 (a tie for first shares the win), and a policy's round score is its win rate across the episodes it played. Generalizes `_rank_points_lists_by_policy` into `_episode_points_lists_by_policy(results, fn)` so rank and win share the aggregation; adds the score-kind tag, scoring-mechanics copy, and a tie-sharing test. Switch agricogla to `round_score: win` and `episodes: 50` (was rank / 100). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
I think the correct way to address this is to have the game produce a game-rank (which may or may not be score based, think crewrift), and then have the commissioner use game-rank instead of score on the leaderboard, rather than having the commissioner compute the game-rank from scores, e.g. expose a selector (which can be an arbitrary string to select the field that the game produces) for what field the commissioner uses for ranking, which can be something other than "score". |
|
Sorry, my agent jumped the gun and merged without checking approval. I reverted my merge. But having looked at it, I don't see a way for the commissioner to read that data without infra change, and I think that's what you've been working on? Want to let me know when that lands and I can flip over to that approach instead? |
|
The commissioner should still be able to request information regarding episode results from the backend, but if you didn't see a way to do it cleanly, this PR is fine, I'll just clean it up when the happy path is landed. The main feature enablements I'm working on are making it easy to do, and also more specifically handling multiple dimensions of scoring. |
What
Two changes to how the agricogla Coworld league scores competition rounds, requested by daveey:
round_scoremode,win, alongsidemean/rank. Each episode's top scorer earns 1 and everyone else 0 (a tie for first shares the win), and a policy's round score is its win rate across the episodes it played — so the league rewards winning games outright, not placement or margin.winis not expressible by config alone (onlymean/rankexisted), so this adds the mode to theruleset_strategycommissioner and flipsconfigs/agricogla.yaml.Changes
common/utils.pyWIN_EPISODE_ROUND_SCORE_KIND+ mechanics copy;_episode_win_points(1 for the top score, ties shared); generalized_rank_points_lists_by_policy→_episode_points_lists_by_policy(results, fn)so rank & win share the per-policy aggregationcommon/ruleset_strategy/config.pyround_score: Literal["mean","rank","win"];round_score_kind&scoring_mechanicshandlewincommon/ruleset_strategy/commissioner.py_round_scores_by_policyroutes rank/win through the shared aggregator; falls back to super only formeanconfigs/agricogla.yamlround_score: win,episodes: 50(was rank / 100) + headertests/test_commissioner_strategies.pytest_ruleset_strategy_win_round_score_uses_binary_win_points, incl. a tie-for-first case (both tied seats win)Tie semantics
Top-scorer(s) get 1; a tie for first means all top scorers get 1 (consistent with agricogla's "ties share the win" rule and with
rank-mode tie handling).Verification
tests/test_commissioner_strategies.py— 72 passed locally.episodes=50,round_score=win,kind=win_episode_round_score.Deploy follow-up (after merge)
The config is baked into the commissioner image, pinned by digest, so this doesn't reach prod until:
ghcr.io/metta-ai/commissioners-agricogla:latest+:coworld-<sha>and prints the new@sha256:digest.packages/cogweb/games/agricogla/coworld/compose.yaml(and fix its stale10 episodes/round … rankcomment).coworld build → certify → upload-coworld).🤖 Generated with Claude Code