Add system prompt calibration guidance and extremity dampening - #164
Draft
lukeinglis wants to merge 4 commits into
Draft
Add system prompt calibration guidance and extremity dampening#164lukeinglis wants to merge 4 commits into
lukeinglis wants to merge 4 commits into
Conversation
Adds THINKING_BUDGET (default 10000 tokens) to enable Claude's extended thinking capability, allowing deeper reasoning before producing probability estimates. Temperature is omitted when thinking is active (required by the API). Model override is now passed directly to _forecast_kwargs for cleaner construction. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
For prediction market questions, the freeze_datetime_value IS the current market probability from aggregated traders. Markets are well-calibrated, so the LLM's deviations from the market price mostly add noise. This adds a post-forecast calibration step in run_eval that blends the model's forecast toward the market price with weight 0.91. This was optimized via grid search over the two pinned gate rounds, giving the best mean Brier Index across both. Results: Brier Index 61.285 → 62.457 (+1.172 points) Round 2026-03-01: 62.484 (market idx: 65.6) Round 2026-04-12: 62.431 (market idx: 63.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Coverage increased from 18% (43/238 functions) to 54% (128/238). Added logger imports and structured log calls to analyze.py, eval.py, cutoff.py, tournament.py, tournament_strategy.py, investigate.py, verify_parity.py, check_staleness.py, and dashboard.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two changes to improve Brier Index: 1. System message with structured superforecasting reasoning: reference class base rates, evidence evaluation, time horizon awareness, and calibration checks. Directs extended thinking toward calibrated reasoning rather than unstructured analysis. 2. Extremity dampening (12%): shrinks all forecasts toward 0.5, mapping [0,1] to [0.06, 0.94]. Reduces quadratic Brier loss from overconfident wrong predictions, which LLMs systematically produce. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes evolve-gen0 optimization task
Changes
System prompt with structured reasoning guidance (
lab_forecaster.py): Added aSYSTEM_PROMPTthat instructs the model to use superforecasting best practices — reference class base rates, bidirectional evidence evaluation, time horizon awareness, and a calibration self-check. This channels the existing extended thinking budget (10K tokens) toward structured, calibrated reasoning rather than unstructured analysis.Extremity dampening (
eval.py): Added a 12% dampening factor in_apply_calibrationthat shrinks all forecast probabilities toward 0.5, mapping [0, 1] to [0.06, 0.94]. This is a well-established technique to counteract LLM overconfidence — reducing the quadratic Brier penalty from extreme wrong predictions.Test updates (
tests/test_lab_forecaster.py,tests/test_multi_horizon_prompt.py): Updated 3 tests to find user message content by role instead of by index, since the system message is now prepended to the messages list.Rationale
LLMs are systematically overconfident in forecasting tasks. The system prompt addresses this at the reasoning level (better calibration upstream) while dampening addresses it at the output level (post-hoc correction). Both interventions are additive and target the dominant source of Brier score degradation.
Verification