An open benchmark for one hard question: how much does a specific resilience project lower a city's borrowing cost?
A city can choose to spend on wildfire hardening, flood control, a seismic retrofit, a water-system upgrade. Everyone agrees these projects reduce physical risk. What nobody can yet answer in a standardized, auditable way is:
Which project buys the city the biggest reduction in future fiscal risk — and therefore the biggest borrowing benefit (lower spread, lower interest cost, more debt capacity)?
The pieces to answer it exist, but they live in separate silos and stop short of each other (see the diagram). The missing middle — turning a project's risk reduction into a citywide fiscal impact — is done today in bespoke, one-off, hard-to-check ways. There is no shared yardstick.
This repo is a first step toward that yardstick — and we're deliberately starting with a benchmark, not a standard.
The diagram above walks left to right:
- Resilience project — e.g. fire hardening, water upgrade.
- Project-level risk reduction — less expected physical damage. (Jupiter, Arcadis, and open tools like CLIMADA are good here.)
- 🔴 Citywide fiscal impact — lower expected revenue loss, emergency cost, service disruption. This is the missing standardized translation.
- Credit assessment — the rating agencies (Moody's, S&P, Fitch) already do this at the city level.
- Borrowing benefit — lower spread / interest cost, more debt capacity.
The hard, unowned step is 2 → 3. Steps 4 → 5 are relatively familiar market machinery once step 3 exists.
It's tempting to jump straight to writing a standard — "here is the official method every city shall use to translate a project into a fiscal impact." We think that's premature, and here's the honest reasoning:
-
A standard presumes we already know the right method. We don't — and neither does anyone else. That translation is exactly the unsolved, contested part. Writing a standard now would be legislating an answer to an open question. It invites years of committee debate and gets little adoption.
-
A benchmark inverts that. It doesn't dictate how you compute the answer. It defines the task and how answers are scored. Anyone — a city, a consultancy, a rating analyst, an AI model — can produce an estimate however they like, with whatever data and tools they trust, and see how it holds up.
-
The standard should be discovered, not decreed. Run the benchmark for a while and the methods that consistently produce defensible, well-calibrated, auditable estimates will become the emerging best practice. That is a far stronger foundation for an eventual standard (or spec) than a whiteboard committee. The benchmark is how the community earns its way to a standard.
So: benchmark now → shared conventions emerge → standard/spec later. Each stage is a precursor to the next, and we don't skip ahead.
Be upfront about it. Unlike a chess engine, there is no single measured "true" number for how much one project moved a city's bond spread. You can never fully isolate one flood project's marginal effect from everything else happening to a city's finances. So we can't score answers against a clean, objective ground truth the way you'd check a math problem.
That is not a reason to give up — it's a reason to grade the reasoning, not just the number. The right tool for a partly-subjective judgment is a rubric: an explicit, written checklist of what a defensible estimate must do — which fiscal channels it accounted for, whether its assumptions are stated, whether its uncertainty is honest, whether an expert would sign off on the logic.
And crucially, the rubric is open and evolving. It lives in this repo. When a municipal advisor, a bond analyst, or a public-works planner says "you're missing X" or "that assumption is wrong for coastal cities," that becomes a proposed change to the rubric. Over time the rubric accumulates the field's hard-won judgment.
An evolving, open rubric is how a benchmark captures collective intelligence. No single person has to be right. The rubric becomes the shared memory of what "a good answer" means — refined by everyone who contributes. That is the real asset here: not the code, but the accumulated, auditable judgment.
(This mirrors how medical-AI benchmarks grade against doctor-written rubrics rather than a single "correct" answer — the rubric, and its provenance, is the thing that matters.)
One benchmark task is a small, concrete decision:
Setting: City X must choose among 3–5 candidate resilience projects.
Given: - each project's cost and what hazard it addresses
- the city's fiscal profile (revenues, reserves, exposure)
Question: Rank the projects by their expected reduction in the city's
future borrowing cost — and show your work.
An answer must include, per project:
- estimated fiscal channels affected (avoided damage, emergency
spending, revenue volatility, reserve draws, recovery costs...)
- the estimated effect on borrowing cost
- the assumptions behind it (hazard model, effectiveness, aid,
insurance, time horizon)
- an honest uncertainty range
An answer is then scored against the open rubric — how complete, transparent, calibrated, and defensible is the reasoning? Two cities using two different AI models and two different datasets can both be scored on the same footing.
A benchmark is only as good as its golden answers — the worked, expert-reviewed reference cases everyone is scored against. The plan is to start narrow and real rather than broad and synthetic:
- one city
- one hazard (e.g. wildfire or flood)
- one project class
- a handful of candidate projects
- reviewed by people who actually live in municipal finance and public works
San Francisco is the first target: an SF public-works / public-finance case makes a strong first golden set precisely because it's concrete, defensible, and reviewable by domain experts. (Details of the first pilot are being worked out with municipal-finance practitioners.)
Everything after that is contribution: another city, another hazard, another expert improving the rubric.
- Municipal finance and public-works teams deciding which resilience project to fund.
- Municipal advisors, underwriters, and analysts who price this risk.
- Researchers and toolmakers (hazard modeling, climate-fiscal risk) who have half the chain and want a shared target for the other half.
The hazard/physical-risk and benefit-cost layers are mature and mostly open — e.g. CLIMADA, OS-Climate physrisk, NIST IN-CORE, FEMA's BCA toolkit. The credit layer is owned by the rating agencies. The muni-bond–climate link is well documented in the academic literature (physical risk measurably moves municipal bond yields). The unowned middle — a shared, auditable way to connect a specific project to a citywide fiscal impact — is what this benchmark is for.
Too early for a formal process. If you work in municipal finance, public works, resilience modeling, or credit analysis and this resonates (or you think it's wrong) — open an issue. Early disagreement from real practitioners is exactly what shapes the rubric.
Where this is headed: Next steps →
MIT (see LICENSE).