[AIG-867] Cost/budget governance + spend attribution su OTel/multitenancy - #55
Conversation
Cost-governance layer that derives spend (tokens + estimated USD) from the
existing aistack.llm.chat usage signal, attributes it per
tenant/workspace/project/agent-pattern, and enforces optional budget caps
with a warn -> block kill-switch.
- src/governance/: types, price-table (fail-open USD estimate), CostAggregator
(append-only cost_ledger on shared SQLite), BudgetEnforcer (warn/block +
idempotent audit events), GovernanceService facade, optional SpanProcessor.
- Spawner wiring: pre-call checkBudget (may block) + post-call recordSpend at
the single llm.usage.* site (no double counting). No-op when disabled.
- REST /api/v1/governance/{status,budgets,spend} + aistack governance CLI.
- Config: governance schema/types; enabled=false and enforce.block=false by
default (opt-in, observe-only first).
- Public API exports + docs/GOVERNANCE.md.
- Unit + integration tests (token aggregation, tenant attribution,
warn->block, default-disabled no-op, fail-open pricing).
SECURITY DEFAULT: module is a no-op unless governance.enabled=true, and hard
blocking requires enforce.block=true (coherent with guardrails AIG-868).
|
Warning Review limit reached
More reviews will be available in 16 minutes and 48 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (22)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- governance/index.ts: import RecordSpendInput as a top-level type instead of an inline import() type expression. - enforcer.test.ts: capture audit() calls via vi.hoisted so the array is initialised before the hoisted vi.mock factory runs.
A re-export (export type { ... } from) does not bind the name within the
re-exporting module, so BudgetEvaluation.budget referencing CostBudget failed
to compile. Add a local 'import type { CostBudget }' alongside the re-export.
This was the single root cause of the Build / Type Check / docker-helm
integration failures.
1 participant
Summary
Adds a cost-governance layer on top of the existing OpenTelemetry usage signal (
src/observability/tracing.ts) and multi-tenancy (src/multitenancy/). It derives spend (tokens + estimated USD) from each LLM call, attributes it per tenant / workspace / project / agent-pattern, and enforces optional budget caps with a two-stage kill-switch (warn → block).The OTel span
aistack.llm.chatalready exposesllm.usage.*; the spawner already callsresourceService.recordApiCallright after it — that is the single point where token usage is known, so spend is recorded there (avoiding double counting from review-loop / consensus spans).Security default (opt-in, observe-only first)
config.governance.enableddefaults to false → the whole module is a no-op (mirrors guardrails AIG-868, audit AIG-635, tracing — all off by default).config.governance.enforce.blockdefaults to false → even when enabled with a budget defined, an over-budget call is never blocked; it is only accounted and acost.budget.warn/cost.budget.blockaudit event is emitted.enforce.block: true. A mis-configured budget can never take agents offline unless an operator opts in.Acceptance criteria
config.governance.budgets[]with hierarchical scope{ tenant, workspace, project, agentPattern }(glob on agent type); most-specific match wins; no budget = unlimited.CostAggregatorpersists one append-onlycost_ledgerrow per call on the shared SQLite store; tenant/workspace read from agent metadata (AIG-649); reports grouped by tenant/workspace/project/agent.BudgetEnforcer: warn atwarnThresholdPercent(default 80%) → block at 100%; emitscost.budget.warn/cost.budget.blockvia the hash-chained audit log (idempotent per budget+window); throwsCostBudgetExceededErrorbefore the LLM call only whenenforce.block.GET /api/v1/governance/{status,budgets,spend?dimension=tenant|workspace|project|agent&from&to}+aistack governance {status,budgets,report}CLI (JSON + table).tests/unit/governance/{price-table,aggregator,enforcer}.test.tsandtests/integration/governance.test.ts(token aggregation, tenant attribution, warn→block, default-disabled no-op, fail-open pricing, block opt-in).Design decisions (defaults documented in docs/GOVERNANCE.md)
(provider, model)glob (USD per 1M tokens). Unknown model → USD 0 (fail-open) but tokens still aggregated + warn logged. Pricing is never a reason to block.MultitenancyContext) + project + agent-pattern; resolution by specificity.cost_ledgeronconfig.memory.path, inline idempotent schema (mirrorsTenantService); windowday|week|month|total(defaultmonth).audit/tenant.llm.usage.*point). An optionalGovernanceSpanProcessoris provided as a secondary path but not wired (no in-process SpanProcessor is cabled today); it must be mutually exclusive with the in-code path to avoid double counting.Known limitations (see docs)
chatResponse.usagearen't tracked.cost_ledgeris append-only; retention/rollup is a follow-up (indexed by(tenant, workspace, ts)).Files (22)
New:
src/governance/{index,types,price-table,aggregator,enforcer,service,span-adapter}.ts,src/web/routes/governance.ts,src/cli/commands/governance.ts,tests/unit/governance/{price-table,aggregator,enforcer}.test.ts,tests/integration/governance.test.ts,docs/GOVERNANCE.md.Modified:
src/agents/spawner.ts,src/types.ts,src/utils/config.ts,src/index.ts,src/web/routes/index.ts,src/web/server.ts,src/cli/commands/index.ts,src/cli/index.ts.CI note
This branch does not touch
.github/workflows/ci.yml(changed in parallel by another branch). The new tests run under the existingvitestunit + integration suites — no new CI step is required. Validation (typecheck / test / lint) runs in CI on this PR (the dev environment has no Node).Manual smoke-test (relevant to AIG-866 sibling)
governance.enabled: trueinaistack.config.jsonwith a tenant-scopedlimitUsdbudget andenforce.block: false.aistack governance report --dimension tenantshows tokens + USD.cost.budget.warnthencost.budget.blockinaistack audit export(with audit enabled), and that agents are not blocked.enforce.block: true; verify the next over-budget call throwsCostBudgetExceededErrorbefore the LLM call.Closes AIG-867