Skip to content

feat(infra): enable Claude/Anthropic in the gateway with agent-default alias (ADR 09) #131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JustAGhosT merged 1 commit into from
Jul 3, 2026
Merged

feat(infra): enable Claude/Anthropic in the gateway with agent-default alias (ADR 09) #131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/deploy-environment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,8 @@ on:
required: false
MOONSHOT_API_KEY:
required: false
ANTHROPIC_API_KEY:
required: false

env:
TF_BACKEND_RG: ${{ secrets.TF_BACKEND_RG }}
Expand Down Expand Up @@ -208,6 +210,7 @@ jobs:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY || '' }}
FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY || '' }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY || '' }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY || '' }}
run: |
set -euo pipefail

Expand All @@ -233,6 +236,9 @@ jobs:
add_provider "openrouter" "OPENROUTER_API_KEY" "${OPENROUTER_API_KEY}"
add_provider "fireworks" "FIREWORKS_API_KEY" "${FIREWORKS_API_KEY}"
add_provider "moonshot" "MOONSHOT_API_KEY" "${MOONSHOT_API_KEY}"
# ADR 09: env_name is LITELLM_ANTHROPIC_API_KEY (the var the LiteLLM
# model_list reads), sourced from the ANTHROPIC_API_KEY secret.
add_provider "anthropic" "LITELLM_ANTHROPIC_API_KEY" "${ANTHROPIC_API_KEY}"

if [ "${providers}" != "{}" ]; then
echo "TF_VAR_extra_providers=${providers}" >> "$GITHUB_ENV"
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ jobs:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY || '' }}
FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY || '' }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY || '' }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY || '' }}
run: |
set -euo pipefail

Expand All @@ -193,6 +194,9 @@ jobs:
add_provider "openrouter" "OPENROUTER_API_KEY" "${OPENROUTER_API_KEY}"
add_provider "fireworks" "FIREWORKS_API_KEY" "${FIREWORKS_API_KEY}"
add_provider "moonshot" "MOONSHOT_API_KEY" "${MOONSHOT_API_KEY}"
# ADR 09: env_name is LITELLM_ANTHROPIC_API_KEY (the var the LiteLLM
# model_list reads), sourced from the ANTHROPIC_API_KEY secret.
add_provider "anthropic" "LITELLM_ANTHROPIC_API_KEY" "${ANTHROPIC_API_KEY}"

if [ "${providers}" != "{}" ]; then
echo "TF_VAR_extra_providers=${providers}" >> "$GITHUB_ENV"
Expand Down Expand Up @@ -363,3 +367,4 @@ jobs:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
84 changes: 58 additions & 26 deletions docs/architecture/09-claude-code-via-sluice.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,26 @@
# Routing Claude Code CLI Through Sluice

Status: Open — awaiting decision
Date: 2026-04-30
Status: Accepted — 2026-07-03
Date: 2026-04-30 (analysis) · 2026-07-03 (decision)

## Decision

CI and other **non-interactive** agents route Claude Code (and any Anthropic-SDK
caller) through Sluice on API-key billing, requesting the **`agent-default`**
alias. Sluice resolves `agent-default` to a concrete Claude model (currently
`claude-haiku-4-5`, degrading to `claude-sonnet-4-6` then Azure `premium`), so
the model can be re-routed centrally without touching any CI workflow.
**Interactive** Claude Code use backed by a Claude Max subscription stays on
**direct-Anthropic** auth — the flat-fee Max economics dominate per-token API
rates for human-driven sessions (see the analysis and recommendation below,
which this decision adopts unchanged).

Implemented via: an `anthropic` entry in the LiteLLM `model_list`
(`infra/modules/sluice_aca/main.tf`), the `LITELLM_ANTHROPIC_API_KEY` secret
wired through the standard `extra_providers` path, provider activation in the
deploy workflows (`ANTHROPIC_API_KEY` GitHub secret → `TF_VAR_extra_providers`),
and a dedicated `mystira-ci` virtual key allow-listing `agent-default`
(`scripts/keys.yaml`).

## Question

Expand Down Expand Up @@ -109,30 +128,43 @@ spending subscription quota through a self-hosted gateway.
on subscription-first (Claude Code in particular). Routing through API
could miss features for weeks.

## If we proceed — what setup looks like

### Operator (one-time)

1. Generate an Anthropic Console API key (pay-per-token billing on the
org's Console account)
2. Store it in KV: `az keyvault secret set --vault-name pvc-prod-sluice-kv
--name anthropic-api-key --value <key>`
3. Add a `model_list` entry to the LiteLLM YAML in
`infra/modules/sluice_aca/main.tf` — e.g.
```yaml
- model_name: claude-sonnet-4-6
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.environ/LITELLM_ANTHROPIC_API_KEY
model_info:
input_cost_per_token: 0.000003
output_cost_per_token: 0.000015
```
4. Mount `LITELLM_ANTHROPIC_API_KEY` from the `anthropic-api-key` KV
secret on the LiteLLM container
5. Update each consumer's `keys.yaml` entry to include the new Claude
model in `models: [...]`
6. `python scripts/manage_keys.py apply` — re-applies model allowlists
## Setup (implemented)

Steps 2–5 of the original analysis are now **in code** — the `model_list`
entries, the `LITELLM_ANTHROPIC_API_KEY` env wiring (via the standard
`extra_providers` path), provider activation in the deploy workflows, and the
`mystira-ci` allow-list all ship in this repo. The remaining operator work is
placing the key and running one deploy.

### Operator (one-time — the single manual step to go live)

1. Generate an Anthropic Console API key (pay-per-token billing on the org's
Console account).
2. Make the key available to Terraform. Two equivalent options:
- **Preferred — GitHub environment secret** (matches how every other
provider key is supplied): set `ANTHROPIC_API_KEY` as a `prod` environment
secret. The `Configure optional model providers` step folds it into
`TF_VAR_extra_providers` as
`{ anthropic = { api_key = <key>, env_name = "LITELLM_ANTHROPIC_API_KEY" } }`,
and Terraform then **creates** the KV secret `provider-anthropic-key` and
mounts it as the `LITELLM_ANTHROPIC_API_KEY` env var on the LiteLLM
container.
- **Direct KV** (matches the runbook convention below):
```
az keyvault secret set --vault-name pvc-prod-sluice-kv \
--name anthropic-api-key --value <key>
```
Comment on lines +153 to +156

@coderabbitai coderabbitai Bot Jul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Label the shell snippet.

The fenced command block at Line 153 is missing a language identifier, which violates the Markdown docs guideline and trips markdownlint. Add bash (or sh) to the fence.

Suggested fix 
-```
+```bash

As per coding guidelines, docs/**/*.md: Include code blocks with language identifiers in Markdown documentation.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```
az keyvault secret set --vault-name pvc-prod-sluice-kv \
--name anthropic-api-key --value <key>
```
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 153-153: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/architecture/09-claude-code-via-sluice.md` around lines 153 - 156, The
fenced command block in the documentation is missing a language identifier,
which causes markdownlint violations. Update the shell snippet in the affected
markdown section to use a shell language tag such as bash or sh, and keep the
existing command content unchanged so the fenced block is correctly labeled.

Sources: Coding guidelines, Linters/SAST tools

Comment on lines +152 to +156

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove the unwired direct-KV go-live path

If an operator follows this Direct KV option, the deploy will still be Azure-only: the deploy workflows only populate TF_VAR_extra_providers from the GitHub ANTHROPIC_API_KEY secret, and Terraform only creates/mounts the provider-anthropic-key secret from that variable. Nothing in the inspected workflows reads anthropic-api-key, so local.has_anthropic remains false and the Claude models plus agent-default alias are not rendered; the runbook should not present this command as an equivalent go-live path unless the missing wiring is added.

Useful? React with 👍 / 👎.

Use this only if wiring `TF_VAR_extra_providers` from that KV secret rather
than a GitHub secret; the `extra_providers`-managed secret is
`provider-anthropic-key`, so keep the two out of sync deliberately.
3. Deploy: run the `deploy` workflow (or merge to trigger it) so Terraform
provisions `provider-anthropic-key`, injects the env var, and renders the
Claude models + `agent-default` alias into the LiteLLM config.
4. `python scripts/manage_keys.py apply` — applies the `mystira-ci` allow-list
(and any other `keys.yaml` changes) on the running proxy.

CI consumers then set the two env vars from the per-consumer snippet below,
using `vkey-mystira-ci`, and request `model="agent-default"`.
Comment on lines +166 to +167
Comment on lines +166 to +167

@coderabbitai coderabbitai Bot Jul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Split the CI and interactive paths.

Line 166 says CI consumers should use vkey-mystira-ci, but the section below still hard-codes vkey-claude-code-cli and describes Claude Code CLI routing through Sluice. That conflicts with the decision above that interactive Max-backed Claude Code stays direct-Anthropic, and it will send operators to the wrong auth flow. Please separate the CI runbook from the interactive one, or swap the example key if this section is CI-only.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/architecture/09-claude-code-via-sluice.md` around lines 166 - 167, The
runbook text mixes CI and interactive Claude Code guidance, causing conflicting
auth/routing instructions. Update the surrounding section so the CI path in the
relevant documentation uses the CI-specific key and agent-default model, and
move or rewrite the Claude Code CLI/Sluice example under a separate interactive
section. Use the existing “CI consumers” guidance and the Claude Code CLI/Sluice
example as the symbols to locate the conflicting text.


### Per-consumer (each service / user)

Expand Down
3 changes: 3 additions & 0 deletions infra/env/prod/terraform.tfvars
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ enable_litellm_db = true
# fireworks = { api_key = "...", env_name = "FIREWORKS_API_KEY" }
# # Phase 3 — Kimi K2 direct Moonshot path (optional; Groq path uses GROQ_API_KEY above)
# moonshot = { api_key = "sk-...", env_name = "MOONSHOT_API_KEY" }
# # ADR 09 — Claude Code CLI / CI agents via Sluice (env_name is the var the
# # LiteLLM model_list reads, NOT the GitHub secret name).
# anthropic = { api_key = "sk-ant-...", env_name = "LITELLM_ANTHROPIC_API_KEY" }
# }

# Custom domain for the gateway. DNS (CNAME + asuid TXT) lives in the
Expand Down
98 changes: 90 additions & 8 deletions infra/modules/sluice_aca/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,72 @@ locals {
},
] : []

# ── Anthropic / Claude models (ADR 09) ─────────────────────────────────
# Enables Claude Code CLI and other Anthropic-SDK callers to route through
# Sluice on API-key billing (ADR 09: "API-key billing = through Sluice").
# The `agent-default` alias is what non-interactive CI agents request; it
# maps to the cheap Claude tier as primary (order 1) and degrades to the
# mid tier via the fallback chain, so Sluice picks the concrete model and
# can re-route later without touching CI workflows. Activated by supplying
# an "anthropic" entry in extra_providers (env_name = LITELLM_ANTHROPIC_API_KEY).
has_anthropic = contains(local.active_provider_names, "anthropic")
anthropic_models = local.has_anthropic ? [
# Concrete Claude models (current Anthropic model ids).
# Cheap tier — Claude Haiku 4.5 ($1 / $5 per 1M tokens).
{
model_name = "claude-haiku-4-5"
litellm_params = {
model = "anthropic/claude-haiku-4-5"
api_key = "os.environ/LITELLM_ANTHROPIC_API_KEY"
}
model_info = {
input_cost_per_token = 0.000001
output_cost_per_token = 0.000005
}
},
# Mid tier — Claude Sonnet 4.6 ($3 / $15 per 1M tokens).
{
model_name = "claude-sonnet-4-6"
litellm_params = {
model = "anthropic/claude-sonnet-4-6"
api_key = "os.environ/LITELLM_ANTHROPIC_API_KEY"
}
model_info = {
input_cost_per_token = 0.000003
output_cost_per_token = 0.000015
}
},
# agent-default group — CI agents request model="agent-default".
# Cheap Claude tier is primary (order 1) so Sluice picks the concrete
# model; degrades to the mid tier (order 2) then to `premium` (Azure,
# always present) via the fallback chain below.
{
model_name = "agent-default"
litellm_params = {
model = "anthropic/claude-haiku-4-5"
api_key = "os.environ/LITELLM_ANTHROPIC_API_KEY"
order = 1
}
model_info = {
input_cost_per_token = 0.000001
output_cost_per_token = 0.000005
}
},
# agent-default fallback member — mid Claude tier (order 2).
{
model_name = "agent-default"
litellm_params = {
model = "anthropic/claude-sonnet-4-6"
api_key = "os.environ/LITELLM_ANTHROPIC_API_KEY"
order = 2
}
model_info = {
input_cost_per_token = 0.000003
output_cost_per_token = 0.000015
}
},
] : []

# ── "auto" alias — points to cheapest entry with fallback chain ────────
# auto is a model group with a single entry (Groq gpt-oss-20b if available,
# otherwise Together, otherwise Fireworks). Fallback chains handle the rest.
Expand Down Expand Up @@ -407,9 +473,28 @@ locals {
local.gemini_models,
local.openrouter_models,
local.moonshot_models,
local.anthropic_models,
local.auto_models,
)

# ── Fallback chains ────────────────────────────────────────────────────
# auto/cheap-* chains exist only with an extra provider (ADR 11); the
# agent-default chain exists only with Anthropic (ADR 09). Built
# separately and concatenated so either can appear on its own.
# Each entry is a single-key map(list(string)); tomap() keeps the element
# type uniform so concat() doesn't choke on differing object attribute sets.
extra_fallbacks = local.has_any_extra ? [
tomap({ auto = ["cheap-fast", "cheap-reasoning", "cheap-long-context", "premium"] }),
tomap({ cheap-fast = ["cheap-reasoning", "premium"] }),
tomap({ cheap-reasoning = ["premium"] }),
tomap({ cheap-long-context = ["premium"] }),
] : []
# agent-default degrades cheap Claude → mid Claude → premium (Azure).
anthropic_fallbacks = local.has_anthropic ? [
tomap({ "agent-default" = ["claude-sonnet-4-6", "premium"] }),
] : []
combined_fallbacks = concat(local.extra_fallbacks, local.anthropic_fallbacks)

# Build with nullable optional keys, then filter out the nulls. Terraform's
# ternary requires both branches to share a type, so the cleaner-looking
# `condition ? {extras} : {}` doesn't compile against typed objects.
Expand Down Expand Up @@ -442,14 +527,11 @@ locals {
budget_duration = var.budget_duration != "" ? var.budget_duration : null
rpm_limit = var.rpm_limit > 0 ? var.rpm_limit : null
tpm_limit = var.tpm_limit > 0 ? var.tpm_limit : null
# Multi-provider fallback chains (ADR 11). Only added when at least one
# extra provider is active; otherwise LiteLLM runs with Azure-only.
fallbacks = local.has_any_extra ? [
{ auto = ["cheap-fast", "cheap-reasoning", "cheap-long-context", "premium"] },
{ cheap-fast = ["cheap-reasoning", "premium"] },
{ cheap-reasoning = ["premium"] },
{ cheap-long-context = ["premium"] },
] : null
# Multi-provider fallback chains (ADR 11 + ADR 09). The auto/cheap-*
# chains are added when an extra provider is active; the agent-default
# chain is added when Anthropic is active. Concatenated so either set can
# appear independently; null only when neither is present (Azure-only).
fallbacks = length(local.combined_fallbacks) > 0 ? local.combined_fallbacks : null
context_window_fallbacks = local.has_any_extra ? [
{ cheap-fast = ["cheap-long-context"] },
{ cheap-reasoning = ["cheap-long-context"] },
Expand Down
17 changes: 17 additions & 0 deletions scripts/keys.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,23 @@ keys:
operator: jurie@phoenixvc.tech
use: interactive

# ── Mystira CI (non-interactive agents via Sluice — ADR 09) ──────────────
# Claude Code CLI in CI / automation routes through Sluice on API-key
# billing. Requests model="agent-default"; Sluice maps it to the cheap
# Claude tier and can re-route later without a workflow change. Separate
# from claude-code-cli (interactive, Max-subscription — stays direct).
- alias: mystira-ci
max_budget: 10
budget_duration: 7d
rpm_limit: 120
tpm_limit: 200000
models: [agent-default, claude-haiku-4-5, claude-sonnet-4-6, gpt-4o, text-embedding-3-large, auto, cheap-fast, cheap-reasoning, premium]
Comment on lines +75 to +80

@coderabbitai coderabbitai Bot Jul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Missing cheap-long-context in the mystira-ci allowlist breaks the auto fallback chain.

mystira-ci includes auto in its models list, but the auto fallback chain (infra/modules/sluice_aca/main.tf, extra_fallbacks) degrades auto → cheap-fast → cheap-reasoning → cheap-long-context → premium. Every other key with auto in this file also lists cheap-long-context; this one omits it. Since manage_keys.py sends models straight to LiteLLM's /key/generate as a key-level allowlist, a fallback landing on cheap-long-context for this key would be rejected.

🔧 Proposed fix 
-    models: [agent-default, claude-haiku-4-5, claude-sonnet-4-6, gpt-4o, text-embedding-3-large, auto, cheap-fast, cheap-reasoning, premium]
+    models: [agent-default, claude-haiku-4-5, claude-sonnet-4-6, gpt-4o, text-embedding-3-large, auto, cheap-fast, cheap-reasoning, cheap-long-context, premium]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
models: [agent-default, claude-haiku-4-5, claude-sonnet-4-6, gpt-4o, text-embedding-3-large, auto, cheap-fast, cheap-reasoning, premium]
models: [agent-default, claude-haiku-4-5, claude-sonnet-4-6, gpt-4o, text-embedding-3-large, auto, cheap-fast, cheap-reasoning, cheap-long-context, premium]
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/keys.yaml` at line 80, The mystira-ci allowlist in keys.yaml is
missing cheap-long-context, which breaks the auto fallback chain when LiteLLM
resolves auto through cheap-fast and cheap-reasoning. Update the models list for
mystira-ci to include cheap-long-context alongside the existing auto-related
models so manage_keys.py can pass a complete allowlist to /key/generate and
allow the fallback path used by infra/modules/sluice_aca/main.tf
extra_fallbacks.

metadata:
project: mystira-workspace
service: ci
use: non-interactive
contact: jurie@phoenixvc.tech

# ── Dashboard Test Playground (small, public-key style) ──────────────────
- alias: dashboard-playground
max_budget: 1
Expand Down