docs: Agent Gateway, API key usage dashboard, agents, and LLM Serving - #21
docs: Agent Gateway, API key usage dashboard, agents, and LLM Serving#21tmvfb wants to merge 23 commits into
Conversation
…M Serving Rebased onto the versioned-docs restructure (#16); all content lives under docs/versions/development/ per that tree's own convention for unreleased documentation, and docs/versions/1.9/ is untouched. - Flesh out Agent Gateway overview: path families (/ai, /serving, /mcp, /a2a, /sandbox), public vs. internal traffic, and admin-only External Model Providers (AI Gateway) workflow. - Add a Usage Dashboard section to API Keys covering the per-workspace Usage tab (time windows, cohorts, token/cost estimates, usage-by-key table). - Flesh out Agents: the Secret -> Model Configuration -> Agent -> MCP tools -> chat-test workflow, and tool-calling caveats for internal models. - Add a new LLM Serving page (deploy, advanced configuration, tool-call parsers, testing, editing, external access via Agent Gateway), wired into the sidebar for the development version only via a new per-item "versions" filter in config.mts (the page does not exist in the frozen 1.9 snapshot, so it must not appear in the 1.9 sidebar). - Correct /ai and /serving path-family ownership against the actual route builders in pkui.
hsteude
left a comment
hsteude
left a comment
There was a problem hiding this comment.
Excelent draft. Thanks @tmvfb ! Could you ask your agent to work on my feedback? Since this significantly improves our docs, could you also make a draft for the AGENT_OPS overview page? Besides, we should create a version for the release candidate we're preparing right now.
|
|
||
| ## When to Use Agents | ||
|
|
||
| Use the Agents page when you need a conversational or task-driven agent that calls an LLM and, optionally, tools exposed through MCP servers or other agents. Use it instead of a standalone script when you want the agent's configuration versioned as Kubernetes resources, observable through pkui, and reachable by other workloads over A2A. |
There was a problem hiding this comment.
I'd get rid of the "optionally". Beeing able to do something is part of what defines Agents.
There was a problem hiding this comment.
In what way is a "script" an alternative? I'd say something in the lines of: Use it when you want your agents to run on the platform....
There was a problem hiding this comment.
Fixed: reworded to "Use it when you want the agent to run on the platform: its configuration versioned as Kubernetes resources, observable through pkui, and reachable by other workloads over A2A." Dropped the script comparison and the "optionally" (2642f3c).
|
|
||
| Building a working agent is a short chain of dependent resources: | ||
|
|
||
| 1. If needed, store an external provider API key as a Kubernetes Secret. |
There was a problem hiding this comment.
Add something like: "If your admin granted your personal or shared workspace access to an exernal model, it will appear automatically (with link to the corresponding docs page.)
There was a problem hiding this comment.
oh i see you mention it later. but the link still makes sense.
There was a problem hiding this comment.
Added the link: step 2 of the workflow list now points to Additional Providers (Administrators) for the admin-granted-model case (2642f3c).
|
|
||
| ### 1. Store the Provider API Key | ||
|
|
||
| Skip this step if you will use an admin-provided Model Configuration or an **Internal** (in-cluster) model. Admin-provided configurations use a centrally managed credential; internal models authenticate over mesh identity. |
There was a problem hiding this comment.
No Human beeing knows how to use semi-colons. Which is why it is a nice "AI-Slop" detector. Also: As a user of this feature I probably don't know what a "mesh identity" is.
There was a problem hiding this comment.
Reworded without the semicolon and without "mesh identity": "Admin-provided configurations use a credential the administrator already manages centrally. Internal models don't need a stored credential at all — the platform authenticates them automatically as in-cluster workloads." (2642f3c)
|
|
||
| Skip this step if you will use an admin-provided Model Configuration or an **Internal** (in-cluster) model. Admin-provided configurations use a centrally managed credential; internal models authenticate over mesh identity. | ||
|
|
||
| Otherwise, for an **external** provider (OpenAI, Anthropic, Gemini), create a Kubernetes Secret first: open the user menu (top right) → **K8s Secrets** → **Add Secret**, and store the provider API key as a key/value pair in the workspace. See [Kubernetes Secrets](../platform/kubernetes.html#kubernetes-secrets) for the full flow. |
There was a problem hiding this comment.
Maybe one or two examples for keys anf values, such as ANTRHOPIC_KEY: sk-....?
There was a problem hiding this comment.
Added concrete examples: ANTHROPIC_API_KEY: sk-ant-... and OPENAI_API_KEY: sk-... (2642f3c).
There was a problem hiding this comment.
This screenshot's shadows are wierd. They seem to be cut off on the left an right.
There was a problem hiding this comment.
Confirmed — checked the corners at full resolution: the drop-shadow is clipped into a hard vertical line on the left/right sides (visible clearly at the bottom corners), unlike the other screenshots in this PR where the shadow fades out with margin on all sides. This is a capture-margin issue (window captured too close to the left/right edges), not something fixable by re-cropping the existing file — recovering a proper fade would need retaking the screenshot with more side margin around the browser window, same as the other AI Gateway/Agents screenshots. I don't have a way to reproduce the live pkui AI Gateway page from here, so I'm leaving this open rather than guessing at a fix. Will need whoever captured the others to redo this one with the same margin.
|
|
||
| ## External Access | ||
|
|
||
| Endpoints shown on the model's detail/API tabs are the workspace-internal serving URL, useful for testing from inside the platform. For external clients (SDKs, CI jobs, applications outside the cluster), call the model through [Agent Gateway](agent_gateway.html) instead, using the `/ai/<workspace>/models/<route-id>/v1/...` path family and an API key scoped to the model. See [API Keys](../platform/api_keys.html). This is the same path family used by models granted through [External Models](../admin/external_models.html). |
There was a problem hiding this comment.
path family? don't really understand as a dumb user
There was a problem hiding this comment.
Reworded away from "path family" to plain "a URL of the form ..." / "this same URL pattern" (2642f3c).
| This page is an early outline. Full Agent Gateway documentation is still being written and will be added here. | ||
| ::: | ||
| Agent Gateway is the shared routing and policy layer for external API traffic in prokube. It gives SDKs, automation, CI jobs, and agent clients a single, API-key-authenticated way to reach model, tool, agent, and sandbox endpoints running in a workspace — without a browser session and without exposing each service through its own ad hoc ingress. | ||
|
|
There was a problem hiding this comment.
Also here i'd like to see the a box for the "upstream references".
There was a problem hiding this comment.
Added an Upstream references box linking agentgateway.dev docs (confirmed via prokube-neo/paas/agentgateway that Agent Gateway is built on the agentgateway project, not Istio/Envoy directly) (2642f3c).
| Agent Gateway is not an agent-only feature. The same routing and policy model fronts classic model-serving endpoints, Knative services, MCP servers, memory stores, kagent A2A agents, and Agent Sandboxes. This page provides an overview of the features enabled by Agent Gateway and explains when it is used. | ||
|
|
||
| It is not only an agent feature. The same gateway model can protect sandbox APIs, MCP servers, agent endpoints, and classic model-serving endpoints. | ||
| ## When to Use Agent Gateway |
There was a problem hiding this comment.
Added a line acknowledging the list is non-exhaustive rather than trying to enumerate every case (2642f3c).
|
|
||
| See [API Keys](../platform/api_keys.html) for the full create/edit/rotate/disable workflow, client authentication formats, and troubleshooting. | ||
|
|
||
| ## Usage Dashboard |
There was a problem hiding this comment.
Checked this against pkui: the Usage tab is NOT admin-only. ai_gateway_keys/routes.py's usage endpoint uses the normal require_namespace_access check (same as list/get key endpoints), not _require_admin. Any workspace member sees their own keys' usage; admins additionally see aggregate internal/unattributed traffic. Added a sentence clarifying that (2642f3c).
| @@ -1,29 +1,75 @@ | |||
| # Agent Gateway | |||
There was a problem hiding this comment.
In general i'm wondering if this whole page should be part of the AGENT_OPS group.
Maybe we can put it into foundations as it is wright now. and just cover the agent ops related bits here. E.g. what it is supposed to do (manage traffic beween agent, mcp and llm, from outside in and also ....). and we could introduce it with a figure such as the one here: "https://prokube.ai/en/sovereign-ai/"
- agents.md: remove awkward 'optionally'/standalone-script phrasing, plain-language rewrite of admin-credential vs internal-model sentence, add Secret key/value example, link admin-granted-model note near workflow step 1, clarify why some providers require an admin, fix tool-calling link to the right llm_serving anchor - llm_serving.md: add upstream references (KServe, vLLM, TEI, faster-whisper), link Kubernetes Deployment/HPA/KEDA docs, replace 'path family' jargon with plain URL-pattern wording - agent_gateway.md: add agentgateway upstream references box, note the use-case list is non-exhaustive, clarify Usage Dashboard is visible to any workspace member (not admin-only)
|
Pushed 2642f3c addressing the concrete line comments (replied inline on each thread): wording fixes in agents.md, upstream-reference boxes and de-jargoning in llm_serving.md and agent_gateway.md, and a fact-check against
Left open, not addressed in this pass:
|
Per review feedback: the generic routing model (path families, public vs. internal traffic, API keys, upstream agentgateway reference) moves to a new Foundation page (platform/agent_gateway.md), since it's used by MLOps and Labs as much as AgentOps. The AgentOps page keeps only the agent-specific content: how traffic moves between agents/MCP/models/sandboxes (new SVG diagram), the agent-specific use-case list, and the External Models routing table. Also: - add Agent Gateway to the Foundation sidebar (development only) - repoint API Keys' and LLM Serving's generic Agent Gateway links to the new Foundation page - link Agent Gateway from Model Serving and Knative Services, since they share the same /serving routing
The curved connectors bent sharply right up to the tip, so the thick stroke's curve showed through the arrowhead marker and made it look lopsided. Added a short straight vertical segment before each tip so the stroke direction matches the marker exactly, on both the caller->gateway and gateway->backend arrows.
Reverted the caller->gateway arrows to the original curves (they were better before). Replaced the two curved dashed peer-to-peer arcs between backends with a single dashed mesh 'bus' line and straight vertical connectors into each box, which avoids the curve/marker distortion entirely since every segment is a plain straight line.
Rewrote sentences using periods, colons, commas, and parentheses instead of em dashes to match the rest of the site's style (these two pages had 11 of the ~20 total em dashes in the whole docs site).
Swept the whole PR diff (not just the Agent Gateway pages) for the same anti-patterns already flagged in review: mid-sentence semicolons joining two independent clauses, and em dash overuse. Fixed instances in agents.md, api_keys.md, external_models.md, and the diagram's alt text/aria-label, rewriting each as two sentences or with a colon/ comma as appropriate. List-item trailing semicolons are left as-is, that's a legitimate enumeration style already used elsewhere.
docs: document MCP ToolHive servers
2 participants
Replaces #20, rebased onto the versioned-docs restructure (#16). All content moved to
docs/versions/development/per that tree's convention for unreleased documentation;docs/versions/1.9/(the frozen release snapshot) is untouched.Summary
docs/versions/development/agentops/agent_gateway.md): fleshed out from a stub — path families (/ai,/serving,/mcp,/a2a,/sandbox), public vs. internal traffic, and a new admin-only External Model Providers section disambiguating the two sidebar entries both labeled AI Gateway (API Keys under Serving vs. provider management under Admin).docs/versions/development/platform/api_keys.md): new Usage Dashboard section documenting the per-workspace Usage tab (time windows, request-attribution cohorts, token/cost estimates, usage-by-key table, caveats).docs/versions/development/agentops/agents.md): fleshed out from a stub — the typical workflow (Secret → Model Configuration → Agent → MCP tools → chat test), the fixed OpenAI/Anthropic/Gemini vs. internal provider list, and the tool-calling caveat for internal models.docs/versions/development/agentops/llm_serving.md, new page): deploying self-hosted models, Advanced Configuration (deployment mode, quantization, autoscaling), the tool-call parser table for agent tool calling, testing/editing, and external access via Agent Gateway. Cross-linked from Agents, Agent Gateway, and Model Serving.docs/.vitepress/config.mts: added a per-sidebar-itemversionsfilter so the new LLM Serving nav entry only appears under/development/— it doesn't exist in the frozen1.9snapshot, so it must not show up (as a dead link) in the 1.9 sidebar. Verified in the built HTML that 1.9's rendered sidebar excludes it while development's includes a working link./aivs/servingpath-family ownership in Agent Gateway docs against the actual pkui route builders (both are shared families, not one-to-one with a single feature)._staticscreenshot paths (../_static/...→../../../_static/...) for content now nested three levels deeper underdocs/versions/development/.All facts were verified directly against
~/pkuifrontend/backend source (route builders, module registrations, exact UI copy) and closed product-decision issues (prokube/pkui#2115, #2266) for the LLM Serving / AgentOps placement question, rather than assumed.Two screenshot slots in API Keys use the existing text-placeholder convention (no image yet):
api-key-usage-tab.png,api-key-usage-by-key-table.png.Verification
npm run build— passes cleanly. Additionally verified indist/: all cross-page anchor links resolve, screenshot references resolve (bundled by the Vite asset pipeline), and the version-scoped sidebar entry renders only under/development/and not/1.9/.