ai-assistant: docs/skills: Add skills proposal#886
Open
illume wants to merge 1 commit into
Open
Conversation
illume
force-pushed
the
docs/ai-assistant-skills-proposal
branch
from
b6c4516 to
3fb6440
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a comprehensive design proposal documenting a “skills subsystem” for the Headlamp AI Assistant, covering skill types (prompt/MCP/tool), sources (local/Git/Helm), security considerations (STRIDE), and routing strategies.
Changes:
- Introduces a new skills design proposal document describing formats, loading/distribution, and trust model.
- Documents an MVP plan (local prompt skills +
SKILL.mdmetadata + UI) and phased roadmap for Git/MCP/tool skills. - Describes skill routing approaches (keyword vs embeddings vs LLM router) and proposed best practices.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| 1. **Progressive loading (agentskills.io pattern).** The [agentskills.io specification](https://agentskills.io/specification) recommends that agents read only the skill `name` and `description` fields until a skill is triggered. Full content is loaded on demand. This is the most important pattern — it means routing can operate on metadata alone without reading multi-KB skill bodies. | ||
|
|
||
| 2. **Limit injected skills to 3–5 per query.** Research and production experience consistently show that LLMs perform best with a focused set of instructions. The [LangChain agents guide](https://docs.langchain.com/oss/python/langchain/agents) recommends limiting tools to the most relevant subset. Our `SkillRouterConfig.maxSkills` defaults to 5. |
Comment on lines
+779
to
+782
| ### Current implementation: `SkillRouter` | ||
|
|
||
| The current implementation (`ai-common/src/skills/SkillRouter.ts`) uses **keyword/TF matching** — the simplest approach that requires zero external dependencies: | ||
|
|
Comment on lines
+800
to
+809
| ### Integration (completed) | ||
|
|
||
| The skills system is fully wired into the LLM pipeline: | ||
|
|
||
| 1. **`SkillManager.getRoutedSkillsPromptText(query, config, routerConfig?)`** — Async method that loads enabled skills, routes them for the query using the configured strategy (embedding or keyword), and returns formatted prompt text. | ||
| 2. **`LangChainManager.setSkillManager(skillManager, skillsConfig)`** — Configures skills on the LLM manager. Once set, both `userSend()` and `userSendStream()` automatically compute routed skills per-message and inject them into the system prompt. | ||
| 3. **`EmbeddingRouter`** — Embedding-based router using LangChain's `Embeddings` abstraction. Uses cosine similarity on skill metadata vectors. Falls back to keyword routing (`SkillRouter`) on failure. | ||
| 4. **`SkillManager.setEmbeddingRouter(router)`** — Strategy pattern: pass an `EmbeddingRouter` for semantic routing, or `null` to use keyword-only routing. | ||
|
|
||
| Skills are routed **per-message** (not globally) so each query gets the most relevant skills injected. Errors in skill loading or routing never block the main LLM call. |
Comment on lines
+707
to
+708
| - **[LangGraph.js: ToolNode](https://langchain-ai.github.io/langgraphjs/reference/classes/langgraph.prebuilt.ToolNode.html)** — The prebuilt node for executing tool calls in a LangGraph agent, with parallel execution and error handling. Important because Headlamp's `ToolOrchestrator` performs similar grouping of read-only vs write tools. | ||
| - **[@langchain/mcp-adapters](https://www.npmjs.com/package/@langchain/mcp-adapters)** — The adapter that bridges MCP servers to LangChain tools. Important because this is the exact package Headlamp uses — MCP skills ultimately flow through this adapter. |
|
|
||
| The embedding-based router is implemented in `EmbeddingRouter.ts` using LangChain's `Embeddings` abstraction: | ||
|
|
||
| **Required packages:** The `EmbeddingRouter` uses `@langchain/core`'s `Embeddings` abstraction, which is already a dependency of `ai-common`. No additional packages are needed for the core embedding routing. The optional `@langchain/community` package would only be needed for advanced vector store integrations (e.g., HNSWLib, FAISS) if the in-memory approach becomes insufficient at scale. |
Comment on lines
+297
to
+314
| // skills/cost-analyzer/CostAnalyzerTool.ts | ||
| import { ToolBase } from '@headlamp-k8s/ai/langchain'; | ||
| import { z } from 'zod'; | ||
|
|
||
| export class CostAnalyzerTool extends ToolBase { | ||
| readonly config = { | ||
| name: 'cost_analyzer', | ||
| description: 'Analyze Kubernetes resource costs', | ||
| schema: z.object({ | ||
| namespace: z.string().optional(), | ||
| }), | ||
| }; | ||
|
|
||
| async handler(args: { namespace?: string }) { | ||
| // ... implementation | ||
| return { content: 'Cost analysis results...' }; | ||
| } | ||
| } |
illume
force-pushed
the
docs/ai-assistant-skills-proposal
branch
from
3fb6440 to
840b518
Compare
Comment on lines
+779
to
+782
| ### Current implementation: `SkillRouter` | ||
|
|
||
| The current implementation (`ai-common/src/skills/SkillRouter.ts`) uses **keyword/TF matching** — the simplest approach that requires zero external dependencies: | ||
|
|
Comment on lines
+800
to
+809
| ### Integration (completed) | ||
|
|
||
| The skills system is fully wired into the LLM pipeline: | ||
|
|
||
| 1. **`SkillManager.getRoutedSkillsPromptText(query, config, routerConfig?)`** — Async method that loads enabled skills, routes them for the query using the configured strategy (embedding or keyword), and returns formatted prompt text. | ||
| 2. **`LangChainManager.setSkillManager(skillManager, skillsConfig)`** — Configures skills on the LLM manager. Once set, both `userSend()` and `userSendStream()` automatically compute routed skills per-message and inject them into the system prompt. | ||
| 3. **`EmbeddingRouter`** — Embedding-based router using LangChain's `Embeddings` abstraction. Uses cosine similarity on skill metadata vectors. Falls back to keyword routing (`SkillRouter`) on failure. | ||
| 4. **`SkillManager.setEmbeddingRouter(router)`** — Strategy pattern: pass an `EmbeddingRouter` for semantic routing, or `null` to use keyword-only routing. | ||
|
|
||
| Skills are routed **per-message** (not globally) so each query gets the most relevant skills injected. Errors in skill loading or routing never block the main LLM call. |
Comment on lines
+811
to
+813
| ### Embedding-based routing with LangChain (implemented) | ||
|
|
||
| The embedding-based router is implemented in `EmbeddingRouter.ts` using LangChain's `Embeddings` abstraction: |
Design proposal for the skills subsystem. Recommends three skill types loaded into the assistant system prompt: - Prompt skills (Markdown documents): default type, suitable for community contributions, loaded from Git repos, Helm values, or local directories. - MCP skills (MCP server endpoints): expose external tools as skills. - Tool skills (TypeScript classes): extend the assistant with custom code. MCP and tool skills require explicit admin approval before activation. Prompt skills are sandboxed to the system prompt and cannot call code. Documents the YAML schema for skill definitions, the `SkillLoader` I/O adapters (filesystem, HTTP, zip), `SkillConfigManager` persistence, and the proposed Headlamp UI for managing skill sources. Signed-off-by: René Dudfield <renedudfield@microsoft.com> Co-authored-by: Ashu Ghildiyal <aghildiyal@microsoft.com> Signed-off-by: Ashu Ghildiyal <aghildiyal@microsoft.com>
illume
force-pushed
the
docs/ai-assistant-skills-proposal
branch
from
840b518 to
5640c73
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
docs/skills/skills-proposal.md— the design proposal for the skills subsystem: how skill YAML files are structured, loaded from local paths, HTTP endpoints, and zip archives, and injected into the assistant system prompt.Assisted by Copilot.
Part of the ai-assistant skills documentation series.