diff --git a/infrastructure/ansible/files/claude-skills/README.md b/infrastructure/ansible/files/claude-skills/README.md new file mode 100644 index 00000000..530b9ba8 --- /dev/null +++ b/infrastructure/ansible/files/claude-skills/README.md @@ -0,0 +1,44 @@ +# claude-skills — curated command + skill bundle + +Curated snapshot of the molyanov-ai-dev methodology bundle, deployed by the +`telegram-ai-agent` Ansible role into every HOME a `claude` process may use: + +1. `/var/lib/telegram-ai-agent/.claude` — bot-spawned engine sessions +2. `/home/op/.claude` — Symphony-spawned agents + operator SSH sessions + +This directory is the **single source of truth** for deployed skills and +commands. Do not also symlink the raw upstream clone +(`/opt/molyanov-ai-dev`) into `~/.claude/skills/` — that produced +duplicate-skill-name conflicts (removed 2026-07). + +## Naming convention — why every command exists twice + +`commands/` intentionally ships two spellings of each multi-word command: + +| Variant | Example | Why | +|---------|---------|-----| +| dash (canonical) | `do-task.md` → `/do-task` | Matches molyanov-ai-dev docs and Claude Code CLI convention | +| underscore (Telegram alias) | `do_task.md` → `/do_task` | Telegram bot commands only allow `[a-z0-9_]{1,32}` — a dash command **cannot be typed as a Telegram command** | + +Both files must stay byte-identical (CI-friendly check: +`diff commands/do-task.md commands/do_task.md`). When editing a command, +edit the dash file and copy it over the underscore twin. + +`skills/` ships **dash names only**. Skills are invoked by the agent via the +Skill tool using the frontmatter `name:` — they are never typed as Telegram +commands, so no underscore alias is needed. Never add an underscore skill +directory: two directories with the same frontmatter `name:` make skill +resolution unreliable in every session (this was the pre-2026-07 bug behind +"agent does not recognize the command at once"). + +## Adding a new command + +1. Create `commands/.md` (frontmatter: `description`, + optional `allowed-tools`). +2. If the name contains a dash, copy it to `commands/.md`. +3. If it delegates to a skill, reference the skill by its dash name + (`Use the `task-decomposition` skill.`). +4. Redeploy: `ansible-playbook playbooks/deploy.yml --tags telegram-ai-agent` + (the sync is additive; renames/removals also need a purge task — see + `telegram_ai_agent_legacy_underscore_skills` in the role defaults for the + pattern). diff --git a/infrastructure/ansible/files/claude-skills/skills/code_reviewing/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/code_reviewing/SKILL.md deleted file mode 100644 index b6aebebe..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/code_reviewing/SKILL.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -name: code-reviewing -description: | - Code review methodology and quality standards for comprehensive code analysis. - Use to understand WHAT and HOW to review code: 11 review dimensions, process, quality standards. - - Use when: "проверь код", "code review", "ревью кода", "review this code", "check code quality" ---- - -# Code Review Methodology - -Comprehensive code review methodology for ensuring production-ready quality and maintainable architecture. - -## Review Dimensions - -Perform systematic analysis across these 11 dimensions: - -### 1. Architectural Patterns - -- Evaluate adherence to established architectural patterns (MVC, MVVM, Clean Architecture, etc.) -- Assess design patterns usage (Factory, Strategy, Observer, etc.) -- Verify layer separation and dependency direction -- Check for architectural anti-patterns (circular dependencies, god objects, tight coupling) - -### 2. Separation of Concerns - -- Validate single responsibility principle compliance -- Examine module boundaries and cohesion -- Review business logic vs presentation logic separation -- Assess data layer abstraction and persistence logic isolation - -**Good practices:** -- One file = one responsibility (UserService in one file, PaymentService in another) -- Functions < 50 lines; if larger, break into smaller functions -- Maximum 3 levels of nesting; use early returns to reduce nesting -- High-level modules should not depend on low-level details - -### 3. Code Readability & Maintainability - -- Evaluate naming conventions (variables, functions, classes) -- Assess code organization and file structure -- Check for appropriate use of comments and documentation -- Review complexity metrics (cyclomatic complexity, nesting depth) -- Verify consistent code style and formatting - -**Good practices:** -- Meaningful comments focus on "why" rather than obvious "what" -- DRY principle: extract repeated code into functions/modules -- Readable > clever: clear code is better than short but cryptic code -- No magic numbers: extract to named constants (`MAX_UPLOAD_SIZE` not `5242880`) - -### 4. Error Handling & Logging - -- Examine error propagation strategy -- Verify appropriate use of try-catch blocks -- Check error messages clarity and actionability -- Assess graceful degradation and fallback mechanisms - -**Good practices (error handling):** -- Always use try-catch for operations that can fail (API calls, DB operations, file I/O) -- Don't swallow errors: always re-throw after logging (unless explicitly handling) -- Fail fast: validate inputs early; throw errors immediately when invalid -- User-friendly errors: show generic message to users, log details internally - -**Logging review checklist:** -- Key operations have logs (external calls, auth events, state transitions, business operations) -- Structured format used (JSON / logger library), not string concatenation or `console.log` -- Every log includes context: userId, action, resourceId (not just a bare message) -- Correlation/request ID propagated through call chain -- Log levels used correctly (info for success, warn for recoverable, error for failures) -- Error logs include stack traces -- No secrets or PII in logs (passwords, tokens, API keys, emails, phone numbers) -- No empty catch blocks (`catch (e) {}` — silent error swallowing) -- No logging inside tight loops (generates thousands of duplicate lines) - -**Automatic severity mappings:** - -| Pattern | Severity | -|---------|----------| -| Secrets or PII logged (tokens, passwords, emails in plaintext) | critical | -| Empty catch block — error swallowed without logging | major | -| External call (API, DB) without any logging | major | -| Missing correlation/request ID in service handling requests | minor | -| `console.log` / `print` used instead of structured logger | minor | - -### 5. Type Safety (TypeScript/typed languages) - -For TypeScript or other typed codebases: - -- Validate type definitions completeness and accuracy -- Check for inappropriate use of `any` type (TypeScript) or equivalent loose typing -- Assess interface and type alias design -- Review generic type usage and constraints -- Verify null/undefined handling and optional chaining -- Check for type assertions and their justification - -### 6. Testing Coverage - -- Evaluate unit test presence and quality -- Assess test coverage for critical paths -- Review test organization and naming -- Check for integration and E2E test needs -- Verify mocking strategies and test isolation -- Assess edge case and error scenario coverage - -**Good practices:** -- Tests needed for: business logic, validations, transforms, error handling -- Tests not needed for: simple getters/setters, one-line configs, trivial updates -- Rule: if mocking >3 dependencies → wrong test type, use integration test - -### 7. Dependencies Management - -- Review new dependencies necessity and appropriateness -- Check for dependency version conflicts -- Assess bundle size impact -- Verify security vulnerabilities (outdated packages) -- Evaluate licensing compatibility - -**Good practices:** -- Verify imports exist before using: read source files to confirm exports match expected usage -- Check function signatures: ensure signatures match how you're calling them -- Prefer well-maintained packages: check npm/PyPI activity, security advisories -- Pin major versions: use `^` (caret) for npm to allow patch updates - -### 8. Security Considerations - -- Check for security vulnerabilities (injection, XSS, CSRF) -- Verify secrets management (no hardcoded credentials) -- Assess input validation and sanitization -- Review authentication and authorization logic -- Check for sensitive data exposure - -**Good practices:** -- Never hardcode secrets: use environment variables (`.env`) for all sensitive data -- Always validate input: check types, formats, ranges before processing -- Sanitize user data: before database operations, API calls, or displaying -- Add to .gitignore: `.env`, `*.key`, `credentials.json`, `secrets/` - -### 9. Performance Implications - -- Identify potential performance bottlenecks -- Review algorithmic complexity -- Check for unnecessary re-renders (React) or recomputations -- Assess memory leak risks -- Evaluate database query efficiency - -**Good practices:** -- Avoid N+1 queries: use batch operations, eager loading, or caching -- Cache expensive computations: use memoization for functions -- Prevent memory leaks: clean up event listeners, timers, subscriptions in cleanup functions -- Use pagination for large datasets: don't load all records at once -- Profile before optimizing: measure actual bottlenecks before making changes - -### 10. Cross-File Consistency - -For the code under review, verify correctness of function/class usage: - -**Process:** -1. When code CALLS a function from another file → Read that file, verify signature matches -2. When code USES a class/method → Read class definition, verify method exists and signature matches -3. When code IMPORTS something → Verify import path is correct - -**What to check:** -- Function called with correct arguments -- Method exists on the class -- Import paths are valid -- Types match (if TypeScript) - -**Report as issue if:** -- Function called with wrong arguments (runtime crash) -- Method doesn't exist (runtime crash) -- Import path broken (load failure) - -Read the source files where functions/classes are defined to verify signatures match. - -### 11. Resource Management - -- Identify heavy resources: ML models, database connection pools, browser instances, API clients, large caches -- Check if heavy resources are created as singletons (one instance shared) or duplicated across files/components -- When code creates a heavy resource (`new Model()`, `ModelClass(...)`, `create_pool()`): search the project for other instantiations of the same class -- Verify resource lifecycle: who creates, who consumes, when disposed -- Check for resource leaks: opened connections/files/handles that are never closed - -**Automatic severity mappings:** - -| Pattern | Severity | -|---------|----------| -| Same heavy resource class instantiated in multiple files without shared instance | major | -| Heavy resource created inside a loop or per-request handler | critical | -| Resource opened but never closed (connection, file handle, cursor) | major | - -## Dimension Prioritization - -Focus on dimensions based on code context: - -| Context | Prioritize | Reason | -|---------|------------|--------| -| Auth/login code | Security (8), Error Handling (4) | Auth vulnerabilities are critical | -| User input handling | Security (8), Type Safety (5) | Input validation prevents attacks | -| Database queries | Security (8), Performance (9) | SQL injection, N+1 queries | -| New feature | Architecture (1), Testing (6) | Foundation for future changes | -| Refactoring | Cross-File (10), Testing (6) | Avoid breaking existing code | -| Performance fix | Performance (9), Dependencies (7) | Target the actual bottleneck | -| Typed codebase | Type Safety (5), Cross-File (10) | Type errors cause runtime crashes | -| ML/AI pipeline | Resource Mgmt (11), Performance (9) | Heavy models duplicated waste memory | -| Microservice init | Resource Mgmt (11), Architecture (1) | Connection pools and clients should be shared | - -## Review Process - -1. **Initial Scan**: Quick overview to understand scope and context -2. **Deep Analysis**: Systematic review of each dimension listed above -3. **Cross-Reference**: Compare implementation against userspec, techspec, and project standards -4. **Issue Categorization**: Classify findings by severity: - - **critical** → blocking issues that must be fixed - - **major** → significant concerns that should be addressed - - **minor** → improvements that are valuable but optional -5. **Recommendation Formulation**: Provide specific, actionable suggestions - -## Quality Standards - -Be thorough but pragmatic: -- Focus on issues that materially impact code quality, security, or maintainability -- Distinguish between critical problems and stylistic preferences -- Provide constructive feedback with specific examples -- Acknowledge good practices when present -- Consider project context and constraints from project documentation (if available) -- Balance idealism with practical delivery needs - -## Communication Style - -- Be direct and specific - avoid vague feedback -- Use technical precision appropriate for senior developers -- Provide code examples in recommendations when helpful -- Explain the "why" behind each issue, not just the "what" -- Maintain professional, respectful tone -- Prioritize actionability over completeness - -Goal: ensure production-ready code that is secure, maintainable, and aligned with project standards. Be thorough in analysis but efficient in communication. diff --git a/infrastructure/ansible/files/claude-skills/skills/code_writing/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/code_writing/SKILL.md deleted file mode 100644 index 9be3da45..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/code_writing/SKILL.md +++ /dev/null @@ -1,139 +0,0 @@ ---- -name: code-writing -description: | - Universal quality coding process: plan, TDD, reviews. - Use whenever code needs to be written — ad-hoc or as part of a task. - - Use when: "напиши код", "закодь", "реализуй", "write code", "implement" - - For planning tasks → tech-spec-planning skill. For specs → user-spec-planning skill. ---- - -# Code Writing - -## Phase 1: Preparation - -1. **Parse Requirements** - - Extract what needs to be built from user message or passed acceptance criteria - - Clarify ambiguities — ask user if unclear - - Formulate acceptance criteria (what "done" looks like) - -2. **Read Project Context (Graceful)** - - **Working on a task?** Read all files listed in the task's "Context" section — it already specifies everything needed. - - **Standalone (no task file)?** Read (skip if missing): - - `.claude/skills/project-knowledge/references/project.md` — project overview - - `.claude/skills/project-knowledge/references/architecture.md` — system structure - - `.claude/skills/project-knowledge/references/patterns.md` — project conventions - - Then read `.claude/skills/project-knowledge/SKILL.md` (if exists). - Consider which domain-specific guides are relevant to your task and read those - (e.g., `architecture.md` Data Model section for DB work, `ux-guidelines.md` for UI tasks). - - **No project patterns?** Apply baseline from [universal-patterns.md](references/universal-patterns.md) — naming, error handling, structure. - -3. **Analyze & Review Approach** - - Before coding, output your findings: - - Grep for usages of code to be modified - - Read all files that will be changed - - Verify solution follows project patterns (or universal patterns) - - Identify existing code that can be reused - - If modifying existing code, run existing tests for the area to establish baseline - - If concerns → discuss with user before proceeding. - -**Checkpoint:** List completed preparation steps before moving to implementation. - -## Phase 2: Implementation (TDD) - -1. **Write Tests First** - - **Before writing tests**, read [testing-guide.md](references/testing-guide.md) — when to write which test type, test structure. - - - Write tests for: business logic, validations, transforms, error handling. Skip trivial code without logic (simple getters, one-liners, configs) - - Write tests for requirements and edge cases - - Tests should fail initially (no implementation yet) - - One test = one scenario, test behavior not implementation - - If mocking >3 dependencies → wrong test type, use integration test - -2. **Write Code** - - Implement to pass tests - - Follow project patterns (from Phase 1) or apply baseline from [universal-patterns.md](references/universal-patterns.md) - - Use env vars for secrets, validate inputs at boundaries - - Handle edge cases, comment WHY not WHAT - -3. **Run Tests** - - All new tests pass - - Fix any failures - -**Checkpoint:** List implemented functionality and test results. - -## Phase 3: Post-work - -1. **Run Lint/Format** - - Run project's linter and formatter before reviews - -2. **Run Relevant Tests** - - Tests for files changed - - Tests mentioned in task (if applicable) - - Save full test suite for end of feature - -3. **Smoke Verification** (if task has Verification Steps → Smoke or User) - - Execute each command from the Smoke section. Record results in decisions.md Verification section. - If a check fails — fix the code before proceeding to reviews. - If the task has User checks — ask the user to verify, wait for confirmation. - - Smoke catches integration bugs that mocked tests miss: - real API responses, library initialization, config validity. - -4. **Run Reviews** (launch in parallel) - - **Reviewer selection:** - - Working on a task file → run reviewers from the task's "Reviewers" section - - Standalone (no task file) → default: code-reviewer, security-auditor, test-reviewer - - For each reviewer: - 1. Spawn subagent via Task tool (subagent_type = reviewer name, e.g. `code-reviewer`) - 2. Pass: git diff of changes, path to task file, path to tech-spec, path to user-spec - 3. Reviewer loads its own skill automatically (via agent frontmatter `skills:`) - 4. Report path: from the task's "Reviewers" section (or `logs/working/` if standalone) - - Reviewers write JSON reports to `logs/working/task-{N}/{reviewer-name}-{round}.json`. - `{N}` = task number from task file; `"standalone"` if no task file. - On re-review: new file with incremented round number, old file stays. - -5. **Process Findings** - - Evaluate each finding on merit — severity is metadata, not a filter. - A valid minor fix still improves quality. Reason: skipping valid findings - silently degrades the codebase over time. - - For each finding: - - **Valid, improves code** → apply (any severity: critical, major, minor, low) - - **Disagree or uncertain** → discuss with user (explain reasoning) - - **Out of scope** → skip, note in findings log - - Produce a findings log: - | # | Source | Severity | Finding | Action | Reason | - Each finding appears in this table — transparent decision trail. - - After applying fixes → re-run tests → re-run the reviewer(s) that reported them. - Limit: 3 review iterations. If findings remain after round 3 → ask user. - Reason: fixes can introduce new issues — a second pass catches regressions. - -**Checkpoint:** List post-work steps completed. - -## Self-Verification - -Verify each item before marking complete. If any item fails, return to the relevant phase. - -- [ ] All phases completed (Preparation, Implementation, Post-work) -- [ ] Tests pass -- [ ] Smoke verification executed (if task had Smoke/User checks) -- [ ] Each reviewer finding evaluated and logged -- [ ] Findings log table produced -- [ ] Review JSON reports saved to `logs/working/task-{N}/` - diff --git a/infrastructure/ansible/files/claude-skills/skills/code_writing/references/testing-guide.md b/infrastructure/ansible/files/claude-skills/skills/code_writing/references/testing-guide.md deleted file mode 100644 index 0c6759c0..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/code_writing/references/testing-guide.md +++ /dev/null @@ -1,117 +0,0 @@ -# Testing Guide for Code Writing - -Condensed testing rules for TDD workflow. For full testing strategy, see `~/.claude/skills/test-master/SKILL.md`. - ---- - -## Test Quality Rules (apply when writing any test) - -### Litmus Test -Before finishing any test, ask: "If I remove the core logic line being tested, does this test still pass?" -If yes — test is useless. Rewrite. - -### Real Data Over Mocks -Priority order: -1. Real dependencies (test DB, real file system) → integration test -2. Minimal mocks (1-2 external services) → unit test -3. Heavy mocking (3+) → wrong test type, switch to integration - -### No Mock-Return Pattern -```typescript -// BAD — tests mock, not code -mockService.getData.mockReturnValue(42) -const result = await handler() -expect(result).toBe(42) - -// GOOD — tests actual computation -const result = await calculateTotal(100, 0.2) -expect(result).toBe(80) -``` - -### Test the Contract -```typescript -// BAD — tests implementation detail -expect(db.query).toHaveBeenCalledWith('SELECT * FROM users WHERE id = 1') - -// GOOD — tests what goes in → what comes out -const user = await getUser(1) -expect(user.name).toBe('Alice') -``` - ---- - -## Decision: Do I Need Tests? - -**YES — write tests for:** -- Business logic (calculations, validations, transforms) -- Decision-making code (if/else, switch, state machines) -- Data processing and formatting -- Error handling logic - -**NO — skip tests for:** -- Simple getters/setters -- One-line text or config changes -- Trivial updates with no logic - -## Decision: Which Test Type? - -| Type | When to Use | Mock Strategy | -|------|-------------|---------------| -| **Unit** | Business logic, pure functions, validations | Mock DB, APIs, file system, time | -| **Integration** | API endpoints, DB operations, external services | Real DB (test), mock external services | -| **E2E** | Critical user journeys (auth, payment, core flows) | Real everything (use sandbox/test mode) | - -**Rule:** If mocking >3 dependencies → wrong test type. Use integration or E2E instead. - -**When to Prioritize E2E over Unit:** -- UI apps → E2E + Integration > Unit -- Browser extensions → E2E (real browser) > Unit -- CLI tools → Unit + Integration -- API/Backend → Unit + Integration - -## TDD Flow - -``` -1. Write tests for requirements + edge cases -2. Run tests → all should FAIL (no implementation) -3. Write code to make tests pass -4. Run tests → all should PASS -5. Refactor if needed (tests stay green) -``` - -## Writing Good Tests - -**DO:** -- One test = one scenario (happy path, error case, edge case — separate tests) -- Test behavior, not implementation details -- Use descriptive names: `test_user_creation_fails_when_email_invalid` -- Test actual results: `expect(calculateTotal(100, 0.2)).toBe(80)` -- Test real state changes: `cart.add({id: 1}); expect(cart.items).toHaveLength(1)` -- Keep tests fast: unit < 100ms, integration < 1s - -**Write quality tests:** -- Write meaningful assertions that verify actual behavior -- Verify results, not just that mock was called -- Add assertions when rendering components -- Keep tests independent — each test sets up its own state -- Isolate test state — each test manages its own data - -## Test Structure Pattern - -``` -// Arrange — set up test data and conditions -// Act — execute the function/action being tested -// Assert — verify the result -``` - -Group tests by feature or function. Use `describe` blocks for grouping, `it`/`test` for individual cases. - -## What test-reviewer Checks - -The test-reviewer agent will verify: -- Tests have meaningful assertions (not trivial) -- Tests cover requirements and edge cases -- Test names are descriptive -- Mocking is appropriate (not excessive) -- Tests are independent and isolated -- Test pyramid is balanced for the feature diff --git a/infrastructure/ansible/files/claude-skills/skills/code_writing/references/universal-patterns.md b/infrastructure/ansible/files/claude-skills/skills/code_writing/references/universal-patterns.md deleted file mode 100644 index 6c519de4..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/code_writing/references/universal-patterns.md +++ /dev/null @@ -1,114 +0,0 @@ -# Code Patterns & Best Practices - -Universal coding standards for generating high-quality code. Always applied as baseline for all projects. - ---- - -## Naming Conventions -- **Functions/Methods**: verbs (`createUser`, `fetchData`, `validateEmail`) -- **Variables**: descriptive nouns (`userData`, `totalPrice`, `isActive`) -- **Constants**: UPPER_SNAKE_CASE (`API_KEY`, `MAX_RETRIES`) -- **Files**: kebab-case for JS/TS, snake_case for Python -- **Classes**: PascalCase (`UserService`, `PaymentProcessor`) -- **Avoid abbreviations** unless universally known (`id`, `url` OK; `usr`, `calc` NOT) - -## Code Organization -- **One file = one responsibility** (UserService in one file, PaymentService in another) -- **Functions should be small**: < 50 lines; if larger, break into smaller functions -- **Limit nesting**: Maximum 3 levels deep; use early returns to reduce nesting -- **Group related code**: Put imports together, constants together, functions by feature -- **Dependency direction**: High-level modules should not depend on low-level details - -## Dependency Management -- **Verify imports exist before using**: Read source files to confirm exports match expected usage -- **Check function signatures**: Ensure function/method signatures match how you're calling them -- **Use Context7 for library docs**: Get up-to-date API documentation for correct usage patterns -- **Prefer well-maintained packages**: Check npm/PyPI activity, security advisories, last update date -- **Pin major versions**: Use `^` (caret) for npm to allow patch updates, avoid breaking changes - -## Separation of Concerns -Extract from code into separate files: -- **Configuration**: `.env` file (API keys, URLs, timeouts, feature flags) -- **All UI text**: Never hardcode user-facing strings — extract to separate files (`messages/`, `locales/`, `constants/`) for easy updates, translations, and consistency -- **LLM/Agent text content**: Never hardcode prompts, templates, and other text content for LLMs/agents in code — extract to separate files (`prompts/`, `templates/`) for easy iteration, review, and reuse -- **Business logic**: Keep separate from framework code (routes, controllers) - -## Security -- **Store all secrets in environment variables** (`.env`) — API keys, passwords, tokens -- **Validate all input**: Check types, formats, ranges before processing -- **Sanitize user data**: Before database operations, API calls, or displaying -- **Add to .gitignore**: `.env`, `*.key`, `credentials.json`, `secrets/` -- **Create .env.example**: With empty/dummy values for documentation - -## Validation -- **Validate at API boundaries**: Check input in controllers, API routes, function entry points -- **Use schema validation libraries**: Zod, Yup, io-ts for runtime type checking and validation -- **Validate on BOTH frontend AND backend**: Defense in depth - never trust client-side validation alone -- **Sanitize before database operations**: Prevent SQL injection, NoSQL injection attacks -- **Fail fast with clear errors**: Return specific validation errors to help users fix input - -## Error Handling -- **Use try-catch** for operations that can fail (API calls, DB operations, file I/O) -- **Log with context**: Include user_id, action, resource_id, error message, stack trace -- **Don't swallow errors**: Always re-throw after logging (unless explicitly handling) -- **Fail fast**: Validate inputs early; throw errors immediately when invalid -- **User-friendly errors**: Show generic message to users, log details internally - -## Logging -- **Use structured logging format**: JSON with consistent fields for easy parsing and searching -- **Include context in every log**: userId, action, resourceId, timestamp, error message -- **Use appropriate log levels**: - - `debug`: Development-only details (verbose, disabled in production) - - `info`: Key operations completed (user login, order created, payment processed) - - `warn`: Recoverable issues (retry succeeded, deprecated API used, rate limit approaching) - - `error`: Failures requiring attention (API call failed, database error, auth failure) -- **Log errors with stack traces**: Helps debug production issues quickly - -**What to log:** -- Entry/exit of key business operations (`order.created`, `payment.processed`) -- External calls (API, DB, queues) with duration: `api.call.completed { service: "stripe", duration_ms: 230 }` -- Authentication and authorization events (login, logout, permission denied, token refresh) -- State transitions (order status changes, workflow steps) -- Startup/shutdown: config loaded (without secret values), connections established, service ready - -**What to exclude from logs:** -- Passwords, API keys, tokens, session IDs -- PII at info level and above: email, phone, full name, IP address (use hashed/masked versions) -- Full request/response bodies (log summary or truncated version instead) -- High-frequency events without sampling (health checks, heartbeats) - -**Correlation ID**: Propagate requestId/correlationId through the entire call chain. Every log entry for the same user request should share one ID for easy tracing. - -**Anti-patterns:** -- Logging inside tight loops (thousands of identical log lines) -- `console.log` / `print` in production code (use a proper logger) -- `catch (e) {}` — empty catch blocks that swallow errors silently -- Logging large objects with `JSON.stringify(entireObject)` (log relevant fields only) - -## Testing -- **Test public APIs**, not internal implementation -- **Mock external services**: API calls, database, file system -- **One test = one scenario**: happy path, errors, edge cases separately -- **Descriptive test names**: `test_user_creation_fails_when_email_invalid` -- **Keep tests fast**: unit < 100ms, integration < 1s - -## Performance -- **Avoid N+1 queries**: Use batch operations, eager loading, or caching instead of loops with queries -- **Cache expensive computations**: Use memoization for functions, Redis for shared state across requests -- **Be mindful of bundle size**: Check impact of new dependencies on frontend load time -- **Prevent memory leaks**: Clean up event listeners, timers, intervals, subscriptions in cleanup functions -- **Use pagination for large datasets**: Don't load all records at once, implement cursor or offset pagination -- **Profile before optimizing**: Measure actual performance bottlenecks before making changes (don't guess) - -## Code Quality -- **Write meaningful comments** (in English): - - Focus on "why" and "what for" rather than obvious "what" - - For complex logic, "what it does" is also valuable - - **When to write comments:** Complex business logic, non-obvious decisions, constraints, edge cases, security areas - - **When NOT to write comments:** Obvious self-documenting code, every function, repeating type information - - **Format:** JSDoc/TSDoc for public APIs, inline comments for complex logic - - When updating code → update comments too -- **DRY principle**: Extract repeated code into functions/modules -- **Readable > clever**: Clear code is better than short but cryptic code -- **Consistent formatting**: Use project's linter/formatter settings -- **No magic numbers**: Extract to named constants (`MAX_UPLOAD_SIZE` not `5242880`) diff --git a/infrastructure/ansible/files/claude-skills/skills/deploy_pipeline/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/deploy_pipeline/SKILL.md deleted file mode 100644 index c8fbe9eb..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/deploy_pipeline/SKILL.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -name: deploy-pipeline -description: | - Sets up CI/CD pipelines, deployment configuration, and automated deploy workflows. - GitHub Actions, platform-specific deploy (Vercel, Railway, Fly.io, AWS, VPS), - secrets management in CI. - - Use when: "подготовь деплой", "настрой автодеплой", "настрой CI/CD", - "setup deploy", "configure deployment", "настрой пайплайн" ---- - -# Deploy Pipeline - -## Gathering Deployment Context - -Read project-knowledge to understand the deployment target: -- `.claude/skills/project-knowledge/references/deployment.md` -- `.claude/skills/project-knowledge/references/architecture.md` -- `.claude/skills/project-knowledge/references/patterns.md` - -If deployment target is not documented, ask the user: -- Target platform (Vercel, Railway, Fly.io, AWS ECS, VPS, NPM, Chrome Web Store) -- Environment details (URLs, project/service IDs, server access) -- Required secrets and where to obtain them - -After gathering answers, immediately update `deployment.md` before proceeding with setup. - -## CI/CD Convention - -Create `.github/workflows/ci.yml` following this structure: - -```yaml -name: CI -on: - push: - branches: [main] - pull_request: - branches: [main] - -jobs: - check-skip: - runs-on: ubuntu-latest - outputs: - should_skip: ${{ steps.check.outputs.should_skip }} - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 2 - - id: check - run: | - FILES=$(git diff --name-only HEAD~1 HEAD 2>/dev/null || git diff --name-only HEAD) - if echo "$FILES" | grep -vqE '\.(md|txt)$|^\.claude/|^\.spec/|^docs/'; then - echo "should_skip=false" >> $GITHUB_OUTPUT - else - echo "should_skip=true" >> $GITHUB_OUTPUT - fi - - test: - needs: check-skip - if: needs.check-skip.outputs.should_skip != 'true' - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - # setup, install, lint, type-check, test, build - - deploy: - needs: test - if: github.ref == 'refs/heads/main' && github.event_name == 'push' - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - # platform-specific deploy action -``` - -Adapt: add language setup, install steps, platform-specific deploy action. - -## Platform Selection - -| Platform | Choose when | -|----------|------------| -| Vercel | Next.js, React, static sites, serverless | -| Railway | Full-stack apps needing managed DB | -| Fly.io | Docker containers, global edge | -| AWS ECS | Enterprise, full infrastructure control | -| Custom VPS | Persistent sessions, multi-device | -| NPM | Node.js packages or CLI tools | -| Chrome Web Store | Browser extensions | - -For VPS deployments: server-specific details (IPs, SSH keys, paths) go to `deployment.md`. - -## Secrets Convention - -Document all required secrets in `.claude/skills/project-knowledge/references/deployment.md`. For each secret: -- Name (GitHub Actions key) -- Where to obtain value (dashboard URL or CLI command) -- Which workflow uses it - -Guide user to add secrets in GitHub repository settings. Create `.env.example` with application-level variable names. - -## Documentation Updates - -After configuring, update project-knowledge references. Append to existing content. - -**deployment.md:** deploy target, pipeline overview, required secrets table, manual deploy command, rollback steps. - -**patterns.md (Git Workflow section):** CI triggers, pipeline jobs, skip logic pattern, PR workflow. - -## Decision Framework - -**Add deploy job?** -YES if: deployment target defined, user requests it, stable main branch. -NO if: early development, manual deploys preferred, manual review step needed (Chrome Web Store). - -**Use matrix strategy?** -YES if: NPM package, cross-platform library. -NO if: single-environment app, internal tool. - -**Add staging?** -YES if: project uses main + dev branches (default workflow). -NO if: Vercel preview deploys sufficient. diff --git a/infrastructure/ansible/files/claude-skills/skills/documentation_writing/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/documentation_writing/SKILL.md deleted file mode 100644 index cfa3caca..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/documentation_writing/SKILL.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -name: documentation-writing -description: | - Maintain project documentation in .claude/skills/project-knowledge/: audit, edit, check consistency, track status. - - Use when: "проверь документацию", "обнови документацию", "аудит документации", - "check docs", "audit documentation", "update docs", "проверь базу знаний проекта" - - For creating documentation from scratch use project-planning skill. - For reading docs or explaining concepts, read project-knowledge skill directly. ---- - -# Documentation Management - -Maintain project documentation in `.claude/skills/project-knowledge/references/`. Audit for bloat, edit files, check consistency, track status. - -For creating documentation from scratch (new project or empty docs), use `project-planning` skill. - -## Documentation Principles - -These rules apply to ALL documentation operations (audit, edit, create). - -**Goal: open docs → understand the project without reading code.** What is this project, how it's structured, what it does, where to find key things, how to deploy, where are logs. A high-level navigation guide. - -**Describe what exists, what it does, and why.** High-level overview of components, how they work together, decisions made (why this stack, why this architecture), operational details (server addresses, deploy procedures, log locations, env var names). Skip what's obvious from reading the code itself — function signatures, implementation details, generic framework behavior. - -**No code blocks, no pseudocode.** Link to source files: `[auth.ts:45-67](src/auth/jwt.ts#L45-L67)`. Code in docs gets outdated and bloats context. - -**No duplication between files.** Information lives in ONE place. Cross-reference: "See deployment.md for env vars." - -**patterns.md: only project-specific patterns.** Universal coding standards live in `~/.claude/skills/code-writing/references/universal-patterns.md`. Project patterns.md contains only what's unique to THIS project. - -## File Structure - -**4 core files** in `.claude/skills/project-knowledge/references/`: - -| File | Contains | -|------|----------| -| project.md | Overview, audience, problem, 3-5 key features, out of scope | -| architecture.md | Tech stack (with WHY), project structure, dependencies, integrations, data model | -| patterns.md | Project-specific code patterns, git workflow, testing methods, business rules | -| deployment.md | Platform, env var names, CI/CD triggers, rollback, monitoring | - -**Optional:** -- **ux-guidelines.md** — only for projects with significant UI -- **{custom}.md** — domain-specific (bot.md, vault.md, api.md) - -Templates with placeholder structure are in `~/.claude/shared/templates/new-project/.claude/skills/project-knowledge/references/`. The templates are self-documenting — each section has comments explaining what to write. - -## Workflows - -### 1. Audit - -**Trigger:** User asks to audit, check quality, or find bloat. - -1. Read all files from `.claude/skills/project-knowledge/references/` + CLAUDE.md + README.md -2. Flag issues: - - Code blocks → replace with file references - - Generic framework knowledge (belongs in official docs) → remove - - Function-level details → suggest moving to code comments - - Bloated sections (>3-5KB per file is suspicious) → condense - - Duplication across files → consolidate to one place - - Placeholder text (`[Project Name]`, `TODO`) → fill or remove - - Inconsistent terminology → standardize - - Outdated info (files/functions that no longer exist) → update or remove - - Universal patterns in patterns.md → flag for removal (those belong in code-writing skill) -3. Preserve operational details: server addresses, SSH configs, deploy commands, log paths, env var names, monitoring URLs. These belong in docs even if they seem "obvious" — they can't be read from code. -4. Create audit report with issues by file -5. Ask user which to fix → apply approved changes → verify consistency - -### 2. Edit - -**Trigger:** User asks to edit or update specific documentation. - -1. Identify target file/section (ask if unclear) -2. Read current content -3. Apply changes following documentation principles -4. Check if changes affect other files (e.g., tech stack mentioned elsewhere) → update related files - -### 3. Check Consistency - -**Trigger:** User asks to verify terminology or check for mismatches. - -1. Read all project-knowledge files -2. Extract: tech stack names/versions, service names, DB names, env var names, platform names -3. Find inconsistencies (e.g., "PostgreSQL" vs "Postgres" vs "postgres") -4. Report → ask user for correct terminology → standardize across all files - -### 4. Show Status - -**Trigger:** User asks about documentation completeness. - -1. Check each file: exists? filled or template? size? last modified? -2. Classify: filled / partially filled / template / missing -3. Show status report with recommendations - -## Root Project Files - -CLAUDE.md and README.md are entry points, not documentation. Keep them minimal — they point to project-knowledge, not contain information. - -**CLAUDE.md** (for AI agents): project name, one-line description, reference to project-knowledge skill, backlog path, default branch. Template: `~/.claude/shared/templates/new-project/CLAUDE.md`. - -**README.md** (for humans, in Russian): project title, purpose, folder structure overview, link to references/. Template: `~/.claude/shared/templates/new-project/README.md`. - -When auditing, verify that CLAUDE.md and README.md stay minimal — detailed info belongs in project-knowledge. - -## Custom Domain Files - -Add when project has a significant domain not covered by 4 core files (bot.md, vault.md, trading.md). - -Process: create in `references/` → update project-knowledge SKILL.md to list it → update CLAUDE.md and README.md if they list doc files. diff --git a/infrastructure/ansible/files/claude-skills/skills/feature_execution/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/feature_execution/SKILL.md deleted file mode 100644 index 4f9b395e..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/feature_execution/SKILL.md +++ /dev/null @@ -1,228 +0,0 @@ ---- -name: feature-execution -description: | - Orchestrate feature delivery as team lead: spawn agents by wave, - manage review cycles (max 3 rounds), commit per wave. - - Use when: "выполни фичу", "do feature", "execute feature", "запусти фичу", - "выполни все задачи", "execute all tasks" ---- - -# Feature Execution - -## Role Boundary - -You are a dispatcher, not a doer. Your job: spawn agents, wait for results, update status, commit orchestration changes, communicate with user. - -Allowed actions: -- Read specs and task frontmatter (for planning) -- Spawn teammates and reviewers (Agent tool) -- Send/receive messages (SendMessage) -- Update frontmatter status fields and checkpoint.yml (Edit) -- Write execution-plan.md (Write) -- Git commits for status/orchestration changes (Bash) - -Forbidden actions: -- Writing or editing source code, tests, configs, prompts -- Running tests, builds, linters, deploys -- Calling MCP tools for task work (Telegram, browsers, APIs) -- Debugging errors or "quickly fixing" anything -- Reading full task content during execution (frontmatter only) - -Reason: when the lead does task work, it pollutes context with implementation details, loses orchestration state, and breaks the parallel execution model. Every task — no matter how trivial — gets a spawned teammate. - -## Phase 1: Initialization - -0. Check `work/{feature}/logs/checkpoint.yml`: - - `last_completed_wave > 0` → this is a resume after context compaction. - Read checkpoint, then read `work/{feature}/decisions.md` to confirm what was actually completed. - For tasks in the resumed wave: if a task has a decisions.md entry, it completed — update its - frontmatter to `done` and skip it. Only re-execute tasks without a decisions.md entry. - Check if `~/.claude/teams/{team_name}/config.json` exists: if yes, team is alive; if no, - recreate via TeamCreate. Skip to Phase 2 starting from `next_wave`. - Report to user: "Resuming from wave {N}. Waves 1-{N-1} completed." - - `last_completed_wave: 0` → fresh start, proceed below. - -1. Read `work/{feature}/tech-spec.md` and `work/{feature}/user-spec.md` -2. Read frontmatter of all task files in `work/{feature}/tasks/` — extract fields: - - | Field | Purpose | - |-------|---------| - | `status` | planned → in_progress → done | - | `wave` | Parallel execution group number | - | `depends_on` | Task numbers that must be done first | - | `skills` | Skills the teammate loads | - | `reviewers` | Reviewer agents to spawn (source of truth) | - | `teammate_name` | Agent name for team spawning (optional) | - | `verify` | Verification types: [smoke], [user], [smoke, user], or [] (optional) | - - Build waves: group tasks by `wave` field. Within a wave, all tasks run in parallel. - -3. Build execution plan following template at `~/.claude/shared/work-templates/execution-plan.md.template` -4. Save to `work/{feature}/logs/execution-plan.md` -5. Show plan to user, wait for approval -6. Create team via TeamCreate -7. Update `work/{feature}/logs/checkpoint.yml`: set `total_waves` from the execution plan. - -**Checkpoint:** execution plan approved, team created, checkpoint initialized. - -## Phase 2: Execute Wave - -1. Find tasks for current wave: `status: planned`, all `depends_on` tasks are `done` -2. Update frontmatter: `status: planned` → `status: in_progress`. Read only frontmatter (`limit=15`), then Edit the status field. Do not read full task content. -3. For each task, spawn **teammate + reviewers** (if task has reviewers): - - Use `teammate_name` from task frontmatter as the agent name. If not set — pick a descriptive name based on the task. - - **Teammate** — `subagent_type: "general-purpose"`, `model: "opus"`, `team_name: "{team}"` - - Prompt template: - - ``` - You are "{name}" executing task {N}. - - Read task: {feature_dir}/tasks/{N}.md - Load skills listed in task frontmatter. If skills listed — follow the loaded skill workflow. - If no skills listed — follow the task instructions directly (the task file contains detailed steps). - - If the task requires user actions — send the instruction to team lead via SendMessage. - Team lead will forward to user and return confirmation. - - {reviewers_block} - - After task complete: - - Write entry to {feature_dir}/decisions.md (follow template at ~/.claude/shared/work-templates/decisions.md.template). - Summary: 1-3 sentences describing what was done and key decisions. Link JSON reports for review details. - - Message team lead: "Task {N} complete. decisions.md updated." - - Feature dir: {feature_dir} - ``` - - **{reviewers_block}** — include only when task has reviewers (not `reviewers: none`): - - ``` - Your reviewers: {reviewer_names} (list of teammate names). - - Review process — after task is complete, follow this review process (overrides review steps from loaded skills): - 1. Run `git diff -- ` and collect the list of changed files + full diff output. - 2. Send each reviewer via SendMessage: list of changed files + full diff output. - 3. Reviewers will perform review, write JSON report to `{feature_dir}/logs/working/task-{N}/{reviewer_name}-round{round}.json`, and send report path back to you. - 4. Read reports, fix findings. After fixes: send updated diff to reviewers for next round. - 5. Max 3 review rounds. Reason: diminishing returns — if 3 rounds cannot resolve findings, the issue requires human judgment. If unresolved after 3 → message team lead to escalate. - - Commit flow: - 1. After implementation complete (tests pass): git commit `feat|fix: task {N} — {brief description}` - 2. Send diff to reviewers for review. - 3. After each round of fixes (tests pass): git commit `fix: address review round {M} for task {N}` - 4. After all reviews pass (or max 3 rounds): git commit review reports with message `chore: review reports for task {N}` - ``` - - If task has `reviewers: none` — skip reviewer spawning. The teammate works independently, commits code with message `feat|fix: task {N} — {brief description}` (tests pass), and reports completion directly to team lead. - - Every task gets a spawned teammate — even tasks with no skills and no reviewers (operational tasks, MCP interactions, benchmarks, manual steps). The lead never executes task work directly. - - **Each reviewer** (when present) — `subagent_type: "general-purpose"`, `model: "sonnet"`, `team_name: "{team}"` - - Reviewer skill mapping (reviewer name → skill to load): - - `code-reviewer` → `code-reviewing` - - `security-auditor` → `security-auditor` - - `test-reviewer` → `test-master` - - `prompt-reviewer` → `prompt-master` - - `deploy-reviewer` → `deploy-pipeline` - - `infrastructure-reviewer` → `infrastructure-setup` - - `skill-checker` → `skill-master` - - `documentation-reviewer` → `documentation-writing` - - Prompt template: - - ``` - You are reviewer "{name}" for task {N}. - - Load your review methodology: Skill(skill="{reviewer_skill}") - Read specs: {feature_dir}/user-spec.md, {feature_dir}/tech-spec.md - Read task: {feature_dir}/tasks/{N}.md - - Wait for a message from teammate "{teammate_name}" with git diff of changes. - - When you receive it: - 1. Review changes following the loaded skill methodology - 2. Write JSON report to: {feature_dir}/logs/working/task-{N}/{reviewer_name}-round{round}.json - 3. Send report path to teammate "{teammate_name}" via SendMessage - - The teammate may send updated diffs for subsequent rounds (max 3). - Review each round the same way. After the final round, shut down. - ``` - -4. All agents work in parallel. Lead waits for teammates to report "Task complete." - -### Audit Wave tasks - -Audit Wave tasks (Code Audit, Security Audit, Test Audit) have `reviewers: none` — each auditor teammate IS the review. Spawn them as standard teammates (general-purpose, opus), each loads its methodology skill. - -Each auditor: -- Reads decisions.md to understand what was done in each task -- Reads all source files listed in tech-spec "Files to modify" across all implementation tasks -- Reviews the final state of code holistically (full files, not diffs) -- Writes report to `{feature_dir}/logs/working/audit/{auditor-name}.json` -- Writes decisions.md entry, reports to lead - -After all 3 reports: -- All clean → proceed to Final Wave -- Issues found → spawn a fixer teammate (ad-hoc, code-writing skill), assign the auditors who found issues as reviewers, standard review protocol (max 3 rounds). After approval → proceed to Final Wave. If unresolved after 3 rounds → escalate (see Escalation). - -### Ad-hoc agents - -When lead spawns an agent outside the original execution plan (to fix audit findings, handle escalations, complete missing work): - -1. Lead assigns a skill and reviewers matching the type of work: - - Code changes → skill: `code-writing`, reviewers: code-reviewer, security-auditor, test-reviewer - - Prompt changes → skill: `prompt-master`, reviewers: prompt-reviewer - - Skill changes → skill: `skill-master`, reviewers: skill-checker - - Deploy/CI changes → skill: `deploy-pipeline`, reviewers: deploy-reviewer - - Infrastructure changes → skill: `infrastructure-setup`, reviewers: infrastructure-reviewer, security-auditor - - Other tasks (research, config, manual steps) → no skill, no reviewers. Agent follows lead's instructions directly. -2. The ad-hoc agent writes a decisions.md entry (same template as planned tasks) -3. Standard review protocol: agent commits → sends diff to reviewers → fix → max 3 rounds -4. Lead verifies decisions.md entry exists before considering ad-hoc work complete - -**Checkpoint:** all teammates reported "Task complete", decisions.md entries written. - -## Phase 3: Wave Transition - -1. If task had Smoke/User verification steps — confirm teammate reported verification results. Missing results without explanation → ask user whether to proceed. -2. Update task frontmatter: `status: in_progress` → `status: done`. Read only frontmatter (`limit=15`), then Edit the status field. Do not read full task content. -3. Git commit: `chore: complete wave {N} — update task statuses and decisions`. Code is already committed by teammates. -4. Update `work/{feature}/logs/checkpoint.yml`: set `last_completed_wave`, update task statuses, set `next_wave`. -5. Next wave → Phase 2 - -**Checkpoint:** all wave tasks done, committed, checkpoint updated. - -## Phase 4: User Review - -All waves done including Final Wave (QA, deploy if applicable, post-deploy verification if applicable). - -1. Show results: what was built, key decisions, QA report summary -2. Describe what to check manually (from execution plan "user checks" section) -3. Issues found → spawn ad-hoc agent to fix (see "Ad-hoc agents" in Phase 2) → review → commit (max 3 rounds). If unresolved → escalate (see Escalation). -4. All ok → finalize, shutdown team, delete `work/{feature}/logs/checkpoint.yml` - -## Escalation - -Call user when: -- 3 review/fix iterations exhausted with remaining findings -- Teammate reports blocker or ambiguous requirement -- Task depends on unavailable MCP tool or external service - -When escalating: -1. Stop all work on the blocked task/wave -2. Report to user: what failed, what was tried (all 3 attempts), what remains unresolved -3. Write decisions.md entry: summary of attempts + unresolved findings -4. Git commit: `chore: escalate task {N} — unresolved after 3 fix rounds` -5. Wait for user decision before continuing - -## Self-Verification - -- [ ] Execution plan created and approved -- [ ] All tasks executed, reviewed where applicable (max 3 iterations each), decisions.md filled -- [ ] All waves committed (including Final Wave) -- [ ] User reviewed and approved diff --git a/infrastructure/ansible/files/claude-skills/skills/infrastructure_setup/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/infrastructure_setup/SKILL.md deleted file mode 100644 index 2140b361..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/infrastructure_setup/SKILL.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -name: infrastructure-setup -description: | - Sets up dev infrastructure for new projects: framework init, folder structure, - Docker, pre-commit hooks (gitleaks), testing infrastructure, .gitignore. - - Use when: "настрой инфраструктуру", "подготовь проект", "настрой тесты", - "настрой проверки при коммите", "настрой проверки при пуше", "setup infrastructure" ---- - -# Infrastructure Setup - -## Gathering Project Context - -Read project-knowledge references: -- `.claude/skills/project-knowledge/references/architecture.md` — tech stack, framework -- `.claude/skills/project-knowledge/references/patterns.md` — code conventions, branching strategy, testing -- `.claude/skills/project-knowledge/references/deployment.md` — deployment strategy - -If files lack needed info, search other project-knowledge references — info may exist under different names. If missing entirely, ask the user and immediately update the relevant doc. - -**Autonomous decisions** (based on project-knowledge): -- Framework init commands, folder structure, test framework, .gitignore patterns - -**Ask user:** -- Docker: needed? Local dev, production, or both? -- Pre-commit strictness: gitleaks only, or add lint/format? - -## Phase 1: Framework Initialization - -Init framework from `architecture.md`. Use Context7 for up-to-date init commands and flags. Verify it starts. - -**Checkpoint:** dev server starts successfully. - -## Phase 2: Folder Structure - -Convention — separate concerns by purpose: - -- **Web Apps:** `src/{components, services, lib, config}` + `tests/{unit, integration, e2e}` -- **APIs:** `src/{routes, services, models, middleware, config}` + `tests/{unit, integration}` -- **CLI tools:** `src/{commands, services, config}` + `tests/{unit, integration}` - -Add `src/prompts/` if project uses LLM prompts. Add `src/messages/` if project uses i18n. - -**Checkpoint:** structure created, matches project type. - -## Phase 3: Docker (conditional) - -Set up only if specified in project-knowledge or user confirms. - -**Checkpoint:** `docker build` succeeds, container starts. - -## Phase 4: .gitignore - -Security patterns (always add): -``` -.env -.env.* -!.env.example -*.key -*.pem -credentials.json -secrets/ -``` - -Add framework-specific patterns from `architecture.md`. -Create `.env.example` with required variable names (no values). - -**Checkpoint:** `git check-ignore .env` returns `.env`. - -## Phase 5: Pre-commit Hooks - -Convention: gitleaks for secret scanning. Target: total pre-commit time under 10 seconds. - -Pre-commit scope (fast, staged files only): -- gitleaks (~2-5 seconds) -- Lint staged files -- Format check - -Full test suites, integration tests, builds belong in CI. - -**Checkpoint:** commit a file containing `AKIA1234567890EXAMPLE` — gitleaks blocks it. - -## Phase 6: Testing Infrastructure - -Set up test framework, create smoke test: 1-2 tests verifying setup works (import main module, check environment). - -**Checkpoint:** test command passes. - -## Phase 7: Documentation & Commit - -Update project-knowledge references (append, don't overwrite): -- `deployment.md` — required environment variables -- `patterns.md` (Git Workflow section) — pre-commit hooks and what they check - -Commit: -``` -chore: setup project infrastructure - -- Initialize [framework] project -- Setup pre-commit hooks (gitleaks) -- Create folder structure -- Add testing infrastructure -- Configure .gitignore and .env.example -[- Setup Docker (if applicable)] -``` - -Verify before commit: `git status` shows no `.env` files (only `.env.example`). - -## Final Validation - -- [ ] Framework runs locally -- [ ] Folder structure matches convention -- [ ] gitleaks blocks test secret -- [ ] `.gitignore` covers `.env`, `*.key`, secrets -- [ ] `.env.example` exists (if project uses env vars) -- [ ] Smoke test passes -- [ ] Documentation updated -- [ ] All infrastructure committed -- [ ] Docker works (if applicable) diff --git a/infrastructure/ansible/files/claude-skills/skills/post_deploy_qa/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/post_deploy_qa/SKILL.md deleted file mode 100644 index f16039c1..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/post_deploy_qa/SKILL.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -name: post-deploy-qa -description: | - Post-deploy verification: execute AVP from tech-spec on live environment, - verify all acceptance criteria (user-spec + tech-spec), pick up deferred - criteria from pre-deploy QA report. Uses MCP tools (Telegram MCP, Playwright, curl, bash). - - Use when: "пост-деплой проверка", "post-deploy verification", "проверь после деплоя", - "MCP verification", "верификация на живом окружении", "проверь деплой", - "запусти AVP", "agent verification plan" ---- - -# Post-deploy QA - -## Input Requirements - -Read before starting: -- `user-spec.md` — acceptance criteria -- `tech-spec.md` — Agent Verification Plan (AVP) section + technical acceptance criteria -- `decisions.md` — deviations from plan (if exists) -- Pre-deploy QA report (`logs/working/qa-report.json`) — check `deferredToPostDeploy` section for criteria that could not be verified pre-deploy -- Confirmation that deploy is complete and environment is live - -If tech-spec has no AVP section — still proceed with acceptance criteria verification. - -## Verification Methodology - -Two verification directions (both required): - -### 1. Agent Verification Plan (AVP) - -Execute each step from AVP section in tech-spec: - -1. Read AVP section — it lists verification steps with expected outcomes and MCP tools to use -2. For each step: - - Use the specified MCP tool (Telegram MCP, Playwright, curl, bash, etc.) - - Perform the described action on live environment - - Compare result with expected outcome - - Record: tool used, step performed, result -3. If MCP tool is unavailable — mark step as `not_verifiable`, continue with remaining steps - -**Checkpoint:** All AVP steps executed or marked `not_verifiable`. Proceed to acceptance criteria. - -### 2. Acceptance Criteria Verification - -Verify all acceptance criteria from user-spec and tech-spec on live environment. This catches criteria that pre-deploy QA could not verify without a live system. - -1. Read pre-deploy QA report (`logs/working/qa-report.json`) — check `deferredToPostDeploy` section -2. For each deferred criterion — follow the verification steps specified in the pre-deploy report -3. Also re-check all acceptance criteria from user-spec and tech-spec against live behavior -4. For each criterion: - - **passed** — verified on live environment, evidence provided - - **failed** — live behavior does not meet the criterion - - **blocked** — cannot verify due to external conditions (no data, third-party service down). Provide a concrete manual verification plan for the user: what to check, when, how - -A `blocked` criterion requires user follow-up before the feature can count as fully verified. - -**Checkpoint:** All acceptance criteria verified, or marked `blocked` with manual verification plan. - -## Severity Classification - -- **critical** — verification step failed, live functionality broken, data integrity at risk -- **major** — works but with significant issues visible in production -- **minor** — cosmetic, non-functional discrepancies - -## Output Format - -Return findings as JSON. Reason: orchestrator parses this to decide pass/fail and log findings. - -```json -{ - "status": "passed | failed", - "summary": { - "totalSteps": 0, - "passed": 0, - "failed": 0, - "blocked": 0, - "notVerifiable": 0, - "criticals": 0, - "majors": 0, - "minors": 0 - }, - "agentVerification": [ - { - "step": "Send /start to bot", - "tool": "telegram_mcp", - "status": "passed | failed | not_verifiable", - "details": "Bot responded with welcome message" - } - ], - "acceptanceCriteria": [ - { - "id": "US-5", - "criterion": "Titles generated with correct declensions", - "source": "user-spec | tech-spec | deferred-from-pre-deploy", - "status": "passed | failed | blocked", - "evidence": "Checked live output, title 'В компании X работает...' uses correct declension", - "manualVerificationPlan": "Only if blocked — what the user should check, when, how" - } - ], - "findings": [ - { - "severity": "critical | major | minor", - "title": "Bot does not respond to /start", - "expected": "Welcome message within 3 seconds", - "actual": "No response after 10 seconds", - "reproduction": "Steps to reproduce..." - } - ] -} -``` - -Status decision: `passed` if zero criticals, `failed` if one or more criticals. - -## Guidelines - -- Every finding includes concrete reproduction: steps, expected vs actual, tool output. -- If all MCP tools are unavailable — report that automated verification is not possible, suggest manual steps. -- Empty findings array = clean verification. - -## Final Check - -Before finishing, verify: -- [ ] All AVP steps executed or marked `not_verifiable` -- [ ] All deferred criteria from pre-deploy QA report addressed -- [ ] All acceptance criteria from user-spec and tech-spec verified on live environment -- [ ] Every `blocked` criterion has a manual verification plan (what to check, when, how) -- [ ] Output JSON is valid, status reflects critical count diff --git a/infrastructure/ansible/files/claude-skills/skills/pre_deploy_qa/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/pre_deploy_qa/SKILL.md deleted file mode 100644 index 6d630b1f..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/pre_deploy_qa/SKILL.md +++ /dev/null @@ -1,154 +0,0 @@ ---- -name: pre-deploy-qa -description: | - Pre-deploy acceptance testing methodology: run test suite (unit/integration/E2E), - verify acceptance criteria from user-spec and tech-spec. Does not require live environment. - - Use when: "приёмочное тестирование", "pre-deploy qa", "проверь перед деплоем", - "run tests and check AC", "запусти qa", "проверь acceptance criteria", - "тестирование фичи", "qa", "проверь фичу" ---- - -# Pre-deploy QA - -## Input Requirements - -Read before starting: -- `user-spec.md` — acceptance criteria -- `tech-spec.md` — technical acceptance criteria -- `decisions.md` — deviations from plan (if exists) -- Project Knowledge — architecture.md, patterns.md (incl. Testing & Git Workflow sections) - -If user-spec or tech-spec missing — request before proceeding. - -## Verification Directions - -Three verification directions (order doesn't matter): - -### Test Suite - -Run all tests (unit, integration, E2E). All must pass. - -- Identify test runner from project config (package.json, pyproject.toml, Makefile, etc.) -- Run full test suite -- Record: total tests, passed, failed, skipped - -### Acceptance Criteria - -Check every criterion from user-spec and tech-spec: - -- **passed** — criterion met, evidence provided -- **failed** — the feature exists but does not meet the criterion -- **not_verifiable** — cannot be checked without live environment, external service, or MCP tool (scope of post-deploy-qa) - -For each criterion — provide evidence (test name, code path, log output). - -### Coverage Verification - -After test suite passes, verify that tests actually exercise the feature: - -- For each file in the feature's scope (from tech-spec "Files to modify"): verify a corresponding test exists. Feature code without any test → severity `critical` -- If project has coverage tooling configured (jest --coverage, pytest --cov, vitest --coverage) — run it. Coverage of feature files dropping below project threshold → severity `critical` -- For each acceptance criterion with status `passed` — verify the linked test actually exercises the relevant code path, not just an import check or mock-only test. Test that doesn't actually test the feature behavior → severity `major` -- Edge cases mentioned in user-spec (error handling, boundary values, empty states) — verify they have corresponding tests. Missing edge case test for M/L features → severity `major` - -## Severity Classification - -- **critical** — acceptance criterion failed, tests fail, core functionality broken -- **major** — works but with significant issues (edge cases, UX bugs, degraded behavior). Escalate to critical if it affects data integrity or core user workflow. -- **minor** — cosmetic, inaccuracies, improvements - -## Output - -### JSON report → `logs/working/qa-report.json` - -Full report saved to file. Reason: orchestrator parses this to decide pass/fail. - -```json -{ - "status": "passed | failed", - "summary": { - "totalChecks": 0, - "passed": 0, - "failed": 0, - "notVerifiable": 0, - "criticals": 0, - "majors": 0, - "minors": 0 - }, - "testSuite": { - "status": "passed | failed", - "details": "All 42 tests passed" - }, - "acceptanceCriteria": [ - { - "criterion": "User can login with email", - "status": "passed | failed | not_verifiable", - "evidence": "Test login_test.py::test_email_login passes" - } - ], - "findings": [ - { - "severity": "critical | major | minor", - "title": "Login fails for emails with + sign", - "expected": "Login succeeds", - "actual": "400 Bad Request", - "reproduction": "Steps to reproduce..." - } - ] -} -``` - -Status decision: `passed` if zero criticals, `failed` if one or more criticals. - -### decisions.md entry — concise summary only - -Write a brief entry to decisions.md following the template (`~/.claude/shared/work-templates/decisions.md.template`). Link to `logs/working/qa-report.json` for the full report. - -Example: -``` -## Task 9: Pre-deploy QA - -**Status:** Done -**Agent:** qa-runner -**Summary:** QA passed. 391 tests green, 28 acceptance criteria checked (25 passed, 3 not_verifiable). No blockers. -**Deviations:** Нет. - -**Verification:** -- Full report: [logs/working/qa-report.json] -``` - -## Guidelines - -- Work from specs only (user-spec, tech-spec, decisions.md). Task files (tasks/*.md) are already verified by reviewers and are outside QA scope. -- Account for decisions.md — deviations from original plan may be justified. -- Every finding includes concrete reproduction: steps, expected vs actual. -- Criteria requiring live environment or MCP tools — mark as `not_verifiable`, note that post-deploy verification is needed. -- Empty findings array = clean audit. - -### Deferred to Post-deploy - -If any acceptance criteria are marked `not_verifiable` — add a `deferredToPostDeploy` section to the JSON report. This section is the handoff contract: post-deploy QA reads it and verifies each deferred criterion on live environment. - -For each deferred criterion, specify: -- Which criterion (ID and text) -- Why it cannot be verified pre-deploy -- What conditions are needed to verify it (live data, MCP tool, user action) -- Concrete verification steps for post-deploy agent - -Example in JSON report: -```json -"deferredToPostDeploy": [ - { - "criterion": "US-5: Titles generated with correct declensions", - "reason": "Requires live LLM call with real data", - "verificationCondition": "New survey entry processed after deploy", - "verificationSteps": "Run a survey entry through the bot, check generated title for grammar and naturalness" - } -] -``` - -Also mention deferred criteria in the decisions.md entry: -``` -**Deferred to post-deploy:** 3 criteria require live verification (US-5, US-8, US-10). See deferredToPostDeploy in qa-report.json. -``` diff --git a/infrastructure/ansible/files/claude-skills/skills/project_planning/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/project_planning/SKILL.md deleted file mode 100644 index 47f88350..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/project_planning/SKILL.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -name: project-planning -description: | - Plan new projects: adaptive interview, tech decisions, - fill all project documentation (project-knowledge) in one session. - - Use when: "сделай описание проекта", "запиши описание проекта в документацию", - "проведи со мной интервью для описания проекта", "заполни документацию проекта", - "начни планирование проекта", "давай опишем проект", "plan a new project", - "fill project documentation" ---- - -# Project Planning - -Conduct adaptive interview → make tech decisions → fill all project documentation in one session. - -## Output Files - -**Project Knowledge** (`.claude/skills/project-knowledge/references/`): -- **project.md** — overview, audience, problem, key features, scope -- **architecture.md** — tech stack, project structure, dependencies, data model -- **patterns.md** — git workflow (code patterns, testing, business rules are filled later during development) -- **deployment.md** — platform, environment, CI/CD, monitoring -- **ux-guidelines.md** — only if project has significant UI - -## Interview Methodology - -**One question at a time.** Ask one question, wait for the answer, then form the next question based on the response. - -**Build on answers.** If user mentioned a domain — ask domain-relevant follow-ups. If they said something vague — clarify that specific point. - -**Confirm understanding.** After 3-5 questions, briefly summarize what you understood. Catches misunderstandings early. - -**Help when stuck.** When user says "not sure" or "don't know": -1. Say it's OK -2. Offer 2-3 common approaches for their type of project -3. Ask which is closer -4. If still uncertain and optional — mark TBD, move on -5. If still uncertain and required — break into simpler sub-questions - -**Recount on scope changes.** If user suddenly adds many features or reveals unexpected complexity — stop and recount total scope. Show the updated list, confirm you understood correctly. - -**If code exists.** Scan the codebase in parallel with the interview to pre-fill technical decisions and ask more targeted questions. - -## Phase 1: Project Discovery - -### 1.1 Interview - -Verify that project-knowledge directory and CLAUDE.md exist. If missing — tell user to run `/init-project` first. - -Ask user to describe the project in free form. Let them say as much or as little as they want. - -Then ask adaptive questions to cover three areas: - -**Project Overview:** -- What the project does (one-line + context) -- Who uses it and why (target audience + use case) -- What problem it solves (core pain point) -- 3-5 key features (high-level only) -- Scope boundaries (explicit exclusions) - -**Features & MVP:** -- Key features with descriptions -- What's included in MVP (launch scope) -- What comes later (post-launch ideas) — note these for the backlog -- Priority for each: Critical / Important / Nice-to-have - -**Development Approach:** -- All at once or phased? -- If phased: how to group features, what's MVP -- If migration: current system, data migration, risks, rollback plan - -### 1.2 Checkpoint - -Move to Phase 2 when you can: -- Write a clear, non-vague project.md -- List key features with priorities and MVP scope -- Describe the development approach - -TBD is acceptable for optional aspects. - -## Phase 2: Technical Decisions - -### 2.1 New Project (no code) - -1. **Propose tech stack** based on Phase 1: frontend, backend, database, key dependencies -2. **Verify choices** against current docs (Context7 if available). Update if you find deprecations or better alternatives. -3. **Propose deployment:** platform, CI/CD approach, environments -4. **Present proposal** to user with rationale for each choice. Iterate until user approves. - -### 2.2 Existing Code - -1. **Extract stack** from the codebase: package files, configs, directory structure -2. **Verify** against current docs (Context7 if available) -3. **Confirm with user:** show what you found, ask about gaps (deployment, missing pieces) -4. Iterate until confirmed. - -### 2.3 Checkpoint - -Move to Phase 3 when: -- Tech stack (frontend, backend, database, key dependencies) approved by user -- Deployment platform and CI/CD approach agreed -- No open questions on technical choices - -## Phase 3: Fill Documentation - -Documentation goal: someone opens these files and understands the project without reading code. Describe what exists, what it does, and why. Record decisions, operational details (server addresses, deploy procedures, log locations), high-level component overview. Write in prose, link to source files for code details. Each fact lives in one file only. - -Use Edit tool to replace template placeholders with real content. Content language: English. - -### 3.1 Project Knowledge Files - -**project.md** — from Phase 1 interview: -- Project overview, target audience, core problem -- Key features with priorities and MVP scope -- Post-launch ideas (if discussed) -- Out of scope - -**architecture.md** — from Phase 2 decisions + codebase analysis: -- Tech stack with "why" for each choice -- Project structure (directory tree) -- Key dependencies (only critical ones, not everything) -- External integrations -- Data flow -- Data model (fill if known, leave template sections if TBD) - -**patterns.md** — fill git workflow section: -- Branch structure (main + dev) -- Testing requirements -- Security gates (pre-commit, pre-push) -- Leave code patterns, testing methods, and business rules sections minimal — filled during development as patterns emerge - -**deployment.md** — from Phase 2 decisions: -- Platform, type, rationale -- Deployment triggers (what deploys where) -- Environments and URLs -- Environment variables (reference .env.example) -- Monitoring: fill if configured, note "not yet configured" if not - -**ux-guidelines.md** — only if project has significant UI. Skip entirely for CLIs, APIs, bots without custom UI. - -### 3.2 Backlog (if applicable) - -If post-launch features were discussed during the interview, offer to save them to a backlog. Ask user where to create the backlog file. - -### 3.3 Checkpoint - -All output files from "Output Files" section created. No template placeholders remain. - -## Phase 4: Review & Commit - -### 4.1 Self-Verify - -Before presenting to user, verify: -- project.md contains all key features discussed in interview -- architecture.md tech stack matches user-approved decisions from Phase 2 -- No template placeholders remain - -Fix any issues before proceeding. - -### 4.2 Documentation Review - -Run `documentation-reviewer` agent (Task tool, sonnet) on the project. Fix critical and major findings. Minor findings — fix or leave at your discretion. - -### 4.3 Show Files - -Show user the list of created files with links. Include ux-guidelines.md and backlog file if they were created. Ask if everything is correct or needs changes. - -### 4.4 Iterate - -- Changes requested → edit files → show updated list → repeat -- Questions → answer → continue waiting for approval -- Repeat until user approves - -### 4.5 Commit - -After approval, ask user if they want to commit. If yes — commit all created documentation files. - -Final message: "Документация заполнена! Можно начинать разработку." diff --git a/infrastructure/ansible/files/claude-skills/skills/prompt_master/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/prompt_master/SKILL.md deleted file mode 100644 index d290ac68..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/prompt_master/SKILL.md +++ /dev/null @@ -1,196 +0,0 @@ ---- -name: prompt-master -description: | - Guide for writing effective prompts for LLMs. - - Use when: "напиши промпт", "улучши промпт", "prompt engineering", "проверь промпт" ---- - -# Prompt Engineering for Reasoning Models - -Based on Anthropic and OpenAI guidelines (2025-2026). Every principle here has a motivation — when you understand WHY something works, you follow it more reliably. - -## Core Principles - -### The model is already smart - -Add only what the model lacks: domain context, constraints, success criteria. Every sentence should justify its token cost. The context window is a shared resource — prompts compete for attention with conversation history, tool outputs, and the model's own reasoning. - -### Clarity over cleverness - -Most prompt failures stem from ambiguity, not model limitations. Test: show the prompt to a colleague with no context. If they're confused about what to do, the model will be too. - -### Motivation over emphasis - -Explain WHY a rule matters. One motivated sentence outperforms ten capitalized words. - -When every instruction screams for attention (ALL CAPS, "CRITICAL", "NEVER", "ALWAYS", "MUST"), nothing stands out. Emphasis words signal a poorly written instruction — rewrite it instead of raising the volume. Over-thorough language ("Be THOROUGH", "Make sure you have the FULL picture") also hurts — it inflates token cost without adding signal. - -``` -Before: - CRITICAL: You MUST ALWAYS validate input. NEVER skip validation. - IMPORTANT: ALWAYS check for edge cases. This is MANDATORY. - -After: - Validate all input before processing. - Reason: unvalidated input causes pipeline crashes in production. -``` - -### Positive framing - -Default to stating what you want, not what to avoid. Models follow positive instructions more reliably. Long prohibition lists get ignored — they add tokens without adding clarity. - -``` -Rewrite when positive form is sufficient: - Before: Don't use bullet points. Never include code examples. - After: Write in prose paragraphs, 2-3 sentences each. - -Keep negatives for hard boundaries where positive rewrite loses the prohibition: - "Return data as JSON. Do not include markdown fences around it." - — positive alone ("Return raw JSON") may not prevent the common mistake. - -Test: does the positive rewrite fully convey the prohibition? - Yes → rewrite positively. - No → keep negative + add reason why it matters. -``` - -### Examples over rules - -1-3 canonical examples transfer knowledge more efficiently than paragraphs of description. Show the desired output — let the model generalize from the pattern. - -### Compress - -Remove filler ("could you please", "I would like you to", "make sure to"). Shorter prompts often perform equally well or better — less noise means stronger signal per token. - -### Degrees of freedom - -Match specificity to the task's fragility. Over-specifying creative tasks stifles the model's reasoning. Under-specifying fragile tasks leads to format errors and broken parsing. - -Fragile tasks (parsing specific formats, following exact protocols): prescribe steps. -Creative tasks (writing, analysis, design): give constraints and let the model find its path. - -### High-level guidance - -Provide constraints and success criteria rather than step-by-step micro-instructions. The model's own reasoning often exceeds prescriptive procedures. Describe WHAT success looks like — the model figures out HOW. - -## Techniques That Work - -### Context over decoration - -Provide concrete context (audience, use case, constraints) instead of decorative phrasing. The model gains nothing from flattery. - -``` -Before: - You are an incredibly brilliant and talented expert programmer - who writes the most amazing code in the world. Please write - a function that validates emails. - -After: - Write an email validation function. - Context: TypeScript, used in a signup form, must handle - international domains. Return {valid: boolean, reason: string}. -``` - -### Structure with XML tags - -Separate instructions, data, and examples with XML tags. This prevents the model from confusing context with instructions. Consistent tag names make handoffs between chained prompts clean. - -``` -Before: - Here's some customer feedback. Also, the format should be - JSON. And please analyze sentiment. The feedback is: - "Great product but shipping was slow." - -After: - - Analyze sentiment. Output JSON: {text, sentiment, confidence}. - - - Great product but shipping was slow. - -``` - -### Examples over explanations - -Instead of describing the desired output format in paragraphs, show 1-3 examples. The model generalizes from patterns faster than from rules. - -``` -Before: - When summarizing articles, create a summary that includes - the main topic as a short phrase, then 2-3 key points as - bullet items, then a one-sentence practical takeaway. - -After: - Summarize articles in this format: - - - Topic: Remote work productivity - - Async communication reduces meetings by 40% - - Written documentation improves onboarding speed - Takeaway: Teams that default to async work ship faster. - -``` - -### Prompt chaining over mega-prompts - -Break complex tasks into focused steps. Each step gets the model's full attention. A chain of 3 focused prompts outperforms one overloaded prompt — empirically shown to reduce error rates. - -``` -Before (single mega-prompt): - Analyze this contract for risks, then draft an email - to the vendor with concerns, then review for tone. - -After (3-step chain): - Step 1: Analyze for risks. Output in tags. - Step 2: Using , draft vendor email with proposed changes. - Step 3: Review for tone. Grade A-F with suggestions. -``` - -### Define success criteria - -Tell the model what good output looks like — not just what to do. Include format, length, audience, and evaluation criteria. - -``` -Before: - Write a good product description. - -After: - Write a product description for noise-canceling headphones. - Audience: Tech-savvy millennials on a comparison shopping site. - Format: 80-120 words, no superlatives, focus on specs and - use cases. End with one differentiating fact vs. competitors. -``` - -## Refinement - -How to improve a prompt that isn't working well: - -1. **Draft** a first version focused on context and success criteria -2. **Test** against 3-5 representative inputs (include edge cases) -3. **Observe** the output — collect specific problems, not vague impressions -4. **Diagnose** each problem: is it ambiguity? missing context? conflicting instructions? wrong format? -5. **Plan changes**: map each problem to a specific edit (what failed → why → what to change) -6. **Refine** with targeted edits — change one thing at a time, test each separately -7. **Metaprompt**: show the model a problematic output and ask "What in this prompt causes this behavior? Suggest a revision." - -### When to add instructions vs. examples - -| Problem | Fix | -|---------|-----| -| Model misunderstands the task | Add context (audience, use case, constraints) | -| Output missing required sections | Add an example showing all sections | -| Wrong format or style | Add 1-2 output examples | -| Model uses wrong context | Restructure with XML tags to separate data from instructions | -| Contradictory behavior | Audit prompt for conflicting rules and resolve them | - -## Quick Reference - -| Instead of | Do this | Why | -|------------|---------|-----| -| Emphasis words (CAPS, "NEVER", "ALWAYS") | One sentence explaining motivation | When everything screams, nothing stands out | -| Lists of prohibitions | State desired behavior; keep negatives only for hard boundaries + motivation | Positive framing is followed more reliably | -| "Act as a world-class expert..." | Provide context + success criteria | Flattery adds tokens without adding signal | -| One mega-prompt with many tasks | Chain of 3-4 focused prompts | Each step gets full model attention | -| Filler ("please", "make sure", "I want you to") | Direct instruction | Fewer tokens = less noise | -| Paragraphs describing format | 1-3 examples of desired output | Models generalize from examples faster than from rules | -| Step-by-step micro-instructions | Constraints + success criteria | Model's own reasoning exceeds prescriptive procedures | diff --git a/infrastructure/ansible/files/claude-skills/skills/security_auditor/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/security_auditor/SKILL.md deleted file mode 100644 index 2eb80e5a..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/security_auditor/SKILL.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -name: security-auditor -description: | - Comprehensive security analysis against OWASP Top 10 standards. - Use after code-reviewer for code handling: authentication, user input, database queries, external APIs. - - AUTOMATIC TRIGGER - Invoke when user says ANY of: - "проверь безопасность", "security audit", "найди уязвимости", "check security" - - Do NOT use for: general code review (use code-reviewer), testing (use test-reviewer) ---- - -# Security Auditor - -Elite security analysis with deep expertise in OWASP Top 10 and modern vulnerability assessment. - -## Core Responsibilities - -1. **Comprehensive Security Analysis**: - - SQL Injection (parameterized queries, ORM usage, raw SQL) - - Cross-Site Scripting (XSS) - stored, reflected, DOM-based - - Cross-Site Request Forgery (CSRF) protection - - Authentication (password storage, session management, MFA) - - Authorization and access control (RBAC, ABAC, privilege escalation) - - Input validation and sanitization (server-side validation, type checking) - - Cryptography (algorithms, key management, secure random) - - Dependency vulnerabilities (npm audit, outdated packages, CVEs) - - Rate limiting and DoS protection - - CORS configuration - - Security headers (CSP, HSTS, X-Frame-Options) - - Hardcoded secrets (API keys, tokens, passwords, connection strings in source code) - - SSRF (server-side request forgery — user-controlled URLs in server-side requests) - - Insecure design (missing threat modeling, business logic flaws) - - Software and data integrity (deserialization attacks, CI/CD integrity) - - Security logging and monitoring (audit trails, security event logging) - -2. **Risk Assessment** - Classify by severity: - - **Critical**: Immediate exploitation, severe impact (data breach, RCE) - - **High**: Significant risk requiring urgent attention (auth bypass, injection) - - **Medium**: Notable concerns needing timely fixes (weak crypto, missing headers) - - **Low**: Best practice violations (information disclosure) - -3. **Dependency Analysis**: npm audit (or equivalent), analyze: - - Direct and transitive dependency vulnerabilities - - Outdated packages with known security issues - - Recommended upgrade paths - -## Operational Protocol - -**Input Requirements**: -1. List of files to audit -2. User specifications (requirements, expected functionality) -3. Technical specifications (architecture, frameworks, dependencies) - -If any missing, request them before proceeding. - -**Analysis Methodology**: -1. Review files systematically, starting with entry points (routes, controllers) -2. Trace data flow from input to output, identifying trust boundaries -3. Check auth at each protected endpoint -4. Examine all database queries for injection -5. Analyze user input handling and output encoding -6. Review cryptographic implementations -7. Verify security headers and CORS policies -8. Run dependency vulnerability scans -9. Cross-reference with OWASP Top 10 - -**Quality Assurance**: -- Provide specific line numbers and code snippets -- Explain attack vector and potential impact -- Avoid false positives by understanding full context -- Consider defense-in-depth already in place - -## Guidelines - -- **Thorough But Precise**: No false positives, no missed real vulnerabilities -- **Context Matters**: Consider full application context -- **Prioritize Actionability**: Every finding must have implementable fix -- **Stay Current**: Reference OWASP Top 10 (2021+) and current CVE databases -- **Explain Impact**: Make risks concrete with realistic attack scenarios -- **Provide Examples**: Include secure code in recommendations -- **Dependencies First**: Always include npm audit results -- **No Assumptions**: Flag uncertain framework protections for manual review - -## Escalation - -Flag immediately: -- Critical vulnerabilities in production -- Signs of existing compromise or malicious code -- Systemic architecture issues requiring redesign -- Compliance violations (GDPR, PCI-DSS) diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_master/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/skill_master/SKILL.md deleted file mode 100644 index 2b58dbec..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_master/SKILL.md +++ /dev/null @@ -1,478 +0,0 @@ ---- -name: skill-master -description: | - Guide for creating/updating skills with specialized knowledge and workflows. - - Use when: "создай скилл", "измени скилл", "гайд по скиллам", "обнови скилл", "улучши скилл", - "create skill", "update skill", "skill guide", "new skill", "how to write a skill" ---- - -# Skill Creator - -## About Skills - -Skills are modular, self-contained packages that extend Claude's capabilities by providing specialized knowledge, workflows, and tools. Think of them as "onboarding guides" for specific domains or tasks—they transform Claude from a general-purpose agent into a specialized agent equipped with procedural knowledge that no model can fully possess. - -### What Skills Provide - -1. Specialized workflows - Multi-step procedures for specific domains -2. Tool integrations - Instructions for working with specific file formats or APIs -3. Domain expertise - Company-specific knowledge, schemas, business logic -4. Bundled resources - Scripts, references, and assets for complex and repetitive tasks - -## Skill Types - -There are two types of skills based on how they guide Claude's work. - -### Procedural Skills - -Use when the task requires a strict sequence of steps where order matters. Phase 2 depends on Phase 1 completing correctly. Skipping or reordering steps would break the workflow. - -Examples: code-writing (Plan → TDD → Review), project-planning (Interview → Features → Roadmap), tech-spec-planning. - -These skills have explicit phases with checkpoints after each phase to verify completion before proceeding. - -**Creating a procedural skill?** Read [procedural-skills.md](references/procedural-skills.md) — phase structure, checkpoints, verification patterns. - -### Informational Skills - -Use when providing methodology, knowledge, or guidelines without a strict execution order. The agent reads relevant sections and applies them to the situation. There's no "Phase 1 must complete before Phase 2" — sections are independent. - -Examples: security-auditor (what to check), testing (when to use which test type), company-info (domain knowledge), database-schemas. - -These skills organize content into logical sections with decision frameworks (YES if / NO if) to help the agent choose what applies. - -**Creating an informational skill?** Read [informational-skills.md](references/informational-skills.md) — section organization, knowledge structure. - -## 1. Discovery - -For new skills or major changes — run discovery interview: -- What problem does the skill solve? -- What phrases should trigger it? -- What should the skill NOT do? -- Concrete usage examples - -**When running user interview**, read [interview-guide.md](references/interview-guide.md) — process overview, example questions for each phase, handling "I don't know" answers. - -**Checkpoint:** Requirements gathered. Problem, triggers, scope, and examples documented. - -## 2. Skill Structure - -### Anatomy of a Skill - -Every skill consists of a required SKILL.md file and optional bundled resources: - -``` -skill-name/ -├── SKILL.md (required) -│ ├── YAML frontmatter metadata (required) -│ │ ├── name: (required) -│ │ └── description: (required) -│ └── Markdown instructions (required) -└── Bundled Resources (optional) - ├── scripts/ - Executable code (Python/Bash/etc.) - ├── references/ - Documentation intended to be loaded into context as needed - └── assets/ - Files used in output (templates, icons, fonts, etc.) -``` - -### Frontmatter - -**`name`** (required): -- kebab-case (lowercase, hyphens) -- ≤64 characters -- Unique identifier - -**`description`** (required): -- Third person ("Analyzes code...", NOT "I analyze...") -- Include both WHAT the skill does AND WHEN to use it -- ≤1024 characters - -#### Description Best Practices - -Claude uses description to decide when to auto-invoke the skill. Be specific and include key terms. - -**Template:** -```yaml -description: | - [What the skill does — be specific, include key terms] - - Use when: [trigger conditions — specific phrases users say] -``` - -**Rules:** -1. **Be specific** — Include key terms that match user requests -2. **List trigger phrases** — Real phrases users actually say (5-10 phrases) -3. **Include variations** — "техспек" AND "составь тз" (different ways to say same thing) - -**Bad:** -```yaml -description: This skill helps with documents. Use when user wants to work with docs. -``` -Why bad: Vague phrases ("work with docs"), no specific triggers. - -**Good:** -```yaml -description: | - Manage .claude/skills/project-knowledge/ docs: create, check, update. - - Use when: "заполни документацию", "создай документацию", "проверь документацию", "обнови документацию" -``` -Why good: Specific actions, concrete trigger phrases. - -**How to gather trigger phrases:** -1. Think: "What would I actually say to invoke this skill?" -2. Ask: "How would different users phrase this request?" -3. Include: Common typos, informal variants, both Russian and English if applicable - -#### Undertriggering Problem - -Claude tends to undertrigger skills — not use them when they'd be useful. To combat this, make descriptions slightly "pushy": explicitly list contexts and keywords that should activate the skill, even non-obvious ones. - -**Instead of:** -```yaml -description: How to build a dashboard to display data. -``` - -**Write:** -```yaml -description: | - How to build a dashboard to display data. Use this skill whenever - the user mentions dashboards, data visualization, internal metrics, - or wants to display any kind of data, even if they don't explicitly - ask for a "dashboard". -``` - -**Need argument-hint, disable-model-invocation, or model override?** Read [frontmatter-options.md](references/frontmatter-options.md) — optional fields and when to use each. - -### Body - -Every SKILL.md body consists of: -- **Core workflow** — main instructions that are always needed -- **Links to references** — for optional/detailed information -- Keep under 500 lines (otherwise → split to references) - -**When defining output format**, read [output-patterns.md](references/output-patterns.md) — template pattern, examples pattern. - -**Checkpoint:** SKILL.md created with frontmatter, body, and references. Skill structure complete. - -### Bundled Resources - -A skill contains only SKILL.md and these three optional directories — nothing else (no README, CHANGELOG, etc.). - -#### Scripts (`scripts/`) - -Executable code (Python/Bash/etc.) for tasks that require deterministic reliability or are repeatedly rewritten. - -- **When to include**: When the same code is being rewritten repeatedly or deterministic reliability is needed -- **Example**: `scripts/rotate_pdf.py` for PDF rotation tasks -- **Benefits**: Token efficient, deterministic, may be executed without loading into context -- **Note**: Scripts may still need to be read by Claude for patching or environment-specific adjustments - -**Concrete example:** When building a `pdf-editor` skill for queries like "Help me rotate this PDF": -1. Rotating a PDF requires re-writing the same code each time -2. A `scripts/rotate_pdf.py` script solves this — write once, execute many times - -**How to spot script candidates:** After running test cases, read the transcripts. If all test runs independently wrote similar helper code (e.g., each created a `create_docx.py`), that's a strong signal to bundle that script. Write once, use on every invocation. - -#### References (`references/`) - -Content needed in some execution paths, not all. If the skill branches (multiple operations, domains, modes) — each branch's details go to a reference. Content needed on every execution stays in SKILL.md. - -**Example:** Task-management skill handles "create" and "edit". Each operation's workflow → separate reference. Task file format used by both → stays in SKILL.md. - -- **No duplication**: Content lives in either SKILL.md or references, not both - -**How to link references in SKILL.md:** - -Embed references in workflow where they're logically needed. Two linking patterns, ranked by strength: - -**Pattern A: Action-embedded (strong)** — the workflow step's action IS applying the reference content. The agent cannot complete the step without loading the file. - -```markdown -3. Write tests following patterns from [testing-guide.md](references/testing-guide.md) - (test structure, naming, what to skip) - -4. Apply audit criteria from [principles.md](references/principles.md) to each file - (code examples, obvious content, generic explanations) -``` - -Why it works: "follow patterns from X" or "apply criteria from X" makes the reference part of the action, not a separate read-then-do instruction. - -**Pattern B: Condition + contents (basic)** — for optional references needed only in specific scenarios. Each link explains WHEN to read and WHAT's inside. - -```markdown -**For tracked changes**, see [REDLINING.md] — revision marks, accept/reject. -**First time with docx-js?** Read [DOCX-JS.md] — setup, examples, pitfalls. -``` - -Use Pattern A for references that contain rules/patterns the agent must follow during a step. Use Pattern B for references that are only relevant in certain branches of the workflow. - -**Anti-pattern: Resource catalog at end of file.** A passive list of references separated from the workflow. The agent reads the workflow top-down, gets instructions, and treats the catalog as optional appendix. - -```markdown -❌ Bad — passive catalog (ignored): -## Resources -### references/structure.md -Complete description of all files... -### references/principles.md -Quality principles... - -✅ Good — embed each reference into the workflow step where it's needed: -4. Apply audit criteria from [principles.md](references/principles.md) to each file -``` - -**Bad** (passive, no trigger): -- `Detailed guide: [X.md]` -- `See [X.md] for details` -- `Finance: [finance.md]` (no context why to read) - -**Good** (embedded in action or conditional): -- `3. Write tests following patterns from [testing-guide.md]` (action-embedded) -- `**Working with finance?** Read [finance.md] — P&L rules, ARR formulas` (conditional) -- `4. Apply criteria from [principles.md] to each file` (action-embedded) - -#### Assets (`assets/`) - -Files not intended to be loaded into context, but rather used within the output Claude produces. - -- **When to include**: When the skill needs files that will be used in the final output -- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates, `assets/frontend-template/` for HTML/React boilerplate -- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents that get copied or modified -- **Benefits**: Separates output resources from documentation, enables Claude to use files without loading them into context - -**Concrete example:** When designing a `frontend-webapp-builder` skill for queries like "Build me a todo app": -1. Writing a frontend webapp requires the same boilerplate HTML/React each time -2. An `assets/hello-world/` template with boilerplate project files solves this — copy and customize - -## 3. Writing Guidelines - -### Concise is Key - -The context window is a public good. Skills share the context window with everything else Claude needs: system prompt, conversation history, other Skills' metadata, and the actual user request. - -**Default assumption: Claude is already very smart.** Only add context Claude doesn't already have. Challenge each piece of information: "Does Claude really need this explanation?" and "Does this paragraph justify its token cost?" - -Prefer concise examples over verbose explanations. - -### Keep it Lean - -Remove things that aren't pulling their weight. After running test cases, read the transcripts — not just the final outputs. If the skill is making the model waste time doing unproductive things, remove those parts of the skill. - -Every instruction has a cost. If removing an instruction doesn't degrade results, it was dead weight. - -### Generalize, Don't Overfit - -Skills are used across many different prompts and contexts. When iterating on a skill based on test results, resist fiddly changes targeted at specific examples. Rather than oppressively constrictive rules, try branching out — use different metaphors, recommend different patterns of working. It's cheap to try and you might land on something better. - -If a skill works only for its test cases, it's useless at scale. - -### Degrees of Freedom - -Match the level of specificity to the task's fragility and variability: - -**High freedom (text-based instructions)**: Use when multiple approaches are valid, decisions depend on context, or heuristics guide the approach. - -**Medium freedom (pseudocode or scripts with parameters)**: Use when a preferred pattern exists, some variation is acceptable, or configuration affects behavior. - -**Low freedom (specific scripts, few parameters)**: Use when operations are fragile and error-prone, consistency is critical, or a specific sequence must be followed. - -Think of Claude as exploring a path: a narrow bridge with cliffs needs specific guardrails (low freedom), while an open field allows many routes (high freedom). - -### Progressive Disclosure - -Skills use a three-level loading system to manage context efficiently: - -1. **Metadata (name + description)** — Always in context (~100 words) -2. **SKILL.md body** — When skill triggers (<5k words) -3. **Bundled resources** — As needed by Claude (unlimited, scripts execute without reading) - -Keep SKILL.md body under 500 lines. Split content into separate files when approaching this limit. When splitting, reference them from SKILL.md and describe clearly when to read them. - -**Key principle:** When a skill supports multiple variations, frameworks, or options, keep only the core workflow and selection guidance in SKILL.md. Move variant-specific details into separate reference files. - -**Pattern 1: High-level guide with references** - -```markdown -# PDF Processing - -## Quick start -Extract text with pdfplumber: -[code example] - -## Advanced features - -**For form filling?** Read [FORMS.md](FORMS.md) — interactive fields, validation, PDF/A. - -For complete API reference, see [REFERENCE.md](REFERENCE.md) — all methods with examples. -``` - -Claude loads FORMS.md or REFERENCE.md only when needed. - -**Pattern 2: Domain-specific organization** - -For skills with multiple domains, organize content by domain: - -``` -bigquery-skill/ -├── SKILL.md (overview and navigation) -└── references/ - ├── finance.md (revenue, billing metrics) - ├── sales.md (opportunities, pipeline) - └── product.md (API usage, features) -``` - -In SKILL.md, link each domain with description: - -**When working with finance data**, read [finance.md](references/finance.md) — P&L rules, revenue calculations, ARR formulas. - -For sales data analysis, see [sales.md](references/sales.md) — opportunity stages, pipeline calculations, account hierarchies. - -**Working with product metrics?** Read [product.md](references/product.md) — API usage tracking, feature adoption, user segments. - -**Pattern 3: Conditional details** - -```markdown -# DOCX Processing - -## Creating documents -Use docx-js for basic operations. - -**First time with docx-js?** Read [DOCX-JS.md](DOCX-JS.md) — setup, examples, pitfalls. - -## Editing documents -For simple edits, modify XML directly. - -For tracked changes, see [REDLINING.md](REDLINING.md) — revision marks, accept/reject logic. -``` - -**Important guidelines:** -- Keep references one level deep from SKILL.md -- For files longer than 100 lines, include a table of contents at the top - -### Writing Approach - -Start by writing a draft, then look at it with fresh eyes and improve. Use theory of mind — make the skill general, not super-narrow to specific examples. Try to explain to the model why things are important in lieu of heavy-handed constraints. - -### Positive over Negative - -Default to positive instructions — they're followed more reliably. Rewrite negatives when the positive form fully conveys the meaning. - -**Rewrite when positive form is sufficient:** -- "Don't use bullet points" → "Write in prose paragraphs" -- "Don't use var" → "Use const/let" - -**Keep negatives for hard boundaries** where the positive rewrite loses the prohibition: -- Security: "Store secrets in .env" alone doesn't convey "never commit them to git" — you need both -- Irreversible damage: "Don't use `--force` on shared branches" — the cost of violation is high -- Disambiguation: "Use `Array.from()`, not spread for NodeList" — negative clarifies which similar option is wrong -- Scope limits: "This skill does not handle deployment" — defines boundary - -**Test:** "Does the positive rewrite fully convey the prohibition?" If no → keep the negative + add motivation (WHY it matters). - -### Explain the Why - -Today's LLMs are smart. They have good theory of mind and when given a good harness can go beyond rote instructions. Try to explain the **why** behind everything you're asking the model to do. Even if user feedback is terse, try to actually understand the task and why the user wrote what they wrote, then transmit this understanding into the instructions. - -**Bad:** "Always return JSON format." -**Good:** "Return findings as JSON — orchestrator parses this automatically, invalid JSON crashes pipeline." - -When explaining is impractical, keep the rule as-is. But default to reasoning over commanding. - -### Avoid Emphasis Words - -Words like CRITICAL, MANDATORY, NEVER, IMPORTANT, MUST are anti-patterns in skills. - -**Why they don't work:** -- Every instruction in a skill is already important — if it wasn't, it shouldn't be there -- When everything is emphasized — nothing stands out -- Emphasis words signal poorly written instructions that need rewriting, not shouting - -**What to do instead:** -- Write clear, specific instructions -- Explain why something matters (see "Explain the Why" above) -- Use structure (steps, checkpoints) to ensure compliance - -If you find yourself writing ALWAYS or NEVER in all caps, that's a yellow flag — reframe and explain the reasoning so that the model understands why the thing you're asking for is important. - -**Hard limit:** Maximum one emphasis word per skill. Ideal: zero. - -### Delegating Heavy Work - -If skill has context-heavy tasks (reviews, research, validation): -- Keep each skill focused on a single methodology -- Delegate heavy subtasks to agents with fresh context -- Orchestrator calls agents → they work isolated → return results - -**When to use subagents:** -- **Reviews** — code-reviewer, security-auditor, test-reviewer check work with fresh context -- **Research** — exploring codebase, reading documentation, searching information -- **Debugging** — isolated context for error diagnosis and root cause analysis -- **Validation** — checking schemas, formats, requirements compliance -- **Parallel tasks** — multiple independent investigations simultaneously -- **High-volume output** — tests, logs, reports that would bloat main context - -**Two approaches:** - -1. **Inline prompts** — for simple, one-off tasks (<50 lines): - ``` - Use general-purpose/explore/plan subagent to find all TypeScript files importing {module} - ``` - -2. **Skill + Agent pattern** — for complex, reusable tasks (>50 lines): - - **Skill** holds methodology (WHAT to do, HOW to analyze) - - **Agent** adds isolation + output format (runs in isolated context) - - Agent uses `skills:` field to preload methodology - - Reference by name: "Use `code-reviewer` agent" - -**Key principle:** Keep detailed agent prompts out of SKILL.md. Large prompts bloat the skill and waste context. Store specialized agent definitions separately; the skill just invokes them. - -**Delegating work to subagent?** Read [agents.md](references/agents.md) — inline prompts, dedicated agents, output contracts. - -**Checkpoint:** Writing guidelines applied. Skill is concise, well-structured, references linked properly. - -## 4. Validation - -### Run skill-checker - -After self-check — run validation: - -``` -Use skill-checker subagent to validate the skill at {path}. -If issues found → fix them → run skill-checker again. -``` - -skill-checker is defined in `~/.claude/agents/skill-checker.md` and has skill-master preloaded. - -### Test the Skill - -After creating or significantly updating a skill, suggest to the user to run skill-tester on it. skill-tester will design test cases, run them with and without the skill, test description triggering accuracy, and produce a report with specific improvement recommendations. - -### Self-Check Before Validation - -**Universal (all skills):** -- [ ] name in kebab-case, ≤64 chars -- [ ] description < 1024 chars, includes "Use when:" with trigger phrases -- [ ] SKILL.md < 500 lines -- [ ] All referenced files exist -- [ ] No extra docs (README, CHANGELOG) -- [ ] References contain only conditional content (not needed on every execution path) -- [ ] References linked as action steps or with condition + contents (no passive links, no resource catalogs at end of file) -- [ ] Defaults to positive instructions. Negatives only for hard boundaries (security, irreversible damage, disambiguation, scope limits) with motivation -- [ ] No emphasis words (CRITICAL, MANDATORY, NEVER) — max one allowed - -**Identify skill type:** procedural or informational? - -**If Procedural:** -- [ ] Has explicit phases with numbered steps -- [ ] Has checkpoints after each phase -- [ ] Has self-verification section at end -- [ ] Uses subagent verification for critical operations (if applicable) - -**If Informational:** -- [ ] Sections organized by logic, not sequence -- [ ] Decision frameworks present (YES if / NO if) where applicable -- [ ] No forced sequential structure - -**Functional (all skills):** -- [ ] Run skill-checker and fix all issues - diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/agents.md b/infrastructure/ansible/files/claude-skills/skills/skill_master/references/agents.md deleted file mode 100644 index 7a584870..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/agents.md +++ /dev/null @@ -1,291 +0,0 @@ -# Skill + Agent Pattern - -Subagents handle context-heavy subtasks for orchestrator skills. Each runs in isolated context, performs work, and returns results (or modifies files directly). - -## Why Subagents - -The orchestrator's context window is limited. Loading a skill, conversation history, and project context already consumes significant space. If the orchestrator opens many files, runs extensive analysis, or generates verbose output, context fills up and quality degrades. - -**Solution:** Delegate heavy work to subagents. Each runs in isolated context, performs its task, and returns a structured result. The orchestrator receives only what it needs. - -**Impact:** According to Anthropic research, multi-agent systems with Claude Opus orchestrator and Claude Sonnet subagents outperform single-agent Claude Opus by 90.2% on research tasks. - -## Orchestration Rules - -Subagents cannot call other subagents — Claude Code supports only one level of orchestration. Nested calls fail silently: - -``` -Orchestrator (main skill) - ├── code-reviewer (subagent) ✓ - ├── security-auditor (subagent) ✓ - └── test-reviewer (subagent) ✓ - -code-reviewer - └── another-agent ✗ FORBIDDEN -``` - -If subagent needs more work → return to orchestrator → orchestrator launches another subagent. - -## When to Use Subagents - -| Task Type | Why Subagent Helps | Example | -|-----------|-------------------|---------| -| Reviews | Fresh context for objective assessment | code-reviewer, security-auditor | -| Research | Extensive file reading stays isolated | Exploring codebase, reading docs | -| Debugging | Isolated diagnosis without polluting main context | Error analysis, root cause | -| Validation | Schema/format checking with clean slate | skill-checker, schema-validator | -| Parallel work | Multiple independent directions | Research 3 modules simultaneously | -| High-volume output | Tests, logs don't bloat main context | Running test suite, log analysis | - -## Inline Agents (Ad-hoc Tasks) - -For simple, one-off tasks — use Task tool with built-in subagent types: - -```markdown -Use Explore subagent to find all files related to authentication -Use general-purpose subagent to analyze the error and suggest fixes -Use Plan subagent to design implementation approach for {feature} -``` - -The orchestrator calls Task tool with arbitrary prompt and `subagent_type`. No agent file needed. - -**Built-in subagent types:** -- `Explore` — fast codebase exploration, file search, pattern matching -- `general-purpose` — flexible tasks, research, analysis -- `Plan` — designing implementation approaches - -**When to use:** -- Simple research/exploration -- One-off file operations -- Tasks under 50 lines of instructions -- No reuse needed - -## Dedicated Agents (Skill + Agent Pattern) - -For complex, reusable tasks — create **Skill + Agent pair**: - -1. **Skill** — holds methodology (WHAT to do, HOW to analyze) - - Usable inline via `/skill-name` - - Contains knowledge - -2. **Agent** — adds isolation + output contract - - Uses `skills:` to preload methodology - - Defines output: JSON, file changes, or actions - - Runs in isolated context - -**Example:** - -```yaml -# skills/code-reviewing/SKILL.md — methodology ---- -name: code-reviewing -description: Code review methodology and quality standards. ---- -## What to Check -- Architecture, error handling, edge cases... - -## Severity Levels -- Critical, Major, Minor... -``` - -```yaml -# agents/code-reviewer.md — isolation + format ---- -name: code-reviewer -description: Review code quality after implementation. -color: blue -skills: - - code-reviewing # Full SKILL.md content loaded -allowed-tools: Read, Glob, Grep ---- -Follow code-reviewing methodology. - -## Output -{ "findings": [...], "summary": {...} } -``` - -**Benefits:** -- Methodology usable inline (`/code-reviewing`) OR in isolation (via agent) -- Multiple agents can run in parallel -- No methodology duplication — skill is single source of truth -- Agent adds structure (output contract) without bloating skill - -## Agent File Format - -Agent files use YAML frontmatter + Markdown body. Store in `~/.claude/agents/{name}.md`. - -```yaml ---- -name: agent-name -description: | - When Claude should delegate to this agent. Include: - - Purpose and capabilities - - Example triggers - - What NOT to use it for -color: blue -skills: - - methodology-skill -allowed-tools: Read, Glob, Grep ---- - -# Agent Instructions - -## Input -[What the agent receives from the orchestrator] - -## Process -[Step-by-step methodology — or reference preloaded skill] - -## Output -[Output contract: JSON schema, file changes, or actions] -``` - -### Required Fields - -| Field | Description | -|-------|-------------| -| `name` | Unique identifier (kebab-case) | -| `description` | When/why to use — Claude reads this to decide delegation | -| `color` | Badge color for visual identification (see below) | -| `skills` | Skill(s) to preload — agent must have methodology from skill | - -### Color Recommendations - -All agents must have a color for visual identification. Valid values: `red`, `blue`, `green`, `yellow`, `purple`, `orange`, `pink`, `cyan`. - -| Color | Agent Type | -|-------|------------| -| blue/cyan | Analysis, review (code-reviewer, test-reviewer) | -| red | Security, critical (security-auditor) | -| yellow | Validation, caution (skill-checker, schema-validator) | -| green | Success-oriented, exploration (Explore) | -| purple/pink | Creative, generation, research | -| orange | Infrastructure, deployment | - -### Optional Fields - -| Field | Default | Description | -|-------|---------|-------------| -| `model` | `inherit` | Always use `inherit` to match orchestrator's model | -| `allowed-tools` | All tools | Restrict to necessary tools (e.g., `Read, Glob, Grep`) | -| `permissionMode` | `default` | Permission handling: `default`, `acceptEdits`, `bypassPermissions`, `plan` | -| `hooks` | None | Lifecycle hooks for validation | - -## Output Contracts - -Agents always return JSON report — even if they modify files or execute commands. Work is the process, output is the report. - -**Analysis agents** — findings and recommendations: -```json -{ - "status": "approved" | "changes_required", - "findings": [...], - "summary": "..." -} -``` - -**Executor agents** — report of changes made: -```json -{ - "status": "success" | "partial" | "failed", - "files_modified": ["path/to/file.ts", ...], - "files_created": ["path/to/new.ts", ...], - "summary": "Created 2 files, modified 3 files" -} -``` - -**Automation agents** — report of actions taken: -```json -{ - "status": "success" | "failed", - "actions": ["ran tests", "deployed to staging"], - "results": {...}, - "errors": [] -} -``` - -## Resuming Agents - -After agent completes, orchestrator receives `agentId`. Use it to continue work with same context: - -``` -Resume agent {agentId} to ask follow-up question about findings -``` - -**When to resume:** -- Need clarification on agent's findings -- Iterative refinement (agent found X, now do Y based on X) - -**When NOT to resume (start fresh):** -- Different task, unrelated to previous -- Context would confuse agent -- Previous work is complete, new work begins - -## Writing Effective Descriptions - -The `description` field is critical — Claude uses it to decide when to delegate. Include: - -1. **Purpose** — what the agent does -2. **Triggers** — when to use (with examples) -3. **Exclusions** — what NOT to use it for - -Example from `code-reviewer`: -```yaml -description: | - Use this agent when code has been written or modified and needs quality assessment. - - **Examples of when to use:** - - After implementing a feature - - After refactoring code - - Before committing changes - - **Proactive usage**: Invoke automatically after any code implementation task. -``` - -## Invoking from Skills - -Reference agents by name in skill workflow: - -```markdown -## Post-work - -1. **Run Reviews** (launch in parallel) - - `code-reviewer` — quality, architecture, patterns - - `security-auditor` — OWASP Top 10, vulnerabilities - -2. **Process Findings** - Evaluate each finding on merit — severity is metadata, not a filter. - - Valid, improves result → apply (any severity) - - Disagree or uncertain → discuss with user - Log each finding with action taken. -``` - -For agents needing specific input: - -```markdown -Use `code-reviewer` subagent with: -- files: {list of modified files} -- userspec: {user requirements document} -- techspec: {technical specifications} -``` - -## Best Practices - -1. **Define clear output contract** — JSON for analysis, file changes for executors -2. **Restrict tools** — Most agents only need `Read, Glob, Grep` -3. **Use `model: inherit`** — Ensures maximum quality from orchestrator's model -4. **Always preload skill** — Agent must have methodology, not just output format -5. **Include examples in description** — Helps Claude know when to invoke -6. **One level of orchestration** — Subagents cannot call other subagents - -## Example Agents - -See existing agents for full examples: -- `~/.claude/agents/code-reviewer.md` — Detailed methodology with review dimensions -- `~/.claude/agents/security-auditor.md` — OWASP-based security analysis -- `~/.claude/agents/skill-checker.md` — Skill validation against standards - -## References - -- [Create custom subagents - Claude Code Docs](https://code.claude.com/docs/en/sub-agents) -- [Multi-agent research system - Anthropic](https://www.anthropic.com/engineering/multi-agent-research-system) diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/frontmatter-options.md b/infrastructure/ansible/files/claude-skills/skills/skill_master/references/frontmatter-options.md deleted file mode 100644 index 1875dd6d..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/frontmatter-options.md +++ /dev/null @@ -1,86 +0,0 @@ -# Optional Frontmatter Fields - -These fields are NOT required for most skills. Use only when needed. - -## Field Reference - -| Field | Default | Description | -|-------|---------|-------------| -| `argument-hint` | None | Autocomplete hint shown after skill name | -| `disable-model-invocation` | `false` | If `true`, skill only triggers manually via `/skill-name` | -| `user-invocable` | `true` | If `false`, skill hidden from `/` menu, only Claude can invoke | -| `allowed-tools` | All tools | Restrict which tools the skill can use | -| `model` | `inherit` | Override model: `sonnet`, `opus`, `haiku`, `inherit` | - -## When to Use Each Field - -### argument-hint - -Shows hint in autocomplete to guide user input. - -```yaml ---- -name: fix-issue -argument-hint: "[issue-number]" ---- -``` - -User sees: `/fix-issue [issue-number]` - -### disable-model-invocation - -Prevents Claude from auto-triggering the skill. Only manual `/skill-name` works. - -```yaml ---- -name: dangerous-operation -disable-model-invocation: true ---- -``` - -Use for: destructive operations, expensive API calls, operations requiring explicit user consent. - -### user-invocable - -Hides skill from `/` menu. Only Claude can invoke it programmatically. - -```yaml ---- -name: internal-helper -user-invocable: false ---- -``` - -Use for: helper skills that shouldn't appear in user-facing menu, internal utilities. - -### allowed-tools - -Restricts which tools the skill can access. - -```yaml ---- -name: read-only-analyzer -allowed-tools: Read, Grep, Glob ---- -``` - -Use for: read-only analysis skills, security-conscious operations, preventing accidental edits. - -### model - -Overrides the model used for this skill. - -```yaml ---- -name: quick-lookup -model: haiku ---- -``` - -Options: -- `inherit` — use orchestrator's model (default, recommended) -- `sonnet` — fast, good for most tasks -- `opus` — best quality, use for complex reasoning -- `haiku` — fastest, use for simple lookups - -Use sparingly. `inherit` is usually best. diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/informational-skills.md b/infrastructure/ansible/files/claude-skills/skills/skill_master/references/informational-skills.md deleted file mode 100644 index 3dc75641..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/informational-skills.md +++ /dev/null @@ -1,93 +0,0 @@ -# Informational Skills - -Informational skills provide methodology, knowledge, or guidelines without strict execution order. The agent reads relevant sections and applies them to the current situation. - -## Section Organization - -Organize content by logical grouping, not by sequence. Each section should be independently useful — the agent may read any section based on what's relevant to the task. - -**Common section types:** -- **Core concepts** — what the skill covers, key definitions -- **Guidelines** — rules and principles to follow -- **Decision frameworks** — when to use what (YES if / NO if tables) -- **Escalation criteria** — when to flag issues or involve the user - -## Knowledge Structure Patterns - -### Pattern 1: Methodology Skill - -For skills that describe HOW to do something (analysis, review, audit): - -```markdown -# Security Auditor - -## Core Responsibilities -What to analyze: SQL injection, XSS, CSRF, authentication... - -## Risk Assessment -How to classify findings: Critical, High, Medium, Low - -## Operational Protocol -Input requirements, analysis methodology, quality assurance - -## Guidelines -Principles to follow during analysis - -## Escalation -When to flag immediately -``` - -See [security-auditor](../../security-auditor/SKILL.md) for a real example. - -### Pattern 2: Decision Guide Skill - -For skills that help choose between options: - -```markdown -# Testing Strategy - -## Overview -Test pyramid, when to use this skill - -## When to Use Each Test Type -Smoke tests → purpose, use cases -Unit tests → purpose, use cases -Integration tests → purpose, use cases -E2E tests → purpose, use cases - -## Decision Framework -Should I write unit tests? YES if / NO if -Should I write integration tests? YES if / NO if - -## Key Principles -Best practices that apply regardless of test type -``` - -See [test-master](../../test-master/SKILL.md) for a real example. - -### Pattern 3: Knowledge Container Skill - -For skills that provide domain-specific information (company info, schemas, APIs): - -```markdown -# Company Knowledge - -## About -Company overview, mission, key products - -## Domain Terms -Glossary of company-specific terminology - -## Key Systems -Internal systems and their purposes - -## Contacts -Who to ask about what -``` - -## Tips for Informational Skills - -1. **No forced sequence** — avoid "Step 1, Step 2" structure unless steps are truly dependent -2. **Self-contained sections** — each section should make sense on its own -3. **Decision frameworks** — use "YES if / NO if" tables when the agent needs to choose -4. **Link to references** — for detailed procedures, link to separate files rather than bloating SKILL.md diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/interview-guide.md b/infrastructure/ansible/files/claude-skills/skills/skill_master/references/interview-guide.md deleted file mode 100644 index 895adb78..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/interview-guide.md +++ /dev/null @@ -1,91 +0,0 @@ -# Skill Discovery Interview Guide - -Run this interview when creating a NEW skill. Skip for editing existing skills. - -## Process Overview - -1. Check for existing interview (resume if found) -2. Phase 1: Skill Overview (name, purpose, triggers, NOT-for) -3. Phase 2: Usage Scenarios (examples, edge cases, errors) -4. Phase 3: Output & Resources (format, bundled resources) -5. Proceed to skill creation with gathered info - -## Starting the Interview - -### Check for Existing Interview - -```bash -ls ~/.claude/tmp/interview-skill-*.yml 2>/dev/null -``` - -If found: -- Read file, show recap: "Нашёл незавершённое интервью по скиллу {name}" -- Ask: "Продолжить или начать заново?" -- If continue: resume from current state -- If restart: archive old file, create new - -### Create New Interview - -```bash -TIMESTAMP=$(date +%Y%m%d_%H%M%S) -cp ~/.claude/shared/interview-templates/skill.yml ~/.claude/tmp/interview-skill-$TIMESTAMP.yml -``` - -Set `interview_metadata.started` to current timestamp. - -## Iterative Interview Loop - -**Repeat for each phase:** - -1. **Find next gap:** Look at interview plan, find item with score < 70% -2. **Ask ONE question** about that gap -3. **Listen** to user's answer -4. **Update interview plan immediately:** - - Add to `conversation_history` - - Update `interview_metadata.last_updated` - - Update score, value, gaps, status for the item - - **SAVE the plan file** -5. **Check stop:** All required items >= 70%? → Move to next phase - -## Example Questions - -### Phase 1: Skill Overview - -- "Как называется скилл? Предложи описательное имя." -- "Какую проблему решает этот скилл? Зачем он нужен?" -- "Это пошаговый процесс с чёткой последовательностью (процедурный скилл) или набор знаний/методология без строгого порядка (информационный скилл)?" -- "Когда скилл должен активироваться? Какие фразы пользователя его триггерят?" -- "Чего скилл НЕ должен делать? Что выходит за рамки?" - -### Phase 2: Usage Scenarios - -- "Приведи 2-3 конкретных примера использования скилла." -- "Какие граничные случаи могут быть? Что если пользователь даст неполную информацию?" -- "Что может пойти не так? Как скилл должен обрабатывать ошибки?" - -### Phase 3: Output & Resources - -- "Что скилл должен производить в результате? Файлы, сообщения, действия?" -- "Нужны ли скиллу вспомогательные ресурсы: скрипты, референсы, ассеты?" -- "Какие внешние инструменты нужны? MCP серверы, API, CLI?" - -## Handling "Не знаю" - -If user doesn't know: -1. Explain why this matters -2. Offer 2-3 examples from similar skills -3. Ask which is closer to their situation -4. If still uncertain and optional: mark as TBD, move on -5. If still uncertain and required: break down into simpler questions - -## After Interview Complete - -Proceed to Step 2 (Planning Reusable Skill Contents) with gathered information. - -The interview plan file serves as the source of requirements for skill creation. - -## Cleanup - -After skill is created and user is satisfied: -- Delete interview file: `rm ~/.claude/tmp/interview-skill-*.yml` -- Or keep for audit trail (shows how requirements were gathered) diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/output-patterns.md b/infrastructure/ansible/files/claude-skills/skills/skill_master/references/output-patterns.md deleted file mode 100644 index d727778c..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/output-patterns.md +++ /dev/null @@ -1,82 +0,0 @@ -# Output Patterns - -Use these patterns when skills need to produce consistent, high-quality output. - -## Template Pattern - -Provide templates for output format. Match the level of strictness to your needs. - -**For strict requirements (like API responses or data formats):** - -```markdown -## Report structure - -Use this exact template structure (deviation breaks automated parsing): - -# [Analysis Title] - -## Executive summary -[One-paragraph overview of key findings] - -## Key findings -- Finding 1 with supporting data -- Finding 2 with supporting data -- Finding 3 with supporting data - -## Recommendations -1. Specific actionable recommendation -2. Specific actionable recommendation -``` - -**For flexible guidance (when adaptation is useful):** - -```markdown -## Report structure - -Here is a sensible default format, but use your best judgment: - -# [Analysis Title] - -## Executive summary -[Overview] - -## Key findings -[Adapt sections based on what you discover] - -## Recommendations -[Tailor to the specific context] - -Adjust sections as needed for the specific analysis type. -``` - -## Examples Pattern - -For skills where output quality depends on seeing examples, provide input/output pairs: - -```markdown -## Commit message format - -Generate commit messages following these examples: - -**Example 1:** -Input: Added user authentication with JWT tokens -Output: -``` -feat(auth): implement JWT-based authentication - -Add login endpoint and token validation middleware -``` - -**Example 2:** -Input: Fixed bug where dates displayed incorrectly in reports -Output: -``` -fix(reports): correct date formatting in timezone conversion - -Use UTC timestamps consistently across report generation -``` - -Follow this style: type(scope): brief description, then detailed explanation. -``` - -Examples help Claude understand the desired style and level of detail more clearly than descriptions alone. diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/procedural-skills.md b/infrastructure/ansible/files/claude-skills/skills/skill_master/references/procedural-skills.md deleted file mode 100644 index 85653710..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_master/references/procedural-skills.md +++ /dev/null @@ -1,83 +0,0 @@ -# Procedural Skills - -Procedural skills guide Claude through a strict sequence of phases where order matters. This reference covers patterns specific to procedural skills. - -## Explicit Steps + Phase Checkpoints - -"Analyze appropriately" — bad. Claude doesn't know what specifically to do. - -Break into explicit steps. For each phase — checkpoint: - -```markdown -## Phase 1: Preparation -1. Read requirements file -2. Check existing tests - -**Checkpoint:** Did I complete all steps? [List what was done] - -## Phase 2: Implementation -1. Write code -2. Add error handling - -**Checkpoint:** Did I complete all steps? [List what was done] -``` - -Checkpoints work because: -- Agent must verify before moving to next phase -- Creates pause points throughout, not just at end -- All phases are equal — no "attention drift" - -## Self-Verification at End - -Add self-check section at the end of skill: - -```markdown -## Final Check - -Before finishing, verify: -- [ ] All phases completed -- [ ] Output matches expected format -- [ ] No errors in generated files -``` - -This is the last checkpoint — agent verifies everything was done correctly. - -## Subagent Verification for Critical Operations - -Self-checks work for routine verification. But for **critical operations** where mistakes are costly, add subagent verification: - -**When to use:** -- Security-sensitive code (auth, user input, DB queries, APIs) -- Files that must follow strict standards (configs, schemas, contracts) -- High-impact changes (payment processing, data migrations) - -**Pattern:** - -```markdown -## Phase N: Verification - -1. **Run Reviews** (launch in parallel) - - `code-reviewer` — quality, architecture, patterns - - `security-auditor` — OWASP Top 10, vulnerabilities - -2. **Handle Findings** - - Agree → fix immediately - - Disagree → discuss with user before proceeding -``` - -**Why subagents work better than checklists for critical checks:** -- Fresh context — no "attention drift" from long conversations -- Specialized focus — agent examines only what it's designed to check -- Structured output — JSON findings are actionable, not vague - -**Include in Final Check:** If your skill uses verification subagents, add them to the self-verification checklist: - -```markdown -## Final Check -- [ ] Run `code-reviewer` and address findings -- [ ] Run `security-auditor` and address findings -``` - -This ensures verification agents are actually invoked, not skipped. - -**Creating verification agents?** Read [agents.md](agents.md) — dedicated agents, output contracts. diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_tester/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/skill_tester/SKILL.md deleted file mode 100644 index 0a5900ae..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_tester/SKILL.md +++ /dev/null @@ -1,288 +0,0 @@ ---- -name: skill-tester -description: | - Test skills end-to-end: design test cases, run with/without skill, grade results, - test description triggering accuracy, produce improvement report. - - Use when: "протестируй скилл", "запусти тесты для скилла", "проверь скилл", - "run skill tests", "test this skill", "skill eval", "оцени скилл", - "придумай тесты для скилла", "создай сценарии тестирования" ---- - -# Skill Tester - -Design tests, run them, grade results, evaluate description triggering, produce -actionable report. All in one workflow — no separate test-design step needed. - -You are team lead, test designer, user actor, and analyst — all in one. - -## Phase 1: Understand & Design - -### 1a. Read the target skill - -1. User provides skill name or path -2. Read the target skill's SKILL.md + ALL referenced files completely -3. Map out: - - Skill type: procedural / informational - - Input: what does the skill expect? (user message, task file, structured data) - - Output: what should the skill produce? (files, messages, actions, decisions) - - Phases: list all phases/steps with their checkpoints - - References: list all files the skill tells agents to read - - Decision points: where does the skill branch based on input? - - Dialogue points: where does the skill ask the user questions? - -### 1b. Design test prompts - -Design test prompts applying criteria from [test-design-guide.md](references/test-design-guide.md) -(realistic prompts, assertion design, persona setup): - -1. Propose 2-3 test prompts: - - 1 happy-path: the most common, standard use case - - 1-2 edge cases: where the skill might break or behave unexpectedly -2. For each prompt, propose assertions — binary, observable checks. - Categories: - - **[Process]**: Did the agent follow the skill's workflow? - - **[Outcome]**: Is the result correct and complete? - - **[Compliance]**: Did the agent obey all skill instructions? - -### 1c. Design trigger eval queries - -Design trigger eval queries applying patterns from [trigger-eval-guide.md](references/trigger-eval-guide.md): - -1. Generate 15-20 trigger eval queries: - - 8-10 should-trigger: varied phrasings of the skill's intended use - - 8-10 should-not-trigger: near-misses that share keywords but need - something different -2. These will be used in Phase 4 to test description accuracy - -### 1d. Confirm with user - -Present the full test plan: -- Test prompts with assertions -- Trigger eval queries -- Proposed model for runners -- Persona (use default, modify only if user requests) - -"Here are the test cases and trigger queries I've prepared. Do these look -right, or do you want to adjust anything?" - -Wait for confirmation before proceeding. - -**Checkpoint:** User confirmed test plan. All prompts have assertions. -Trigger eval queries prepared. - -## Phase 2: Execute Tests - -### 2a. Setup - -1. Create workspace: `~/.claude/skill-tests/{skill-name}/iteration-{N}/` - (N = 1 for first run, increment for re-runs) -2. Save test plan to workspace: - - `evals.json` with all prompts and assertions - - `trigger-evals.json` with all trigger queries -3. TeamCreate(team_name="skill-test-{skill-name}") -4. Plan runners: per scenario = 2 with-skill + 1 baseline without skill - -Show plan to user: "I'll run {N} scenarios, {M} runners total. -Model: {model}. Proceed?" - -### 2b. Spawn runners - -For each scenario, spawn all runners in parallel: - -**With-skill runners (2 per scenario):** -- Prompt = scenario's task prompt (natural, as user would write) -- Each runner loads the tested skill: `Skill(skill="{tested-skill-name}")` -- Model: as confirmed with user -- Use `run_in_background: true` - -**Baseline runner (1 per scenario):** -- Same task prompt, same model -- Receives no skill to load -- Use `run_in_background: true` - -Save each runner's task_id — needed for grader agents to retrieve transcripts. - -Scenarios run sequentially. Runners within a scenario run in parallel. - -### 2c. Interact as user persona - -If runners send questions, answer in character per the scenario's persona. -Rules: -- Stay in character: answer as the user would -- Be consistent: same question from different runners → same answer -- Answer naturally — without guidance toward any specific behavior -- Keep conversation purely about the task itself -- Baseline runner may ask different questions (no skill to guide it) — - this is expected, answer them too - -### 2d. Capture timing data - -When each runner completes, immediately save timing data: - -```json -{ - "total_tokens": 84852, - "duration_ms": 23332, - "total_duration_seconds": 23.3 -} -``` - -This is the only opportunity to capture this data — it comes through the -task notification and isn't persisted elsewhere. Process each notification -as it arrives rather than trying to batch them. - -Save to `timing.json` in each runner's result directory. - -**Checkpoint:** All runners completed. Timing data captured. - -## Phase 3: Grade & Analyze - -### 3a. Grade via grader agents - -When all runners for a scenario finish, spawn grader agents — one per runner. -Delegate transcript analysis to grader agents — transcripts are large and -reading them directly would exhaust the lead agent's context, leaving no room -for report compilation. - -Each grader receives instructions from -[grading-guide.md](references/grading-guide.md) and: - -1. The runner's task_id (grader calls `TaskOutput(task_id)` for transcript) -2. The scenario's assertions (copy the criteria list into the prompt) -3. The skill's SKILL.md path (grader reads it for compliance check) -4. Whether this is a skill-runner or baseline - -Spawn all graders in parallel. Wait for all to return. - -### 3b. Compile results per scenario - -Using only grader outputs (not transcripts): - -1. Build results table (assertions × runners) -2. Cross-runner consistency: where did skill-runners diverge? - - Divergence on a criterion = ambiguous instruction in the skill -3. Baseline comparison: - - Passed by skill-runners ONLY → skill adds value - - Passed by ALL → criterion too easy or skill doesn't help here - - Failed by ALL → criterion may be unrealistic - - Passed by baseline ONLY → skill might be harmful for this case - -Clean up runners for this scenario before moving to the next one. - -### 3c. Benchmark aggregation - -Across all scenarios, compute: - -1. **Pass rate** per assertion per config (with-skill / baseline) -2. **Timing comparison**: tokens and duration per config -3. **Overall skill value**: how many assertions improve vs baseline - -### 3d. Analyst pass - -Surface patterns the aggregate stats might hide: - -- **Non-discriminating assertions**: pass regardless of whether skill is - used. These don't prove the skill helps — consider removing or replacing - with harder assertions. -- **High-variance assertions**: one skill-runner passes, other fails on - same criterion. Usually means the skill's instruction is ambiguous — - identify the specific instruction and quote it. -- **Time/token tradeoffs**: skill adds value but costs 2x tokens? Flag it. - The user should know the cost of improvement. -- **Repeated code in transcripts**: if multiple runners independently wrote - similar helper scripts, flag this as a candidate for bundling in - the skill's `scripts/` directory. - -**Checkpoint:** All scenarios graded. Benchmark computed. Analyst observations -recorded. - -## Phase 4: Test Description Triggering - -### 4a. Evaluate trigger accuracy - -For each trigger eval query from `trigger-evals.json`: -- Assess whether the skill's current description would cause Claude to - invoke the skill for this query -- Consider: does the query's intent match the description's keywords and - contexts? Would Claude see this as the skill's domain? - -Categorize each query: -- **True positive**: should-trigger → would trigger -- **True negative**: should-not-trigger → would not trigger -- **False negative**: should-trigger → would NOT trigger (undertriggering) -- **False positive**: should-not-trigger → would trigger (overtriggering) - -### 4b. Calculate trigger accuracy - -``` -Trigger accuracy = (true positives + true negatives) / total queries -False negative rate = false negatives / should-trigger queries -False positive rate = false positives / should-not-trigger queries -``` - -False negatives (undertriggering) are the most costly — users won't discover -the skill exists. False positives waste time but are less harmful. - -### 4c. Suggest improved description - -If trigger accuracy < 85% or false negative rate > 20%: -- Analyze which queries fail and why -- Draft an improved description that would trigger correctly -- Show before/after comparison in the report - -**Checkpoint:** Trigger accuracy calculated. Description improvement -suggested if needed. - -## Phase 5: Report - -Structure the report according to -[report-template.md](references/report-template.md). - -The report includes: -1. **Results per scenario** — assertions × runners table with evidence -2. **Skill compliance** — phase-by-phase execution check -3. **Benchmark summary** — pass_rate, tokens, time per config -4. **Analyst observations** — non-discriminating, high-variance, cost analysis -5. **Description trigger accuracy** — accuracy metrics + suggested improvement -6. **Scripts to bundle** — if repeated code found across transcripts -7. **Recommendations** — priority-ordered specific fixes for skill-master - -Save to: `~/.claude/skill-tests/{skill-name}/reports/{timestamp}-report.md` - -Show report to user: "Here's the test report. Key findings: [summary]. -The report is at [path] — you can share it with skill-master to apply fixes." - -TeamDelete after report delivery. - -## Improving the Skill (Iteration) - -If the user wants to iterate after receiving the report: - -1. User (or skill-master) applies fixes to the skill -2. Run skill-tester again → results go to `iteration-{N+1}/` -3. Previous iteration results are available for comparison -4. Report shows delta: what improved, what regressed - -When iterating, keep these principles in mind: -- **Generalize from feedback**: resist fiddly changes targeted at specific - test cases. If a skill works only for its test cases, it's useless at scale. -- **Keep the prompt lean**: read transcripts. If the skill makes the model - waste time doing unproductive things, remove those parts. -- **Explain the why**: rather than adding rigid ALWAYS/NEVER rules, explain - reasoning so the model understands the intent. - -## Self-Verification - -- [ ] Target skill fully read (SKILL.md + all references) -- [ ] All scenarios executed (2 skill-runners + 1 baseline each) -- [ ] Grader agents used for transcript analysis (not read by lead directly) -- [ ] Every assertion graded with cited evidence from tool call transcripts -- [ ] Skill compliance checked per runner (phase-by-phase) -- [ ] Baseline comparison completed per assertion -- [ ] Benchmark aggregated (pass_rate, tokens, time) -- [ ] Analyst pass completed (non-discriminating, high-variance, cost, repeated code) -- [ ] Trigger eval queries tested against description -- [ ] Description improvement suggested if accuracy < 85% -- [ ] Report saved and shown to user -- [ ] Team deleted after report delivery diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/grading-guide.md b/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/grading-guide.md deleted file mode 100644 index e0b415ab..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/grading-guide.md +++ /dev/null @@ -1,120 +0,0 @@ -# Grading Guide - -Instructions for grader agents that evaluate test run transcripts. - -## Grader Agent Prompt Template - -When spawning a grader, include these instructions in the prompt: - ---- - -You are a grader evaluating a test run of a Claude Code skill. Your job is -to determine whether each assertion passed or failed based on evidence from -the transcript. - -### Input - -1. **Runner task_id**: Call `TaskOutput(task_id)` to get the full transcript - with every tool call (Read, Grep, Write, WebFetch, Bash, Skill, etc.) -2. **Assertions to check**: [list provided by skill-tester] -3. **Skill path**: Read the skill's SKILL.md to understand expected behavior -4. **Runner type**: skill-runner or baseline - -### Grading Rules - -- **PASS** requires clear evidence: a specific tool call, file content, or - message. Quote it directly. No inference, no "probably did it." -- **FAIL** when no evidence found, evidence contradicts the assertion, or - only surface compliance (correct format but wrong substance). -- **When uncertain: FAIL.** Burden of proof is on the assertion. If you - can't find clear evidence, it didn't happen. - -### Evidence Types by Category - -**[Process] assertions** — cite specific tool calls with arguments: -- "Tool call #3: WebFetch(url='https://...')" → PASS -- "No WebFetch calls found in transcript" → FAIL - -**[Outcome] assertions** — cite file content (read the created files): -- "File vault/note.md contains frontmatter: {type: 'note', tags: [...]}" → PASS -- "File exists but missing 'tags' field in frontmatter" → FAIL - -**[Compliance] assertions** — cite sequence of actions: -- "Phase 1 completed (calls #1-#5), checkpoint message at #6, Phase 2 - started at #7" → PASS -- "Jumped directly to implementation at call #2, skipped Phase 1" → FAIL - -### Skill Compliance Check - -Beyond individual assertions, check overall skill compliance: - -1. **Phase execution**: For procedural skills, did the agent follow phases - in order? List each phase with YES/NO and evidence. -2. **References read**: List each reference file the skill mentions. Did the - agent read it? Cite the Read tool call. -3. **Checkpoints hit**: Did the agent pause at checkpoints? Cite messages. - -### Output Format - -Return this exact structure: - -``` -## Assertion Results - -| # | Assertion | Category | Verdict | Evidence | -|---|-----------|----------|---------|----------| -| 1 | [text] | Process | PASS | "Tool call #3: WebFetch(url='...')" | -| 2 | [text] | Outcome | FAIL | "File created but missing required field 'tags'" | - -## Skill Compliance - -| Phase | Executed | Evidence | -|-------|----------|----------| -| 1. Preparation | YES | calls #1-#5 | -| 2. Implementation | YES | calls #7-#15 | -| 3. Self-Review | NO | no review messages found | - -## References Read - -- patterns.md: YES (Read call #4) -- architecture.md: NO (never read) - -## Files Created - -- path/to/file.md — contents: {summary of key fields/structure} - -## Summary - -Pass rate: X/Y assertions passed. -Key issues: [list any notable failures or concerns] -``` - ---- - -## Grader Spawning - -Spawn one grader per runner. For a scenario with 2 skill-runners + 1 baseline, -spawn 3 graders in parallel. - -Each grader runs in isolated context — it reads only its assigned runner's -transcript. This prevents cross-contamination between evaluations. - -## Processing Grader Results - -After all graders return, the skill-tester compiles results WITHOUT reading -transcripts. Use only grader outputs to: - -1. Build the cross-runner results table -2. Identify divergences between skill-runners -3. Compare skill-runners vs baseline -4. Feed into benchmark aggregation - -### Grading for Programmatic Assertions - -Some assertions can be checked programmatically — file exists, field present, -test passes. For these, write and run a script rather than relying on the -grader's judgment. Scripts are faster, more reliable, and can be reused -across iterations. - -Example: "Output file contains valid JSON with field 'name'" -→ Write a quick script that reads the file, parses JSON, checks field. diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/report-template.md b/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/report-template.md deleted file mode 100644 index 878af8be..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/report-template.md +++ /dev/null @@ -1,204 +0,0 @@ -# Report Template - -## Structure - -```markdown -# Skill Test Report: {skill-name} - -**Date:** {date} -**Skill path:** {path} -**Model:** {model} -**Iteration:** {N} -**Scenarios:** {count} ({M} happy-path + {K} edge-cases) -**Runners per scenario:** 2 with skill + 1 baseline - ---- - -## Scenario: {name} ({type}) - -**Task:** {prompt text or summary} -**Persona modifications:** {none / list of changes} - -### Results Table - -| # | Assertion | Category | Runner 1 | Runner 2 | Baseline | Evidence | -|---|-----------|----------|----------|----------|----------|----------| -| 1 | Asked about stack | Process | PASS | PASS | FAIL | R1: "Какой стек?" msg #3; R2: "Технологии?" msg #2; BL: did not ask | -| 2 | Loaded patterns.md | Compliance | PASS | PASS | FAIL | R1: Read call for patterns.md; R2: same; BL: skipped | -| 3 | Tests before code | Process | PASS | FAIL | FAIL | R1: test.py #7 → api.py #9; R2: api.py #5 → test.py #8 (wrong order) | - -### Skill Compliance - -| Phase | Runner 1 | Runner 2 | Baseline | -|-------|----------|----------|----------| -| 1. Preparation | YES | YES | skipped | -| 2. TDD | YES | partial | skipped | -| 3. Implementation | YES | YES | YES | -| 4. Self-Review | YES | NO | NO | - -References read: -- patterns.md: R1 yes, R2 yes, BL no -- architecture.md: R1 yes, R2 no, BL no - -### Cross-Runner Consistency - -Runner 1 and Runner 2 diverged on assertion #3 (TDD order) and phase 4 -(Self-Review). This suggests the skill's TDD instruction may be ambiguous. -Specifically: [quote the ambiguous instruction from the skill]. - -### Baseline Comparison - -Assertions passed ONLY by skill-runners (skill adds value): -- #1 (asked about stack), #2 (loaded patterns) - -Assertions passed by ALL (skill doesn't help): -- none - -Assertions failed by ALL: -- none - -Assertions passed by baseline ONLY (skill might be harmful): -- none - ---- - -(repeat for each scenario) - ---- - -## Benchmark Summary - -### Pass Rates - -| Config | Pass Rate | Assertions Passed | -|--------|-----------|-------------------| -| With skill (avg) | 85% ± 7% | 17/20 | -| Baseline | 40% | 8/20 | -| **Delta** | **+45%** | **+9** | - -### Timing - -| Config | Avg Tokens | Avg Duration | -|--------|-----------|--------------| -| With skill | 84,000 ± 12,000 | 45s ± 8s | -| Baseline | 52,000 | 28s | -| **Delta** | **+32,000 (+62%)** | **+17s (+61%)** | - -### Per-Assertion Breakdown - -| Assertion | With Skill | Baseline | Discriminating? | -|-----------|-----------|----------|-----------------| -| Asked about stack | 100% | 0% | YES — skill's key value | -| Tests pass | 100% | 100% | NO — passes without skill too | -| Loaded patterns.md | 100% | 0% | YES | - -## Analyst Observations - -### Non-Discriminating Assertions -- "Tests pass" passes regardless of skill. This doesn't prove the skill helps. - Consider replacing with a harder assertion or removing. - -### High-Variance Assertions -- "Tests before code" (assertion #3): Runner 1 passes, Runner 2 fails. - The skill's TDD instruction at line 45 says "Write tests first" — possibly - too vague. Consider: "Write test file BEFORE creating implementation file. - Verify test file timestamp is earlier." - -### Time/Token Tradeoffs -- Skill adds +45% pass rate but costs +62% tokens. Acceptable tradeoff for - a coding skill where correctness matters more than speed. - -### Repeated Code in Transcripts -- All 3 runners wrote similar `setup_db.py` helper scripts. Consider - bundling this as `scripts/setup_db.py` in the skill. - -## Description Trigger Accuracy - -**Current description:** -``` -{current description text} -``` - -**Trigger accuracy:** {X}% ({N}/{M} correct) -**False negative rate:** {Y}% ({K} queries that should trigger but wouldn't) -**False positive rate:** {Z}% ({J} queries that shouldn't trigger but would) - -### Failed Queries - -| # | Query | Expected | Actual | Why | -|---|-------|----------|--------|-----| -| 3 | "нужно визуализировать данные..." | trigger | no trigger | Missing "visualization" keyword | -| 14 | "explain how git branches work" | no trigger | trigger | Description too broad | - -### Suggested Description - -```yaml -description: | - {improved description} -``` - -Changes from current: -- Added: {what was added} -- Removed: {what was removed} -- Expected new accuracy: ~{X}% - -## Recommendations - -Priority-ordered list of fixes for skill-master: - -1. **[High]** Fix ambiguous TDD instruction - - Where: SKILL.md line 45 - - Before: "Write tests first" - - After: "Create test file BEFORE implementation file" - - Why: Runners diverged — R1 passed, R2 failed (assertion #3) - -2. **[High]** Update description for better triggering - - Where: SKILL.md frontmatter - - Before: "{old description}" - - After: "{new description}" - - Why: Trigger accuracy {X}%, false negative rate {Y}% - -3. **[Medium]** Bundle repeated helper script - - What: `setup_db.py` written independently by all runners - - Action: Create `scripts/setup_db.py` in skill directory - - Why: Saves tokens on every invocation - -4. **[Low]** Remove non-discriminating assertion from future tests - - What: "Tests pass" passes with and without skill - - Action: Replace with harder assertion or drop - - Why: Doesn't test skill value - -## Verdict: {Ready / Needs Fixes / Broken} - -**Ready** — all key assertions pass consistently, skill adds clear value, -description triggers accurately. - -**Needs Fixes** — some assertions fail or description undertriggers, but -fixable with specific changes listed above. - -**Broken** — fundamental issues, major rewrite needed. -``` - -## Iteration Comparison (iteration 2+) - -When this is not the first iteration, add a delta section: - -```markdown -## Changes Since Iteration {N-1} - -### Fixes Applied -- [list what was changed in the skill] - -### Impact -| Metric | Iteration {N-1} | Iteration {N} | Delta | -|--------|----------------|---------------|-------| -| Pass rate (with skill) | 70% | 85% | +15% | -| Trigger accuracy | 65% | 90% | +25% | -| Tokens (avg) | 90,000 | 84,000 | -6,000 | - -### Regressions -- [any assertions that got worse — flag these] - -### Still Failing -- [assertions that failed before and still fail — needs different approach] -``` diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/test-design-guide.md b/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/test-design-guide.md deleted file mode 100644 index 47d7a642..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/test-design-guide.md +++ /dev/null @@ -1,157 +0,0 @@ -# Test Design Guide - -How to create effective test prompts and assertions for skill testing. - -## Writing Test Prompts - -### Realism is Everything - -Test prompts must be realistic — exactly what a real user would actually type. -Not abstract requests, but concrete and specific with enough detail to feel real. - -**Bad prompts** (too abstract, test nothing): -- "Format this data" -- "Extract text from PDF" -- "Create a chart" -- "Help me with code" - -**Good prompts** (realistic, detailed, natural): -- "ok so my boss just sent me this xlsx file (its in my downloads, called - something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add - a column that shows the profit margin as a percentage. The revenue is in - column C and costs are in column D i think" -- "Хочу добавить авторизацию через Google в мобильное приложение. Сейчас - у нас React Native, бэкенд на FastAPI. Нужно чтобы юзер мог логиниться - через гугл и мы сохраняли его профиль" -- "у меня проект на next.js, нужно добавить страницу /settings где юзер - может поменять email и пароль. бд — postgres через prisma" - -### What Makes a Prompt Realistic - -- **File paths, names, context**: mention real-looking files, columns, values -- **Personal backstory**: brief context about the user's situation -- **Mix of formality**: some formal, some casual with typos and abbreviations -- **Various lengths**: short direct requests and longer ones with context -- **Non-obvious skill use**: cases where user doesn't name the skill but - clearly needs it - -### Prompt Categories - -1. **Happy path** (1 per skill): The most common, standard use case. - This is what 80% of users will ask for. - -2. **Edge cases** (1-2 per skill): Where the skill might break: - - Ambiguous user input ("ну сделай что-нибудь") - - Missing context (no project files available) - - Contradictory requirements - - Unusually large or small scope - - Input that triggers rarely-used branches - - User changing mind mid-dialogue - -### Prompt Types by Skill Type - -**One-shot skill** (task-manager): -"Поставь задачу на завтра: купить продукты в 10:00, уведомление за 30 минут." - -**Coding skill** (code-writing): -"Реализуй задачу: ~/.claude/skill-tests/code-writing/scenarios/task-1.md" - -**Dialogue skill** (user-spec-planning): -"Хочу добавить авторизацию через Google в мобильное приложение." - -**Informational skill** (methodology): -"Как правильно организовать работу с ветками в git?" - -## Writing Assertions - -### What Good Assertions Look Like - -Each assertion must be: -- **Binary**: pass or fail, no "partially" -- **Observable**: checkable from agent's messages, tool calls, and created files -- **Specific**: no "did a good job", no "output is correct" -- **Skill-focused**: tests skill behavior, not general agent quality -- **Descriptively named**: reads clearly so someone glancing at results - immediately understands what each one checks - -### Assertion Categories - -**[Process]** — Did the agent follow the skill's workflow? -- "Asked about tech stack before writing code" -- "Loaded reference file patterns.md" -- "Ran tests before implementation (TDD order)" - -**[Outcome]** — Is the result correct and complete? -- "Created file contains required frontmatter fields" -- "Tests pass when executed" -- "Output matches expected format" - -**[Compliance]** — Did the agent obey skill instructions? -- "Followed all phases in order" -- "Performed self-review at end" -- "Used subagent for code review" - -### Assertion Quality - -Good assertions are objectively verifiable. Subjective quality ("good writing -style", "clean code") is better evaluated qualitatively by the user — don't -force assertions onto things that need human judgment. - -The number depends on skill complexity — simple skills may have 5, -complex procedural skills can have 15+. - -## Persona - -Default persona (do not change unless user explicitly asks): - -> Предприниматель, занимается vibe-coding через Claude Code. Не программист — -> не знает синтаксис, библиотеки, алгоритмы. Есть техническое образование, -> понимает продукты и архитектуру на уровне "что делает что". Общается прямо, -> без воды. - -**Edge-case persona modifications** (only for edge-case scenarios): -- Даёт противоречивые ответы -- Меняет требования в середине -- Отвечает "ну сделай как-нибудь" даже на продуктовые вопросы -- Даёт размытые, неконкретные ответы -- Перескакивает между темами - -## evals.json Format - -```json -{ - "skill_name": "example-skill", - "evals": [ - { - "id": 1, - "name": "descriptive-name-here", - "type": "happy-path", - "prompt": "User's task prompt — natural, as user would type", - "expected_output": "Description of expected result", - "files": [], - "persona_modifications": [], - "model": "opus", - "assertions": [ - { - "id": "process-1", - "category": "Process", - "text": "Agent asked about tech stack before writing code", - "verification": "Check message history for tech stack question" - }, - { - "id": "outcome-1", - "category": "Outcome", - "text": "Created file contains required frontmatter fields", - "verification": "Read created file, check for name, description fields" - }, - { - "id": "compliance-1", - "category": "Compliance", - "text": "Agent followed all phases in order", - "verification": "Check message sequence matches skill phase order" - } - ] - } - ] -} -``` diff --git a/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/trigger-eval-guide.md b/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/trigger-eval-guide.md deleted file mode 100644 index 579ca784..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/skill_tester/references/trigger-eval-guide.md +++ /dev/null @@ -1,139 +0,0 @@ -# Trigger Eval Guide - -How to design queries that test whether a skill's description triggers correctly. - -## Why This Matters - -The description field in SKILL.md frontmatter is the primary mechanism that -determines whether Claude invokes a skill. Claude sees the skill's name + -description in its available_skills list and decides whether to consult it. - -Claude tends to **undertrigger** skills — not use them when they'd be useful. -This means false negatives (skill should trigger but doesn't) are the most -costly failure mode: users won't discover the skill exists. - -## Designing Trigger Eval Queries - -### Should-Trigger Queries (8-10) - -Think about **coverage** — different phrasings of the same intent: -- Some formal, some casual -- Cases where user doesn't explicitly name the skill or file type but - clearly needs it -- Uncommon use cases that still fall within the skill's domain -- Cases where this skill competes with another but should win -- Different levels of detail (brief request vs. detailed backstory) -- Different languages if applicable (Russian + English) - -### Should-NOT-Trigger Queries (8-10) - -The most valuable are **near-misses** — queries that share keywords or -concepts with the skill but actually need something different: -- Adjacent domains -- Ambiguous phrasing where naive keyword match would trigger but shouldn't -- Cases touching on something the skill does but in a context where another - tool/skill is more appropriate - -Avoid obviously irrelevant queries — "Write a fibonacci function" as a -negative test for a PDF skill is too easy. It doesn't test anything. The -negative cases should be genuinely tricky. - -### Query Quality - -Queries must be realistic — something a real user would actually type: -- File paths, personal context, column names, company names, URLs -- Some backstory -- Mix of lowercase, abbreviations, typos, casual speech -- Different lengths -- Concrete and specific, not abstract - -**Bad query:** -``` -"Format this data" -``` - -**Good query:** -``` -"ok so my boss just sent me this xlsx file (its in my downloads, called -something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add -a column that shows the profit margin as a percentage" -``` - -### Substantive Queries - -Skills trigger for tasks Claude can't easily handle on its own. Simple, -one-step queries like "read this PDF" may not trigger a skill even if the -description matches perfectly — Claude handles them directly with basic tools. - -Complex, multi-step, or specialized queries reliably trigger skills when the -description matches. So eval queries should be substantive enough that Claude -would actually benefit from consulting a skill. - -## trigger-evals.json Format - -```json -[ - { - "id": 1, - "query": "the user prompt — realistic, detailed", - "should_trigger": true, - "rationale": "Why this should/shouldn't trigger the skill" - } -] -``` - -## Evaluating Trigger Accuracy - -### Assessment Method - -For each query, assess whether the skill's current description would cause -Claude to invoke it. Consider: -- Does the query's intent match the description's keywords? -- Would Claude see this as the skill's domain based on description alone? -- Is the query substantive enough to warrant a skill consultation? - -### Metrics - -``` -Trigger accuracy = (true positives + true negatives) / total queries -False negative rate = false negatives / should-trigger queries -False positive rate = false positives / should-not-trigger queries -``` - -**Target: ≥85% trigger accuracy, ≤20% false negative rate.** - -### Improving the Description - -When accuracy is below target, analyze failure patterns: - -1. **False negatives cluster** — missing keywords? Add them. - The description should explicitly list contexts that activate the skill. -2. **False positives cluster** — description too broad? Add scope limits. - "This skill does NOT handle X" can help disambiguate. -3. **Mixed failures** — description may need restructuring. Try the - "pushy" approach: list specific scenarios explicitly rather than - relying on general terms. - -### Before/After Format - -When suggesting an improved description, show: - -``` -BEFORE: -description: | - [current description] - -AFTER: -description: | - [improved description] - -Changes: -- Added: [keywords/contexts added] -- Removed: [overly broad terms removed] -- Clarified: [disambiguations added] - -Expected impact: -- False negatives fixed: queries #3, #7, #9 -- False positives fixed: query #14 -- New accuracy: ~95% (was 70%) -``` diff --git a/infrastructure/ansible/files/claude-skills/skills/task_decomposition/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/task_decomposition/SKILL.md deleted file mode 100644 index 6fa04787..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/task_decomposition/SKILL.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -name: task-decomposition -description: | - Decompose approved tech-spec into atomic task files with parallel creation and validation. - - Use when: "разбей на задачи", "декомпозиция", "decompose tech-spec", - "создай задачи из техспека", "/decompose-tech-spec" ---- - -# Task Decomposition - -Decompose tech-spec Implementation Tasks into individual task files with parallel creation and validation. - -**Input:** `work/{feature}/tech-spec.md` (status: approved) -**Output:** `work/{feature}/tasks/*.md` (validated) -**Language:** Task files in English, communication in Russian - -## Phase 1: Create Tasks - -1. Ask user for feature name if not provided. - -2. Read `work/{feature}/tech-spec.md`. Check frontmatter `status: approved`. - If not approved — tell user: "tech-spec не утверждён. Сначала запусти `/new-tech-spec` и доведи до approved." Stop. - -3. Read `work/{feature}/user-spec.md`. - -4. Note the task template path: `~/.claude/shared/work-templates/tasks/task.md.template` - -5. Read skills/reviewers catalog from [skills-and-reviewers.md](~/.claude/skills/tech-spec-planning/references/skills-and-reviewers.md) — for passing correct skills/reviewers to task-creators. - -6. For each task in Implementation Tasks — launch [`task-creator`](~/.claude/agents/task-creator.md) subagent in parallel. - Pass each task-creator: - - feature_path, task_number, task_name - - template_path: `~/.claude/shared/work-templates/tasks/task.md.template` - - files_to_modify, files_to_read (from tech-spec) - - depends_on, wave, skills, reviewers, verify (from tech-spec) - - teammate_name (if specified in tech-spec, optional) - Each task-creator copies the template to `tasks/{N}.md` first, then edits each section in place. This ensures no sections are skipped. - -7. Confirm each task-creator returned a file path. Skip reading task content — preserve context budget for validation phase. -8. Git commit: `draft(tasks): create {N} tasks from tech-spec for {feature}` - -**Checkpoint:** -- [ ] All `tasks/*.md` files created -- [ ] Each task-creator returned file path -- [ ] Draft committed - -## Phase 2: Validation (up to 3 iterations) - -Tech-spec was already validated by 6 validators. This phase checks only: (1) task-creator correctly expanded tasks by template, (2) no mismatches with real code appeared during detailing. - -### Validators - -Launch both in parallel: - -[`task-validator`](~/.claude/agents/task-validator.md) (sonnet) — Template Compliance + AC/TDD carry-forward: -- Batch: 5 tasks per call -- Pass: feature_path, task_numbers array, batch_number, iteration -- Report: `logs/tasks/template-batch{N}-review.json` - -[`reality-checker`](~/.claude/agents/reality-checker.md) (sonnet) — Reality & Adequacy: -- Batch: 3 tasks per call -- Pass: feature_path, task_numbers array, batch_number, iteration -- Report: `logs/tasks/reality-batch{N}-review.json` - -### Process - -1. Launch both validators in parallel (task-validator in batches of 5, reality-checker in batches of 3). -2. Read JSON reports, collect findings. -3. If issues found — for each task with issues, launch [`task-creator`](~/.claude/agents/task-creator.md) in fix mode: - - Pass: same inputs as creation + `mode: fix` + `findings` from validators - - task-creator reads existing task, applies fixes, overwrites file -4. After each validation round, git commit: `chore(tasks): validation round {N} — {summary}` -5. Re-validate fixed tasks (repeat 1-4). Maximum 3 iterations. -6. If problems remain after 3rd iteration — show user: "Вот что осталось — давай решим вместе." - -### Cross-Task Integration Check - -After individual validation passes, run a final cross-task check: - -1. Launch both validators on ALL tasks in a single batch (not split into smaller batches): - - `task-validator` — focus: shared resource ownership (one owner, consumers depend_on owner), no competing instances in same wave - - `reality-checker` — focus: duplicate heavy resource init, hidden dependencies, inconsistent approaches across tasks - -2. If issues found → launch `task-creator` in fix mode for affected tasks. Re-validate fixed tasks. - -3. Max 2 iterations for cross-task check (on top of the 3 individual iterations). - -**Checkpoint:** -- [ ] Both validators: status=approved OR user resolved remaining issues -- [ ] Cross-task integration check: no cross-task conflicts - -## Phase 3: Present to User - -1. Summary: task count, waves, dependencies, validation results (iterations, issues found/fixed). -2. Wait for user approval. -3. Git commit: `chore(tasks): task decomposition approved for {feature}` -4. Suggest next step: `/do-task` for individual tasks. - -**Checkpoint:** -- [ ] Summary presented to user -- [ ] User approved task decomposition -- [ ] Approval committed - -## Final Check - -- [ ] All phases completed (tasks created, validation passed) -- [ ] All tasks match template (frontmatter: status, depends_on, wave, skills, reviewers, teammate_name) -- [ ] Validation: both validators passed or user confirmed remaining issues diff --git a/infrastructure/ansible/files/claude-skills/skills/tech_spec_planning/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/tech_spec_planning/SKILL.md deleted file mode 100644 index 4b3d3f58..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/tech_spec_planning/SKILL.md +++ /dev/null @@ -1,185 +0,0 @@ ---- -name: tech-spec-planning -description: | - Creates tech-spec.md with architecture, decisions, testing strategy, and implementation plan. - - Use when: "сделай техспек", "составь техспек", "техническая спецификация", - "tech spec", "создай тз", "составь тз", "new-tech-spec", "/new-tech-spec" - - Requires existing user-spec.md as input (create with user-spec-planning skill first if missing). ---- - -# Tech Spec Planning - -Create technical specification through code research, adaptive clarification, and multi-validator review. - -**Input:** `work/{feature}/user-spec.md` + Project Knowledge -**Output:** `work/{feature}/tech-spec.md` (approved) -**Language:** Technical documentation in English, communication in Russian - -## Phase 1: Load Context - -1. Ask user for feature name if not provided. Check `work/{feature}/` exists, create if needed. - -2. Read `work/{feature}/user-spec.md`. If missing — ask user to describe the task or create user-spec first. - Extract `size: S|M|L` from user-spec frontmatter — it determines testing strategy depth in tech-spec. - -3. Read all files in `.claude/skills/project-knowledge/references/` (project.md, architecture.md, patterns.md, deployment.md, ux-guidelines.md, and any custom domain files). Missing files are fine — not all projects have all guides. - -4. user-spec.md is the single input source — all information from interview.yml and code research is already consolidated there. - -**Checkpoint:** -- [ ] Feature folder exists -- [ ] user-spec.md read, size extracted -- [ ] Project Knowledge read - -## Phase 2: Code Research - -Launch `code-researcher` subagent (Task tool, opus) with feature path and user-spec path. The agent reads existing `code-research.md` (from user-spec phase if available) and deepens analysis for implementation. - -After subagent completes — read `{feature_path}/code-research.md`. Use in Phase 3 clarification and Phase 4 spec writing. - -If during later phases a gap is discovered — launch `code-researcher` again with the specific question. - -**Checkpoint:** -- [ ] code-research.md created/updated with implementation-level analysis -- [ ] Research file read by orchestrator - -## Phase 3: Clarification (Adaptive) - -Analyze if additional information is needed based on user-spec and code research. - -- Ask technical questions if gaps exist. No limit on question count — ask as many as needed. -- Focus: technical constraints, integration points, data sources, external dependencies. -- If gaps found in user-spec requirements — discuss with user and update user-spec too (via subagent or directly). -- If requirements are fundamentally unclear — suggest creating user-spec first. - -**Checkpoint:** -- [ ] All technical gaps clarified (or none existed) - -## Phase 4: Create tech-spec - -1. Copy template to feature folder: - ```bash - cp ~/.claude/shared/work-templates/tech-spec.md.template work/{feature}/tech-spec.md - ``` - Then edit sections one by one using Edit tool. This keeps template structure and examples visible while you work. - -2. Fill frontmatter: - - `created`: today's date - - `status`: draft - - `size`: copy from user-spec (S|M|L) - - `branch`: `dev` - -3. Fill all template sections. The template defines section structure — follow it directly. - In Architecture → Shared Resources: list heavy resources (ML models, DB pools, browser instances, API clients) shared across components. Specify owner (who creates), consumers, instance count. If none — write "None". - - **User-spec anchoring:** Every decision in the Decisions section must reference a user-spec requirement it serves (e.g., "Supports US-3: push notifications"). If a decision is purely technical (not derived from any user-spec requirement) — mark it `[TECHNICAL]` with justification. If a decision contradicts or changes a user-spec requirement — document it in the User-Spec Deviations section and mark as `[PENDING USER APPROVAL]`. All deviations must be documented explicitly — this preserves the user's original intent for review. - -4. Fill Implementation Tasks by waves. For each task provide: Description, Skill, Reviewers, Verify-smoke (optional), Verify-user (optional), Files to modify, Files to read. Select skill and reviewers from [skills-and-reviewers.md](references/skills-and-reviewers.md) (execution skills catalog, reviewer agents, default mappings). - - For each task, write `Verify-smoke:` when the task involves: - - External API integration → curl/httpie command to real endpoint with expected response - - Library/model initialization → `python -c` or import check that verifies setup - - Docker/infrastructure → `docker compose build`, `docker run` commands - - LLM/prompt work → spawn agent with prompt + test question, check response - - External service API (OpenRouter, Stripe, etc.) → test API call with expected response - - MCP-verifiable UI/frontend → use Playwright MCP or similar to check rendered page - Write `Verify-user:` when user should check something: UI on localhost, behavior, UX. - Omit both if task is purely internal logic covered by unit tests. - - **Task brevity rules:** - - Tasks are brief scope descriptions (2-3 sentences). Detailed steps, AC, and TDD anchors are created during task-decomposition phase. - - Task Description answers WHAT and WHY, not HOW. No step-by-step instructions, no line numbers, no implementation details. - - All technical decisions belong in the Decisions section, not in task descriptions. If you're writing a decision rationale inside a task — move it to Decisions. - -5. The last two waves are always **Audit Wave** and **Final Wave**, in that order: - - **Audit Wave** (always present) — 3 tasks running in parallel, `reviewers: none`: - - **Code Audit** (skill: `code-reviewing`) — holistic code quality review of all feature code - - **Security Audit** (skill: `security-auditor`) — OWASP Top 10 across all components - - **Test Audit** (skill: `test-master`) — test quality and coverage across all components - - Auditors read all source files from the feature and write reports (analysis only). If issues found — feature-execution lead spawns a fixer agent, auditors become reviewers for the fix. - - **Final Wave:** - - **QA** (skill: `pre-deploy-qa`) — always present. Acceptance testing: run all tests, verify acceptance criteria from user-spec and tech-spec. - - **Deploy** (skill: `deploy-pipeline`) — only if deploy is needed for this feature. - - **Post-deploy verification** (skill: `post-deploy-qa`) — only if live-environment checks are needed (MCP tools listed in Agent Verification Plan → Tools required). - QA is mandatory. Deploy and post-deploy — if applicable. - -6. Fill User-Spec Deviations section. For each element in tech-spec that changes, extends, or contradicts user-spec — add an entry with the requirement ID, what user-spec says, what tech-spec does differently, and why. Mark each entry `[PENDING USER APPROVAL]`. If no deviations — write "None". - -7. Task Count Check: if >15 tasks — propose splitting into MVP + Extension phases. Wait for user decision. - -8. Git commit: `draft(techspec): create tech-spec for {feature}` - -**Checkpoint:** -- [ ] tech-spec.md created in work/{feature}/ with all sections -- [ ] Implementation Tasks include Description (2-3 sentences), skill, reviewers for each task -- [ ] No AC or TDD anchors in tasks (those come from task-decomposition phase) -- [ ] Technical decisions are in Decisions section, not in task descriptions -- [ ] Every Decision references a user-spec requirement or is marked `[TECHNICAL]` -- [ ] User-Spec Deviations section filled (or "None") -- [ ] Final Wave present with QA (mandatory) + Deploy/Post-deploy (if applicable) -- [ ] Task count ≤15 (or user approved larger scope) - -## Phase 5: Validation - -### Run 5 validators in parallel - -Launch all as subagents, each writes JSON report to `logs/techspec/{name}-review.json`: - -| Validator | Agent | Checks | -|-----------|-------|--------| -| Mirage detector | `skeptic` | Non-existent files, APIs, functions, dependencies | -| Completeness + adequacy | `completeness-validator` | Bidirectional traceability, scope creep, overengineering, underengineering, solution depth | -| Security | `security-auditor` | OWASP, input validation, auth, sensitive data | -| Testing strategy | `test-reviewer` | Test plan adequacy for feature size S/M/L | -| Template + wave conflicts | `tech-spec-validator` | All sections filled, frontmatter, format, skills/reviewers, wave conflict detection | - -Pass to each validator: `work/{feature}/tech-spec.md` + `work/{feature}/user-spec.md`. - -### Process findings - -Read all 5 reports. For each finding: -- Fix if clearly valid (typos, missing sections, structural issues) -- Reject with reasoning if disagree — only for findings unrelated to user-spec alignment - -**User-spec alignment findings require user decision.** When completeness-validator reports `gap`, `scope_creep`, `overengineering`, or `shallow_solution` — present each finding to the user with your recommendation (fix / keep / adjust). Reason: these findings mean the tech-spec may contradict what the user asked for — only the user can decide whether the deviation is acceptable. - -### Iterate if needed (up to 3 iterations) - -If fixes were made: -1. Apply targeted fixes directly in `work/{feature}/tech-spec.md`. -2. Git commit: `chore(techspec): validation round {N} — {summary of fixes}` -3. Re-run validators on updated tech-spec. -4. Repeat up to 3 iterations. - -If problems remain after 3 iterations — show user: "Validation didn't pass in 3 iterations. Here's what remains — let's resolve together." - -**Checkpoint:** -- [ ] All 5 validators ran -- [ ] Findings processed (fixed / rejected / discussed) -- [ ] Final tech-spec.md placed in work/{feature}/ - -## Phase 6: User Approval - -1. Show user the full tech-spec.md. -2. Show validation summary: iterations count, issues found and resolved. -3. Wait for explicit approval. -4. If user has comments — fix, re-validate, show again. -5. After approval: update `status: draft` → `status: approved` in tech-spec frontmatter. -6. Git commit: `chore(techspec): approve tech-spec for {feature}` -7. Tell user next step: run `/decompose-tech-spec` to create task files. - -**Checkpoint:** -- [ ] User explicitly approved tech-spec -- [ ] status = approved - -## Final Check - -- [ ] tech-spec.md created with all sections (Implementation Tasks are brief scope descriptions) -- [ ] Validation passed (5 validators) -- [ ] User approved tech-spec -- [ ] status = approved in frontmatter diff --git a/infrastructure/ansible/files/claude-skills/skills/tech_spec_planning/references/skills-and-reviewers.md b/infrastructure/ansible/files/claude-skills/skills/tech_spec_planning/references/skills-and-reviewers.md deleted file mode 100644 index c6f5fcac..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/tech_spec_planning/references/skills-and-reviewers.md +++ /dev/null @@ -1,112 +0,0 @@ -# Skills and Reviewers Catalog - -Single source of truth for selecting skills and reviewers in Implementation Tasks. -Used by: tech-spec-planning (Phase 4), task-decomposition (Phase 1). - -## Execution Skills - -| Skill | What it's for | Typical tasks | -|-------|--------------|---------------| -| `code-writing` | Writing/modifying code, TDD cycle | API endpoints, models, services, components, migrations, tests | -| `infrastructure-setup` | Framework init, folder structure, Docker, pre-commit hooks, testing setup | Dockerfile, pre-commit hooks, folder structure, .gitignore, smoke tests | -| `deploy-pipeline` | CI/CD pipelines, deployment config, automated deploy | GitHub Actions, deploy scripts, platform config, secrets management | -| `documentation-writing` | Documentation, Project Knowledge updates | Architecture docs, API docs, conventions, patterns | -| `skill-master` | Creating/updating skills and agents | New skills, skill modifications | -| `pre-deploy-qa` | Acceptance testing before deploy (tests + acceptance criteria) | QA task in Final Wave | -| `post-deploy-qa` | Live environment verification after deploy via MCP tools | Post-deploy task in Final Wave | -| `prompt-master` | Writing/improving LLM prompts, prompt engineering | System prompts, user prompt templates, few-shot examples, prompt optimization | - -| `code-reviewing` | Full-feature code quality audit | Code Audit in Audit Wave | -| `security-auditor` | Full-feature security audit | Security Audit in Audit Wave | -| `test-master` | Full-feature test quality audit | Test Audit in Audit Wave | - -Tasks without skill (user instructions) — skill not specified, description is in the task itself. Example: "ask user to register a bot in BotFather". - -Prompt tasks (LLM system prompts, user templates) use `prompt-master` skill — they are NOT code-writing tasks. TDD Anchor is replaced by manual verification on sample data. - -## Reviewer Agents - -| Agent | What it checks | Model | -|-------|---------------|-------| -| `code-reviewer` | Code quality: structure, patterns, naming, complexity, error handling | sonnet | -| `security-auditor` | OWASP Top 10, injection, XSS, auth, input validation, secrets | sonnet | -| `test-reviewer` | Test quality: coverage, meaningful assertions, test pyramid balance | sonnet | -| `skill-checker` | Skill compliance: frontmatter, structure, skill-master guidelines | sonnet | -| `prompt-reviewer` | Prompt quality: clarity, positive framing, examples over rules, compression, XML structure, success criteria | sonnet | -| `infrastructure-reviewer` | Infrastructure setup quality: folder structure, pre-commit, Docker, .gitignore, testing | sonnet | -| `deploy-reviewer` | CI/CD pipeline and deployment config quality: workflows, secrets, platform config | sonnet | - -## Skill → Reviewers Mapping - -| Skill | Default reviewers | -|-------|------------------| -| `code-writing` | `code-reviewer`, `security-auditor`, `test-reviewer` | -| `infrastructure-setup` | `code-reviewer`, `security-auditor`, `infrastructure-reviewer` | -| `deploy-pipeline` | `code-reviewer`, `security-auditor`, `deploy-reviewer` | -| `documentation-writing` | `code-reviewer` | -| `skill-master` | `skill-checker` | -| `pre-deploy-qa` | none — QA is its own verification | -| `post-deploy-qa` | none — verification result is the review | -| `prompt-master` | `prompt-reviewer` | -| `code-reviewing` | none — auditor IS the review (Audit Wave) | -| `security-auditor` | none — auditor IS the review (Audit Wave) | -| `test-master` | none — auditor IS the review (Audit Wave) | - -When `reviewers` field is empty in a task — fall back to the default set for that skill. - -## Examples - -### Code task (most common) -```yaml -skills: [code-writing] -reviewers: [code-reviewer, security-auditor, test-reviewer] -``` - -### Infrastructure setup task -```yaml -skills: [infrastructure-setup] -reviewers: [code-reviewer, security-auditor, infrastructure-reviewer] -``` - -### Deploy pipeline task -```yaml -skills: [deploy-pipeline] -reviewers: [code-reviewer, security-auditor, deploy-reviewer] -``` - -### Task handling user input or auth -```yaml -skills: [code-writing] -reviewers: [code-reviewer, security-auditor, test-reviewer] -``` -Security-auditor is already in the default set for code-writing. No extra action needed. - -### Documentation task -```yaml -skills: [documentation-writing] -reviewers: [code-reviewer] -``` - -### Audit task (Audit Wave) -```yaml -skills: [code-reviewing] # or security-auditor, test-master -reviewers: [] -``` - -### QA task (Final Wave) -```yaml -skills: [pre-deploy-qa] -reviewers: [] -``` - -### Post-deploy verification (Final Wave) -```yaml -skills: [post-deploy-qa] -reviewers: [] -``` - -### Prompt task -```yaml -skills: [prompt-master] -reviewers: [prompt-reviewer] -``` diff --git a/infrastructure/ansible/files/claude-skills/skills/test_master/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/test_master/SKILL.md deleted file mode 100644 index 5e675174..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/test_master/SKILL.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -name: test-master -description: | - Testing methodology: when to write which tests, how to ensure test quality, test pyramid strategy. - - Use when: "напиши тесты", "как тестировать", "проанализируй тесты", "проверь качество тестов", "ревью тестов", "тестовая стратегия" ---- - -# Test Master - -**Test Pyramid:** -``` - /\ - /E2E\ <- Few (3-5 critical flows) - /------\ - /Integr.\ <- Some (all endpoints + DB) - /----------\ - / Unit \ <- Many (all business logic) - /--------------\ - / Smoke \ <- Minimal (1-2 basic tests) -/------------------\ -``` - ---- - -## When to Use Each Test Type - -### Smoke Tests -**Purpose:** Verify basic project setup works. - -**Use for:** -- Testing framework is configured -- Environment variables accessible -- Basic imports work -- Infrastructure is functional - -**Written:** During infrastructure setup (once per project) - -**Setting up smoke tests?** Read [smoke-tests.md](references/smoke-tests.md) — CI integration, example templates. - ---- - -### Unit Tests -**Purpose:** Test business logic in isolation. - -**Use for:** -- Functions with calculations, validations, transformations -- Decision-making logic (if/else, switch) -- Data processing and formatting -- Error handling logic - -**Written:** By code-developer during each task (immediately after code) - -**Skip for:** Simple getters/setters, one-line changes, trivial updates - -**Writing unit tests?** Read [unit-tests.md](references/unit-tests.md) — patterns, mocking, examples. - ---- - -### Integration Tests -**Purpose:** Test API endpoints, database, and external services. - -**Use for:** -- All API endpoints (POST/PUT/DELETE especially) -- Database operations (create/update/delete) -- External service integrations (payments, email, webhooks) - -**Written:** As separate task at end of feature (if defined in Tech Spec) - -**Rule:** Every API endpoint and every DB write operation must have a corresponding integration test. Missing integration tests for these is a quality gap. - -**Writing integration tests?** Read [integration-tests.md](references/integration-tests.md) — API testing, DB setup, fixtures. - ---- - -### E2E Tests -**Purpose:** Test critical user journeys end-to-end. - -**Use for:** -- Top 3-5 most critical user flows -- Large features (>5 tasks) -- Critical business processes (auth, payment, core features) - -**Written:** After deploy to dev, before manual testing (if proposed/requested) - -**Writing E2E tests?** Read [e2e-tests.md](references/e2e-tests.md) — Playwright/Cypress setup, page objects, CI. - ---- - -## Decision Framework - -### Should I write unit tests for this? - -**YES if:** -- Function has business logic -- Function makes decisions -- Function transforms data -- Function handles errors -- Task specifies testing - -**NO if:** -- Simple getter/setter -- One-line text change -- Trivial config update -- No code written (research/docs) - -### Should I write integration tests? - -**YES if:** -- Tech Spec specifies integration tests -- Feature has API endpoints -- Feature interacts with database -- Feature calls external services - -**NO if:** -- Tech Spec says "None" -- Feature is purely client-side -- Already covered by E2E tests - -### Should I write E2E tests? - -**YES if:** -- Feature has >5 tasks -- Feature touches critical flows -- Feature has breaking changes -- User explicitly requests -- Tech Spec specifies E2E tests - -**NO if:** -- Small feature (<3 tasks) -- Non-critical functionality -- Well covered by unit + integration tests -- Time/cost constraints - ---- - -## Key Testing Principles - -1. **Write tests immediately** - In the same session as the code, before moving on -2. **Test behavior, not implementation** - Focus on what, not how -3. **Keep tests fast** - Unit: milliseconds, Integration: seconds, E2E: minutes -4. **Isolate tests** - Mock external dependencies in unit tests -5. **One concern per test** - Each test validates one thing -6. **Clear test names** - Describe what's tested and expected outcome -7. **Independent tests** - Each test runs with its own setup, no shared state -8. **Clean state** - Always start with known database state -9. **Tests must verify real behavior** - Assert on actual results, not mock calls -10. **Every test earns its place** - Each test catches a specific failure no other test catches (see below) - ---- - -## Redundant Testing Anti-pattern - -Tests that duplicate coverage waste time and create maintenance burden. - -**Signs of redundant testing:** -- Same behavior verified by both unit test and integration test with no added value -- E2E test that only checks what unit tests already cover -- Multiple test files testing the same function with same scenarios -- "Tests for completeness" that exist without protecting against real regressions - -**Rule:** Each test must justify its existence — it catches a specific failure that no other test catches. If removing the test reduces zero confidence, it belongs nowhere. - ---- - -## Test Quality Requirements - -### What Makes a BAD Test - -**Tests nothing:** -```typescript -expect(true).toBe(true) -``` - -**Tests only that mock was called:** -```typescript -expect(api.call).toHaveBeenCalled() // Without checking result -``` - -**No assertions:** -```typescript -render() // Just renders, checks nothing -``` - -### What Makes a GOOD Test - -**Tests actual result:** -```typescript -expect(calculateTotal(100, 0.2)).toBe(80) -``` - -**Tests real state change:** -```typescript -cart.add({ id: 1 }) -expect(cart.items).toHaveLength(1) -``` - -### Rule: Excessive Mocking = Wrong Test Type - -If mocking 3+ dependencies -> use integration or E2E test instead. - ---- - -## When to Prioritize E2E Over Unit Tests - -For some project types, E2E tests are MORE valuable than unit tests: - -| Project Type | Primary Tests | Why | -|--------------|---------------|-----| -| API/Backend | Unit + Integration | Logic in functions | -| CLI Tools | Unit + Integration | Testable in isolation | -| **UI Apps** | **E2E + Integration** | Logic in UI interaction | -| **Browser Extensions** | **E2E (real browser)** | APIs can't be mocked reliably | -| **Mobile Apps** | **E2E** | Platform APIs need real env | - -**Rule:** If mocking more than testing -> wrong test type. - ---- - -## Mocking Strategy - -### Unit Tests -- **Mock:** Database, API calls, file system, time -- **Why:** Fast, isolated, deterministic -- **How:** Use framework mocking (jest.mock, unittest.mock) - -### Integration Tests -- **Real:** Database (test DB), file system -- **Mock:** External services (payments, email) -- **Why:** Test real interactions, avoid external costs/delays - -### E2E Tests -- **Real:** Everything (use test/sandbox mode for external services) -- **Why:** Test complete real-world scenario - ---- - -## Test Quality Review - -**When reviewing existing tests**, read [test-quality-review.md](references/test-quality-review.md) — categories of bad tests, severity levels, decision criteria. - diff --git a/infrastructure/ansible/files/claude-skills/skills/test_master/references/e2e-tests.md b/infrastructure/ansible/files/claude-skills/skills/test_master/references/e2e-tests.md deleted file mode 100644 index fdb4b1b9..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/test_master/references/e2e-tests.md +++ /dev/null @@ -1,177 +0,0 @@ -# E2E Testing Guide - -**For:** code-developer subagent when E2E tests are requested - -## Table of Contents -- [When E2E Tests Are Written](#when-e2e-tests-are-written) -- [What to Test](#what-to-test) -- [Test Structure](#test-structure) -- [What to Verify](#what-to-verify) -- [Test Execution](#test-execution) -- [Tooling](#tooling) -- [Key Principles](#key-principles) -- [Coverage Guidelines](#coverage-guidelines) -- [Checklist Before Completion](#checklist-before-completion) - -## When E2E Tests Are Written - -### Defined in Tech Spec -- Testing Requirements section specifies E2E needs -- Lists critical user journeys to test - -### Agent Proposes E2E Tests -Agent should propose E2E tests when: -- Feature has **>5 tasks** (large, complex feature) -- Feature touches **critical user flows** (auth, payment, core business logic) -- Feature has **breaking changes** in API or UI -- User explicitly requests E2E tests - -**When to propose:** After deploy to dev, before manual testing (new step in workflow) - -## What to Test - -### Top 3-5 Critical User Journeys -**Not everything** - only the most important flows: - -**Examples:** -- User registration → email verification → first login -- User login → create order → checkout → payment → confirmation -- Admin creates content → publishes → user views content -- User uploads file → processes → downloads result -- Integration: external service webhook → system processes → user notified - -### How to Identify Critical Flows -Defined during User Spec phase: -- What must work for business to function? -- What would cause major problems if broken? -- What do users do most frequently? - -## Test Structure - -### Full User Journey -Test complete flow from start to finish: -1. **Setup** - Clean database, create required data -2. **User actions** - Simulate real user interactions (UI or API) -3. **Verify results** - Check UI state, database, emails, side effects -4. **Cleanup** - Reset system to clean state - -### E2E Test Phases - -**Phase 1: Authentication** -- User registration/login -- Session creation -- Access to protected resources - -**Phase 2: Core Actions** -- Main user actions (create, update, delete) -- Form submissions -- Navigation between pages - -**Phase 3: Business Logic** -- Data processing -- Calculations -- Integrations with external services - -**Phase 4: Verification** -- Success messages shown -- Data persisted correctly -- Emails/notifications sent -- UI updates reflect changes - -## What to Verify - -### UI State -- Correct pages displayed -- Elements visible/hidden as expected -- Forms populated with correct data -- Error messages shown appropriately -- Success confirmations displayed - -### Backend State -- Database records created/updated correctly -- Related records updated (associations) -- Background jobs triggered -- Cache invalidated/updated - -### External Systems -- Emails sent to correct recipients -- Webhooks triggered to external services -- Files uploaded to storage -- Payment processed (test mode) - -## Test Execution - -### Where Tests Run -- **Dev environment** - After deploy to dev -- Real database (separate test DB) -- Real integrations (test/sandbox mode) - -### Test Speed -- E2E tests are **slow** (minutes, not seconds) -- Full browser automation takes time -- Only test critical paths, not every edge case - -### When to Run -1. After deploy to dev -2. Before manual testing (saves time on manual checks) -3. Before merge to main (required for critical features) -4. Optionally on CI/CD for critical flows - -## Tooling - -### Choose Framework Based on Tech Stack - -**Web Apps:** -- Playwright (recommended, modern, fast) -- Cypress (good for React/Vue) -- Selenium (older, more complex) - -**API-First Apps:** -- Postman/Newman (API testing) -- REST Assured (Java) -- SuperTest (Node.js) - -**Mobile Apps:** -- Appium (cross-platform) -- Detox (React Native) -- XCTest/Espresso (native) - -### Configuration -- Use headless mode for CI/CD (faster) -- Use headed mode for debugging (see what's happening) -- Set reasonable timeouts (30s for slow operations) -- Take screenshots on failure (debugging) - -## Key Principles - -1. **Few tests, critical flows** - Only top 3-5 journeys, not every scenario -2. **Test behavior, not UI details** - Don't test exact button position, test functionality -3. **Resilient selectors** - Use data-testid or semantic selectors, not CSS classes -4. **Realistic scenarios** - Simulate real user behavior, not edge cases -5. **Independent tests** - Each test can run alone, no dependencies between tests -6. **Clean state** - Reset database/system between tests - -## Coverage Guidelines - -### ✅ Write E2E for: -- Complete user registration flow -- Payment/checkout process -- Critical business workflows (order creation, document processing) -- Authentication and authorization flows - -### ❌ Don't write E2E for: -- Edge cases (covered by unit tests) -- Error handling (covered by integration tests) -- Every form field validation (covered by unit tests) -- Admin features rarely used (manual testing sufficient) - -## Checklist Before Completion - -- ✅ Top 3-5 critical flows identified (from Tech Spec or User Spec) -- ✅ Each flow tests complete user journey (start to finish) -- ✅ All tests pass on dev environment -- ✅ Tests verify UI, database, and external systems -- ✅ Tests are independent (can run in any order) -- ✅ Tests have proper cleanup (reset state) -- ✅ Screenshots/videos captured on failure (for debugging) - diff --git a/infrastructure/ansible/files/claude-skills/skills/test_master/references/integration-tests.md b/infrastructure/ansible/files/claude-skills/skills/test_master/references/integration-tests.md deleted file mode 100644 index e8bf7268..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/test_master/references/integration-tests.md +++ /dev/null @@ -1,137 +0,0 @@ -# Integration Testing Guide - -**For:** code-developer subagent executing Integration Tests task - -## Table of Contents -- [When Integration Tests Are Written](#when-integration-tests-are-written) -- [What to Test](#what-to-test) -- [Test Structure](#test-structure) -- [What to Verify](#what-to-verify) -- [Coverage Requirements](#coverage-requirements) -- [Key Principles](#key-principles) -- [Test Database](#test-database) -- [External Service Mocking](#external-service-mocking) -- [Checklist Before Completion](#checklist-before-completion) - -## When Integration Tests Are Written - -- Defined in Tech Spec (Testing Requirements section) -- Created as separate task at end of feature -- Executed after all feature tasks are completed -- Before deploy to dev environment - -## What to Test - -### API Endpoints -All HTTP endpoints created/modified in this feature: -- POST requests (create operations) -- PUT/PATCH requests (update operations) -- DELETE requests (delete operations) -- GET requests with complex logic or filtering -- Authentication/authorization on protected endpoints - -### Database Operations -All database interactions: -- Record creation (INSERT queries) -- Record updates (UPDATE queries) -- Record deletion (DELETE queries) -- Complex queries (JOIN, aggregation) -- Data integrity constraints -- Transaction handling - -### External Service Integrations -All third-party service calls: -- Payment gateway integrations -- Email service (SendGrid, Mailgun, etc.) -- Cloud storage (S3, GCS, etc.) -- Webhook handlers -- External APIs (Stripe, Twilio, etc.) - -## Test Structure - -### Setup Phase -1. **Initialize test database** - Clean state for each test -2. **Create fixtures** - Set up test data (users, records, etc.) -3. **Configure test environment** - Set test API keys, URLs - -### Test Phase -1. **Execute API call** - Make HTTP request to endpoint -2. **Verify response** - Check status code, response body -3. **Verify side effects** - Check database state, external calls - -### Cleanup Phase -1. **Rollback or truncate** - Clean database after test -2. **Reset mocks** - Clear any mocked external services -3. **Close connections** - Clean up resources - -## What to Verify - -### API Response -- Correct HTTP status code (200, 201, 400, 404, etc.) -- Response body structure matches expected format -- Response data contains correct values -- Error messages are clear and actionable - -### Database State -- Records created/updated/deleted as expected -- Related records updated (foreign keys, associations) -- Constraints enforced (unique, not null, etc.) -- No orphaned or corrupted data - -### System Behavior -- Emails sent (check email queue or mock) -- Files uploaded (check storage or mock) -- Events triggered (webhooks, background jobs) -- Logs written correctly - -## Coverage Requirements - -Test **all endpoints and integrations** defined in Tech Spec: -- Every API endpoint in the feature -- Every database operation (create/update/delete) -- Every external service call - -**Exception:** Read-only GET endpoints with trivial logic may be skipped if covered by E2E tests. - -## Key Principles - -1. **Real dependencies** - Use test database, not mocks (unlike unit tests) -2. **Isolated tests** - Each test runs independently -3. **Clean state** - Always start with known database state -4. **Fast cleanup** - Rollback transactions or truncate tables -5. **Mock external services** - Don't make real calls to payment/email (use test mode or mocks) -6. **Verify side effects** - Don't just check response, verify database and system state - -## Test Database - -### Setup -- Use separate test database (never production or dev database) -- Run migrations to set up schema -- Optionally seed with minimal required data - -### Per-test -- Create fixtures for test data -- Execute test -- Rollback transaction OR truncate tables - -### Best practices -- Use transactions for fast cleanup (rollback after each test) -- Avoid shared state between tests -- Keep fixture data minimal (only what's needed for test) - -## External Service Mocking - -For external services (Stripe, SendGrid, etc.): -- Use test/sandbox mode if available -- Mock HTTP calls to external APIs -- Verify mocked calls were made with correct parameters -- Don't make real API calls (slow, costs money, unreliable) - -## Checklist Before Completion - -- ✅ All endpoints from Tech Spec are tested -- ✅ All database operations are verified -- ✅ All external integrations are tested (mocked) -- ✅ All tests pass -- ✅ Tests run in reasonable time (seconds, not minutes) -- ✅ Test database cleanup works correctly diff --git a/infrastructure/ansible/files/claude-skills/skills/test_master/references/smoke-tests.md b/infrastructure/ansible/files/claude-skills/skills/test_master/references/smoke-tests.md deleted file mode 100644 index 388c6e3a..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/test_master/references/smoke-tests.md +++ /dev/null @@ -1,257 +0,0 @@ -# Smoke Testing Guide - -**For:** Infrastructure setup, verifying basic project functionality - -## Table of Contents -- [Purpose](#purpose) -- [When to Write Smoke Tests](#when-to-write-smoke-tests) -- [What to Test](#what-to-test) -- [Example Smoke Tests](#example-smoke-tests) -- [Characteristics](#characteristics) -- [CI/CD Integration](#cicd-integration) -- [Key Principles](#key-principles) -- [Common Mistakes](#common-mistakes) -- [Checklist](#checklist) - -## Purpose - -Smoke tests verify minimal system functionality - "is the system alive?" - -**Not for:** -- Testing business logic (use unit tests) -- Testing API endpoints (use integration tests) -- Testing user flows (use E2E tests) - -**For:** -- Verifying project setup works -- Ensuring test infrastructure is functional -- Providing CI/CD with basic health check -- Confirming dependencies are installed - ---- - -## When to Write Smoke Tests - -### During Infrastructure Setup (Step 7) -- Part of testing infrastructure setup -- Created once per project -- Verifies test framework configuration -- Checks environment is set up - -### Add to CI Pipeline -- Run smoke tests first (before unit/integration/E2E) -- If smoke test fails → don't run other tests (fail fast) -- Saves CI time by catching infrastructure problems early - ---- - -## What to Test - -### ✅ Write smoke tests for: -- Key modules/packages can be imported -- **App can start** (renders, server starts, CLI runs) -- Environment variables accessible (`process.env.NODE_ENV`) - -**Good smoke test:** -```typescript -it('should import main module without errors', () => { - expect(() => require('../src/index')).not.toThrow() -}) -``` - -**Bad smoke test - tests nothing:** -```typescript -// ❌ USELESS - delete such tests -it('should pass', () => { - expect(true).toBe(true) -}) -``` - -### ❌ Don't write smoke tests for: -- Business logic (use unit tests) -- API endpoints (use integration tests) -- User workflows (use E2E tests) -- Data processing (use unit tests) - ---- - -## Example Smoke Tests - -### Node.js/TypeScript - -**File:** `tests/smoke.test.ts` - -```typescript -/** - * Smoke tests - verify basic project setup - */ - -describe('Project Setup - Smoke Test', () => { - it('should have NODE_ENV configured', () => { - // Verify environment is set up - expect(process.env.NODE_ENV).toBeDefined(); - }); - - it('should be able to import main module', () => { - // Verify main application code can be imported - expect(() => { - require('../src/index'); - }).not.toThrow(); - }); -}); -``` - -### Python - -**File:** `tests/test_smoke.py` - -```python -""" -Smoke tests - verify basic project setup -""" - -import os - - -def test_environment_configured(): - """Verify environment variables can be accessed.""" - # This should pass even if ENVIRONMENT is not set - # (allows test to pass in minimal CI environments) - env = os.getenv('ENVIRONMENT', 'test') - assert env is not None - - -def test_main_module_import(): - """Verify main application module can be imported.""" - try: - import src.main # Adjust based on your structure - assert True - except ImportError as e: - assert False, f"Failed to import main module: {e}" -``` - ---- - -## Characteristics - -### Speed -- **Target:** Milliseconds -- **Requirement:** <1 second total -- Fastest tests in the pyramid - -### Scope -- **Minimal:** 1-2 tests are sufficient -- Don't test everything, just basics -- If more than 5 smoke tests → probably testing too much - -### When They Run -- **First** in test suite (before all others) -- **Every CI run** (fail fast if infrastructure broken) -- **Locally** when setting up project - -### What They Don't Do -- ❌ Don't test business logic -- ❌ Don't make database queries -- ❌ Don't make API calls -- ❌ Don't test user interactions - ---- - -## CI/CD Integration - -### Run Smoke Tests First - -```yaml -# .github/workflows/ci.yml -jobs: - smoke-test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: '20' - - run: npm ci - - name: Run smoke tests - run: npm test -- tests/smoke.test.ts - - unit-tests: - needs: smoke-test # Only run if smoke passes - runs-on: ubuntu-latest - steps: - - name: Run unit tests - run: npm test -- tests/unit/ -``` - -### Fail Fast Strategy - -If smoke test fails: -- Stop CI pipeline immediately -- Don't run slower tests -- Save CI time and costs -- Infrastructure problem is obvious - ---- - -## Key Principles - -1. **Minimal** - Only 1-2 tests, not comprehensive -2. **Fast** - Must run in milliseconds -3. **Infrastructure-focused** - Tests setup, not logic -4. **Always pass** - If smoke test fails, stop everything and fix setup -5. **Run first** - Before all other test types - ---- - -## Common Mistakes - -### ❌ Too Many Smoke Tests -```typescript -// Bad: Testing too much in smoke tests -describe('Smoke', () => { - it('test database connection', ...); - it('test API endpoint', ...); - it('test user creation', ...); - it('test authentication', ...); - // ... 20 more tests -}); -``` - -**Fix:** Keep minimal (1-2 tests). Move others to appropriate test type. - -### ❌ Slow Smoke Tests -```typescript -// Bad: Smoke test that takes seconds -it('should connect to database', async () => { - await db.connect(); // Slow! - await db.query('SELECT 1'); // Not a smoke test! -}); -``` - -**Fix:** Smoke tests shouldn't make real connections. Just test imports work. - -### ❌ Business Logic in Smoke Tests -```typescript -// Bad: Testing business logic -it('should calculate discount correctly', () => { - expect(calculateDiscount(100, 0.2)).toBe(80); -}); -``` - -**Fix:** Move to unit tests. - ---- - -## Checklist - -Before completing smoke test setup: - -- [ ] 1-2 smoke tests created -- [ ] Tests verify test framework works -- [ ] Tests verify environment configured -- [ ] Tests verify main module imports -- [ ] Tests run in <1 second -- [ ] Tests always pass (if fail → fix infrastructure) -- [ ] Tests added to CI/CD as first job -- [ ] CI configured to fail fast if smoke tests fail - diff --git a/infrastructure/ansible/files/claude-skills/skills/test_master/references/test-quality-review.md b/infrastructure/ansible/files/claude-skills/skills/test_master/references/test-quality-review.md deleted file mode 100644 index 3ef8f856..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/test_master/references/test-quality-review.md +++ /dev/null @@ -1,257 +0,0 @@ -# Test Quality Review Guide - -Methodology for analyzing quality of existing tests. Detects meaningless, ineffective, or poorly designed tests. - -## Table of Contents -- [Core Philosophy](#core-philosophy) -- [Categories of Bad Tests](#categories-of-bad-tests) -- [Severity Levels](#severity-levels) -- [Review Process](#review-process) -- [Status Decision Criteria](#status-decision-criteria) -- [Task Required Decision](#task-required-decision) -- [Litmus Test Methodology](#litmus-test-methodology) -- [Prescriptive Findings](#prescriptive-findings) - -## Core Philosophy - -Tests exist to: -1. Verify that code does what it should -2. Catch regressions when code changes -3. Document expected behavior - -Tests that fail these purposes are worse than no tests - they provide false confidence. - ---- - -## Categories of Bad Tests - -### Category 1: Empty/Meaningless Tests - -Tests that verify nothing: - -```typescript -// BAD - Tests nothing -test('should exist', () => { - expect(true).toBe(true); -}); - -// BAD - No assertions -test('renders component', () => { - render(); -}); - -// BAD - Only checks function exists -test('function defined', () => { - expect(typeof myFunction).toBe('function'); -}); -``` - -### Category 2: Mock-Only Tests - -Tests that only verify mock calls without checking results: - -```typescript -// BAD - Only tests mock was called -test('calls API', async () => { - await fetchUserData(1); - expect(api.get).toHaveBeenCalledWith('/users/1'); - // No assertion on the actual result! -}); - -// BAD - Mocks everything, tests nothing real -test('processes data', () => { - const mockProcessor = jest.fn().mockReturnValue('result'); - expect(mockProcessor()).toBe('result'); // Testing the mock! -}); -``` - -### Mock-Return Anti-pattern - -Agent mocks a dependency to return a value, then asserts the same value: - -```typescript -// Tests mock wiring, not code behavior -const mockUser = { id: 1, name: 'Alice' }; -mockUserService.create.mockResolvedValue(mockUser); -const result = await handler(req, res); -expect(result).toEqual(mockUser); -// Litmus test: delete handler implementation → test still passes -``` - -### Category 3: Missing Coverage - -Code without corresponding tests: - -- Business logic functions without unit tests -- API endpoints without integration tests -- Decision branches (if/else) not covered -- Error handling paths untested -- Edge cases from spec not tested - -### Category 4: Test Pyramid Violations - -Wrong test distribution: - -``` -Expected: -- Many unit tests (fast, isolated) -- Some integration tests (real DB/API) -- Few E2E tests (critical paths only) - -Violations: -- All E2E, no unit tests (slow, brittle) -- Only unit tests for UI app (misses real interactions) -- Integration tests for pure logic (overkill) -``` - -### Category 5: Excessive Mocking - -When mocking defeats the purpose: - -```typescript -// BAD - Mocks 3+ dependencies -test('user service', () => { - const mockDb = jest.mock('database'); - const mockCache = jest.mock('cache'); - const mockEmail = jest.mock('email'); - const mockLogger = jest.mock('logger'); - // At this point, what are we even testing? -}); -``` - -**Rule:** If mocking 3+ dependencies, this should be an integration test. - -### Category 6: Test Anti-patterns - -- **Implementation testing** - Tests break when refactoring without behavior change -- **Snapshot abuse** - Large snapshots nobody reviews -- **Flaky tests** - Random failures due to timing/order -- **Shared state** - Tests depend on each other -- **Magic values** - Unexplained test data - ---- - -## Severity Levels - -### Critical -- No tests at all for business-critical code -- Tests that actively hide bugs (incorrect assertions) -- All tests are empty/meaningless (false coverage) - -### High -- Missing tests for error handling -- Tests verify only mock calls (no result checking) -- Key acceptance criteria not tested - -### Medium -- Excessive mocking (should be integration test) -- Test pyramid violation (wrong test type used) -- Edge cases from spec not covered - -### Low -- Minor best practice violations -- Could be more specific assertions -- Naming improvements needed - ---- - -## Review Process - -1. **Identify Test Files**: Find all test files for reviewed code -2. **Map Coverage**: Match implementation files to test files -3. **Analyze Each Test**: - - Does it have meaningful assertions? - - Does it test real behavior or just mocks? - - Does it cover the right scenarios? -4. **Check Pyramid Balance**: Assess unit/integration/E2E distribution -5. **Find Gaps**: Identify untested code paths -6. **Categorize Findings**: Group by category and severity - ---- - -## Status Decision Criteria - -### passed -- All tests have meaningful assertions -- Critical business logic is tested -- Test pyramid is reasonably balanced -- Minor suggestions only (low severity) - -### needs_improvement -- Some tests need better assertions (medium severity) -- Some coverage gaps exist (non-critical areas) -- Pyramid slightly unbalanced -- No critical issues - -### failed -- Tests are meaningless (empty or mock-only) -- Critical business logic untested -- Tests hide bugs (wrong assertions) -- Test pyramid severely inverted -- Multiple high/critical severity issues - -**Decision matrix:** -- `critical > 0` → failed -- `high >= 3` → failed -- `high >= 1 AND medium >= 3` → needs_improvement -- `medium >= 5` → needs_improvement -- Only low issues → passed -- No issues → passed - ---- - -## Task Required Decision - -Set `taskRequired.needed = true` when: -- status === "failed" -- critical > 0 -- high >= 2 -- Critical business logic has no tests - -Set `taskRequired.needed = false` when: -- status === "passed" -- Only low/medium issues -- Issues can be fixed in current context - ---- - -## Litmus Test Methodology - -For every test touching business logic, ask: - -> "If I remove the core logic line being tested, does this test still pass?" - -**How to apply:** -1. Identify the core logic line (computation, validation, or side effect) -2. Mentally remove it -3. Trace test execution without that line -4. If test still passes → flag as litmus test failure - -**Common patterns that fail:** -- Mock returns X, test asserts X (passes with empty function) -- Test only asserts mock.toHaveBeenCalled() (passes with any call) -- Test uses same hardcoded data for input and expected output - ---- - -## Prescriptive Findings - -Every finding must include a concrete replacement, not just a problem description. - -**Bad finding:** -```json -{ "issue": "Test has no meaningful assertions", "recommendation": "Add assertion that verifies actual behavior" } -``` - -**Good finding:** -```json -{ - "issue": "Mock returns mockUser, test asserts mockUser — tests mock wiring, not code", - "litmusTestFailed": true, - "replacement": { - "approach": "Call real createUser with test data, assert on actual result", - "assertions": ["result.id is defined", "result.email === input.email"], - "mockChange": "Remove mockUserService, use test DB" - } -} -``` diff --git a/infrastructure/ansible/files/claude-skills/skills/test_master/references/unit-tests.md b/infrastructure/ansible/files/claude-skills/skills/test_master/references/unit-tests.md deleted file mode 100644 index 594f61c2..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/test_master/references/unit-tests.md +++ /dev/null @@ -1,150 +0,0 @@ -# Unit Testing Guide - -**For:** code-developer subagent working on a task - -## Table of Contents -- [When to Write Unit Tests](#when-to-write-unit-tests) -- [When Unit Tests Are Wrong Choice](#when-unit-tests-are-wrong-choice) -- [Development Flow](#development-flow) -- [What to Test](#what-to-test) -- [How to Organize Tests](#how-to-organize-tests) -- [Mocking Dependencies](#mocking-dependencies) -- [Key Rules](#key-rules) -- [Coverage Check](#coverage-check) - -## When to Write Unit Tests - -Write unit tests **when justified**: - -### ✅ Write tests for: -- Functions with business logic (calculations, validations, transformations) -- Functions that make decisions (if/else, switch, conditions) -- Data processing and formatting -- Error handling logic -- Edge cases and boundary conditions - -### ❌ Skip tests for: -- Simple getters/setters without logic -- One-line text changes or UI copy updates -- Trivial configuration changes -- Tasks without code (research, documentation, GitHub issues) - -**Rule of thumb:** If the function transforms data or makes a decision → test it. - -## When Unit Tests Are Wrong Choice - -### Excessive Mocking = Wrong Test Type - -If you need to mock 3+ dependencies to test something, consider: -- Integration test (test with real dependencies) -- E2E test (test in real environment) - -**Bad - testing mocks:** -```typescript -jest.mock('../db') -jest.mock('../api') -jest.mock('../cache') -it('should process', () => { - process() - expect(db.save).toHaveBeenCalled() // Tests mock, not behavior -}) -``` - -**Better:** Integration test with real dependencies. - -### UI Components with Complex State - -Don't unit test React/Vue components with mocked hooks and context. -Use integration tests or E2E instead. - -**Bad:** -```typescript -jest.mock('../hooks/useAuth') -jest.mock('../context/CartContext') -it('renders', () => render()) -``` - -**Better:** E2E test that actually clicks through checkout flow. - -## Development Flow - -1. **Read the task** - Understand requirements and acceptance criteria -2. **Write the code** - Implement the functionality -3. **Write tests immediately** - Don't defer, write in the same session -4. **Run tests** - Verify they pass -5. **If tests fail** - Fix code or tests, repeat step 4 -6. **Return to orchestrator** - Only when all tests pass - -## What to Test - -### Test the task requirements -All functionality described in the task must be covered: -- Main happy path (expected behavior) -- Edge cases mentioned in task -- Error handling specified in task -- Validation rules from task - -### Test business logic -- Input validation (valid/invalid inputs) -- Calculations (correct results, edge values) -- Transformations (data format changes) -- Conditional logic (all branches) -- Error conditions (exceptions, failures) - -## How to Organize Tests - -### Structure: Arrange → Act → Assert - -1. **Arrange** - Set up test data and preconditions -2. **Act** - Execute the function being tested -3. **Assert** - Verify the result matches expectations - -### One test = one concern -- Each test validates one specific behavior -- Don't test multiple unrelated things in one test -- Makes failures easier to diagnose - -### Clear test names -Name tests to describe what they test: -- `test_calculate_total_with_discount` -- `test_validate_email_rejects_invalid_format` -- `test_parse_date_handles_null_input` - -## Mocking Dependencies - -### What to mock -- Database calls -- API requests to external services -- File system operations -- Time/dates (for consistent tests) -- Random number generators - -### Why mock -- Tests run fast (milliseconds) -- Tests are isolated (no external dependencies) -- Tests are reliable (no network/DB failures) -- Tests are repeatable (same result every time) - -### How to mock -Use mocking libraries appropriate to your tech stack: -- Mock external service responses -- Mock database query results -- Inject mocked dependencies into functions - -## Key Rules - -1. **Fast execution** - Unit tests must run in milliseconds -2. **Isolated** - No real database, API, or file system access -3. **Deterministic** - Same input → same output, always -4. **Independent** - Tests don't depend on each other -5. **Single assertion** - One test = one thing to verify -6. **Immediate** - Write tests right after code, not later - -## Coverage Check - -Before returning to orchestrator, verify: -- ✅ All business logic from task is tested -- ✅ All tests pass -- ✅ No skipped or commented-out tests -- ✅ Edge cases are covered -- ✅ Error handling is tested diff --git a/infrastructure/ansible/files/claude-skills/skills/user_spec_planning/SKILL.md b/infrastructure/ansible/files/claude-skills/skills/user_spec_planning/SKILL.md deleted file mode 100644 index e4a378ae..00000000 --- a/infrastructure/ansible/files/claude-skills/skills/user_spec_planning/SKILL.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -name: user-spec-planning -description: | - Creates user-spec.md through adaptive interview with codebase scanning and dual validation. - - Use when: "сделай юзер спек", "проведи интервью для юзер спека", - "создай юзерспек", "user spec", "detailed planning", "хочу продумать фичу", - "опиши требования к фиче", "сделай описание фичи", "/new-user-spec" - - For tech planning use tech-spec-planning. For project planning use project-planning. ---- - -# User Spec Planning - -Thorough adaptive interview → codebase scan → user-spec.md → dual validation → user approval. -Output: `work/{feature}/user-spec.md` with status `approved`. - -## Interview Style - -Conduct interview in Russian. Be thorough and opinionated — an engaged co-thinker who actively proposes solutions and challenges weak answers. - -**How to interview:** -- 3-4 questions per batch. Run as many batches as needed until the cycle's items are fully covered. -- Propose solutions based on Project Knowledge: "В architecture.md описан паттерн X — думаю, здесь нужно Y. Согласен?" -- Challenge with substance — concrete counterexamples, code references, unexplored scenarios: "А что если пользователь сделает Z? В коде модуль Q не обрабатывает этот случай." -- Accept the answer after one substantive challenge and move on to the next gap. -- When user says "не знаю": help think through it (examples, common patterns). Optional item → mark TBD. Required item → break into simpler questions. - -**Interview depth** depends on feature size (S/M/L in interview metadata): -- S (1-3 files, local fix): focused interview, core behavior -- M (several components): moderate depth, integration questions -- L (new architecture): deep interview, thorough edge cases and risk analysis - -## Process - -### Phase 0: Init - -1. Check for existing interview: look in `work/*/logs/userspec/interview.yml` for `metadata.status: in_progress`. If found — load, show discussed topics summary, resume. If multiple found — show list, let user choose. -2. Get task description: "Опиши, что хочешь сделать." -3. Determine work_type (feature / bug / refactoring) from description. -4. Propose feature name (kebab-case), get user confirmation. -5. Run `~/.claude/shared/scripts/init-feature-folder.sh {name}` — creates folder structure with interview.yml. -6. Update interview.yml: set metadata.started, metadata.status: in_progress, phase1_feature_overview.feature_name, phase1_feature_overview.work_type. - -**Checkpoint:** interview.yml exists with status in_progress, feature name confirmed. - -### Phase 1: Study Project Knowledge - -Read ALL files from `.claude/skills/project-knowledge/references/`. If directory missing or empty — warn user, suggest running project-planning skill (or `/init-project-knowledge` command). - -These files are your context for the entire interview. Reference them when asking questions and proposing solutions. - -### Phase 2: Cycle 1 — General Understanding - -**Scope:** `phase1_feature_overview` items in interview.yml. - -1. Score user's initial description against all items (detailed 80-95%, brief 50-70%, vague 20-40%, not mentioned 0%). -2. Run interview loop (see below) on phase1_feature_overview items. -3. During this cycle — determine feature size S/M/L and agree on testing strategy: - - S: integration/E2E usually not needed — state why - - M: propose whether integration tests make sense, explain reasoning - - L: propose specific integration and E2E scope with justification - -### Phase 3: Code Scanning - -Launch `code-researcher` subagent (Task tool, opus) with feature path and feature description from Cycle 1. - -After subagent completes — read `{feature_path}/code-research.md`. Use findings in Cycle 2 questions. - -If during later phases a gap is discovered — launch `code-researcher` again with the specific question to investigate. - -### Phase 4: Cycle 2 — Code-Informed Refinement - -**Scope:** `phase2_user_experience` + `phase3_integration` items. - -1. Summarize understanding: "Я понял задачу так: [X]. Делать планирую так: [Y, based on code]." -2. Questions based on code findings: "Нашёл модуль X, который делает Y — переиспользуем?" -3. Cover deploy and user actions (items `deploy_approach`, `manual_user_actions`): - - "Нужны ли ручные шаги для запуска? (создать бота, получить API ключи, настроить сервис, зарегистрироваться где-то)" - - "Как деплоить? Что нужно настроить? (уже есть CI/CD, нужно настроить, ручной деплой)" - - "Как проверить что работает после деплоя? (MCP-инструменты, curl, ручная проверка)" - - "Что можно проверить прямо во время разработки, без деплоя? (вызвать внешний API, запустить локально, проверить конфиг, потыкать UI на localhost, протестировать промпт)" -4. Run interview loop on phase2 + phase3 items. - -### Phase 5: Cycle 3 — Review & Finalize - -**Scope:** ALL items across all phases still below threshold. - -Cleanup pass: revisit anything not fully covered in Cycles 1-2. Deepen edge cases and error scenarios — probe for scenarios user hasn't considered, even if items formally passed threshold. - -Run interview loop on remaining gaps. - -### Phase 6: Completeness Check - -Launch `interview-completeness-checker` subagent (Task tool, sonnet) with feature path. It reviews interview.yml against PK files and code-research.md. - -- `needs_more` → ask the suggested questions, re-run checker -- `complete` → proceed to Phase 7 - -### Phase 7: Create User Spec - -1. Copy template to working file: - - Copy `~/.claude/shared/work-templates/user-spec.md.template` → `work/{feature}/user-spec.md` - - Edit sections one by one using Edit tool, replacing placeholders with interview data - Reason: agent sees template structure and comments while editing each section, preventing drift from template format. -2. Content rules: - - "Что делаем" — self-contained, understandable without the interview - - "Зачем" — concrete user value, not "улучшить UX" - - Acceptance criteria — testable, no "работает корректно" - - Every discussed topic from interview must appear in the spec -3. If feature seems large (>10 criteria, >3 user flows, >5 integrations) — suggest splitting. - -Git commit: `draft(userspec): create user-spec for {feature}` - -### Phase 8: Validation - -Run 2 validators in parallel (Task tool): -- `userspec-quality-validator` (sonnet) — document structure, template compliance, formal completeness. Returns JSON with per-check pass/fail and findings list. -- `userspec-adequacy-validator` (opus) — feasibility, over/underengineering, better alternatives. Returns JSON with findings by category and severity. - -**Handling findings:** -- Obvious issue → fix silently -- Borderline → discuss with user -- Disagree with finding → reject with reasoning -- Conflict between validators → userspec-adequacy-validator takes priority (substance over form) - -After each validation round (validators wrote reports + you applied fixes), git commit: `chore(userspec): validation round {N} — {summary of fixes}`. Re-run validators. Max 3 iterations, then show remaining issues to user. - -### Phase 9: User Approval - -Show user-spec.md link + validation summary. If changes requested — edit and show again. - -When approved: -1. Set user-spec.md frontmatter `status: approved` -2. Set interview.yml `metadata.status: completed` -3. Git commit: `chore(userspec): approve user-spec for {feature}` -4. Suggest `/new-tech-spec {feature-name}` - -## Interview Loop - -Runs inside each cycle. Repeats until the cycle's scope is fully covered. - -``` -1. Find gaps: required items in current scope with score < 85%. Lowest first. -2. Ask 3-4 questions about different gaps. Reference PK and code findings. -3. User responds. -4. Update interview.yml: - - conversation_history: add full Q&A entry - - Item: score, value, gaps, status - - metadata: last_updated, current_question_num - - Save immediately -5. Check stop criteria (BOTH must be true): - a) All required items in scope score >= 85% - b) Structural: every required item has non-empty value, - no TBD in value, gaps empty or only conscious limitations -6. Not done → step 1. Done → exit cycle. -``` - -Scoring: detailed answer 80-95%, brief 50-70%, vague 20-40%, not mentioned 0%. - -Optional items: cover when user mentions relevant context or when naturally connected to required items. - -## Work Type Adaptations - -All three cycles apply to any work_type, but focus shifts: - -**Bug:** Cycle 1 → reproduction steps, expected vs actual, severity, when it broke. Code scanning → find bug location and root cause. Cycle 2 → fix approach, regression risks. - -**Refactoring:** Cycle 1 → current problems, target architecture, stability guarantees. Code scanning → current structure, dependencies, test coverage. Cycle 2 → migration path, backward compatibility. - -## Scope Changes - -If understanding changes significantly during interview: -- Update affected scores downward, add new gaps -- Reassess feature size (S/M/L) -- If work_type changes (was feature, actually bug) — pivot items accordingly -- Note the change in interview.yml notes section - -## Self-Verification - -- [ ] All cycles completed, completeness checker passed -- [ ] user-spec.md filled with real content (no placeholders) -- [ ] Both validators passed (or issues resolved with user) -- [ ] User approved, frontmatter status: approved -- [ ] interview.yml metadata.status: completed -- [ ] Suggested `/new-tech-spec` as next step diff --git a/infrastructure/ansible/roles/molyanov/README.md b/infrastructure/ansible/roles/molyanov/README.md index 6ff35449..f0fabaf3 100644 --- a/infrastructure/ansible/roles/molyanov/README.md +++ b/infrastructure/ansible/roles/molyanov/README.md @@ -4,7 +4,7 @@ Install molyanov-ai-dev slash commands and Project Knowledge guard hook for codi ## Responsibilities -1. **molyanov-ai-dev skills bundle**: Clone `pavel-molyanov/molyanov-ai-dev` (MIT) at pinned tag (v0.3.0) to `/opt/molyanov-ai-dev`, then symlink `skills/` to `~/.claude/skills/molyanov-ai-dev` so Claude Code discovers the bundle. +1. **molyanov-ai-dev skills bundle**: Clone `pavel-molyanov/molyanov-ai-dev` (MIT) at pinned tag (v0.3.0) to `/opt/molyanov-ai-dev` for operator reference and upstream diffing. The clone is NOT symlinked into `~/.claude/skills/` anymore: the nested `~/.claude/skills/molyanov-ai-dev//` layout was either undiscovered by Claude Code (skills resolve at `~/.claude/skills//SKILL.md`) or duplicated the curated flat bundle that the `telegram-ai-agent` role syncs (`infrastructure/ansible/files/claude-skills/`), producing duplicate-skill-name conflicts. The curated flat bundle is the single source of truth for deployed skills/commands; the role removes the legacy symlink on deploy. 2. **Global configuration**: Render `~/.fabric/molyanov/global.yml` with defaults and references 3. **PK guard pre-write hook**: Install `~/.fabric/molyanov/hooks/pk-guard.sh` (mode 0755) 4. **Ops-notify wrapper**: Install `~/.fabric/molyanov/hooks/ops-notify.sh` (mode 0755) diff --git a/infrastructure/ansible/roles/molyanov/molecule/verify.yml b/infrastructure/ansible/roles/molyanov/molecule/verify.yml index 6b74fafc..1947905d 100644 --- a/infrastructure/ansible/roles/molyanov/molecule/verify.yml +++ b/infrastructure/ansible/roles/molyanov/molecule/verify.yml @@ -3,17 +3,29 @@ hosts: all become: true tasks: - - name: Check molyanov skills bundle installed + - name: Check molyanov bundle clone installed ansible.builtin.stat: - path: "{{ molyanov_operator_home }}/.claude/skills/molyanov-ai-dev" + path: "{{ molyanov_install_dir | default('/opt/molyanov-ai-dev') }}/skills" register: molyanov_skills_stat - - name: Assert molyanov skills directory exists + - name: Assert molyanov bundle skills tree exists in install dir ansible.builtin.assert: that: - molyanov_skills_stat.stat.exists - molyanov_skills_stat.stat.isdir - fail_msg: "molyanov-ai-dev skills bundle not installed" + fail_msg: "molyanov-ai-dev bundle not cloned to install dir" + + - name: Check legacy nested skills symlink is gone + ansible.builtin.stat: + path: "{{ molyanov_operator_home }}/.claude/skills/molyanov-ai-dev" + follow: false + register: molyanov_legacy_link_stat + + - name: Assert legacy nested skills symlink removed + ansible.builtin.assert: + that: + - not molyanov_legacy_link_stat.stat.exists + fail_msg: "legacy ~/.claude/skills/molyanov-ai-dev symlink still present (duplicate-skill source)" - name: Check pk-guard hook installed ansible.builtin.stat: diff --git a/infrastructure/ansible/roles/molyanov/tasks/main.yml b/infrastructure/ansible/roles/molyanov/tasks/main.yml index 8d55a1bc..10f378ef 100644 --- a/infrastructure/ansible/roles/molyanov/tasks/main.yml +++ b/infrastructure/ansible/roles/molyanov/tasks/main.yml @@ -46,17 +46,21 @@ become_user: "{{ molyanov_operator_user }}" register: molyanov_skills_clone -- name: Symlink molyanov skills tree into operator's Claude Code skills dir - # Upstream layout (v0.3.0+): top-level skills/ directory contains the bundle. - # Symlinking (rather than copying) preserves a single source of truth and - # lets `git pull` reflect immediately under ~/.claude/skills/. +- name: Remove legacy nested skills symlink (~/.claude/skills/molyanov-ai-dev) + # Until 2026-07 the role symlinked the whole upstream bundle as ONE nested + # dir: ~/.claude/skills/molyanov-ai-dev//SKILL.md. Claude Code + # resolves personal skills at ~/.claude/skills//SKILL.md, so the + # nested layout was either invisible (skills never discovered under the + # names the operator expected — "alias is different") or, where nested + # discovery kicked in, a full duplicate of the curated flat bundle that + # the telegram-ai-agent role syncs into the same tree — duplicate skill + # names either way. The flat curated bundle + # (infrastructure/ansible/files/claude-skills/) is the single source of + # truth now; the clone at {{ molyanov_install_dir }} remains for operator + # reference and upstream diffing only. ansible.builtin.file: - src: "{{ molyanov_install_dir }}/skills" - dest: "{{ molyanov_operator_home }}/.claude/skills/molyanov-ai-dev" - state: link - force: true - owner: "{{ molyanov_operator_user }}" - group: "{{ molyanov_operator_user }}" + path: "{{ molyanov_operator_home }}/.claude/skills/molyanov-ai-dev" + state: absent become: true become_user: "{{ molyanov_operator_user }}" @@ -87,19 +91,17 @@ mode: '0755' notify: Molyanov hook installed -- name: Verify molyanov skills symlink exists and points into install dir +- name: Verify molyanov bundle clone exists ansible.builtin.stat: - path: "{{ molyanov_operator_home }}/.claude/skills/molyanov-ai-dev" - follow: false + path: "{{ molyanov_install_dir }}/skills" register: molyanov_skills_stat -- name: Assert molyanov skills installed (symlink → install dir) +- name: Assert molyanov bundle cloned (skills tree present in install dir) ansible.builtin.assert: that: - molyanov_skills_stat.stat.exists - - molyanov_skills_stat.stat.islnk - - molyanov_skills_stat.stat.lnk_source == molyanov_install_dir + "/skills" - fail_msg: "molyanov-ai-dev skills bundle installation failed (expected symlink → {{ molyanov_install_dir }}/skills)" + - molyanov_skills_stat.stat.isdir + fail_msg: "molyanov-ai-dev bundle clone failed (expected {{ molyanov_install_dir }}/skills)" - name: Verify pk-guard hook installed ansible.builtin.stat: diff --git a/infrastructure/ansible/roles/telegram-ai-agent/README.md b/infrastructure/ansible/roles/telegram-ai-agent/README.md index d616205d..3d4ddd22 100644 --- a/infrastructure/ansible/roles/telegram-ai-agent/README.md +++ b/infrastructure/ansible/roles/telegram-ai-agent/README.md @@ -326,6 +326,46 @@ ansible-playbook playbooks/deploy.yml -i inventory/hosts.yml --tags telegram-ai- ## Troubleshooting +### Claude auth reset / quota exhausted ("relogin") + +The bot's claude subprocesses authenticate from `/etc/telegram-ai-agent/.env` +(`CLAUDE_CODE_OAUTH_TOKEN` primary, `ANTHROPIC_API_KEY` fallback) — **not** +from `claude /login` browser credentials (the service HOME is isolated). +There is no in-chat relogin; credential rotation + service restart is the +relogin. The role installs a one-command runbook: + +```bash +# Plain reset: kill stray engine processes, restart with same credentials +sudo bot-claude-reset + +# Also drop session-resume state (next message starts a fresh Claude session) +sudo bot-claude-reset --wipe-sessions + +# Subscription quota burned / token expired: rotate the OAuth token +# (generate on laptop: `claude setup-token`) +sudo bot-claude-reset --oauth-token sk-ant-oat01-... + +# Or switch to pay-per-token while the subscription quota recovers +sudo bot-claude-reset --api-key sk-ant-api03-... +``` + +Tokens rotated live are overwritten by the next Ansible deploy — persist +them in `infrastructure/secrets/secrets.sops.yml` +(`claude_code_oauth_token` / `anthropic_api_key`). + +To reduce quota burn without redeploying the template, override the model +globally (`telegram_ai_agent_claude_model: "sonnet"` — CLI aliases track the +latest release) or per topic (`model:` key in `telegram_ai_agent_topics`). + +### Slash commands from Telegram (naming convention) + +Telegram bot commands only allow `[a-z0-9_]` — dash commands cannot be typed +in a Telegram chat. The deployed bundle therefore ships every multi-word +command twice: `/do-task` (canonical, CLI) and `/do_task` (Telegram alias). +Skills ship dash-only names; underscore skill duplicates were removed +2026-07 because identical frontmatter `name:` values made skill resolution +unreliable. See `infrastructure/ansible/files/claude-skills/README.md`. + ### Service fails to start ```bash journalctl -u telegram-ai-agent -n 50 -e diff --git a/infrastructure/ansible/roles/telegram-ai-agent/defaults/main.yml b/infrastructure/ansible/roles/telegram-ai-agent/defaults/main.yml index 6192dc36..70bdd61a 100644 --- a/infrastructure/ansible/roles/telegram-ai-agent/defaults/main.yml +++ b/infrastructure/ansible/roles/telegram-ai-agent/defaults/main.yml @@ -41,6 +41,39 @@ telegram_ai_agent_default_cwd: "/home/op/work" # them as a static baseline. telegram_ai_agent_topics: {} +# Model pinning for bot-spawned engine sessions. Rendered into +# /etc/telegram-ai-agent/config.yml per topic; a topic can override with +# its own `model` key in telegram_ai_agent_topics. Aliases ("sonnet", +# "opus") are accepted by the claude CLI and track the latest release — +# prefer them over dated IDs so a quota-driven switch (e.g. opus quota +# burned → drop to sonnet) is a one-var redeploy, not a template edit. +telegram_ai_agent_claude_model: "claude-opus-4-1" +telegram_ai_agent_codex_model: "codex-latest" + +# Skill dirs from the pre-2026-07 bundle that shipped underscore duplicates +# of every dash-named skill (identical frontmatter `name:` → duplicate-skill +# conflicts in Claude Code). Purged from every target HOME on deploy because +# the skills sync is additive. Remove this list (and its purge task) once +# all long-lived VMs have redeployed past 2026-07. +telegram_ai_agent_legacy_underscore_skills: + - code_reviewing + - code_writing + - deploy_pipeline + - documentation_writing + - feature_execution + - infrastructure_setup + - post_deploy_qa + - pre_deploy_qa + - project_planning + - prompt_master + - security_auditor + - skill_master + - skill_tester + - task_decomposition + - tech_spec_planning + - test_master + - user_spec_planning + # Systemd service parameters telegram_ai_agent_restart_policy: "on-failure" telegram_ai_agent_restart_sec: 5 diff --git a/infrastructure/ansible/roles/telegram-ai-agent/files/bot-claude-reset.sh b/infrastructure/ansible/roles/telegram-ai-agent/files/bot-claude-reset.sh new file mode 100644 index 00000000..9dc8727b --- /dev/null +++ b/infrastructure/ansible/roles/telegram-ai-agent/files/bot-claude-reset.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# bot-claude-reset — reset the telegram-ai-agent bot's Claude engine state. +# +# Use when the bot stops answering because the Claude subscription quota is +# exhausted, the OAuth token expired/was revoked, or stale engine sessions +# keep resuming into a broken state ("Thinking..." forever). Installed to +# /usr/local/bin by the telegram-ai-agent Ansible role. +# +# What it does (in order): +# 1. systemctl stop telegram-ai-agent +# 2. kills stray `claude` engine subprocesses and bot MCP servers +# 3. (--wipe-sessions) removes the bot's session-resume state so the next +# message starts a FRESH Claude session instead of resuming a dead one +# 4. (--oauth-token / --api-key) rotates the credential in +# /etc/telegram-ai-agent/.env — this is the "relogin": the bot's claude +# subprocesses authenticate from these env vars, not from +# `claude /login` browser credentials (service HOME is isolated) +# 5. systemctl start telegram-ai-agent + prints status +# +# Typical quota-day flows: +# sudo bot-claude-reset # plain reset (stray procs, same creds) +# sudo bot-claude-reset --wipe-sessions # also drop session resume state +# sudo bot-claude-reset --oauth-token sk-ant-oat01-... # rotate subscription token +# sudo bot-claude-reset --api-key sk-ant-api03-... # switch to pay-per-token key +# +# NOTE: a token rotated here survives until the next Ansible deploy, which +# re-renders .env from sops. Copy the new token into +# infrastructure/secrets/secrets.sops.yml (claude_code_oauth_token / +# anthropic_api_key) to make it permanent. + +set -euo pipefail + +ENV_FILE="/etc/telegram-ai-agent/.env" +BOT_HOME="/opt/telegram-ai-agent" +SERVICE="telegram-ai-agent" + +WIPE_SESSIONS=0 +NEW_OAUTH_TOKEN="" +NEW_API_KEY="" + +usage() { grep '^#' "$0" | sed 's/^# \{0,1\}//' | sed -n '2,30p'; exit "${1:-0}"; } + +while [[ $# -gt 0 ]]; do + case "$1" in + --wipe-sessions) WIPE_SESSIONS=1; shift ;; + --oauth-token) NEW_OAUTH_TOKEN="${2:?--oauth-token requires a value}"; shift 2 ;; + --api-key) NEW_API_KEY="${2:?--api-key requires a value}"; shift 2 ;; + -h|--help) usage 0 ;; + *) echo "unknown argument: $1" >&2; usage 1 ;; + esac +done + +if [[ $EUID -ne 0 ]]; then + echo "must run as root (sudo bot-claude-reset ...)" >&2 + exit 1 +fi + +# Replace KEY=... in the env file, or append if the key is absent. +# Preserves 0600 by editing in place. +set_env_var() { + local key="$1" value="$2" + if grep -q "^${key}=" "$ENV_FILE"; then + sed -i "s|^${key}=.*|${key}=${value}|" "$ENV_FILE" + else + printf '%s=%s\n' "$key" "$value" >> "$ENV_FILE" + fi + chmod 0600 "$ENV_FILE" + echo "updated ${key} in ${ENV_FILE}" +} + +echo "==> stopping ${SERVICE}" +systemctl stop "$SERVICE" + +echo "==> killing stray engine processes" +pkill -9 -f "claude --output-format stream-json" 2>/dev/null || true +pkill -9 -f "mcp-servers/bot" 2>/dev/null || true + +if [[ $WIPE_SESSIONS -eq 1 ]]; then + echo "==> wiping session-resume state" + for f in channel_sessions.json session_mapping.json; do + if [[ -f "${BOT_HOME}/${f}" ]]; then + : > "${BOT_HOME}/${f}" + echo "cleared ${BOT_HOME}/${f}" + fi + done +fi + +if [[ -n "$NEW_OAUTH_TOKEN" ]]; then + set_env_var "CLAUDE_CODE_OAUTH_TOKEN" "$NEW_OAUTH_TOKEN" +fi +if [[ -n "$NEW_API_KEY" ]]; then + set_env_var "ANTHROPIC_API_KEY" "$NEW_API_KEY" +fi + +echo "==> starting ${SERVICE}" +systemctl start "$SERVICE" +sleep 2 +systemctl --no-pager --lines=5 status "$SERVICE" || true + +echo +echo "Done. Send a message to the bot to verify. If auth still fails, check:" +echo " journalctl -u ${SERVICE} -n 50 --no-pager" +echo "Remember: tokens rotated here are overwritten by the next Ansible deploy —" +echo "persist them in infrastructure/secrets/secrets.sops.yml." diff --git a/infrastructure/ansible/roles/telegram-ai-agent/tasks/main.yml b/infrastructure/ansible/roles/telegram-ai-agent/tasks/main.yml index 0d6761b6..f28aa381 100644 --- a/infrastructure/ansible/roles/telegram-ai-agent/tasks/main.yml +++ b/infrastructure/ansible/roles/telegram-ai-agent/tasks/main.yml @@ -523,6 +523,25 @@ - /var/lib/telegram-ai-agent/.claude - /home/op/.claude + - name: Purge underscore-duplicate skill dirs (stale from pre-2026-07 bundle) + # The bundle used to ship every skill twice — dash dir (code-reviewing) + # AND underscore dir (code_reviewing) — with IDENTICAL frontmatter + # `name:`. Claude Code resolves skills by that name, so every session + # saw 17 duplicate-name conflicts and skill/command resolution became + # unreliable ("agent does not recognize the command at once", operator + # report 2026-07-27). Dash dirs are canonical (skills are invoked by + # the agent via the Skill tool, never typed as Telegram commands, so + # no underscore alias is needed at the skill layer). The sync above is + # additive (delete: no), so previously deployed underscore dirs must + # be removed explicitly. + ansible.builtin.file: + path: "{{ item.0 }}/skills/{{ item.1 }}" + state: absent + loop: "{{ ['/var/lib/telegram-ai-agent/.claude', '/home/op/.claude'] + | product(telegram_ai_agent_legacy_underscore_skills) | list }}" + loop_control: + label: "{{ item.0 }}/skills/{{ item.1 }}" + - name: Sync commands/ into each target HOME ansible.posix.synchronize: src: "{{ role_path }}/../../files/claude-skills/commands/" @@ -570,6 +589,22 @@ group: "{{ telegram_ai_agent_group }}" mode: "0755" +- name: Install bot-claude-reset operator helper + # Quota-day / relogin runbook in one command: stop bot, kill stray claude + # engine processes, optionally wipe session-resume state and rotate the + # CLAUDE_CODE_OAUTH_TOKEN / ANTHROPIC_API_KEY in .env, restart. The bot's + # claude subprocesses authenticate from .env vars (service HOME is + # isolated from `claude /login` credentials), so credential rotation + + # service restart IS the relogin. See role README "Claude auth reset". + tags: [telegram-ai-agent, claude-reset] + become: true + ansible.builtin.copy: + src: bot-claude-reset.sh + dest: /usr/local/bin/bot-claude-reset + owner: root + group: root + mode: "0755" + - name: Install and enable systemd unit tags: [telegram-ai-agent, systemd] block: diff --git a/infrastructure/ansible/roles/telegram-ai-agent/templates/config.yml.j2 b/infrastructure/ansible/roles/telegram-ai-agent/templates/config.yml.j2 index 9193cc62..5950bde9 100644 --- a/infrastructure/ansible/roles/telegram-ai-agent/templates/config.yml.j2 +++ b/infrastructure/ansible/roles/telegram-ai-agent/templates/config.yml.j2 @@ -2,8 +2,14 @@ # Generated by Ansible role: telegram-ai-agent (T09) # Source: infrastructure/ansible/roles/telegram-ai-agent/templates/config.yml.j2 +{# Block tags sit at column 0 with NO whitespace-control dashes: Ansible's + template module renders with trim_blocks=True (newline after each block + tag is dropped). The previous `{%- ... %}` form additionally stripped the + newline BEFORE each tag, gluing `topics:` and every `model:` onto the + preceding line whenever the topics map was non-empty (latent — deployed + topic matrix has been {} since 2026-05-24). #} topics: -{%- for topic_name, topic_config in telegram_ai_agent_topics_with_ids.items() %} +{% for topic_name, topic_config in telegram_ai_agent_topics_with_ids.items() %} {{ topic_name }}: chat_id: {{ topic_config.chat_id }} thread_id: {{ topic_config.thread_id }} @@ -12,12 +18,12 @@ topics: mode: {{ topic_config.mode }} exec_mode: {{ topic_config.exec_mode }} stream_mode: {{ topic_config.stream_mode }} - {%- if topic_config.engine == "claude" %} - model: claude-opus-4-1 - {%- elif topic_config.engine == "codex" %} - model: codex-latest - {%- endif %} -{%- endfor %} +{% if topic_config.engine == "claude" %} + model: {{ topic_config.model | default(telegram_ai_agent_claude_model) }} +{% elif topic_config.engine == "codex" %} + model: {{ topic_config.model | default(telegram_ai_agent_codex_model) }} +{% endif %} +{% endfor %} # Per-topic ruflo toggles (autopilot/aidefence/rag_memory/MNEMONIC_MODE) and # memory namespaces were removed 2026-05-20 alongside the ruflo role drop. diff --git a/work/tg-slash-command-gateway/user-spec.md b/work/tg-slash-command-gateway/user-spec.md new file mode 100644 index 00000000..a83efb08 --- /dev/null +++ b/work/tg-slash-command-gateway/user-spec.md @@ -0,0 +1,154 @@ +--- +feature: tg-slash-command-gateway +work_type: feature +size: M +status: draft +created: 2026-07-27 +last_updated: 2026-07-27 +target_repo: git@github.com:mnemonik-dev/telegram-ai-agent.git (own-the-fork, source of truth) +upstream_origin: git@github.com:pavel-molyanov/telegram-ai-agent.git (read-only sync source) +upstream_license: MIT +parent_feature: coding-fabric (consumes via Ansible role T09 — bumps telegram_ai_agent_pin) +note: | + Companion coding-fabric changes landed on branch claude/tg-claude-slash-commands-irtd4k: + - underscore-duplicate skill dirs removed from the deployed bundle (duplicate + frontmatter names broke skill resolution); + - dash/underscore command alias convention documented + (infrastructure/ansible/files/claude-skills/README.md); + - engine model unpinned from the config template + (telegram_ai_agent_claude_model / per-topic model override); + - bot-claude-reset operator helper installed to /usr/local/bin (VM-side + relogin runbook). + Everything below requires changes in the bot fork itself. +--- + +# User Spec — tg-slash-command-gateway + +Make the Telegram bot a full gateway to the agent's slash-command surface: +every command available to a local Claude Code session (native commands, +molyanov methodology commands, any operator-added command file) must be +discoverable and usable from Telegram — including the forum's General chat. + +## 1. Problems (operator report 2026-07-27) + +- **P1 — engine cannot be changed from the General chat.** Engine/model are + fixed per topic in `/etc/telegram-ai-agent/config.yml`; the General chat + (no `thread_id`) has no topic entry at all, so there is no way to switch + claude ↔ codex or change model without SSH + redeploy. +- **P2 — no reset/relogin after Claude quota exhaustion.** When the + subscription quota burns out mid-day, the bot keeps resuming a dead + session. There is no in-chat way to start a fresh session or re-establish + auth; the operator has to SSH into the VM. +- **P3 — native Claude Code slash commands are unavailable from Telegram.** + `/clear`, `/compact`, `/model` etc. either get swallowed by Telegram + command parsing or are passed as literal prompt text to a `-p` session, + where interactive-only commands do nothing. +- **P4 — molyanov command aliases don't correspond.** Molyanov docs say + `/do-task`; Telegram can only send `/do_task` (`[a-z0-9_]` limit); the + engine-side command file may exist under only one spelling. The operator + can't predict which alias works, and the agent "does not recognize it at + once". + +## 2. Scope + +### 2.1 In scope (bot fork changes) + +1. **Command discovery + Telegram registration.** + - On startup (and on `/sync_commands`), scan the engine HOME's + `~/.claude/commands/*.md`, parse frontmatter `description`. + - Register the union of (bot built-ins + discovered commands) via + Telegram `setMyCommands`, names normalized to `[a-z0-9_]{1,32}` + (dashes → underscores, truncate, dedupe). Telegram caps at 100 + commands — prefer bot built-ins, then discovered commands + alphabetically; log what was dropped. + - Result: Telegram UI autocompletes every agent command (the "main + idea": all agent slash commands visible in the TG client). +2. **Alias normalization (fixes P4).** Incoming `/foo_bar args` that is not + a bot built-in resolves against the discovered registry: exact filename + match first, then dash-spelling (`foo-bar`). The resolved canonical + spelling is what gets passed to the engine as the prompt + (`"/foo-bar args"`), so only ONE command file per command is needed + engine-side and the operator may type either spelling. +3. **Native command mapping (fixes P3).** Whitelist of Claude-native + commands meaningful in non-interactive mode, mapped to bot session + operations instead of prompt passthrough: + - `/clear`, `/new` → drop stored `session_id` for the chat/topic + (resume=False on next message); + - `/compact` → run engine with `/compact` against the stored session; + - `/model ` → persist per-chat model override (see P1 + storage below) and confirm; + - `/status` → spawn `claude -p "ping"` probe, report auth/model/quota + errors verbatim to the chat. + Unknown `/commands` fall through to alias normalization (2), then to + plain prompt passthrough with a warning. +4. **Engine switching everywhere, incl. General chat (fixes P1).** + - `/engine claude|codex` and `/model ` persist to the bot's runtime + `topic_config.json` keyed by `(chat_id, thread_id-or-"general")` — the + General chat gets a first-class config key instead of falling through + to static defaults. + - Runtime overrides take precedence over `/etc/telegram-ai-agent/config.yml`; + `/engine reset` clears the override. +5. **In-chat reset/relogin (fixes P2).** + - `/relogin` → same effect as VM-side `bot-claude-reset --wipe-sessions`: + kill stray engine subprocesses for that chat, drop session state, + re-read `.env`, run the `/status` probe, report the outcome. + - On engine failure that matches auth/quota patterns (rate_limit, + invalid api key, oauth revoked), the bot replies with an actionable + message naming `/relogin` and the VM runbook instead of silent + "Thinking...". + - Token VALUES are never accepted via chat (Telegram history is not a + secret store) — rotation stays VM-side via `bot-claude-reset`. + +### 2.2 Out of scope + +- Rust rewrite; tmux/`/tui`/voice changes; MCP server changes. +- Accepting credentials through Telegram messages. +- Multi-operator ACLs (existing single-operator allowlist unchanged). +- coding-fabric Ansible changes beyond bumping `telegram_ai_agent_pin` + (companion changes already landed, see frontmatter note). + +## 3. Acceptance criteria + +- AC1. After deploy, typing `/` in the bot chat shows the discovered command + list (spot-check: `do_task`, `new_user_spec`, `write_code`) with + descriptions from frontmatter. +- AC2. `/do_task ` and (via text) `/do-task ` reach the engine + as the same canonical command; engine transcript shows the command file + was loaded (no "unknown command" fallback). +- AC3. `/engine codex` in the **General chat** switches the engine for + subsequent General-chat messages; `/engine claude` switches back; both + survive bot restart (persisted runtime config). +- AC4. `/model sonnet` changes the model for the current chat/topic without + redeploy; `/status` reports the active engine+model. +- AC5. `/clear` (or `/new`) starts a fresh session — next reply does not + resume the previous session_id. +- AC6. With a deliberately broken `CLAUDE_CODE_OAUTH_TOKEN`, sending a + message yields an actionable auth-error reply (not endless "Thinking..."); + after fixing `.env` on the VM, `/relogin` restores service without + systemctl access. +- AC7. `/sync_commands` re-scans and re-registers after a new command file + is added engine-side; no bot restart needed. +- AC8. Unit tests: registry scan/normalization/dedupe/100-cap; alias + resolution precedence; native-command whitelist dispatch; runtime + override precedence over static config. Integration test: General-chat + engine switch end-to-end with mocked engines. +- AC9. Upstream-sync hygiene: changes isolated to new modules + minimal + handler wiring, documented in fork README (own-the-fork strategy, but + keep rebase surface small). + +## 4. Verification on coding-fabric VM + +1. Deploy fork pin bump via T09 role; `systemctl is-active telegram-ai-agent`. +2. Walk AC1–AC7 from the operator's Telegram client (General chat + one + forum topic). +3. `sudo bot-claude-reset --wipe-sessions` still works and coexists with + `/relogin` (VM-side path remains the credential-rotation path). + +## 5. Risks + +| # | Risk | Mitigation | +|---|------|------------| +| R1 | Telegram 100-command cap truncates the registry | Deterministic priority + log dropped names; `/sync_commands` output lists them | +| R2 | Native-command semantics drift with claude CLI releases | Whitelist is data-driven (single dict); `/status` probe surfaces breakage fast | +| R3 | Runtime overrides vs static config.yml precedence confusion | `/status` always prints effective engine/model/source; `/engine reset` documented | +| R4 | Fork rebase pain against upstream handler refactors | New logic in dedicated modules (command_registry.py, native_commands.py); handlers only call into them |