Skip to content

feat(skillopt): add meta-skill module for optimizer cross-epoch memory - #1271

Draft
osilkin98 wants to merge 1 commit into
mainfrom
factory/run-67b2c051
Draft

feat(skillopt): add meta-skill module for optimizer cross-epoch memory#1271
osilkin98 wants to merge 1 commit into
mainfrom
factory/run-67b2c051

Conversation

@osilkin98

Copy link
Copy Markdown
Collaborator

Closes #1236

Changes

  • New skillopt/ package — adds optimizer-side meta-skill memory for cross-epoch learning
  • Core module (skillopt/optimizer/meta_skill.py): format_meta_skill_context() renders optimizer memory into prompt-ready blocks; run_meta_skill() calls the optimizer model with adjacent-epoch comparison data; load_meta_skill_content() loads previous epoch results with 3-epoch recency window; validate_deployment_gate() catches meta-skill leakage in final SKILL.md; should_generate_meta_skill() enforces epoch-2 minimum and score-delta conditioning
  • Prompt template (skillopt/prompts/meta_skill.md): optimizer-coach system prompt that produces abstract optimizer-facing guidance, not target-facing task instructions
  • Trainer hooks (skillopt/engine/trainer.py): generate_epoch_meta_skill() epoch-boundary hook with resume safety, first-epoch skip, and score-delta conditioning; load_active_meta_skill() epoch-start loading
  • Reflect injection (skillopt/gradient/reflect.py): reflect_on_errors(), reflect_on_successes(), reflect_and_merge() — all accept meta_skill_context parameter for in-memory prepend to optimizer prompts (on-disk analyst templates are never modified)
  • Safety constraints: 3000 token cap, 3-epoch recency window, score-delta conditioning, deployment gate validation, file separation (meta_skill/ directory, never skills/)
  • 49 unit tests covering all key functions including edge cases (malformed JSON, token overflow, recency boundary, negative delta skip)

Mirrors the architecture from the reference implementation at ~/SkillOpt/ but adapted for the factory's module structure and dependency injection pattern (explicit chat_fn parameter instead of global import).

#1236)

Add a new skillopt/ package implementing optimizer-side meta-skill
memory that accumulates cross-epoch learnings and feeds them back
to future optimizer calls.

New modules:
- skillopt/optimizer/meta_skill.py: Core meta-skill generation with
  format_meta_skill_context(), run_meta_skill(), load_meta_skill_content(),
  validate_deployment_gate(), should_generate_meta_skill()
- skillopt/prompts/meta_skill.md: LLM prompt template for meta-skill
  generation (optimizer-coach, not target-facing)
- skillopt/engine/trainer.py: Epoch-boundary hook (generate_epoch_meta_skill)
  and epoch-start loading (load_active_meta_skill)
- skillopt/gradient/reflect.py: Reflect functions with meta_skill_context
  parameter for in-memory prepend to optimizer prompts

Safety constraints:
- 3000 token cap (approximate: len // 4)
- 3-epoch recency window for loading
- Score-delta conditioning (skip negative-delta epochs)
- Deployment gate validation (detect leakage markers in final SKILL.md)
- File separation (meta_skill/ directory, never skills/)

Tests: 49 unit tests covering all key functions.
@github-actions

Copy link
Copy Markdown

Sentrux Quality Report

Absolute

Scanning ....
[scan] git ls-files: 619 total, 607 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 607 files, 107 unique dirs, 99 cache misses, 6.1ms
[resolve] 1097 resolved, 1546 unresolved (of 2643 total specs)
[resolve_imports] project_map 6.2ms, suffix_idx 1.1ms, suffix_resolve 17.7ms, total 25.0ms
[build_graphs] 607 files | maps 2.2ms, imports 25.2ms, calls+inherit 5.9ms, total 33.3ms | 1096 import, 8682 call, 11 inherit edges
sentrux check — 3 rules checked

Quality: 4692

✗ [Error] max_cc: 3 function(s) exceed max cyclomatic complexity of 30
    factory/cli/_ceo_helpers.py:_validate_ceo_flags (cc=43)
    factory/cli/_ceo_helpers.py:_execute_ceo (cc=43)
    factory/cli/run.py:cmd_run (cc=32)

✗ 1 violation(s) found

Diff (vs base branch)

Scanning ....
[scan] git ls-files: 619 total, 607 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 607 files, 107 unique dirs, 99 cache misses, 6.4ms
[resolve] 1097 resolved, 1546 unresolved (of 2643 total specs)
[resolve_imports] project_map 6.5ms, suffix_idx 1.1ms, suffix_resolve 17.6ms, total 25.3ms
[build_graphs] 607 files | maps 2.2ms, imports 25.4ms, calls+inherit 6.6ms, total 34.2ms | 1096 import, 8682 call, 11 inherit edges
sentrux gate — structural regression check

Quality:      4694 -> 4692
Coupling:     0.79 → 0.79
Cycles:       3 → 3
God files:    3 → 3

Distance from Main Sequence: 0.38

✓ No degradation detected

@codecov

codecov Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.24%. Comparing base (1925cdf) to head (e051c72).
⚠️ Report is 138 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1271      +/-   ##
==========================================
- Coverage   88.22%   86.24%   -1.99%     
==========================================
  Files         139      211      +72     
  Lines       15524    23096    +7572     
  Branches     2433     3640    +1207     
==========================================
+ Hits        13696    19919    +6223     
- Misses       1308     2333    +1025     
- Partials      520      844     +324     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SkillOpt: implement meta-skill (optimizer self-improvement across epochs)

2 participants