feat(skillopt): add meta-skill module for optimizer cross-epoch memory - #1271
Draft
osilkin98 wants to merge 1 commit into
Draft
feat(skillopt): add meta-skill module for optimizer cross-epoch memory#1271osilkin98 wants to merge 1 commit into
osilkin98 wants to merge 1 commit into
Conversation
#1236) Add a new skillopt/ package implementing optimizer-side meta-skill memory that accumulates cross-epoch learnings and feeds them back to future optimizer calls. New modules: - skillopt/optimizer/meta_skill.py: Core meta-skill generation with format_meta_skill_context(), run_meta_skill(), load_meta_skill_content(), validate_deployment_gate(), should_generate_meta_skill() - skillopt/prompts/meta_skill.md: LLM prompt template for meta-skill generation (optimizer-coach, not target-facing) - skillopt/engine/trainer.py: Epoch-boundary hook (generate_epoch_meta_skill) and epoch-start loading (load_active_meta_skill) - skillopt/gradient/reflect.py: Reflect functions with meta_skill_context parameter for in-memory prepend to optimizer prompts Safety constraints: - 3000 token cap (approximate: len // 4) - 3-epoch recency window for loading - Score-delta conditioning (skip negative-delta epochs) - Deployment gate validation (detect leakage markers in final SKILL.md) - File separation (meta_skill/ directory, never skills/) Tests: 49 unit tests covering all key functions.
Sentrux Quality ReportAbsoluteDiff (vs base branch) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1271 +/- ##
==========================================
- Coverage 88.22% 86.24% -1.99%
==========================================
Files 139 211 +72
Lines 15524 23096 +7572
Branches 2433 3640 +1207
==========================================
+ Hits 13696 19919 +6223
- Misses 1308 2333 +1025
- Partials 520 844 +324 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1236
Changes
skillopt/package — adds optimizer-side meta-skill memory for cross-epoch learningskillopt/optimizer/meta_skill.py):format_meta_skill_context()renders optimizer memory into prompt-ready blocks;run_meta_skill()calls the optimizer model with adjacent-epoch comparison data;load_meta_skill_content()loads previous epoch results with 3-epoch recency window;validate_deployment_gate()catches meta-skill leakage in final SKILL.md;should_generate_meta_skill()enforces epoch-2 minimum and score-delta conditioningskillopt/prompts/meta_skill.md): optimizer-coach system prompt that produces abstract optimizer-facing guidance, not target-facing task instructionsskillopt/engine/trainer.py):generate_epoch_meta_skill()epoch-boundary hook with resume safety, first-epoch skip, and score-delta conditioning;load_active_meta_skill()epoch-start loadingskillopt/gradient/reflect.py):reflect_on_errors(),reflect_on_successes(),reflect_and_merge()— all acceptmeta_skill_contextparameter for in-memory prepend to optimizer prompts (on-disk analyst templates are never modified)meta_skill/directory, neverskills/)Mirrors the architecture from the reference implementation at
~/SkillOpt/but adapted for the factory's module structure and dependency injection pattern (explicitchat_fnparameter instead of global import).