feat(quality): install-readiness probe — every listed artifact must actually install - #271
Open
adamkrawczyk wants to merge 1 commit into
Open
feat(quality): install-readiness probe — every listed artifact must actually install#271adamkrawczyk wants to merge 1 commit into
adamkrawczyk wants to merge 1 commit into
Conversation
…ctually install
Adam, 2026-08-21: 'not all artifacts are ready to be installed' on
LoopSkill. Builds the deterministic gate that walks the REAL install path
for every hosted artifact kind (skills, personalities, bundles), mirroring
scripts/bundle_validate.py + scripts/personality_validate.py conventions
exactly (same _get() transport shim, 429-aware backoff, WARN-vs-FAIL split,
exit contract, --json shape).
scripts/install_probe.py — per-kind gate ladders:
skills: catalog listing -> detail 200 -> install-resolve 200
-> tarball fetch/non-empty/valid-gzip -> SKILL.md +
skill.toml parse -> related_skills refs exist
personalities: catalog listing -> detail 200 -> non-empty system_prompt
-> recommended/member skill refs exist
bundles: catalog listing -> public well-known install-index 200
-> >=1 member -> every member (local + federated) resolves
A non-free tier correctly 401ing anonymous install is recorded as a
skip (gate working as designed), not a failure — only an install path
that SHOULD resolve and doesn't counts as broken.
Ran read-only against prod (app.loopskill.io): 69 artifacts checked
(57 skills, 2 personalities, 10 bundles), 60 pass, 9 fail, 0 skip.
Failing:
- skill:agentic-os — tier unset/None, anon install 401s despite
looking free-tier in the public catalog
- skill:super-memory — 9 related_skills entries in the shipped
SKILL.md point at slugs not in the catalog
- bundle:agent-marketing, cold-outreach-and-email,
copywriting-and-humanizer, coreys-marketing,
proactive-and-autonomous-agent-ops, seo-and-search-growth,
terraform-and-kubernetes-ops
— federated (ext:) members do not resolve via
/api/federation/filter (reproduced
independently against bundle_validate.py's
existing G1f gate — same root cause, not a
new-script artifact)
tests/test_install_probe.py — 31 tests, RED+GREEN per gate branch,
transport mocked via monkeypatched _get/_get_bytes (same Router pattern
as test_bundle_factory_rails.py / test_personality_factory_rails.py),
including a real in-memory gzip tarball builder for the G4/G5/G6 gates.
README.md — synced the test-file-count claim (425 -> 426) so
tests/test_readme_claims.py stays green with the new test file added.
Local gates green: targeted pytest (77 passed: install_probe +
readme_claims + tier_vocab_audit + bundle/personality factory rails +
route_manifest_regenerates), ruff check + ruff format --check on app/ and
the new files. No routes added, no app/ changes — hygiene-only per
AGENTS.md Phase-G scope.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adam, 2026-08-21: "not all artifacts are ready to be installed" on LoopSkill.
scripts/install_probe.pyis the deterministic gate that answers this directly: it walks the REAL install path for every hosted artifact kind (skills, personalities, bundles) — not the catalog row, the actual bytes an installer would fetch. Mirrorsscripts/bundle_validate.py/scripts/personality_validate.pyconventions exactly: same_get()transport shim, 429-aware backoff, WARN-vs-FAIL separation, exit contract (0 all-pass / 1 failures / 2 infra),--jsonflag.Gate ladders per artifact kind
related_skillsrefs exist in the catalogsystem_prompt→recommended_skills/member_skillsrefs existext:) resolvesA non-free tier correctly 401-ing anonymous install is recorded as a
skip(the gate working as designed) — only an install path that SHOULD resolve and doesn't counts as a failure. This is deliberate: not every "broken" is a bug, and the ledger shouldn't cry wolf on a working gate.Ran read-only against prod (app.loopskill.io)
69 artifacts checked (57 skills, 2 personalities, 10 bundles) → 60 pass, 9 fail, 0 skip.
Failing:
skill:agentic-os—tieris unset/None; anonymous install 401s despite the skill appearing free-tier in the public catalog (/api/statseven buckets it as"uncategorized": 1)skill:super-memory— 9related_skillsentries in the SHIPPED SKILL.md (cognee-kuzu-to-pghybrid-migration, cognee-litellm-proxy-rotation, cognee-api-watchdog, cognee-nightly-ingest-optimization, cognee-retrieval-architecture, cognee-llm-provider-swap, cognee-minor-version-upgrade, cognee-agent-coordination, vault-context-loader) point at slugs that are NOT in the public catalogbundle:agent-marketing,cold-outreach-and-email,copywriting-and-humanizer,coreys-marketing,proactive-and-autonomous-agent-ops,seo-and-search-growth,terraform-and-kubernetes-ops— federated (ext:) members do not resolve via/api/federation/filter. Independently reproduced against the existingbundle_validate.py --slug <these>(its G1f gate hits the identical failures) — confirms this is a real, pre-existing federation-resolution gap, not an artifact of the new script.Tests
tests/test_install_probe.py— 31 tests, RED+GREEN per gate branch, transport mocked via monkeypatched_get/_get_bytes(sameRouterpattern astest_bundle_factory_rails.py/test_personality_factory_rails.py), including a real in-memory gzip tarball builder to exercise the G4/G5/G6 tarball gates without hitting the network.README
Synced the test-file-count claim (425 → 426) so
tests/test_readme_claims.pystays green with the new test file added (CI trap called out in the brief — caught and fixed).Local gates (all green)
pytest tests/test_install_probe.py tests/test_readme_claims.py tests/test_w0_5_tier_vocab_audit.py tests/test_bundle_factory_rails.py tests/test_personality_factory_rails.py tests/test_route_manifest_regenerates.py→ 77 passedruff check app/ scripts/install_probe.py tests/test_install_probe.py→ all checks passedruff format --check app/ scripts/install_probe.py tests/test_install_probe.py→ all formattedNo new routes added (manifest gate untouched), no
app/changes — hygiene/tooling-only per AGENTS.md Phase-G scope. Not merging — for review.