perf(extensions): skip re-bundling pulled extensions on cold catalog#1389
Conversation
…nd workflow run Previously, OTEL traces only covered the outer CLI, method-run, and workflow-run envelopes plus the inner driver execution — leaving 60-80% of wall-clock time invisible. This adds spans for the phases that execute in those gaps: CLI bootstrap: - swamp.cli.bootstrap (telemetry init → command tree construction) - swamp.cli.configure_extension_loaders (catalog setup) - swamp.cli.teardown (datastore sync + telemetry flush) Method run: - swamp.model.method.setup (logging → driver resolution) - swamp.model.method.evaluate_expressions (CEL evaluation) - swamp.model.method.resolve_runtime (vault/env resolution) - swamp.model.method.pre_execute (hash + output save + driver config) - swamp.model.method.post_execute (data artifacts + reports) - swamp.model.method.reports (report execution) Workflow run: - swamp.workflow.setup (marker load → run creation) - swamp.workflow.build_context (model resolver buildContext) - swamp.workflow.teardown (post-job completion) - swamp.workflow.reports (workflow-scope report execution) - swamp.workflow.save_run (run persistence) All spans use try/finally to ensure they end on error paths. Zero overhead when tracing is disabled (no-op tracer). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the extension catalog is wiped (after upgrade, corruption, or first run in a new worktree), the reconcile service previously spawned a Deno bundle subprocess for every pulled extension file — even when cached .js bundles already existed on disk from the original `extension pull`. For a repo with @swamp/aws/ec2 (107 model types), this added ~27s of pure waste on cold start (107 subprocess spawns × ~250ms each). Fix: add a `trustPulledCache` flag to `bundleWithCache` that skips the `deno bundle` subprocess when a cached bundle exists for a pulled extension. The flag is only passed from: - `reconcileExtension()` when `originType === "pulled"` - `load()` cold-catalog bootstrap path The warm-path stale-file detection (`findStaleFiles` → `rebundleAndUpdateCatalog`) does NOT pass the flag, so fingerprint-based change detection still triggers re-bundling when source files are modified. Benchmark (107 extension types, 20 models, 40 data items): - Cold catalog before: 33s - Cold catalog after: 5.7s - Warm runs: unchanged (0.85s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
CLI UX Review
Blocking
None.
Suggestions
None.
Verdict
PASS — This PR touches only internal bundling logic (extension_loader.ts, reconcile_from_disk_service.ts). No commands, renderers, help text, error messages, or output formatters are changed. Users will see the same CLI behavior, just faster (33 s → 5.7 s cold start for large extension catalogs). Nothing to flag from a UX perspective.
There was a problem hiding this comment.
Code Review
Clean, well-scoped performance fix. The change correctly threads trustPulledCache through bundleWithCache → bundleAndIndexOne → ReconcileFromDiskService, short-circuiting the isExpectedBundleFailure filesystem walk and deno bundle subprocess for pulled extensions that already have cached bundles on disk.
Blocking Issues
None.
Suggestions
- Unit test for
trustPulledCachepath —extension_loader_test.tshas no coverage of the newtrustPulledCacheoption. A test that sets up a pulled extension with an existing cached bundle and assertsbundleWithCachereturns{ fromCache: true }without invoking the subprocess would make this path regression-safe. The change is low-risk and covered by integration/UAT tests, so not blocking.
Notes
- No
anytypes, no fire-and-forget promises, no libswamp import boundary violations. isExpectedBundleFailureis a heuristic about bare specifiers / missing deno.json — not a security gate — so bypassing it for pulled extensions with existing bundles is safe.- Fingerprint checks in
reconcileExtension(lines 430-437) still run before the bundle path, so changed sources are re-bundled regardless oftrustPulledCache. - The option stays internal to
ExtensionLoaderandReconcileFromDiskService—bundleAndIndexOneisn't exported throughsrc/libswamp/mod.ts, so no public API impact.
There was a problem hiding this comment.
Adversarial Review
Critical / High
None found.
Medium
src/domain/extensions/extension_loader.ts:156—trustPulledCache: trueis hardcoded inload(), which processes both pulled and user-written extensions.
Theload()method at line 100 is called fromsrc/cli/mod.ts:579with mixed directories (absoluteVaultsDir+ user source dirs + pulled dirs). HardcodingtrustPulledCache: truelooks overly broad at first glance. However, this is safe in practice becausebundleWithCacheat line 962 gates on theisPulledpath check (boundaryDirmust be underpulled-extensions/), so user-written extensions never hit the short-circuit. The flag is effectively a no-op for non-pulled files. Still, passingtrueunconditionally is misleading — a future reader might assume it affects all files. Consider threading the origin info throughload()like the reconcile service does, or adding a comment explaining why the blankettrueis safe.
Low
-
src/domain/extensions/extension_loader.ts:940— Optional parameter bag on internal method. Addingoptions?: { trustPulledCache?: boolean }tobundleWithCachemeans three of the four call sites now pass it and one (rebundleAndUpdateCatalogat line 795) doesn't, relying on the defaultundefined. This is fine and the default behavior (fall through to the subprocess) is correct for the warm-path rebundle case. Just noting the asymmetry — no action needed. -
Theoretical staleness window. If a pulled extension's
.tssource were updated on disk without a corresponding.jsbundle update,trustPulledCachewould serve the stale bundle. I verified that thepulloperation (src/libswamp/extensions/pull.ts:425-427) extracts pre-built bundles alongside sources, so they're always in sync. The only way to hit this would be manual filesystem tampering, which is outside the threat model.
Verdict
PASS — The change is minimal, well-scoped, and correct. The isPulled path check in bundleWithCache prevents the optimization from ever applying to user-written extensions. The pull pipeline guarantees source/bundle consistency for pulled extensions. The warm-path stale detection is untouched. The 6x cold-start improvement with no correctness trade-off is a clean win.
Summary
deno bundlesubprocess for every pulled extension file — even when cached.jsbundles already existed on disk@swamp/aws/ec2(107 model types), this added ~27s of pure waste on cold startBenchmark (107 types, 20 models, 40 data items)
Safety
findStaleFiles) still re-bundles when fingerprints changeTest Plan
🤖 Generated with Claude Code