Skip to content

feat: resolve npm sources via npm pack #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kreneskyp merged 4 commits into from
Jun 28, 2026
Merged

feat: resolve npm sources via npm pack #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bd98f1a
feat: resolve npm sources via npm pack
kreneskyp Jun 27, 2026
3984df6
harden: reject option-like npm package to prevent npm pack flag injec…
kreneskyp Jun 27, 2026
dfb3cd6
harden: robust npm pack parse, fetcher-dir contract, cache hygiene
kreneskyp Jun 28, 2026
37f28ca
fix: stricter npm-pack parse + spec/matrix backsync for npm robustness
kreneskyp Jun 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 56 additions & 19 deletions spec/functional/FR-004-source-resolution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,13 @@ relationships:

The library SHALL export `resolveSource(source, opts)` which synchronously fetches
a source to a local directory and returns a `ResolvedSource` (`dir`, `sha?`,
`ref?`). `git` (via the injectable `GitRunner`, default `defaultGitRunner` using
`execFileSync`) SHALL be the only side effect. `ResolveOptions` carries a
`cacheRoot` (sources cached under `<cacheRoot>/git/<key>`) and an optional `git`
runner. The function SHALL validate the source via `normalizeSource` ([FR-001](./FR-001-typed-source-union.md)) and
`ref?`). A per-source package-manager subprocess SHALL be the only side effect:
`git` for git sources (via the injectable `GitRunner`, default `defaultGitRunner`
using `execFileSync`); `npm pack` + `tar` for `npm` sources (via the injectable
`NpmFetcher`, default `defaultNpmFetcher`). `ResolveOptions` carries a `cacheRoot`
(git sources cached under `<cacheRoot>/git/<key>`, npm sources under
`<cacheRoot>/npm/<key>`), an optional `git` runner, and an optional `npm` fetcher.
The function SHALL validate the source via `normalizeSource` ([FR-001](./FR-001-typed-source-union.md)) and
expand git URLs via `toGitUrl` ([FR-002](./FR-002-git-url-shorthand.md)).

## Behavior
Expand All @@ -31,8 +34,18 @@ expand git URLs via `toGitUrl` ([FR-002](./FR-002-git-url-shorthand.md)).
- **`path` source**: resolve `source.path` to an absolute path; if it does not
exist, throw `SourceError("path source not found: <dir>")`; otherwise return
`{ dir }` (no `sha`, no `ref`).
- **`url` / `npm` source**: throw `UnsupportedSourceError` (message mentions the
type is not yet supported) — these reserved types are not resolved.
- **`url` source**: throw `UnsupportedSourceError` (message mentions the type is
not yet supported) — this reserved type is not resolved.
- **`npm` source**: cache under `<cacheRoot>/npm/<key>` (key sanitized from
`<package>@<version | "latest">`); invoke the `NpmFetcher` to `npm pack` the
package tarball and extract it, then return `{ dir, sha, ref }` where `dir` is
the extracted content root, `sha` is the **resolved published version** (the
durable pin, mirroring a git sha), and `ref` echoes back the requested
`source.version`. An **exact-version** pin (`X.Y.Z[-+…]`) whose extracted
`package.json` and recorded version are already cached is reused without
re-fetching; an unpinned or range spec re-fetches every time to honor "latest".
On a re-fetch (or any cache miss) the npm cache dir is cleared before
re-extracting so stale tarball artifacts do not accumulate (FR-004-AC-13).
- **git sources** (`github`, `git`, `git-subdir`):
- Compute the clone URL via `toGitUrl(source.repo | source.url)` and the cache
dir `<cacheRoot>/git/<sanitized-url>` (non-`[A-Za-z0-9._-]` chars replaced
Expand Down Expand Up @@ -68,24 +81,48 @@ spec.md §14 Known Limitations).
"pipe"] })`; an injected `GitRunner` replaces it so tests run with no real git
(FR-004-AC-7).

**`defaultNpmFetcher`.** The default fetcher runs `npm pack <ref> --pack-destination
<destDir> --json` (where `<ref>` is `<package>@<version>` when a version is given,
else `<package>`, plus `--registry <registry>` when supplied), parses the JSON
metadata for the tarball `filename` and resolved `version`, then extracts the
tarball with `tar -xzf <tarball> -C <destDir>` and returns `{ dir:
<destDir>/package, version }`. The `npm pack` argv is built by the pure, exported
`npmPackArgs(spec, destDir)` helper so the version and `--registry` branches are
unit-testable offline (FR-004-AC-11). Metadata parsing is delegated to the pure,
exported `parseNpmPackJson(out)` helper, which scans past lifecycle-script stdout
noise and returns the first npm-pack metadata object (an object with a string
`filename`), throwing a descriptive `SourceError` on no-array/empty/invalid
output (FR-004-AC-12, FR-004-CON-4). `npm pack` of a **local folder** runs
offline (no registry), which is how the default fetcher is integration-tested
(FR-004-AC-10). An injected `NpmFetcher` replaces the default so tests run with no
real `npm`/network (FR-004-AC-8, -AC-9).

## Constraints

| ID | Constraint | Type | Validation |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------- | ------------- | ------------- |
| FR-004-CON-1 | Git is the sole side effect; resolution performs no other I/O beyond filesystem reads/dir creation and the `git` subprocess. | architectural | Test (TC-011) |
| FR-004-CON-2 | The clone is blobless and no-checkout (`--filter=blob:none --no-checkout`); subdir sources sparse-checkout only the requested path. | performance | Test (TC-008) |
| ID | Constraint | Type | Validation |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ------------- |
| FR-004-CON-1 | The only side effect is the per-source package-manager subprocess (`git` for git sources; `npm pack` + `tar` for npm sources); resolution performs no other network or I/O beyond filesystem reads/dir creation. | architectural | Test (TC-011) |
| FR-004-CON-2 | The clone is blobless and no-checkout (`--filter=blob:none --no-checkout`); subdir sources sparse-checkout only the requested path. | performance | Test (TC-008) |
| FR-004-CON-3 | `normalizeSource` rejects an `npm` `package` beginning with `-`, so it cannot reach `npm pack` as a CLI flag (second-order command-line-injection guard). | security | Test (TC-026) |
| FR-004-CON-4 | When `npm pack --json` output contains no metadata array (no array at all, an empty array, or an array whose first element is not an object with a string `filename`), `parseNpmPackJson` throws a descriptive `SourceError` rather than returning garbage or a confusing downstream failure. | robustness | Test (TC-027) |

## Acceptance Criteria

| ID | Criteria | Verification |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| FR-004-AC-1 | A `path` source for an existing directory returns `{ dir }` pointing at that directory (its contents are readable). | Test (TC-006) |
| FR-004-AC-2 | A `path` source for a non-existent directory throws `SourceError`. | Test (TC-006) |
| FR-004-AC-3 | A `url` source and an `npm` source each throw `UnsupportedSourceError`. | Test (TC-007) |
| FR-004-AC-4 | A `git-subdir` source pinned to a tag returns `dir` ending in the subdir path, contains the subdir's files, and reports the tag's `sha` and the requested `ref`. | Test (TC-008) |
| FR-004-AC-5 | A whole-repo `git` source with no pin resolves to `HEAD` (latest commit); re-resolving the same cached URL at a tag exercises the fetch branch and resolves that tag's sha. | Test (TC-009) |
| FR-004-AC-6 | A `git` source pinned by `sha` checks out exactly that commit. | Test (TC-010) |
| FR-004-AC-7 | A `github` source resolved with an injected `GitRunner` performs no real git: the returned `sha` is the runner's output and the first git argv is `clone`. | Test (TC-011) |
| ID | Criteria | Verification |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| FR-004-AC-1 | A `path` source for an existing directory returns `{ dir }` pointing at that directory (its contents are readable). | Test (TC-006) |
| FR-004-AC-2 | A `path` source for a non-existent directory throws `SourceError`. | Test (TC-006) |
| FR-004-AC-3 | A `url` source throws `UnsupportedSourceError`. | Test (TC-007) |
| FR-004-AC-4 | A `git-subdir` source pinned to a tag returns `dir` ending in the subdir path, contains the subdir's files, and reports the tag's `sha` and the requested `ref`. | Test (TC-008) |
| FR-004-AC-5 | A whole-repo `git` source with no pin resolves to `HEAD` (latest commit); re-resolving the same cached URL at a tag exercises the fetch branch and resolves that tag's sha. | Test (TC-009) |
| FR-004-AC-6 | A `git` source pinned by `sha` checks out exactly that commit. | Test (TC-010) |
| FR-004-AC-7 | A `github` source resolved with an injected `GitRunner` performs no real git: the returned `sha` is the runner's output and the first git argv is `clone`. | Test (TC-011) |
| FR-004-AC-8 | An `npm` source resolved with an injected `NpmFetcher` extracts the tarball content (manifest present at the returned `dir`) and pins the **resolved version** as `sha`, echoing the requested version as `ref`. | Test (TC-022) |
| FR-004-AC-9 | An **exact-version** `npm` pin is served from cache on a second resolve (fetcher invoked once); an **unpinned** spec re-fetches every resolve (fetcher invoked each time) and pins the resolved version returned by the fetcher. | Test (TC-023) |
| FR-004-AC-10 | `defaultNpmFetcher` packs and extracts a **local** package folder via `npm pack` + `tar`, fully offline, returning the package version and an extracted content root containing `package.json`. | Test (TC-024) |
| FR-004-AC-11 | `npmPackArgs` builds the correct argv for a pinned spec (`<pkg>@<version>`), an unpinned spec (`<pkg>`), and includes `--registry <registry>` when a registry is supplied. | Test (TC-025) |
| FR-004-AC-12 | `parseNpmPackJson` robustly parses the `npm pack --json` metadata: it scans past lifecycle-script stdout noise (including stray `[` brackets, empty arrays, and non-metadata noise arrays) and returns the first array whose first element is an npm-pack metadata object (an object with a string `filename`); on no-array/empty/invalid output it throws a descriptive `SourceError`. | Test (TC-027) |
| FR-004-AC-13 | An **unpinned** (or range) `npm` re-fetch clears any prior cached tarball + extraction before re-extracting, so the npm cache dir does not accumulate stale tarball artifacts across "latest" re-fetches. | Test (TC-028) |

## Dependencies

Expand Down
26 changes: 15 additions & 11 deletions spec/non-functional/NFR-003-synchronous-zero-git-hot-path.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,26 +14,30 @@ relationships:

## Statement

All operations SHALL be **synchronous**, with the `git` subprocess (via
`GitRunner`) as the only external side effect; the library SHALL NOT introduce
async/Promise APIs or non-git network calls. A `lazy` reconcile of an
All operations SHALL be **synchronous**, with a per-source package-manager
subprocess — `git` (via `GitRunner`) and, for npm sources, `npm pack` + `tar` (via
`NpmFetcher`) — as the only external side effect; the library SHALL NOT introduce
async/Promise APIs or any other network calls. A `lazy` reconcile of an
already-settled manifest SHALL perform **zero** git invocations so it is safe on a
per-CLI-invocation hot path.

## Measurement and Evaluation

| Metric | Target | Threshold | Method |
| ------------------------------------------------------------- | ------ | --------- | ---------- |
| Git invocations on a 2nd lazy reconcile of a settled manifest | 0 | 0 | Test |
| Promise-returning functions in the public API | 0 | 0 | Inspection |
| Non-git external side effects in `resolveSource` | 0 | 0 | Analysis |
| Metric | Target | Threshold | Method |
| ---------------------------------------------------------------------- | ------ | --------- | ---------- |
| Git invocations on a 2nd lazy reconcile of a settled manifest | 0 | 0 | Test |
| Promise-returning functions in the public API | 0 | 0 | Inspection |
| External side effects in `resolveSource` beyond `git`/`npm pack`+`tar` | 0 | 0 | Analysis |

## Verification

- The reconcile test counts `GitRunner` calls; the second lazy reconcile of a
settled manifest asserts the count is exactly `0` ([FR-007-AC-2](../functional/FR-007-reconcile.md)).
- Inspect the public API in `src/index.ts`: every exported function is synchronous
(no `async`, no `Promise<...>` return types).
- `resolveSource` performs only filesystem reads/dir creation and the injected
`git` subprocess; with an injected fake `GitRunner` it resolves with no real git
([FR-004-AC-7](../functional/FR-004-source-resolution.md)).
- `resolveSource` performs only filesystem reads/dir creation and the per-source
package-manager subprocess (`git`; or `npm pack` + `tar` for npm sources); with
an injected fake `GitRunner` it resolves git sources with no real git
([FR-004-AC-7](../functional/FR-004-source-resolution.md)), and with an injected
fake `NpmFetcher` it resolves npm sources with no real `npm`/network
([FR-004-AC-8](../functional/FR-004-source-resolution.md)).
Loading
Loading