build: support dd-trace package override and filter publish layer check

[joeyzhao2018]

Summary

Two improvements to the layer build/publish flow, mirroring the recent changes to datadog-lambda-python:

1. scripts/build_layers.sh + Dockerfile — add the ability to build a layer with an arbitrary dd-trace package, useful for git-bisect or testing a feature branch of dd-trace-js end-to-end inside the layer. Three new (mutually exclusive, opt-in) env vars on the build script:

   • DD_TRACE_PACKAGE — exact package spec passed straight to npm/yarn (e.g. npm:dd-trace@5.0.0, file:./local-tarball.tgz).
   • DD_TRACE_COMMIT — dd-trace-js commit SHA, expanded to git+https://github.com/DataDog/dd-trace-js.git#<sha>.
   • DD_TRACE_COMMIT_BRANCH — dd-trace-js branch name, expanded the same way.

   The resolved spec is forwarded to docker via --build-arg dd_trace_package=.... When none of those env vars are set, the build-arg is empty and the Dockerfile takes its existing path.

2. scripts/publish_layers.sh — pre-flight check now only requires the zips for layers in LAYERS, not all four entries in LAYER_PATHS. When LAYERS is unset, it still resolves to all available layers, so default invocations behave identically.

About the Dockerfile change (it's a bit ugly — here's why)

The new step in the Dockerfile is a one-line node -e '...' that rewrites package.json to swap the dd-trace entry over to the override spec, gated on the new build-arg:

ARG dd_trace_package
...
RUN if [ -n "$dd_trace_package" ]; then \
    node -e 'const fs = require("fs"); const pkg = JSON.parse(fs.readFileSync("package.json", "utf8")); const spec = process.argv[1]; if (pkg.devDependencies && pkg.devDependencies["dd-trace"]) { pkg.devDependencies["dd-trace"] = spec; } else { pkg.dependencies = pkg.dependencies || {}; pkg.dependencies["dd-trace"] = spec; } fs.writeFileSync("package.json", JSON.stringify(pkg, null, 2) + "\n");' "$dd_trace_package"; \
    fi
RUN yarn install

It's not pretty, but the alternatives were worse:

• yarn add dd-trace@<spec> instead of editing package.json: yarn add triggers a resolve+install during the rewrite step, then yarn install runs again on the next layer — duplicate work and a worse-cached image.
• sed/jq-based JSON edit: sed on JSON is fragile (formatting, quoting, key ordering); jq isn't preinstalled in the node base image.
• Separate helper script copied in: would split the override logic across two files instead of keeping it self-contained in the Dockerfile.

Since node is guaranteed to exist (it's the layer's whole point), an inline node -e rewrite is the most hermetic option even if the one-liner reads rough. The if guard means the entire branch is dead code in default builds.

The script also handles a quirk of the current package.json: dd-trace lives in devDependencies (because yarn packs it into the layer build, not at runtime) — so we update it in place if it's already there, and only fall back to writing into dependencies if it isn't. That preserves the existing dev-vs-prod placement.

Backwards compatibility

• No new env vars set → DD_TRACE_PACKAGE_SPEC is empty → build-arg is empty → Dockerfile if is false → identical to today.
• LAYERS unset on publish → still requires all 4 zips.
• Dockerfile change is a no-op unless the new build-arg is non-empty.

Test plan

• DD_TRACE_COMMIT_BRANCH=<branch> NODE_VERSION=20.19 ./scripts/build_layers.sh produces a layer that uses the override.
• VERSION=N REGIONS=us-east-2 LAYERS=Datadog-Node20-x ./scripts/publish_layers.sh proceeds with only the matching zip built.
• Default ./scripts/build_layers.sh (no override env vars) builds layers byte-identical to current behavior.