From 0a85af879bd8f6509a67234ec763ad5eb62d998c Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sun, 16 Aug 2026 12:24:36 +0700 Subject: [PATCH 1/2] feat(devcontainer): bake the rate-limit-fragile toolchain into a Dockerfile `.devcontainer/` grows a Dockerfile and loses its compose file, so it now reads image -> environment -> services: Dockerfile, devcontainer.json, stacks/. WHY THE DOCKERFILE bun, uv and go-task were Features. All three route through nanolayer's gh-release installer, which lists a release's assets by calling api.github.com with no credentials: nanolayer/installers/gh_release/resolvers/asset_resolver.py:126 urllib.request.urlopen(f"https://api.github.com/repos/{repo}/releases/tags/{tag}") Codespaces build hosts and GitHub-hosted runners share egress IP pools, so the 60 req/hr anonymous limit is routinely exhausted, the call 403s, and one failed Feature fails the whole image build -- after which Codespaces drops the developer into a bare recovery container. Pinning does not help: the pin supplies the tag, but _get_release_assets() still calls the API to list assets. The three are now installed by .devcontainer/Dockerfile from URLs that never touch the API, as pinned ARGs. mise joins them for a different reason: it was an unpinned `curl https://mise.run | sh` in post-create, the only unpinned tool in a template that pins everything else. Every other third-party Feature was checked and kept: devcontainers-extra deno and lukewiwa shellcheck curl releases/download/... directly, and robbert229 postgresql-client is apt. `mise install` deliberately stays in post-create -- four of the six entries in mise.toml use the npm: and pipx: backends, and Node and Python arrive from Features, which layer after the Dockerfile stage. WHY THE COMPOSE MOVE The orchestrator moves to .devcontainer/stacks/compose.yaml so stacks/ is self-contained. That breaks Compose's positional .env discovery, which resolves against the directory of the first -f file. Left alone the failure is silent: every ${VAR:-default} takes its default and COMPOSE_PROFILES reads as empty, so all opt-in stacks vanish without an error. Measured, same file: with --env-file, 10 services resolve; without, 1. So every caller -- startup.sh, the MOTD, and CI -- now names the env file, and the orchestrator pins `name: musher-dev` because Compose would otherwise derive the project name from the stacks/ folder. ENFORCEMENT Re-adding one of those Features looks like a harmless simplification, so it is a check rather than a comment. New `toolchain` policy: TC-01 bun/uv/go-task must not appear in the features block TC-02 every image-baked ARG is an exact pin TC-03 TASK_VERSION matches CI's arduino/setup-task version It runs under the existing governance job, and scripts/verify-toolchain.sh replaces the build job's `echo` so CI asserts the built container reports the pinned versions. No new CI job, so the rulesets and hooks policies are untouched. Verified: shellcheck, markdownlint, yamllint, actionlint, codespell and all five governance policies pass; `compose config` resolves under default and all profiles; each stack's relative bind mounts still resolve against its own folder. The image build and the docker-run version assertion are unrun here -- no Docker daemon -- and land with the first CI run. Co-Authored-By: Claude Opus 5 (1M context) --- .devcontainer/.dockerignore | 14 ++ .devcontainer/.env.example | 10 +- .devcontainer/Dockerfile | 129 ++++++++++++++++++ .devcontainer/compose.yaml | 27 ---- .devcontainer/devcontainer-lock.json | 15 -- .devcontainer/devcontainer.json | 35 +++-- .devcontainer/scripts/lib/base-setup.sh | 13 +- .devcontainer/scripts/lib/motd.sh | 20 ++- .devcontainer/scripts/startup.sh | 30 +++- .devcontainer/scripts/verify-toolchain.sh | 106 ++++++++++++++ .devcontainer/stacks/compose.yaml | 39 ++++++ .github/dependabot.yml | 15 +- .github/workflows/validate.yaml | 22 ++- .repo/README.md | 19 +++ .repo/governance/policies/__init__.py | 3 +- .../governance/policies/toolchain/__init__.py | 5 + .repo/governance/policies/toolchain/check.py | 86 ++++++++++++ .../policies/toolchain/violations.py | 95 +++++++++++++ CONFIGURATION.md | 102 ++++++++++++-- README.md | 28 ++-- 20 files changed, 709 insertions(+), 104 deletions(-) create mode 100644 .devcontainer/.dockerignore create mode 100644 .devcontainer/Dockerfile delete mode 100644 .devcontainer/compose.yaml create mode 100644 .devcontainer/scripts/verify-toolchain.sh create mode 100644 .devcontainer/stacks/compose.yaml create mode 100644 .repo/governance/policies/toolchain/__init__.py create mode 100644 .repo/governance/policies/toolchain/check.py create mode 100644 .repo/governance/policies/toolchain/violations.py diff --git a/.devcontainer/.dockerignore b/.devcontainer/.dockerignore new file mode 100644 index 0000000..17ec350 --- /dev/null +++ b/.devcontainer/.dockerignore @@ -0,0 +1,14 @@ +# Build context for .devcontainer/Dockerfile is `.devcontainer/` itself, and the +# Dockerfile has no COPY instruction -- every tool it bakes is fetched over +# the network from a pinned URL. So the correct context is empty. +# +# This is not merely tidiness. `.devcontainer/.env` is gitignored and holds +# POSTGRES_PASSWORD and the MinIO root credentials. It is never copied, but an +# un-ignored context is still transferred to the Docker daemon and can land in +# the build cache. +# +# Deliberately the context-root form rather than a `Dockerfile.dockerignore` +# sibling: when Features are present the devcontainer CLI synthesizes an +# extended Dockerfile in a temp directory, so BuildKit's sibling lookup would +# miss it. `/.dockerignore` is always found. +* diff --git a/.devcontainer/.env.example b/.devcontainer/.env.example index 6124fdd..689db92 100644 --- a/.devcontainer/.env.example +++ b/.devcontainer/.env.example @@ -5,8 +5,14 @@ # On first container build, `initializeCommand` copies it to # `.devcontainer/.env` (gitignored) on the host. From there: # -# * Docker Compose auto-discovers it (sibling to compose.yaml) -# and interpolates ${VAR:-default} references. +# * Docker Compose reads it because every caller passes +# `--env-file .devcontainer/.env` explicitly, and interpolates +# ${VAR:-default} references from it. Compose's positional +# auto-discovery does NOT apply: the orchestrator lives at +# `.devcontainer/stacks/compose.yaml` and this file is not its +# sibling. Any new caller must pass --env-file too, or every +# value silently falls back to its default and COMPOSE_PROFILES +# reads as empty. # * `runArgs --env-file` loads it into the dev container itself # so shells, runtimes, and `task` runs see the same values. # diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile new file mode 100644 index 0000000..3da1c39 --- /dev/null +++ b/.devcontainer/Dockerfile @@ -0,0 +1,129 @@ +# syntax=docker/dockerfile:1 +# ============================================================================= +# Musher dev container base image. +# +# WHY THIS FILE EXISTS +# +# bun, uv and task are baked here rather than installed via +# ghcr.io/devcontainers-extra/features/{bun,uv,go-task}. All three route through +# nanolayer's gh-release installer, which enumerates release assets against +# api.github.com UNAUTHENTICATED: +# +# nanolayer/installers/gh_release/resolvers/asset_resolver.py +# urllib.request.urlopen(f"https://api.github.com/repos/{repo}/releases/tags/{tag}") +# +# There is no Authorization header and no GITHUB_TOKEN read anywhere in that +# module. Codespaces build hosts and GitHub-hosted Actions runners share egress +# IP pools, so the 60 req/hr anonymous limit is routinely exhausted and the +# build dies with: +# +# urllib.error.HTTPError: HTTP Error 403: rate limit exceeded +# ERROR: Failed to install Bun. No asset patterns matched. +# +# That message is misleading -- the asset pattern was fine, the API call 403'd. +# PINNING THE VERSION DOES NOT HELP: the pin only supplies the tag, and +# _get_release_assets() still calls the API to LIST the release's assets. +# +# One failed Feature fails the whole image build, and Codespaces then falls back +# to a bare recovery container with none of the toolchain -- so the developer +# sees "task: command not found" rather than the real error. +# +# Every installer below fetches its artifact directly (bun -> bun.sh, uv -> +# astral.sh, task -> github.com/.../releases/download, mise -> mise.run). None +# touches api.github.com, so none can be starved by a noisy neighbour. +# +# Features that stay in devcontainer.json were checked individually and are +# clean: devcontainers-extra/deno curls releases/download/... directly, +# lukewiwa/shellcheck does the same, and robbert229/postgresql-client is apt. +# +# This boundary is enforced, not just documented -- see `repo toolchain check` +# (.repo/governance/policies/toolchain/). +# +# WHAT DOES *NOT* BELONG HERE +# +# The rule is: this file bakes version-pinned tools; scripts/post-create.sh owns +# whatever is inherently runtime or self-updating. In particular `mise install` +# cannot run here -- four of the six entries in mise.toml use the npm: and pipx: +# backends, and Node and Python arrive from Features, which layer AFTER this +# stage. Only the mise binary is baked. +# +# Pinned to the 24.04 LTS tag: the floating `:ubuntu` tag rolls forward to +# interim releases (e.g. 25.10) that docker-in-docker does not support. +# ============================================================================= +FROM mcr.microsoft.com/devcontainers/base:ubuntu-24.04 + +# `curl | bash` reports the exit status of bash, so a curl that dies mid-stream +# yields an empty script, a successful shell, and an image missing a tool. That +# silent-success shape is the failure class this file exists to remove, so make +# the pipe honest. +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +# Image-baked tool pins. These ARGs are the single source of truth for the four +# tools below -- there is deliberately no versions file, because nothing outside +# this image consumes them. `repo toolchain check` asserts each is a concrete +# version (TC-02) and that TASK_VERSION matches CI's setup-task pin (TC-03). +ARG BUN_VERSION=1.3.14 +ARG UV_VERSION=0.11.28 +ARG TASK_VERSION=3.52.0 +ARG MISE_VERSION=v2026.8.6 + +RUN set -eux; \ + # bun's installer hard-requires unzip, which the base image ships. Assert + # rather than apt-install a fallback: an unpinned `apt-get install` would be + # the only unpinned thing in this image, and a base that stopped shipping + # unzip is a decision someone should have to make explicitly. + command -v unzip >/dev/null 2>&1 || { echo "unzip missing from the base image; bun's installer requires it" >&2; exit 1; }; \ + \ + # --- bun -> /usr/local/bin/{bun,bunx} (BUN_INSTALL=/usr/local => $BUN_INSTALL/bin) + # Arch and the avx2 baseline variant are chosen by the installer; do not + # hardcode either (Codespaces is x86_64, some dev machines are aarch64). + # HOME is redirected to a scratch dir because the installer unconditionally + # appends PATH exports to ~/.bashrc and ~/.zshrc when they are writable. + mkdir -p /tmp/bun-home; \ + HOME=/tmp/bun-home BUN_INSTALL=/usr/local bash -c \ + "curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://bun.sh/install \ + | bash -s bun-v${BUN_VERSION}"; \ + rm -rf /tmp/bun-home; \ + # `bunx` needs no wrapper: the installer creates it as a symlink to `bun`. + # Do NOT `printf > /usr/local/bin/bunx` to "recreate" it -- that follows the + # symlink and overwrites the ~90 MB bun binary with the wrapper text, after + # which every `bun` call recurses into itself and hangs. + \ + # --- uv -> /usr/local/bin/{uv,uvx} + curl --retry 5 --retry-all-errors --retry-delay 3 -LsSf \ + "https://astral.sh/uv/${UV_VERSION}/install.sh" \ + | env UV_INSTALL_DIR=/usr/local/bin INSTALLER_NO_MODIFY_PATH=1 sh; \ + \ + # --- task -> /usr/local/bin/task + # godownloader script: downloads the release tarball and its checksum file + # from github.com/go-task/task/releases/download and sha256-verifies before + # installing. `-b` sets bindir; the trailing arg is the tag. + curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://taskfile.dev/install.sh \ + | sh -s -- -b /usr/local/bin "v${TASK_VERSION}"; \ + \ + # --- mise -> /usr/local/bin/mise + # Installed system-wide and version-pinned. Previously this was an unpinned + # `curl https://mise.run | sh` in post-create -- the only unpinned tool in a + # repo that pins everything else. The per-user shim dir that remoteEnv PATH + # expects (~/.local/share/mise/shims) is still created at runtime by + # `mise reshim` in base-setup.sh. + MISE_VERSION="${MISE_VERSION}" MISE_INSTALL_PATH=/usr/local/bin/mise \ + bash -c 'curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://mise.run | sh'; \ + \ + # Fail THIS layer, loudly, rather than letting post-create discover it. + # + # Presence-only on purpose. The pinned download URLs already guarantee the + # versions -- a wrong version 404s -- and asserting them by executing the + # binaries in the same layer that installed them is a known hazard under + # BuildKit. The runtime version assertion lives in + # scripts/verify-toolchain.sh, which CI runs against the built container. + test -x /usr/local/bin/bun; \ + test -x /usr/local/bin/bunx; \ + test -x /usr/local/bin/uv; \ + test -x /usr/local/bin/uvx; \ + test -x /usr/local/bin/task; \ + test -x /usr/local/bin/mise + +# No `USER vscode` here on purpose: Features install after this stage and +# `updateRemoteUserUID` expects root at build time. devcontainer.json's +# `remoteUser` owns runtime identity. diff --git a/.devcontainer/compose.yaml b/.devcontainer/compose.yaml deleted file mode 100644 index af6c520..0000000 --- a/.devcontainer/compose.yaml +++ /dev/null @@ -1,27 +0,0 @@ -# Musher Dev Container — Docker Compose orchestrator. -# Each stack lives in its own folder under stacks/ (its compose.yaml plus any -# config it needs); add a new folder there and include its compose.yaml below. -# Services run inside Docker-in-Docker via startup.sh. -# -# Port Allocation: -# 15432 PostgreSQL 15440 MinIO API (obs.) -# 15433 Redis 15441 MinIO Console (obs.) -# 15434 MinIO API 15442 Tempo -# 15435 MinIO Console 15443 Loki -# 15436 OCI Registry 15444 VictoriaMetrics -# 15460 Azimutt 15445 OTel HTTP -# 15446 OTel gRPC -# 15447 Grafana -# 15448 Pyroscope - -include: - - stacks/postgres/compose.yaml - - stacks/redis/compose.yaml - - stacks/minio/compose.yaml - - stacks/registry/compose.yaml - - stacks/azimutt/compose.yaml - - stacks/observability/compose.yaml - -networks: - default: - name: musher-dev diff --git a/.devcontainer/devcontainer-lock.json b/.devcontainer/devcontainer-lock.json index 21168aa..282fcb1 100644 --- a/.devcontainer/devcontainer-lock.json +++ b/.devcontainer/devcontainer-lock.json @@ -1,25 +1,10 @@ { "features": { - "ghcr.io/devcontainers-extra/features/bun:1": { - "version": "1.1.0", - "resolved": "ghcr.io/devcontainers-extra/features/bun@sha256:0624284ecaead9dd4c6654616a7f939cfa4ebcbc60593700a74e35b1767befa5", - "integrity": "sha256:0624284ecaead9dd4c6654616a7f939cfa4ebcbc60593700a74e35b1767befa5" - }, "ghcr.io/devcontainers-extra/features/deno:1": { "version": "1.0.4", "resolved": "ghcr.io/devcontainers-extra/features/deno@sha256:7013bf7726828a33579604fe1aa5c36a253b578d5991e55800b418b56fd2cca5", "integrity": "sha256:7013bf7726828a33579604fe1aa5c36a253b578d5991e55800b418b56fd2cca5" }, - "ghcr.io/devcontainers-extra/features/go-task:1": { - "version": "1.0.6", - "resolved": "ghcr.io/devcontainers-extra/features/go-task@sha256:4d1db153919976cadd3209ca05d655a761a01707767716994dad677b4538dc1b", - "integrity": "sha256:4d1db153919976cadd3209ca05d655a761a01707767716994dad677b4538dc1b" - }, - "ghcr.io/devcontainers-extra/features/uv:1": { - "version": "1.0.2", - "resolved": "ghcr.io/devcontainers-extra/features/uv@sha256:1ac5b9f17a9e9e745933d0ac2ecf758e06ed3da4423cd22c94cce4d482fd2dd8", - "integrity": "sha256:1ac5b9f17a9e9e745933d0ac2ecf758e06ed3da4423cd22c94cce4d482fd2dd8" - }, "ghcr.io/devcontainers/features/common-utils:2": { "version": "2.5.9", "resolved": "ghcr.io/devcontainers/features/common-utils@sha256:cb0c4d3c276f157eed17935747e364178d75fee17f55c4e129966f64633deb3a", diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index f54e4f3..8d6bf4b 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -2,17 +2,24 @@ // Uncomment optional blocks as needed. Comment out what you don't use. { "name": "Musher Dev", - // Pin to the LTS, not the floating :ubuntu tag — it rolls to interim releases - // (e.g. 25.10) that upstream Features like docker-in-docker don't support. - "image": "mcr.microsoft.com/devcontainers/base:ubuntu-24.04", + // The image is defined by .devcontainer/Dockerfile, which bakes the tools + // whose Features cannot be relied on (see its header, and the "Runtimes & + // Tools" section of CONFIGURATION.md). Features below layer on top of it. + // Context is .devcontainer/ and is emptied by .dockerignore — there is no + // COPY instruction, and .env must never reach the daemon. + "build": { + "dockerfile": "Dockerfile", + "context": "." + }, "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/${localWorkspaceFolderBasename},type=bind,consistency=cached", "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}", // Developer tools are pinned here as Features (baked into the image). Tools // with no Feature — the Codex and Lefthook CLIs — live in - // .devcontainer/mise.toml; Claude Code self-updates via its native installer. - // See CONFIGURATION.md → "Runtimes & Tools". + // .devcontainer/mise.toml; bun, uv, task and mise are baked by the Dockerfile; + // Claude Code self-updates via its native installer. + // See CONFIGURATION.md → "Runtimes & Tools" for which tier a tool belongs in. "features": { // --- Platform --- "ghcr.io/devcontainers/features/common-utils:2": { @@ -35,14 +42,16 @@ "installGradle": false, "installMaven": false }, + // deno stays a Feature: its installer curls releases/download/... directly + // and never calls api.github.com. bun, uv and task are NOT here — do not + // re-add them. Their Features route through nanolayer's gh-release + // installer, which lists assets against api.github.com unauthenticated and + // fails the whole build once the shared-IP rate limit is hit. They are + // baked by .devcontainer/Dockerfile instead; enforced by TC-01 in + // .repo/governance/policies/toolchain/. "ghcr.io/devcontainers-extra/features/deno:1": { "version": "2.9.2" }, - "ghcr.io/devcontainers-extra/features/bun:1": { "version": "1.3.14" }, - - // --- Package managers --- - "ghcr.io/devcontainers-extra/features/uv:1": { "version": "0.11.28" }, - // --- Task runner & linting --- - "ghcr.io/devcontainers-extra/features/go-task:1": { "version": "3.52.0" }, + // --- Linting --- "ghcr.io/lukewiwa/features/shellcheck:0": { "version": "v0.11.0" }, // --- Database tooling --- @@ -98,7 +107,9 @@ }, "remoteEnv": { - // mise shims (codex, lefthook) + ~/.local/bin (mise, Claude Code) on PATH. + // mise shims (codex, lefthook) + ~/.local/bin (Claude Code) on PATH. + // mise, bun, uv and task are baked into /usr/local/bin by the Dockerfile + // and are already on the default PATH. "PATH": "/home/vscode/.local/share/mise/shims:/home/vscode/.local/bin:${containerEnv:PATH}" }, diff --git a/.devcontainer/scripts/lib/base-setup.sh b/.devcontainer/scripts/lib/base-setup.sh index 9951d15..c7b915c 100644 --- a/.devcontainer/scripts/lib/base-setup.sh +++ b/.devcontainer/scripts/lib/base-setup.sh @@ -70,6 +70,9 @@ base_fix_nvm_permissions() { # --- mise (pins the CLIs that have no devcontainer Feature) --- +# Fallback only: the image bakes mise at /usr/local/bin/mise, which +# `command -v` finds first. This path covers the base_install_mise fallback, +# which installs per-user. readonly _MISE_BIN="${_HOME}/.local/bin/mise" readonly _MISE_SHIMS="${_HOME}/.local/share/mise/shims" @@ -83,7 +86,13 @@ base_setup_path() { export PATH="${_MISE_SHIMS}:${_HOME}/.local/bin:${PATH}" } -# Installs mise via the official installer if not already present. +# Installs mise if it is not already present. +# +# The dev container image bakes a pinned mise at /usr/local/bin/mise +# (ARG MISE_VERSION in .devcontainer/Dockerfile), so this normally short-circuits. +# The installer below is the fallback for a consuming repo that strips the +# Dockerfile, and is deliberately unpinned because in that case there is no ARG +# to read the pin from. # # Outputs: # Writes progress to stderr via log() @@ -94,7 +103,7 @@ base_install_mise() { log "mise already installed, skipping" return 0 fi - log "Installing mise (https://mise.run)..." + log "mise not baked into the image; falling back to https://mise.run..." retry 3 5 bash -c 'curl -fsSL https://mise.run | sh' } diff --git a/.devcontainer/scripts/lib/motd.sh b/.devcontainer/scripts/lib/motd.sh index 1118942..1b737f7 100644 --- a/.devcontainer/scripts/lib/motd.sh +++ b/.devcontainer/scripts/lib/motd.sh @@ -4,7 +4,7 @@ # This is a library file meant to be sourced, not executed directly. # Requires common.sh (has_cmd, log) to be sourced first. # -# Usage: source "path/to/motd.sh"; show_motd "/path/to/compose.yaml" +# Usage: source "path/to/motd.sh"; show_motd "/path/to/stacks/compose.yaml" "/path/to/.devcontainer" if [[ -z "${_MOTD_SH_LOADED:-}" ]]; then readonly _MOTD_SH_LOADED=1 @@ -77,12 +77,18 @@ _motd_runtimes() { _motd_services() { local compose_file="$1" + local env_file="${2:-}" if [[ -z "$compose_file" ]] || [[ ! -f "$compose_file" ]] || ! has_cmd docker; then return 0 fi + # Name the env file explicitly for the same reason startup.sh does: the + # compose file lives under stacks/ and is no longer a sibling of .env. + local -a env_args=() + [[ -n "$env_file" && -f "$env_file" ]] && env_args=(--env-file "$env_file") + local output - output="$(docker compose -f "$compose_file" ps --format json 2>/dev/null || true)" + output="$(docker compose "${env_args[@]}" -f "$compose_file" ps --format json 2>/dev/null || true)" if [[ -z "$output" ]]; then return 0 fi @@ -136,7 +142,8 @@ _motd_quickref() { echo "" echo " ${_BOLD}Quick Reference${_RESET}" echo " ${_DIM}${sep}${_RESET}" - echo " docker compose -f .devcontainer/compose.yaml up -d / down / logs -f" + echo " docker compose --env-file .devcontainer/.env \\" + echo " -f .devcontainer/stacks/compose.yaml up -d / down / logs -f" echo " git status / log / diff" echo " task Task runner" echo " claude Claude Code AI" @@ -201,8 +208,9 @@ _motd_tips() { # Renders the full MOTD to stdout. # # Arguments: -# $1 — path to compose.yaml (may be empty to skip services) -# $2 — path to .devcontainer/ directory (may be empty to skip env warnings) +# $1 — path to stacks/compose.yaml (may be empty to skip services) +# $2 — path to .devcontainer/ directory (may be empty to skip env warnings); +# also supplies the --env-file the compose file needs # Outputs: # MOTD text to stdout show_motd() { @@ -216,7 +224,7 @@ show_motd() { echo "" _motd_header _motd_runtimes - _motd_services "$compose_file" + _motd_services "$compose_file" "${devcontainer_dir:+${devcontainer_dir}/.env}" _motd_env_warnings "$devcontainer_dir" _motd_quickref _motd_tips diff --git a/.devcontainer/scripts/startup.sh b/.devcontainer/scripts/startup.sh index b56613a..58013e6 100644 --- a/.devcontainer/scripts/startup.sh +++ b/.devcontainer/scripts/startup.sh @@ -2,7 +2,7 @@ # startup.sh — Starts compose services and waits for health checks. # # Executed on every container start to bring up supporting services -# (databases, caches, observability) defined in compose.yaml. +# (databases, caches, observability) defined in stacks/compose.yaml. # # Usage: Called automatically by devcontainer.json postStartCommand. set -euo pipefail @@ -11,8 +11,14 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" readonly SCRIPT_DIR DEVCONTAINER_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" readonly DEVCONTAINER_DIR -COMPOSE_FILE="${DEVCONTAINER_DIR}/compose.yaml" +COMPOSE_FILE="${DEVCONTAINER_DIR}/stacks/compose.yaml" readonly COMPOSE_FILE +# The compose file lives under stacks/, so it is no longer a sibling of .env and +# Compose's positional auto-discovery cannot find it. Name the env file +# explicitly on every invocation -- otherwise ${VAR:-default} silently wins and +# COMPOSE_PROFILES reads as empty, disabling every opt-in stack without an error. +ENV_FILE="${DEVCONTAINER_DIR}/.env" +readonly ENV_FILE # shellcheck source=lib/common.sh source "${SCRIPT_DIR}/lib/common.sh" @@ -38,7 +44,8 @@ trap 'on_error ${LINENO} "${BASH_COMMAND}"' ERR # Arguments: # $1 — timeout in seconds (default: 60) # Globals: -# COMPOSE_FILE — read, path to compose.yaml +# COMPOSE_FILE — read, path to stacks/compose.yaml +# ENV_FILE — read, path to .env (passed to every compose invocation) # Outputs: # Writes progress/warnings to stderr via log() # Returns: @@ -49,7 +56,7 @@ wait_for_healthy() { local elapsed=0 while ((elapsed < timeout)); do local output - output="$(docker compose -f "${COMPOSE_FILE}" ps --format json 2>/dev/null || true)" + output="$(docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" ps --format json 2>/dev/null || true)" # Detect failed services (exited, dead, or unhealthy) local failed="" @@ -62,7 +69,7 @@ wait_for_healthy() { local state state="$(echo "$line" | grep -o '"State":"[^"]*"' | head -1 | cut -d'"' -f4)" log " ${name}: ${state}" - docker compose -f "${COMPOSE_FILE}" logs --tail=10 "$name" 2>/dev/null || true + docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" logs --tail=10 "$name" 2>/dev/null || true done return 1 fi @@ -94,13 +101,22 @@ main() { fi if [[ ! -f "${COMPOSE_FILE}" ]]; then - log "No compose.yaml found, skipping service startup" + log "No stacks/compose.yaml found, skipping service startup" + show_motd "" "${DEVCONTAINER_DIR}" + return 0 + fi + + # --env-file errors out on a missing path. initialize.sh creates .env on the + # host before the container starts, so this only trips if that hook was + # skipped -- say a bare `docker run` outside the devcontainer tooling. + if [[ ! -f "${ENV_FILE}" ]]; then + log "No .env found at ${ENV_FILE}; run scripts/initialize.sh first" show_motd "" "${DEVCONTAINER_DIR}" return 0 fi log "Starting compose services..." - docker compose -f "${COMPOSE_FILE}" up -d --remove-orphans + docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" up -d --remove-orphans wait_for_healthy 60 diff --git a/.devcontainer/scripts/verify-toolchain.sh b/.devcontainer/scripts/verify-toolchain.sh new file mode 100644 index 0000000..0839537 --- /dev/null +++ b/.devcontainer/scripts/verify-toolchain.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +# verify-toolchain.sh — Asserts the image-baked tools report their pinned versions. +# +# The Dockerfile's own assertions are presence-only (`test -x`): executing a +# binary in the same layer that installed it is a known BuildKit hazard, and the +# pinned download URLs already guarantee the version -- a wrong one 404s. This +# script is the runtime half of that split, run against the built container. +# +# The expected versions are read back out of .devcontainer/Dockerfile so the +# ARGs stay the single source of truth. `repo toolchain check` separately +# asserts those ARGs are concrete pins rather than floating tags. +# +# Usage: bash .devcontainer/scripts/verify-toolchain.sh +# (CI runs it as the devcontainers/ci `runCmd`.) +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly SCRIPT_DIR +DOCKERFILE="${SCRIPT_DIR}/../Dockerfile" +readonly DOCKERFILE + +# Reads a pinned ARG default out of the Dockerfile. +# +# Arguments: +# $1 — the ARG name, e.g. BUN_VERSION +# Outputs: +# The pinned value on stdout +# Returns: +# 0 on success, 1 if the ARG is absent or has no default +arg_pin() { + local name="${1}" + local value + value="$(sed -n "s/^ARG ${name}=\\(.*\\)$/\\1/p" "${DOCKERFILE}" | head -1)" + if [[ -z "${value}" ]]; then + echo "ERROR: no 'ARG ${name}=' in ${DOCKERFILE}" >&2 + return 1 + fi + printf '%s\n' "${value}" +} + +# Asserts ` --version` reports the expected version. +# +# Arguments: +# $1 — the binary name, which is also the human-readable tool name +# $2 — expected version (no leading 'v') +# Outputs: +# Writes a pass/fail line to stdout +# Returns: +# 0 on match, 1 on mismatch or missing binary +assert_version() { + local tool="${1}" expected="${2}" + local actual + if ! command -v "${tool}" >/dev/null 2>&1; then + echo " FAIL ${tool}: not on PATH" + return 1 + fi + # Tools disagree on output shape (`task` prints "Task version: v3.52.0", uv + # prints "uv 0.11.28"), so match the expected string anywhere in the first + # line rather than parsing four different formats. + actual="$("${tool}" --version 2>&1 | head -1)" + if [[ "${actual}" != *"${expected}"* ]]; then + echo " FAIL ${tool}: expected ${expected}, got '${actual}'" + return 1 + fi + echo " ok ${tool} ${expected}" +} + +# Entry point: checks every image-baked tool against its Dockerfile pin. +# +# Outputs: +# Writes per-tool results to stdout +# Returns: +# 0 if all tools match, 1 otherwise +main() { + echo "Verifying image-baked toolchain against ${DOCKERFILE}..." + + local bun_v uv_v task_v mise_v + bun_v="$(arg_pin BUN_VERSION)" + uv_v="$(arg_pin UV_VERSION)" + task_v="$(arg_pin TASK_VERSION)" + # MISE_VERSION is pinned with a leading 'v'; `mise --version` prints without. + mise_v="$(arg_pin MISE_VERSION)" + mise_v="${mise_v#v}" + + local failed=0 + assert_version bun "${bun_v}" || failed=1 + assert_version uv "${uv_v}" || failed=1 + assert_version task "${task_v}" || failed=1 + assert_version mise "${mise_v}" || failed=1 + + # bunx is a symlink to bun; assert it survived rather than being overwritten. + if [[ ! -x /usr/local/bin/bunx ]]; then + echo " FAIL bunx: /usr/local/bin/bunx missing or not executable" + failed=1 + else + echo " ok bunx" + fi + + if ((failed)); then + echo "Toolchain verification FAILED" >&2 + return 1 + fi + echo "Toolchain verification passed" +} + +main "$@" diff --git a/.devcontainer/stacks/compose.yaml b/.devcontainer/stacks/compose.yaml new file mode 100644 index 0000000..89b4a66 --- /dev/null +++ b/.devcontainer/stacks/compose.yaml @@ -0,0 +1,39 @@ +# Musher Dev Container — Docker Compose orchestrator. +# +# Each stack lives in its own folder next to this file (its compose.yaml plus +# any config it needs); add a new folder and include its compose.yaml below. +# Services run inside Docker-in-Docker, started by scripts/startup.sh. +# +# This file is NOT the dev container. The dev container itself is defined by +# ../Dockerfile + ../devcontainer.json; these are the supporting services it +# brings up. +# +# `name` is set explicitly because Compose otherwise derives the project name +# from the project directory — which would silently become "stacks". Likewise +# every caller must pass `--env-file ../.env`: this file is no longer a sibling +# of .env, so Compose's positional auto-discovery does not reach it. +# +# Port Allocation: +# 15432 PostgreSQL 15440 MinIO API (obs.) +# 15433 Redis 15441 MinIO Console (obs.) +# 15434 MinIO API 15442 Tempo +# 15435 MinIO Console 15443 Loki +# 15436 OCI Registry 15444 VictoriaMetrics +# 15460 Azimutt 15445 OTel HTTP +# 15446 OTel gRPC +# 15447 Grafana +# 15448 Pyroscope + +name: musher-dev + +include: + - postgres/compose.yaml + - redis/compose.yaml + - minio/compose.yaml + - registry/compose.yaml + - azimutt/compose.yaml + - observability/compose.yaml + +networks: + default: + name: musher-dev diff --git a/.github/dependabot.yml b/.github/dependabot.yml index c224137..c55311f 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -21,11 +21,16 @@ updates: commit-message: prefix: "ci" - # Compose service images. Each stack lives in .devcontainer/stacks// with - # its own standard-named compose.yaml (pulled in via `include:`), so the - # .devcontainer root and every stack folder are scanned. If Dependabot does not - # surface PRs for a stack, verify detection under - # Insights -> Dependency graph -> Dependabot and adjust these directories. + # Docker images. /.devcontainer covers the Dockerfile's `FROM` base image; + # /.devcontainer/stacks/** covers each stack's own standard-named compose.yaml + # (pulled into stacks/compose.yaml via `include:`). The orchestrator itself + # declares no images. If Dependabot does not surface PRs for a stack, verify + # detection under Insights -> Dependency graph -> Dependabot and adjust these + # directories. + # + # Note: this does NOT cover the tool pins in the Dockerfile's ARGs (bun, uv, + # task, mise) -- Dependabot has no ecosystem for those. Bump them by hand; + # `repo toolchain check` keeps them exact and in step with CI. - package-ecosystem: "docker" directories: - "/.devcontainer" diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml index dff9ee6..37b3694 100644 --- a/.github/workflows/validate.yaml +++ b/.github/workflows/validate.yaml @@ -27,12 +27,25 @@ jobs: - name: Validate compose config run: | cp .devcontainer/.env.example .devcontainer/.env - docker compose -f .devcontainer/compose.yaml config --quiet + # --env-file is mandatory, not decorative: the orchestrator lives under + # stacks/ and is no longer a sibling of .env, so Compose's positional + # auto-discovery would silently fall back to ${VAR:-default}. + docker compose --env-file .devcontainer/.env \ + -f .devcontainer/stacks/compose.yaml config --quiet + - name: Validate compose config with every profile enabled + run: | + # `config` respects profiles, so the run above only covers the always-on + # stacks. Enable all of them so an opt-in stack cannot rot unnoticed -- + # and so a broken --env-file surfaces here rather than at someone's + # first `Reopen in Container`. + COMPOSE_PROFILES=redis,minio,registry,azimutt,observability \ + docker compose --env-file .devcontainer/.env \ + -f .devcontainer/stacks/compose.yaml config --quiet - name: Reject floating image tags run: | # Fail on :latest / :main tags that are not digest-pinned (@sha256:...). if grep -rnE 'image:.*(:latest|:main)([[:space:]]|$)' \ - .devcontainer/compose.yaml .devcontainer/stacks/; then + .devcontainer/stacks/; then echo "::error::Floating image tag found. Pin to a version or digest." >&2 exit 1 fi @@ -107,4 +120,7 @@ jobs: - name: Build devcontainer uses: devcontainers/ci@v0.3 with: - runCmd: echo "devcontainer smoke test passed" + # Asserts the Dockerfile-baked tools report their pinned versions -- + # the runtime half of the split the Dockerfile's presence-only + # `test -x` assertions leave open. + runCmd: bash .devcontainer/scripts/verify-toolchain.sh diff --git a/.repo/README.md b/.repo/README.md index cdc83ef..fe8e990 100644 --- a/.repo/README.md +++ b/.repo/README.md @@ -43,6 +43,25 @@ runs `uv tool install ./.repo`). To reinstall after editing it: | `ports` | `PORT-01`..`PORT-05` | The port table, `forwardPorts`/`portsAttributes`, and compose published ports all agree and stay in the reserved range | | `hooks` | `HOOK-01`..`HOOK-04` | Every lefthook job has a CI counterpart and vice versa, or a recorded reason why not | | `rulesets` | `RS-01`..`RS-04` | Committed branch rulesets stay valid and in step with the CI jobs they require | +| `toolchain` | `TC-01`..`TC-03` | The tools the image bakes stay out of the Features block, keep exact pins, and stay in step with CI | + +### Why `toolchain` exists + +`TC-01` is the policy least likely to be guessed from the code it guards. bun, +uv and task are installed by [`.devcontainer/Dockerfile`](../.devcontainer/Dockerfile) +rather than by their `devcontainers-extra` Features, because those Features +resolve release assets through nanolayer, which calls `api.github.com` with no +credentials. On Codespaces build hosts and GitHub-hosted runners — which share +egress IP pools — that hits the 60 req/hr anonymous limit, and one failed +Feature fails the whole image build. Pinning the version does not avoid the +call. Re-adding any of the three looks like a harmless simplification, which is +exactly why it is a check and not a comment. The full diagnosis is in +[`policies/toolchain/violations.py`](governance/policies/toolchain/violations.py). + +`BANNED_FEATURES` in [`policies/toolchain/check.py`](governance/policies/toolchain/check.py) +lists only Features whose installers were read and confirmed to make that call. +`deno`, `shellcheck` and `postgresql-client` were checked and are clean, so they +stay Features — the rule is about the installer's behaviour, not the publisher. ### The one-way-check tables diff --git a/.repo/governance/policies/__init__.py b/.repo/governance/policies/__init__.py index 4547fdc..c689d69 100644 --- a/.repo/governance/policies/__init__.py +++ b/.repo/governance/policies/__init__.py @@ -13,7 +13,7 @@ from collections.abc import Callable -from governance.policies import config, hooks, ports, rulesets +from governance.policies import config, hooks, ports, rulesets, toolchain from governance.reporting import Report POLICIES: dict[str, Callable[[], Report]] = { @@ -21,6 +21,7 @@ "ports": ports.run, "hooks": hooks.run, "rulesets": rulesets.run, + "toolchain": toolchain.run, } __all__ = ["POLICIES"] diff --git a/.repo/governance/policies/toolchain/__init__.py b/.repo/governance/policies/toolchain/__init__.py new file mode 100644 index 0000000..25cc327 --- /dev/null +++ b/.repo/governance/policies/toolchain/__init__.py @@ -0,0 +1,5 @@ +"""Policy: the image-baked toolchain is pinned, and stays out of Features.""" + +from governance.policies.toolchain.check import run + +__all__ = ["run"] diff --git a/.repo/governance/policies/toolchain/check.py b/.repo/governance/policies/toolchain/check.py new file mode 100644 index 0000000..f0ee78c --- /dev/null +++ b/.repo/governance/policies/toolchain/check.py @@ -0,0 +1,86 @@ +"""Detect drift in the tools the container image bakes itself. + +Three tools (bun, uv, task) are installed by .devcontainer/Dockerfile rather +than by a Feature, for a reason that is easy to lose: their Features fail on +rate-limited build hosts. This policy keeps that decision from being quietly +undone, and keeps the pins that replaced them honest. +""" + +from __future__ import annotations + +import re + +from governance import repo +from governance.policies.toolchain import violations as v +from governance.reporting import Report + +DOCKERFILE = ".devcontainer/Dockerfile" +DEVCONTAINER = ".devcontainer/devcontainer.json" +WORKFLOW = ".github/workflows/validate.yaml" + +#: Feature ref (without the version suffix) -> the tool it would install. +#: Only Features whose installers were verified to call api.github.com belong +#: here. deno, shellcheck and postgresql-client were checked and are clean: +#: the first two curl releases/download/... directly, the third is apt. +BANNED_FEATURES = { + "ghcr.io/devcontainers-extra/features/bun": "bun", + "ghcr.io/devcontainers-extra/features/uv": "uv", + "ghcr.io/devcontainers-extra/features/go-task": "task", +} + +#: ARG name -> the tool it pins. Every one of these must exist and be exact. +REQUIRED_ARGS = { + "BUN_VERSION": "bun", + "UV_VERSION": "uv", + "TASK_VERSION": "task", + "MISE_VERSION": "mise", +} + +#: Values that are a moving target rather than a pin. +FLOATING = {"latest", "stable", "main", "master", ""} + +_ARG = re.compile(r"^ARG\s+([A-Z0-9_]+)=(.*)$", re.MULTILINE) + + +def _dockerfile_args() -> dict[str, str]: + """The ARG defaults declared by the Dockerfile.""" + return {m.group(1): m.group(2).strip() for m in _ARG.finditer(repo.read_text(DOCKERFILE))} + + +def _ci_task_version() -> str | None: + """The version arduino/setup-task is pinned to in CI, if it is used.""" + workflow = repo.read_yaml(WORKFLOW) or {} + for job in (workflow.get("jobs") or {}).values(): + for step in (job or {}).get("steps", []) or []: + uses = str((step or {}).get("uses", "")) + if uses.startswith("arduino/setup-task@"): + version = ((step or {}).get("with") or {}).get("version") + if version is not None: + return str(version) + return None + + +def run() -> Report: + report = Report(policy="toolchain") + + features = repo.read_jsonc(DEVCONTAINER).get("features", {}) + for ref in features: + # Feature refs carry a version suffix (`...:1`); compare the name only. + name = ref.rsplit(":", 1)[0] + tool = BANNED_FEATURES.get(name) + if tool: + report.add(v.feature_reintroduced(ref, tool)) + + args = _dockerfile_args() + for name, tool in REQUIRED_ARGS.items(): + if name not in args: + report.add(v.missing_arg(name, tool)) + elif args[name].lower().lstrip("v") in FLOATING: + report.add(v.floating_arg(name, args[name])) + + baked_task = args.get("TASK_VERSION") + ci_task = _ci_task_version() + if baked_task and ci_task and baked_task.lstrip("v") != ci_task.lstrip("v"): + report.add(v.task_version_drift(baked_task, ci_task, WORKFLOW)) + + return report diff --git a/.repo/governance/policies/toolchain/violations.py b/.repo/governance/policies/toolchain/violations.py new file mode 100644 index 0000000..e892be3 --- /dev/null +++ b/.repo/governance/policies/toolchain/violations.py @@ -0,0 +1,95 @@ +"""What can go wrong with the image-baked toolchain, and why each rule exists.""" + +from __future__ import annotations + +from governance.reporting import Violation + +DOCS = "CONFIGURATION.md#runtimes--tools" + +DOCKERFILE = ".devcontainer/Dockerfile" +DEVCONTAINER = ".devcontainer/devcontainer.json" + +#: The diagnosis behind TC-01, stated once. It is long on purpose: the whole +#: point of moving these three tools into the Dockerfile is easy to mistake for +#: fussiness, and the next person to reach for the Feature deserves the actual +#: failure mode rather than "we don't do that here". +_WHY_NOT_A_FEATURE = ( + "The devcontainers-extra Features for bun, uv and go-task all install via " + "nanolayer's gh-release helper, which lists a release's assets by calling " + "api.github.com with no credentials " + "(nanolayer/installers/gh_release/resolvers/asset_resolver.py). Codespaces " + "build hosts and GitHub-hosted Actions runners share egress IP pools, so " + "the 60 req/hr anonymous limit is routinely exhausted and the call 403s. " + "Pinning the version does not help -- the pin only supplies the tag; the " + "asset listing still hits the API. One failed Feature fails the entire " + "image build, and Codespaces then drops the developer into a bare recovery " + "container, so the symptom is 'task: command not found' rather than the " + "real error." +) + + +def feature_reintroduced(feature: str, tool: str) -> Violation: + return Violation( + code="TC-01", + summary=f"{tool} is declared as a Feature; it must be baked by the Dockerfile", + reason=_WHY_NOT_A_FEATURE, + fix=( + f"Remove {feature!r} from the features block and pin {tool} with an " + f"ARG in {DOCKERFILE} instead." + ), + where=DEVCONTAINER, + docs=DOCS, + ) + + +def missing_arg(name: str, tool: str) -> Violation: + return Violation( + code="TC-02", + summary=f"no pinned 'ARG {name}=' for {tool}", + reason=( + "The ARG defaults are the single source of truth for what the image " + "bakes. scripts/verify-toolchain.sh reads them back to assert the " + "built container matches, so an absent or empty pin turns that CI " + "check into a no-op." + ), + fix=f"Add `ARG {name}=` to {DOCKERFILE}.", + where=DOCKERFILE, + docs=DOCS, + ) + + +def floating_arg(name: str, value: str) -> Violation: + return Violation( + code="TC-02", + summary=f"ARG {name} is set to the floating value {value!r}", + reason=( + "Every other tool in this template is pinned to an exact version. A " + "floating tag makes the image irreproducible and silently changes " + "the toolchain under developers who rebuild on different days." + ), + fix=f"Replace {value!r} with an exact version in {DOCKERFILE}.", + where=DOCKERFILE, + docs=DOCS, + ) + + +def task_version_drift(dockerfile_version: str, ci_version: str, ci_path: str) -> Violation: + return Violation( + code="TC-03", + summary=( + f"task pinned to {dockerfile_version} in the Dockerfile but " + f"{ci_version} in CI" + ), + reason=( + "CI installs Task with arduino/setup-task rather than the dev " + "container image, so the two pins are the same decision recorded " + "twice. When they drift, a Taskfile change can pass locally and " + "fail in CI (or the reverse) for reasons no diff explains." + ), + fix=( + f"Set the same version in {DOCKERFILE} (ARG TASK_VERSION) and " + f"{ci_path} (arduino/setup-task `version:`)." + ), + where=ci_path, + docs=DOCS, + ) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index d6dc093..c00fd47 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -18,9 +18,13 @@ Does it configure a linter, formatter, or the git hooks? Does it provision the container itself? → .devcontainer/ (see the branches below) -Runtime, or any tool that has a devcontainer Feature? +Runtime, or a tool with a Feature whose installer avoids api.github.com? → devcontainer.json → features block (pin the version) +Tool whose Feature is rate-limit-fragile, or that must exist before +Features run (bun, uv, task, mise)? + → .devcontainer/Dockerfile → an ARG pin + CLI with no Feature (npm-distributed, e.g. codex, lefthook)? → .devcontainer/mise.toml @@ -28,7 +32,7 @@ Self-updating CLI (e.g. Claude Code)? → scripts/lib/base-setup.sh (native installer) Infrastructure service (DB, cache, queue, storage)? - → stacks//compose.yaml (new folder, add to compose.yaml includes) + → stacks//compose.yaml (new folder, add to stacks/compose.yaml includes) VS Code editor behavior or extension? → devcontainer.json → customizations.vscode block @@ -55,7 +59,7 @@ the first "yes". | --- | --- | --- | --- | | 1 | Can the tool *only* load from the repo root, with no flag to point elsewhere? | Repo root | `Taskfile.yml`, `.gitattributes`, `.gitignore` | | 2 | Does it configure a linter, formatter, or the git hooks? | `.config/` | `lefthook.yml`, `markdownlint.jsonc`, `yamllint.yaml` | -| 3 | Does it provision the container or its services? | `.devcontainer/` | `devcontainer.json`, `mise.toml`, `stacks/*/` | +| 3 | Does it provision the container or its services? | `.devcontainer/` | `Dockerfile`, `devcontainer.json`, `mise.toml`, `stacks/` | | 4 | Does it enforce repo structure? | `.repo/` | The `repo` CLI and its policies | Why `.config/` is dotted: it is repo infrastructure, and it sits alongside the @@ -79,7 +83,8 @@ See [`.config/README.md`](.config/README.md) for the per-file index, and | Category | Need | Canonical Location | | --- | --- | --- | -| **Runtimes & Tools** | Anything with a Feature (Node, Python, Go, Java, Deno, bun, uv, gh, Task, ShellCheck) | `devcontainer.json` → `features` (pinned) | +| **Runtimes & Tools** | Anything with a usable Feature (Node, Python, Go, Java, Deno, gh, ShellCheck, psql) | `devcontainer.json` → `features` (pinned) | +| | Image-baked tools (bun, uv, Task, mise) | `.devcontainer/Dockerfile` → `ARG` (pinned) | | | CLIs with no Feature (Codex, Lefthook) | `.devcontainer/mise.toml` | | | Self-updating CLIs (Claude Code) | `scripts/lib/base-setup.sh` | | **Tooling** | Git hooks | `.config/lefthook.yml` | @@ -99,7 +104,8 @@ See [`.config/README.md`](.config/README.md) for the per-file index, and | | Service credentials (dev-only) | `.devcontainer/.env` | | | Service profiles/toggles | `.devcontainer/.env` → `COMPOSE_PROFILES` | | | Secrets (API keys, tokens) | Host env forwarded via `remoteEnv` — never committed | -| **Services** | Infrastructure services | `.devcontainer/stacks//compose.yaml` | +| **Services** | Stack orchestrator (`include:` list) | `.devcontainer/stacks/compose.yaml` | +| | Infrastructure services | `.devcontainer/stacks//compose.yaml` | | | Service enable/disable | `.devcontainer/.env` → `COMPOSE_PROFILES` | | | Service tuning/config | `.devcontainer/stacks//` (colocated) | | **Networking** | Port allocation (container-side) | `.devcontainer/stacks//compose.yaml` → `ports:` | @@ -119,13 +125,23 @@ See [`.config/README.md`](.config/README.md) for the per-file index, and | | Codex CLI (pinned) | `.devcontainer/mise.toml` | | | AI CLI config persistence | `devcontainer.json` → `mounts` (named volumes) | | **Security** | Container capabilities | `devcontainer.json` → `capAdd` / `securityOpt` | +| | Docker build context exclusions | `.devcontainer/.dockerignore` | | | Network binding | Compose files → all ports bound to `127.0.0.1` | --- ## Runtimes & Tools -Tools land in one of three places. **Prefer a Feature** — if one exists, pin its version there. +Tools land in one of four places. Ask these in order and stop at the first "yes". + +| # | Question | Home | +| --- | --- | --- | +| 1 | Is there a Feature, **and** does its installer avoid `api.github.com`? | `devcontainer.json` → `features` | +| 2 | Is the Feature rate-limit-fragile, or must the tool exist before Features run? | `.devcontainer/Dockerfile` → an `ARG` pin | +| 3 | No Feature, and only needed at runtime? | `.devcontainer/mise.toml` | +| 4 | Does the tool update itself? | `scripts/lib/base-setup.sh` | + +**Prefer a Feature.** Tier 2 exists because of one specific, verified failure — not as a general escape hatch. ### 1. Tools with a Feature → `devcontainer.json` @@ -142,7 +158,44 @@ Runtimes and any CLI that ships a devcontainer Feature are pinned in the `featur Comment out any tool you don't need (and its matching VS Code extension). Pin an exact version where the Feature supports it; a couple track a major line instead (`java: 17`, `postgresql-client: 16`). -### 2. CLIs with no Feature → `.devcontainer/mise.toml` +Before adding a third-party Feature, read its `install.sh`. If it delegates to +`ghcr.io/devcontainers-extra/features/gh-release` (directly or via nanolayer), it belongs in tier 2 — see below. + +### 2. Image-baked tools → `.devcontainer/Dockerfile` + +`bun`, `uv`, `task` and `mise` are installed by the Dockerfile as pinned `ARG`s rather than by Features: + +```dockerfile +ARG BUN_VERSION=1.3.14 +ARG UV_VERSION=0.11.28 +ARG TASK_VERSION=3.52.0 +ARG MISE_VERSION=v2026.8.6 +``` + +The first three have Features, and those Features are the problem. All of them resolve release assets through +nanolayer's `gh-release` helper, which lists a release's assets by calling `api.github.com` **with no credentials**. +Codespaces build hosts and GitHub-hosted Actions runners share egress IP pools, so the 60 req/hr anonymous limit is +routinely exhausted, the call 403s, and one failed Feature fails the entire image build — after which Codespaces +drops you into a bare recovery container. Pinning the version does not help: the pin supplies the tag, but the asset +listing still hits the API. `mise` is here for a different reason — it was previously an unpinned `curl | sh` in +post-create, the only unpinned tool in a template that pins everything else. + +`repo toolchain check` enforces all of this: `TC-01` fails if one of those Features comes back, `TC-02` fails on a +floating pin, and `TC-03` fails if `TASK_VERSION` drifts from CI's `arduino/setup-task` version. + +Two constraints on what can go here: + +- **Features layer *after* this stage**, so the Dockerfile cannot use anything a Feature provides. This is why + `mise install` stays in post-create — four of the six entries in `mise.toml` use the `npm:` and `pipx:` backends, + and Node and Python come from Features. +- **Runtime identity is not the Dockerfile's job.** There is no `USER` instruction; `updateRemoteUserUID` expects + root at build time and `remoteUser` owns identity afterwards. Anything touching the named-volume mount points + (`~/.claude`, `~/.config/gh`, `~/.codex`) belongs in post-create. + +The build context is `.devcontainer/`, emptied by `.dockerignore` — the Dockerfile has no `COPY` instruction, and +`.devcontainer/.env` must never reach the Docker daemon. + +### 3. CLIs with no Feature → `.devcontainer/mise.toml` npm-distributed CLIs like Codex and Lefthook have no Feature, so [mise](https://mise.jdx.dev) pins and installs them. Add a line under `[tools]`: @@ -155,7 +208,7 @@ Add a line under `[tools]`: `mise install` runs in post-create; re-run it (or `task tools:install`) after editing. -### 3. Self-updating CLIs → `scripts/lib/base-setup.sh` +### 4. Self-updating CLIs → `scripts/lib/base-setup.sh` CLIs that manage their own updates (Claude Code) install via their native installer. Add a function and call it from `base_setup()`: @@ -249,7 +302,7 @@ The template follows a three-state grammar: ### Enabling/Disabling Services -All services are included in `compose.yaml`. Optional services are gated by Compose profiles: +All services are included in `.devcontainer/stacks/compose.yaml`. Optional services are gated by Compose profiles: | Service | Profile | Always On? | | --- | --- | --- | @@ -269,7 +322,8 @@ COMPOSE_PROFILES=redis,minio,observability ### Adding a New Service 1. Create `stacks/myservice/compose.yaml` -2. Add `- stacks/myservice/compose.yaml` to `compose.yaml` `include:` +2. Add `- myservice/compose.yaml` to the `include:` list in `stacks/compose.yaml` + (paths are relative to that file) 3. Optionally add `profiles: [myservice]` if it should be opt-in 4. Add port forwarding in `devcontainer.json` → `forwardPorts` and `portsAttributes` 5. Use `${VAR:-default}` for any credentials, and add them to `.env.example` @@ -282,6 +336,7 @@ the stack folder (e.g. `./init`, `./config/...`): ```text stacks/ + compose.yaml The orchestrator: `include:` one line per stack postgres/ compose.yaml init/ SQL init scripts (mounted at ./init) @@ -290,6 +345,23 @@ stacks/ config/ OTel, Grafana, Tempo, Loki configs (mounted at ./config) ``` +Relative paths inside a stack's `compose.yaml` resolve against *that file's* folder, not the orchestrator's, so a +stack stays self-contained. + +### Why every caller passes `--env-file` + +Compose discovers `.env` in the project directory, which defaults to the folder holding the first `-f` file. The +orchestrator lives in `stacks/` and `.env` lives in `.devcontainer/`, so that discovery does not reach it. Every +caller — `scripts/startup.sh`, the MOTD, and CI — therefore names it explicitly: + +```bash +docker compose --env-file .devcontainer/.env -f .devcontainer/stacks/compose.yaml up -d +``` + +Omitting it does not error. Every `${VAR:-default}` quietly takes its default and `COMPOSE_PROFILES` reads as empty, +so all opt-in stacks silently vanish. The orchestrator also sets `name: musher-dev` explicitly, because Compose +would otherwise derive the project name from the `stacks/` folder. + --- ## Networking @@ -448,6 +520,7 @@ settings across container rebuilds. policies/__init__.py The policy registry -- the only wiring a policy needs policies/config/ .config/ layout, index, and no shadowing root config policies/ports/ Port table ↔ forwardPorts ↔ compose parity + policies/toolchain/ Image-baked pins; banned rate-limit-fragile Features policies/hooks/ lefthook ↔ CI job parity policies/rulesets/ Branch rulesets ↔ CI job-name parity .github/ @@ -458,12 +531,14 @@ taskfiles/ Task modules included by the root Taskfile.yml Taskfile.yml Task entry point (cannot move — root-only discovery) .gitattributes Line-ending policy (`* text=auto eol=lf`) .devcontainer/ + Dockerfile The image: bun, uv, task, mise as pinned ARGs + .dockerignore Empties the build context (`*`); keeps .env off the daemon devcontainer.json Features, extensions, settings, mounts, ports - mise.toml CLIs without a Feature (single source of tool versions) - compose.yaml Stack orchestrator (includes stacks//compose.yaml) + mise.toml Runtime-only CLIs with no Feature .env.example Environment template (copy to .env) .env Local overrides (gitignored) - stacks/ One folder per stack: its compose.yaml + colocated config + stacks/ The services, and the orchestrator that includes them + compose.yaml Stack orchestrator (`include:` + `name: musher-dev`) postgres/ compose.yaml PostgreSQL with pgvector (always on) init/ @@ -490,6 +565,7 @@ Taskfile.yml Task entry point (cannot move — root-only discov initialize.sh Host-side bootstrap (runs before docker run) post-create.sh One-time setup entry point startup.sh Every-start service launcher + verify-toolchain.sh Asserts baked tools match the Dockerfile ARGs (CI) lib/ base-setup.sh Reusable tool installer (mise CLIs + Claude) common.sh Shared utilities diff --git a/README.md b/README.md index cb673c6..3475136 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,9 @@ need. ## What You Get -- Ubuntu base with zsh/oh-my-zsh -- Node, Python, Go, Java, Deno, bun, uv — pinned Features +- Ubuntu 24.04 LTS base with zsh/oh-my-zsh +- Node, Python, Go, Java, Deno — pinned Features +- bun, uv, Task, mise — baked into the image by `.devcontainer/Dockerfile` (pinned `ARG`s) - Docker-in-Docker - Git + GitHub CLI - Claude Code + Codex CLI + Task runner + Lefthook @@ -27,8 +28,9 @@ need. All local-dev state is contained under `.devcontainer/`. On first build, `initializeCommand` copies `.env.example` → `.env` (gitignored). The same file feeds: -- **Docker Compose** — auto-discovered as the sibling `.env` next to `compose.yaml`, used to interpolate - `${VAR:-default}` references. +- **Docker Compose** — passed explicitly as `--env-file .devcontainer/.env` by every caller, used to interpolate + `${VAR:-default}` references. It is not auto-discovered: the orchestrator lives at + `.devcontainer/stacks/compose.yaml` and `.env` is not its sibling. - **The dev container itself** — loaded via `runArgs --env-file`, so shells and runtimes inside the container see the same values. @@ -49,18 +51,21 @@ The startup MOTD also warns about drift or unfilled required keys. ## Customize - Comment out unneeded features/extensions in `devcontainer.json` -- Change a tool version → `devcontainer.json` (Features), or `.devcontainer/mise.toml` for CLIs without a Feature - (AI CLIs, lefthook, linters) +- Change a tool version → `devcontainer.json` (Features), `.devcontainer/Dockerfile` (bun, uv, Task, mise), or + `.devcontainer/mise.toml` for runtime-only CLIs (AI CLIs, lefthook, linters). The four-tier rule is in + [CONFIGURATION.md](CONFIGURATION.md#runtimes--tools) and enforced by `repo toolchain check`. - Change a lint rule → the matching file in `.config/` (see [`.config/README.md`](.config/README.md)) - Add project setup to `scripts/post-create.sh` (runs after `base_setup`) -- Enable optional services via `COMPOSE_PROFILES` in `.devcontainer/.env` (redis, minio, registry, azimutt, observability) +- Add or enable a service → `.devcontainer/stacks/` (one folder per stack, registered in `stacks/compose.yaml`); + toggle with `COMPOSE_PROFILES` in `.devcontainer/.env` (redis, minio, registry, azimutt, observability) - Full reference → [CONFIGURATION.md](CONFIGURATION.md) ## Included CI `.github/workflows/validate.yaml` runs seven jobs: ShellCheck, Compose config validation, devcontainer lockfile -freshness, `.env` template sync, the lint gates, the repo structure policies, and a devcontainer build. Lint tool -versions resolve from `.devcontainer/mise.toml` — the same file the container uses — so CI and local cannot drift. +freshness, `.env` template sync, the lint gates, the repo structure policies, and a devcontainer build that also +asserts the image-baked tools report their pinned versions. Lint tool versions resolve from +`.devcontainer/mise.toml` — the same file the container uses — so CI and local cannot drift. The same lint and structure checks run pre-commit via [`.config/lefthook.yml`](.config/lefthook.yml), and `repo hooks check` fails the build if the two ever disagree. @@ -75,8 +80,9 @@ will tell you why. ### CRLF / WSL line ending issues -The `postCreateCommand` automatically strips `\r` from all scripts before running them. If you add new scripts, ensure -they're under `.devcontainer/scripts/` to be included. +`.gitattributes` (`* text=auto eol=lf`) normalizes every tracked file, so scripts arrive with LF on every platform +and no fixup step is needed. The one exception is `.devcontainer/.env`, which is gitignored and therefore out of +`.gitattributes`' reach — `scripts/initialize.sh` strips `\r` from it host-side before the container starts. ### Stale containers From 2f82183e0bc95966efac75a18d92d220051b01ab Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sun, 16 Aug 2026 13:11:11 +0700 Subject: [PATCH 2/2] refactor(repo): cut comments to signal, and enforce the convention The Dockerfile added in the previous commit opened with 52 comment lines before its first instruction -- twice the next-longest block in the repo, and 8% of every comment line under .devcontainer/. For a template read before it is run, the first file a new developer opens read like an incident report. The goal is not fewer comments; it is that each one earns its line, and each rationale is written once. MEASURED FIRST Across 22 files (2,225 lines, 643 comment lines) the repo averages 0.49 comment:code. The Dockerfile was 3.03. Block sizes are bimodal -- 43% are one line, a secondary mode sits at 4-8, and only four blocks anywhere exceeded 20 lines, all of them file headers. Separately, one decision was explained six times: the nanolayer/api.github.com diagnosis occupied ~1,461 words across Dockerfile, devcontainer.json, the toolchain policy, CONFIGURATION.md and .repo/README.md. WHAT CHANGED CONFIGURATION.md is now the single account (the asset_resolver call, why the pin does not help, the recovery-container symptom, and which Features were cleared). Everything else carries a one-line summary and a pointer. The depth belongs in the Violation `docs` field, which reporting.py already provides for exactly this -- _WHY_NOT_A_FEATURE was 102 words against a repo norm of 37-49, and is now 44. Dockerfile: 129 -> 65 lines, ratio 3.03 -> 0.97, longest block 52 -> 13. The four hazards that are destructive or silent when ignored stay, at 1-2 lines each: the bun installer's HOME rewrite, the bunx symlink, presence-only assertions, and the absent USER. Also trimmed: stacks/compose.yaml (the 12-line port table duplicated CONFIGURATION.md and was the one copy `repo ports check` never read -- a fourth unverified copy free to drift), .env.example, initialize.sh, .dockerignore, mise.toml, devcontainer.json. Removed the ~13 comments that restate the line below them, in motd.sh, startup.sh and postgres/compose.yaml, plus three common.sh dividers guarding a single function each. Fixed startup.sh's function header, the only one in the repo ordering Arguments before Globals; Google's order is Globals, Arguments, Outputs, Returns. The Google-style headers on every function in scripts/lib/ are untouched -- Google mandates them for libraries regardless of length, and they are why the shell libs sit at ~1.0 rather than 0.2. ENFORCEMENT New `comments` policy, under the existing governance job: CMT-01 contiguous comment block over 20 lines CMT-02 an ALLOWED_LONG_BLOCKS entry that no longer excuses anything CMT-03 a docs: pointer whose anchor no longer resolves The limit is 20 because the data has a gap there -- nothing legitimate sits between 16 and 22 -- so it catches outliers without fighting lefthook.yml's dense 16-line header. ALLOWED_LONG_BLOCKS ships empty. CMT-03 is what makes the rest safe: replacing prose with pointers only works while pointers resolve. All five existing anchors resolve, so it starts green as a regression guard. Markdown is excluded from the scan -- `#` is a heading there -- as are shebangs and shellcheck/syntax directives. Verified: all three CMT codes negative-tested by sabotaging what they guard; shellcheck, markdownlint, yamllint, actionlint, codespell and all six policies pass; compose resolves under default and all profiles. Net across the touched files: 361 -> 233 comment lines, 0.67 -> 0.44. Co-Authored-By: Claude Opus 5 (1M context) --- .devcontainer/.dockerignore | 18 +-- .devcontainer/.env.example | 30 ++-- .devcontainer/Dockerfile | 108 +++------------ .devcontainer/devcontainer.json | 28 ++-- .devcontainer/mise.toml | 23 ++- .devcontainer/scripts/initialize.sh | 25 ++-- .devcontainer/scripts/lib/common.sh | 6 - .devcontainer/scripts/lib/motd.sh | 6 - .devcontainer/scripts/startup.sh | 6 +- .devcontainer/stacks/compose.yaml | 31 ++--- .devcontainer/stacks/postgres/compose.yaml | 2 - .repo/README.md | 21 +-- .repo/governance/policies/__init__.py | 3 +- .../governance/policies/comments/__init__.py | 5 + .repo/governance/policies/comments/check.py | 131 ++++++++++++++++++ .../policies/comments/violations.py | 57 ++++++++ .repo/governance/policies/toolchain/check.py | 6 +- .../policies/toolchain/violations.py | 21 +-- CONFIGURATION.md | 70 +++++++++- 19 files changed, 348 insertions(+), 249 deletions(-) create mode 100644 .repo/governance/policies/comments/__init__.py create mode 100644 .repo/governance/policies/comments/check.py create mode 100644 .repo/governance/policies/comments/violations.py diff --git a/.devcontainer/.dockerignore b/.devcontainer/.dockerignore index 17ec350..8d1de1a 100644 --- a/.devcontainer/.dockerignore +++ b/.devcontainer/.dockerignore @@ -1,14 +1,10 @@ -# Build context for .devcontainer/Dockerfile is `.devcontainer/` itself, and the -# Dockerfile has no COPY instruction -- every tool it bakes is fetched over -# the network from a pinned URL. So the correct context is empty. +# The build context is `.devcontainer/` and the Dockerfile has no COPY +# instruction, so the correct context is empty. # -# This is not merely tidiness. `.devcontainer/.env` is gitignored and holds -# POSTGRES_PASSWORD and the MinIO root credentials. It is never copied, but an -# un-ignored context is still transferred to the Docker daemon and can land in -# the build cache. +# Not just tidiness: `.env` is gitignored and holds POSTGRES_PASSWORD and the +# MinIO root credentials. An un-ignored context still reaches the Docker daemon +# and can land in the build cache. # -# Deliberately the context-root form rather than a `Dockerfile.dockerignore` -# sibling: when Features are present the devcontainer CLI synthesizes an -# extended Dockerfile in a temp directory, so BuildKit's sibling lookup would -# miss it. `/.dockerignore` is always found. +# Context-root form, not a `Dockerfile.dockerignore` sibling — with Features +# present the devcontainer CLI builds from a temp dir, so the sibling is missed. * diff --git a/.devcontainer/.env.example b/.devcontainer/.env.example index 689db92..a771442 100644 --- a/.devcontainer/.env.example +++ b/.devcontainer/.env.example @@ -1,28 +1,16 @@ # ============================================================ # Dev Container Environment Template # ============================================================ -# This file is the source of truth for local-dev environment vars. -# On first container build, `initializeCommand` copies it to -# `.devcontainer/.env` (gitignored) on the host. From there: +# Source of truth for local-dev environment vars. On first build +# `initializeCommand` copies this to `.devcontainer/.env` (gitignored), which +# feeds both Docker Compose (via an explicit --env-file) and the dev container +# itself (via runArgs --env-file). See CONFIGURATION.md → "Environment +# Variables"; `task env:check` reports drift between the two files. # -# * Docker Compose reads it because every caller passes -# `--env-file .devcontainer/.env` explicitly, and interpolates -# ${VAR:-default} references from it. Compose's positional -# auto-discovery does NOT apply: the orchestrator lives at -# `.devcontainer/stacks/compose.yaml` and this file is not its -# sibling. Any new caller must pass --env-file too, or every -# value silently falls back to its default and COMPOSE_PROFILES -# reads as empty. -# * `runArgs --env-file` loads it into the dev container itself -# so shells, runtimes, and `task` runs see the same values. -# -# Drift checks: `task env:check` compares this file against your -# local `.env` and flags missing keys. -# -# Convention (three states): -# 1. Filled defaults — `VAR=value` safe demo values; override only if needed. -# 2. Required (empty) — `VAR=` must be filled in; container warns at startup. -# 3. Optional overrides — `# VAR=value` uncomment to enable. +# Three states, by convention: +# 1. Filled default — `VAR=value` safe demo value; override if needed. +# 2. Required (empty) — `VAR=` must be filled; startup warns. +# 3. Optional override — `# VAR=value` uncomment to enable. # ============================================================ diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index 3da1c39..e278994 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -1,93 +1,42 @@ # syntax=docker/dockerfile:1 -# ============================================================================= # Musher dev container base image. # -# WHY THIS FILE EXISTS +# bun, uv and task are installed here rather than as Features: their Features +# resolve release assets through an unauthenticated api.github.com call, which +# the shared egress IPs used by Codespaces and CI rate-limit, failing the whole +# build. mise is here because it was otherwise the one unpinned tool. # -# bun, uv and task are baked here rather than installed via -# ghcr.io/devcontainers-extra/features/{bun,uv,go-task}. All three route through -# nanolayer's gh-release installer, which enumerates release assets against -# api.github.com UNAUTHENTICATED: +# Full rationale: CONFIGURATION.md → "Runtimes & Tools" +# Enforced by: repo toolchain check (TC-01..TC-03) # -# nanolayer/installers/gh_release/resolvers/asset_resolver.py -# urllib.request.urlopen(f"https://api.github.com/repos/{repo}/releases/tags/{tag}") -# -# There is no Authorization header and no GITHUB_TOKEN read anywhere in that -# module. Codespaces build hosts and GitHub-hosted Actions runners share egress -# IP pools, so the 60 req/hr anonymous limit is routinely exhausted and the -# build dies with: -# -# urllib.error.HTTPError: HTTP Error 403: rate limit exceeded -# ERROR: Failed to install Bun. No asset patterns matched. -# -# That message is misleading -- the asset pattern was fine, the API call 403'd. -# PINNING THE VERSION DOES NOT HELP: the pin only supplies the tag, and -# _get_release_assets() still calls the API to LIST the release's assets. -# -# One failed Feature fails the whole image build, and Codespaces then falls back -# to a bare recovery container with none of the toolchain -- so the developer -# sees "task: command not found" rather than the real error. -# -# Every installer below fetches its artifact directly (bun -> bun.sh, uv -> -# astral.sh, task -> github.com/.../releases/download, mise -> mise.run). None -# touches api.github.com, so none can be starved by a noisy neighbour. -# -# Features that stay in devcontainer.json were checked individually and are -# clean: devcontainers-extra/deno curls releases/download/... directly, -# lukewiwa/shellcheck does the same, and robbert229/postgresql-client is apt. -# -# This boundary is enforced, not just documented -- see `repo toolchain check` -# (.repo/governance/policies/toolchain/). -# -# WHAT DOES *NOT* BELONG HERE -# -# The rule is: this file bakes version-pinned tools; scripts/post-create.sh owns -# whatever is inherently runtime or self-updating. In particular `mise install` -# cannot run here -- four of the six entries in mise.toml use the npm: and pipx: -# backends, and Node and Python arrive from Features, which layer AFTER this -# stage. Only the mise binary is baked. -# -# Pinned to the 24.04 LTS tag: the floating `:ubuntu` tag rolls forward to -# interim releases (e.g. 25.10) that docker-in-docker does not support. -# ============================================================================= +# Pinned to the LTS tag: floating `:ubuntu` rolls to interim releases that +# docker-in-docker does not support. FROM mcr.microsoft.com/devcontainers/base:ubuntu-24.04 -# `curl | bash` reports the exit status of bash, so a curl that dies mid-stream -# yields an empty script, a successful shell, and an image missing a tool. That -# silent-success shape is the failure class this file exists to remove, so make -# the pipe honest. +# Without pipefail a truncated `curl | bash` reports bash's exit status, giving +# a successful build and an image with no tool in it. SHELL ["/bin/bash", "-o", "pipefail", "-c"] -# Image-baked tool pins. These ARGs are the single source of truth for the four -# tools below -- there is deliberately no versions file, because nothing outside -# this image consumes them. `repo toolchain check` asserts each is a concrete -# version (TC-02) and that TASK_VERSION matches CI's setup-task pin (TC-03). +# The single source of truth for what this image bakes. verify-toolchain.sh +# reads them back to assert the built container matches. ARG BUN_VERSION=1.3.14 ARG UV_VERSION=0.11.28 ARG TASK_VERSION=3.52.0 ARG MISE_VERSION=v2026.8.6 RUN set -eux; \ - # bun's installer hard-requires unzip, which the base image ships. Assert - # rather than apt-install a fallback: an unpinned `apt-get install` would be - # the only unpinned thing in this image, and a base that stopped shipping - # unzip is a decision someone should have to make explicitly. command -v unzip >/dev/null 2>&1 || { echo "unzip missing from the base image; bun's installer requires it" >&2; exit 1; }; \ \ - # --- bun -> /usr/local/bin/{bun,bunx} (BUN_INSTALL=/usr/local => $BUN_INSTALL/bin) - # Arch and the avx2 baseline variant are chosen by the installer; do not - # hardcode either (Codespaces is x86_64, some dev machines are aarch64). - # HOME is redirected to a scratch dir because the installer unconditionally - # appends PATH exports to ~/.bashrc and ~/.zshrc when they are writable. + # --- bun -> /usr/local/bin/{bun,bunx} + # HOME is a scratch dir because the installer appends PATH exports to + # ~/.bashrc and ~/.zshrc whenever they are writable. mkdir -p /tmp/bun-home; \ HOME=/tmp/bun-home BUN_INSTALL=/usr/local bash -c \ "curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://bun.sh/install \ | bash -s bun-v${BUN_VERSION}"; \ rm -rf /tmp/bun-home; \ - # `bunx` needs no wrapper: the installer creates it as a symlink to `bun`. - # Do NOT `printf > /usr/local/bin/bunx` to "recreate" it -- that follows the - # symlink and overwrites the ~90 MB bun binary with the wrapper text, after - # which every `bun` call recurses into itself and hangs. + # Never write to /usr/local/bin/bunx: it is a symlink to bun, so doing so + # overwrites the binary and every `bun` call then recurses into itself. \ # --- uv -> /usr/local/bin/{uv,uvx} curl --retry 5 --retry-all-errors --retry-delay 3 -LsSf \ @@ -95,28 +44,16 @@ RUN set -eux; \ | env UV_INSTALL_DIR=/usr/local/bin INSTALLER_NO_MODIFY_PATH=1 sh; \ \ # --- task -> /usr/local/bin/task - # godownloader script: downloads the release tarball and its checksum file - # from github.com/go-task/task/releases/download and sha256-verifies before - # installing. `-b` sets bindir; the trailing arg is the tag. curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://taskfile.dev/install.sh \ | sh -s -- -b /usr/local/bin "v${TASK_VERSION}"; \ \ # --- mise -> /usr/local/bin/mise - # Installed system-wide and version-pinned. Previously this was an unpinned - # `curl https://mise.run | sh` in post-create -- the only unpinned tool in a - # repo that pins everything else. The per-user shim dir that remoteEnv PATH - # expects (~/.local/share/mise/shims) is still created at runtime by - # `mise reshim` in base-setup.sh. + # System-wide; base-setup.sh still runs `mise reshim` to build the per-user + # shim dir that remoteEnv's PATH expects. MISE_VERSION="${MISE_VERSION}" MISE_INSTALL_PATH=/usr/local/bin/mise \ bash -c 'curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://mise.run | sh'; \ \ - # Fail THIS layer, loudly, rather than letting post-create discover it. - # - # Presence-only on purpose. The pinned download URLs already guarantee the - # versions -- a wrong version 404s -- and asserting them by executing the - # binaries in the same layer that installed them is a known hazard under - # BuildKit. The runtime version assertion lives in - # scripts/verify-toolchain.sh, which CI runs against the built container. + # Presence-only: the runtime version check lives in verify-toolchain.sh. test -x /usr/local/bin/bun; \ test -x /usr/local/bin/bunx; \ test -x /usr/local/bin/uv; \ @@ -124,6 +61,5 @@ RUN set -eux; \ test -x /usr/local/bin/task; \ test -x /usr/local/bin/mise -# No `USER vscode` here on purpose: Features install after this stage and -# `updateRemoteUserUID` expects root at build time. devcontainer.json's -# `remoteUser` owns runtime identity. +# No USER: Features install after this stage and updateRemoteUserUID expects +# root at build time. devcontainer.json's remoteUser owns runtime identity. diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index 8d6bf4b..6dc2c4d 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -2,11 +2,8 @@ // Uncomment optional blocks as needed. Comment out what you don't use. { "name": "Musher Dev", - // The image is defined by .devcontainer/Dockerfile, which bakes the tools - // whose Features cannot be relied on (see its header, and the "Runtimes & - // Tools" section of CONFIGURATION.md). Features below layer on top of it. - // Context is .devcontainer/ and is emptied by .dockerignore — there is no - // COPY instruction, and .env must never reach the daemon. + // Dockerfile bakes the tools whose Features cannot be relied on; the + // Features below layer on top of it. Context is emptied by .dockerignore. "build": { "dockerfile": "Dockerfile", "context": "." @@ -15,11 +12,7 @@ "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/${localWorkspaceFolderBasename},type=bind,consistency=cached", "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}", - // Developer tools are pinned here as Features (baked into the image). Tools - // with no Feature — the Codex and Lefthook CLIs — live in - // .devcontainer/mise.toml; bun, uv, task and mise are baked by the Dockerfile; - // Claude Code self-updates via its native installer. - // See CONFIGURATION.md → "Runtimes & Tools" for which tier a tool belongs in. + // Which tier a tool belongs in: CONFIGURATION.md → "Runtimes & Tools". "features": { // --- Platform --- "ghcr.io/devcontainers/features/common-utils:2": { @@ -42,13 +35,9 @@ "installGradle": false, "installMaven": false }, - // deno stays a Feature: its installer curls releases/download/... directly - // and never calls api.github.com. bun, uv and task are NOT here — do not - // re-add them. Their Features route through nanolayer's gh-release - // installer, which lists assets against api.github.com unauthenticated and - // fails the whole build once the shared-IP rate limit is hit. They are - // baked by .devcontainer/Dockerfile instead; enforced by TC-01 in - // .repo/governance/policies/toolchain/. + // deno is safe as a Feature; bun, uv and task are not — do not re-add + // them. They are baked by the Dockerfile (TC-01 fails the build if they + // reappear here). Why: CONFIGURATION.md → "Runtimes & Tools". "ghcr.io/devcontainers-extra/features/deno:1": { "version": "2.9.2" }, // --- Linting --- @@ -107,9 +96,8 @@ }, "remoteEnv": { - // mise shims (codex, lefthook) + ~/.local/bin (Claude Code) on PATH. - // mise, bun, uv and task are baked into /usr/local/bin by the Dockerfile - // and are already on the default PATH. + // mise shims (codex, lefthook) + ~/.local/bin (Claude Code). The baked + // tools live in /usr/local/bin and are already on PATH. "PATH": "/home/vscode/.local/share/mise/shims:/home/vscode/.local/bin:${containerEnv:PATH}" }, diff --git a/.devcontainer/mise.toml b/.devcontainer/mise.toml index 18732a1..be59813 100644 --- a/.devcontainer/mise.toml +++ b/.devcontainer/mise.toml @@ -1,20 +1,13 @@ -# Developer tools that have no devcontainer Feature. +# Developer CLIs that have no devcontainer Feature and are only needed at +# runtime. Tools with a Feature are pinned in devcontainer.json; bun, uv, task +# and mise are baked by the Dockerfile. +# Tier rules: CONFIGURATION.md → "Runtimes & Tools". # -# Everything with a Feature is pinned in devcontainer.json (baked into the -# image); the CLIs below are npm-distributed with no Feature, so mise installs -# and version-pins them instead. Claude Code is the one other exception — it -# self-updates via its native installer (scripts/lib/base-setup.sh). +# This file is the single source of these versions: post-create runs +# `mise install` against it, and CI resolves the same pins via jdx/mise-action. +# Edit a version here, then `task tools:install`. # -# Change a version here, then run `task tools:install`. Devcontainer post-create -# runs `mise install` against this file. Docs: https://mise.jdx.dev -# See CONFIGURATION.md → "Runtimes & Tools". - -# This file is the SINGLE source of tool versions. CI resolves the same pins -# from here via jdx/mise-action (MISE_GLOBAL_CONFIG_FILE), so a version is -# never stated twice and local and CI cannot drift. -# -# Backends are fully qualified (npm:, pipx:, aqua:) rather than short names so -# resolution does not depend on mise's registry. +# Backends are fully qualified so resolution does not depend on mise's registry. [tools] # --- AI + git hooks ------------------------------------------------------ diff --git a/.devcontainer/scripts/initialize.sh b/.devcontainer/scripts/initialize.sh index aad2d88..e0af018 100644 --- a/.devcontainer/scripts/initialize.sh +++ b/.devcontainer/scripts/initialize.sh @@ -1,25 +1,18 @@ #!/usr/bin/env bash # initialize.sh — Host-side bootstrap for the dev container. # -# Runs on the host (via devcontainer.json `initializeCommand`) BEFORE -# `docker run` is invoked. Because `runArgs --env-file` is evaluated at -# `docker run` time, the .env file must exist on the host before the -# container starts — that's why this work lives here, not in -# post-create.sh. +# Runs on the host via `initializeCommand`, before `docker run` — it has to, +# because `runArgs --env-file` is evaluated at `docker run` time, so .env must +# already exist. post-create.sh would be too late. # -# Responsibilities: -# * Create .devcontainer/.env from .env.example on first clone. -# * Touch an empty .env if no example exists, so --env-file never hard-fails. -# * Strip CRLF from .env (Windows/WSL safety — docker --env-file -# rejects files with CRLF line endings). -# -# Why this CRLF guard survives while the postCreateCommand one did not: -# .gitattributes (`* text=auto eol=lf`) normalizes every file Git checks -# out, which covers scripts/ and made the old `fix-crlf` step redundant. -# It cannot cover .env — that file is gitignored, generated locally, and -# hand-edited, so a Windows editor can reintroduce CR at any time. +# The CRLF guard here is not redundant with .gitattributes. That normalizes +# every file Git checks out, which is why the old postCreate fix-crlf step +# could go; it cannot reach .env, which is gitignored, generated locally and +# hand-edited, so a Windows editor can reintroduce CR at any time. Docker +# rejects an --env-file containing CRLF. # # Idempotent: safe to run on every container start. + set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" diff --git a/.devcontainer/scripts/lib/common.sh b/.devcontainer/scripts/lib/common.sh index b11149b..3b63b71 100644 --- a/.devcontainer/scripts/lib/common.sh +++ b/.devcontainer/scripts/lib/common.sh @@ -8,8 +8,6 @@ # and tool verification functions used by all setup scripts. set -euo pipefail -# --- Logging --- - # Logs a timestamped message to stderr. # # Arguments: @@ -105,8 +103,6 @@ setup_config_dirs() { done } -# --- NVM helpers --- - # Fixes NVM directory ownership to the current user. # # Globals: @@ -121,8 +117,6 @@ fix_nvm_permissions() { fi } -# --- NPM install helper --- - # Installs an npm package globally with retry logic. # # Arguments: diff --git a/.devcontainer/scripts/lib/motd.sh b/.devcontainer/scripts/lib/motd.sh index 1b737f7..dd5d2ad 100644 --- a/.devcontainer/scripts/lib/motd.sh +++ b/.devcontainer/scripts/lib/motd.sh @@ -59,17 +59,14 @@ _motd_runtimes() { echo " ${_BOLD}Runtimes${_RESET}" echo " ${_DIM}${sep}${_RESET}" - # Row 1: node + python _motd_runtime_entry node "node" "node -v" _motd_runtime_entry python3 "python" "python3 -c 'import platform; print(platform.python_version())'" echo "" - # Row 2: go + java _motd_runtime_entry go "go" "go version | grep -oP '\\d+\\.\\d+\\.\\d+'" _motd_runtime_entry java "java" "java -version 2>&1 | head -1 | grep -oP '\\d+[\\d.]+'" echo "" - # Row 3: deno + bun _motd_runtime_entry deno "deno" "deno -v | head -1 | awk '{print \$2}'" _motd_runtime_entry bun "bun" "bun -v" echo "" @@ -107,18 +104,15 @@ _motd_services() { state="$(echo "$line" | grep -oP '"State"\s*:\s*"\K[^"]+' | head -1)" health="$(echo "$line" | grep -oP '"Health"\s*:\s*"\K[^"]+' | head -1)" - # Extract published host port ports="$(echo "$line" | grep -oP '"PublishedPort"\s*:\s*\K\d+' | head -1)" [[ -z "$name" ]] && continue - # Build display name local display_name="$name" if [[ -n "$ports" ]] && [[ "$ports" != "0" ]]; then display_name="${name} (${ports})" fi - # Determine status label and color local status_label color if [[ -n "$health" ]] && [[ "$health" != "" ]]; then status_label="$health" diff --git a/.devcontainer/scripts/startup.sh b/.devcontainer/scripts/startup.sh index 58013e6..8f57a06 100644 --- a/.devcontainer/scripts/startup.sh +++ b/.devcontainer/scripts/startup.sh @@ -41,11 +41,11 @@ trap 'on_error ${LINENO} "${BASH_COMMAND}"' ERR # Polls compose services until all report healthy or timeout elapses. # -# Arguments: -# $1 — timeout in seconds (default: 60) # Globals: # COMPOSE_FILE — read, path to stacks/compose.yaml # ENV_FILE — read, path to .env (passed to every compose invocation) +# Arguments: +# $1 — timeout in seconds (default: 60) # Outputs: # Writes progress/warnings to stderr via log() # Returns: @@ -58,7 +58,6 @@ wait_for_healthy() { local output output="$(docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" ps --format json 2>/dev/null || true)" - # Detect failed services (exited, dead, or unhealthy) local failed="" failed="$(echo "$output" | grep -E '"(exited|dead|unhealthy)"' || true)" if [[ -n "$failed" ]]; then @@ -74,7 +73,6 @@ wait_for_healthy() { return 1 fi - # Count services still starting local starting starting="$(echo "$output" | grep -c '"starting"' || true)" if [[ "$starting" -eq 0 ]]; then diff --git a/.devcontainer/stacks/compose.yaml b/.devcontainer/stacks/compose.yaml index 89b4a66..e90a0b6 100644 --- a/.devcontainer/stacks/compose.yaml +++ b/.devcontainer/stacks/compose.yaml @@ -1,28 +1,15 @@ -# Musher Dev Container — Docker Compose orchestrator. +# Musher dev container — Docker Compose orchestrator. # -# Each stack lives in its own folder next to this file (its compose.yaml plus -# any config it needs); add a new folder and include its compose.yaml below. -# Services run inside Docker-in-Docker, started by scripts/startup.sh. +# One folder per stack, each holding its own compose.yaml plus any config it +# needs. Services run inside Docker-in-Docker, started by scripts/startup.sh. +# This is not the dev container itself — that is ../Dockerfile + ../devcontainer.json. # -# This file is NOT the dev container. The dev container itself is defined by -# ../Dockerfile + ../devcontainer.json; these are the supporting services it -# brings up. +# Every caller must pass `--env-file ../.env`: this file is not a sibling of +# .env, so Compose's positional discovery does not reach it and every value +# would silently fall back to its default. `name` is explicit for the same +# reason — Compose would otherwise derive the project name from this folder. # -# `name` is set explicitly because Compose otherwise derives the project name -# from the project directory — which would silently become "stacks". Likewise -# every caller must pass `--env-file ../.env`: this file is no longer a sibling -# of .env, so Compose's positional auto-discovery does not reach it. -# -# Port Allocation: -# 15432 PostgreSQL 15440 MinIO API (obs.) -# 15433 Redis 15441 MinIO Console (obs.) -# 15434 MinIO API 15442 Tempo -# 15435 MinIO Console 15443 Loki -# 15436 OCI Registry 15444 VictoriaMetrics -# 15460 Azimutt 15445 OTel HTTP -# 15446 OTel gRPC -# 15447 Grafana -# 15448 Pyroscope +# Ports: CONFIGURATION.md → "Port Allocation" (parity enforced by `repo ports check`). name: musher-dev diff --git a/.devcontainer/stacks/postgres/compose.yaml b/.devcontainer/stacks/postgres/compose.yaml index c95f759..39111c7 100644 --- a/.devcontainer/stacks/postgres/compose.yaml +++ b/.devcontainer/stacks/postgres/compose.yaml @@ -1,5 +1,3 @@ -# PostgreSQL with pgvector extension. - services: postgres: image: pgvector/pgvector:0.8.5-pg17 diff --git a/.repo/README.md b/.repo/README.md index fe8e990..8567fff 100644 --- a/.repo/README.md +++ b/.repo/README.md @@ -44,24 +44,15 @@ runs `uv tool install ./.repo`). To reinstall after editing it: | `hooks` | `HOOK-01`..`HOOK-04` | Every lefthook job has a CI counterpart and vice versa, or a recorded reason why not | | `rulesets` | `RS-01`..`RS-04` | Committed branch rulesets stay valid and in step with the CI jobs they require | | `toolchain` | `TC-01`..`TC-03` | The tools the image bakes stay out of the Features block, keep exact pins, and stay in step with CI | +| `comments` | `CMT-01`..`CMT-03` | Comment blocks stay short, the allowlist stays honest, and every `docs:` pointer still resolves | ### Why `toolchain` exists -`TC-01` is the policy least likely to be guessed from the code it guards. bun, -uv and task are installed by [`.devcontainer/Dockerfile`](../.devcontainer/Dockerfile) -rather than by their `devcontainers-extra` Features, because those Features -resolve release assets through nanolayer, which calls `api.github.com` with no -credentials. On Codespaces build hosts and GitHub-hosted runners — which share -egress IP pools — that hits the 60 req/hr anonymous limit, and one failed -Feature fails the whole image build. Pinning the version does not avoid the -call. Re-adding any of the three looks like a harmless simplification, which is -exactly why it is a check and not a comment. The full diagnosis is in -[`policies/toolchain/violations.py`](governance/policies/toolchain/violations.py). - -`BANNED_FEATURES` in [`policies/toolchain/check.py`](governance/policies/toolchain/check.py) -lists only Features whose installers were read and confirmed to make that call. -`deno`, `shellcheck` and `postgresql-client` were checked and are clean, so they -stay Features — the rule is about the installer's behaviour, not the publisher. +`TC-01` is the policy least likely to be guessed from the code it guards: bun, uv and +task are installed by [`.devcontainer/Dockerfile`](../.devcontainer/Dockerfile) rather +than by their Features, because those Features fail on rate-limited build hosts. +Re-adding one looks like a harmless simplification, which is exactly why it is a check. +Full account: [`CONFIGURATION.md`](../CONFIGURATION.md) → "Runtimes & Tools". ### The one-way-check tables diff --git a/.repo/governance/policies/__init__.py b/.repo/governance/policies/__init__.py index c689d69..0d4892a 100644 --- a/.repo/governance/policies/__init__.py +++ b/.repo/governance/policies/__init__.py @@ -13,7 +13,7 @@ from collections.abc import Callable -from governance.policies import config, hooks, ports, rulesets, toolchain +from governance.policies import comments, config, hooks, ports, rulesets, toolchain from governance.reporting import Report POLICIES: dict[str, Callable[[], Report]] = { @@ -22,6 +22,7 @@ "hooks": hooks.run, "rulesets": rulesets.run, "toolchain": toolchain.run, + "comments": comments.run, } __all__ = ["POLICIES"] diff --git a/.repo/governance/policies/comments/__init__.py b/.repo/governance/policies/comments/__init__.py new file mode 100644 index 0000000..7dc1120 --- /dev/null +++ b/.repo/governance/policies/comments/__init__.py @@ -0,0 +1,5 @@ +"""Policy: comments stay short, and the pointers they lean on still resolve.""" + +from governance.policies.comments.check import run + +__all__ = ["run"] diff --git a/.repo/governance/policies/comments/check.py b/.repo/governance/policies/comments/check.py new file mode 100644 index 0000000..9bc34ba --- /dev/null +++ b/.repo/governance/policies/comments/check.py @@ -0,0 +1,131 @@ +"""Detect comment blocks that have outgrown their file, and dead doc pointers. + +The rule this enforces is written in CONFIGURATION.md -> "Comments": say the +non-obvious, say it once, and point at it from everywhere else. CMT-01 caps the +volume; CMT-03 keeps the pointers that make the trade possible from rotting. +""" + +from __future__ import annotations + +import ast +import re +from pathlib import Path + +from governance import repo +from governance.policies.comments import violations as v +from governance.reporting import Report + +#: Longest contiguous run of comment lines a file may carry. The natural block +#: size in this repo is 4-8 lines and nothing legitimate sits between 16 and 22, +#: so the limit lands in a real gap rather than on an arbitrary round number. +MAX_BLOCK = 20 + +#: Files whose long block is the point of the file, and why. Empty by design -- +#: an entry here is a standing exemption, so it has to earn its place. +ALLOWED_LONG_BLOCKS: dict[str, str] = {} + +#: Where comments sit next to code. Markdown is absent on purpose: `#` starts a +#: heading there, not a comment. +SCAN_GLOBS = ( + ".devcontainer/Dockerfile", + ".devcontainer/.dockerignore", + ".devcontainer/.env.example", + ".devcontainer/*.toml", + ".devcontainer/**/*.sh", + ".devcontainer/**/*.yaml", + ".config/*.yml", + ".config/*.yaml", + ".github/**/*.yml", + ".github/**/*.yaml", + "taskfiles/*.yml", + "Taskfile.yml", + ".repo/**/*.py", +) + +#: Lines that start with `#` but are instructions to a tool, not prose. They sit +#: inside blocks and must not inflate them. +_DIRECTIVE = re.compile( + r"^#\s*(!|shellcheck\b|syntax=|type:\s|noqa\b|pylint:|mypy:|fmt:|nosec\b)" +) + +#: `docs="CONFIGURATION.md#anchor"` and the DOCS constants the policies share. +_DOC_REF = re.compile(r'"([A-Za-z0-9_./-]+\.md#[^"]+)"') + +_HEADING = re.compile(r"^#{1,6}\s+(.*)$", re.MULTILINE) + + +def _slug(heading: str) -> str: + """GitHub's heading-anchor slug: lowercase, drop punctuation, spaces to dashes.""" + text = heading.strip().lower() + text = re.sub(r"[^\w\s-]", "", text) + return re.sub(r"\s", "-", text) + + +def _anchors(rel: str) -> set[str]: + return {_slug(m.group(1)) for m in _HEADING.finditer(repo.read_text(rel))} + + +def _blocks(path: Path) -> list[tuple[int, int]]: + """Contiguous comment runs as (start_line, length), directives excluded.""" + found: list[tuple[int, int]] = [] + start = length = 0 + for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + stripped = raw.strip() + is_comment = stripped.startswith("#") and not _DIRECTIVE.match(stripped) + if is_comment: + start = start or number + length += 1 + continue + # A directive interrupts prose without ending the block it sits in, so + # only a genuine line of code closes one. + if stripped.startswith("#"): + continue + if start: + found.append((start, length)) + start = length = 0 + if start: + found.append((start, length)) + return found + + +def _scanned_files() -> list[Path]: + seen: dict[str, Path] = {} + for pattern in SCAN_GLOBS: + for path in repo.glob(pattern): + if path.is_file(): + seen[repo.rel(path)] = path + return [seen[key] for key in sorted(seen)] + + +def run() -> Report: + report = Report(policy="comments") + + used: set[str] = set() + for path in _scanned_files(): + rel = repo.rel(path) + for line, length in _blocks(path): + if length <= MAX_BLOCK: + continue + if rel in ALLOWED_LONG_BLOCKS: + used.add(rel) + continue + report.add(v.block_too_long(rel, line, length, MAX_BLOCK)) + + for rel in sorted(set(ALLOWED_LONG_BLOCKS) - used): + report.add(v.stale_allowance(rel)) + + # Pointers are read statically: the violation factories take arguments, so + # calling them just to inspect `docs` would mean inventing fixture values. + for path in repo.glob(".repo/**/*.py"): + source = repo.rel(path) + for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))): + if not isinstance(node, ast.Constant) or not isinstance(node.value, str): + continue + match = _DOC_REF.fullmatch(f'"{node.value}"') + if not match: + continue + target, _, anchor = node.value.partition("#") + if not repo.exists(target) or anchor not in _anchors(target): + report.add(v.dead_pointer(f"{source}:{node.lineno}", node.value, target)) + + return report diff --git a/.repo/governance/policies/comments/violations.py b/.repo/governance/policies/comments/violations.py new file mode 100644 index 0000000..0bd47cf --- /dev/null +++ b/.repo/governance/policies/comments/violations.py @@ -0,0 +1,57 @@ +"""What can go wrong with comment volume, and why each rule exists.""" + +from __future__ import annotations + +from governance.reporting import Violation + +DOCS = "CONFIGURATION.md#comments" + + +def block_too_long(path: str, line: int, length: int, limit: int) -> Violation: + return Violation( + code="CMT-01", + summary=f"comment block of {length} lines (limit {limit})", + reason=( + "This template is read before it is run, so a header that buries the " + "file is a cost paid by every reader. Blocks here run 4-8 lines; past " + "the limit the content is almost always rationale, which belongs in " + "CONFIGURATION.md with a pointer left behind." + ), + fix=( + "Move the detail into CONFIGURATION.md and leave a one-line summary " + "and a pointer, or register the block in ALLOWED_LONG_BLOCKS with a " + "reason." + ), + where=f"{path}:{line}", + docs=DOCS, + ) + + +def stale_allowance(path: str) -> Violation: + return Violation( + code="CMT-02", + summary=f"ALLOWED_LONG_BLOCKS still excuses {path}, which no longer needs it", + reason=( + "An allowlist that outlives what it excused stops describing the repo " + "and starts hiding the next violation." + ), + fix=f"Drop the {path!r} entry from ALLOWED_LONG_BLOCKS.", + where=".repo/governance/policies/comments/check.py", + docs=DOCS, + ) + + +def dead_pointer(source: str, ref: str, target: str) -> Violation: + return Violation( + code="CMT-03", + summary=f"docs pointer {ref!r} does not resolve", + reason=( + "Comments here reference a single canonical explanation instead of " + "repeating it. That trade is only safe while the references hold: a " + "renamed heading turns the reasoning into something nobody can find, " + "and the next reader writes the explanation out again." + ), + fix=f"Point at a heading that exists in {target}, or restore the heading.", + where=source, + docs=DOCS, + ) diff --git a/.repo/governance/policies/toolchain/check.py b/.repo/governance/policies/toolchain/check.py index f0ee78c..9e7b577 100644 --- a/.repo/governance/policies/toolchain/check.py +++ b/.repo/governance/policies/toolchain/check.py @@ -19,9 +19,9 @@ WORKFLOW = ".github/workflows/validate.yaml" #: Feature ref (without the version suffix) -> the tool it would install. -#: Only Features whose installers were verified to call api.github.com belong -#: here. deno, shellcheck and postgresql-client were checked and are clean: -#: the first two curl releases/download/... directly, the third is apt. +#: Only Features whose installers were read and confirmed to call +#: api.github.com belong here; deno, shellcheck and postgresql-client were +#: checked and are clean. See CONFIGURATION.md -> "Runtimes & Tools". BANNED_FEATURES = { "ghcr.io/devcontainers-extra/features/bun": "bun", "ghcr.io/devcontainers-extra/features/uv": "uv", diff --git a/.repo/governance/policies/toolchain/violations.py b/.repo/governance/policies/toolchain/violations.py index e892be3..16b284d 100644 --- a/.repo/governance/policies/toolchain/violations.py +++ b/.repo/governance/policies/toolchain/violations.py @@ -9,22 +9,13 @@ DOCKERFILE = ".devcontainer/Dockerfile" DEVCONTAINER = ".devcontainer/devcontainer.json" -#: The diagnosis behind TC-01, stated once. It is long on purpose: the whole -#: point of moving these three tools into the Dockerfile is easy to mistake for -#: fussiness, and the next person to reach for the Feature deserves the actual -#: failure mode rather than "we don't do that here". +#: The short form; the full diagnosis is in CONFIGURATION.md, which `docs` below +#: points at. _WHY_NOT_A_FEATURE = ( - "The devcontainers-extra Features for bun, uv and go-task all install via " - "nanolayer's gh-release helper, which lists a release's assets by calling " - "api.github.com with no credentials " - "(nanolayer/installers/gh_release/resolvers/asset_resolver.py). Codespaces " - "build hosts and GitHub-hosted Actions runners share egress IP pools, so " - "the 60 req/hr anonymous limit is routinely exhausted and the call 403s. " - "Pinning the version does not help -- the pin only supplies the tag; the " - "asset listing still hits the API. One failed Feature fails the entire " - "image build, and Codespaces then drops the developer into a bare recovery " - "container, so the symptom is 'task: command not found' rather than the " - "real error." + "The devcontainers-extra Features for bun, uv and go-task resolve release " + "assets through an unauthenticated api.github.com call. Codespaces and CI " + "share egress IPs, so that call is rate-limited, and one failed Feature " + "fails the entire image build. Pinning the version does not avoid the call." ) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index c00fd47..f149ce9 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -173,16 +173,38 @@ ARG MISE_VERSION=v2026.8.6 ``` The first three have Features, and those Features are the problem. All of them resolve release assets through -nanolayer's `gh-release` helper, which lists a release's assets by calling `api.github.com` **with no credentials**. -Codespaces build hosts and GitHub-hosted Actions runners share egress IP pools, so the 60 req/hr anonymous limit is -routinely exhausted, the call 403s, and one failed Feature fails the entire image build — after which Codespaces -drops you into a bare recovery container. Pinning the version does not help: the pin supplies the tag, but the asset -listing still hits the API. `mise` is here for a different reason — it was previously an unpinned `curl | sh` in -post-create, the only unpinned tool in a template that pins everything else. +nanolayer's `gh-release` helper, which lists a release's assets by calling `api.github.com` **with no credentials** — +`nanolayer/installers/gh_release/resolvers/asset_resolver.py`: + +```python +response = urllib.request.urlopen( + f"https://api.github.com/repos/{repo}/releases/tags/{tag}" +) # nosec +``` + +There is no `Authorization` header and no `GITHUB_TOKEN` read anywhere in that module. Codespaces build hosts and +GitHub-hosted Actions runners share egress IP pools, so the 60 req/hr anonymous limit is routinely exhausted and the +call 403s. One failed Feature fails the entire image build, after which Codespaces drops you into a bare recovery +container — so the symptom a developer reports is `task: command not found`, not a rate limit. + +**Pinning the version does not help.** The pin only supplies the tag; `_get_release_assets()` still calls the API to +*list* the assets. The build log reads `Using Bun version: bun-v1.3.14` and then 403s, which makes the failure look +like a bad version pin when the pin was fine. + +`mise` is baked for a different reason — it was previously an unpinned `curl | sh` in post-create, the only unpinned +tool in a template that pins everything else. + +**Features that were checked and cleared**, and stay Features: `devcontainers-extra/deno` and `lukewiwa/shellcheck` +build a `releases/download/...` URL directly; `robbert229/postgresql-client` is apt. The rule is about the +installer's behaviour, not the publisher — so read a third-party Feature's `install.sh` before adding it. `repo toolchain check` enforces all of this: `TC-01` fails if one of those Features comes back, `TC-02` fails on a floating pin, and `TC-03` fails if `TASK_VERSION` drifts from CI's `arduino/setup-task` version. +Version assertions in the Dockerfile are presence-only (`test -x`). The pinned download URLs already guarantee the +version — a wrong one 404s — and executing a binary in the layer that installed it is a known BuildKit hazard. The +runtime assertion lives in `scripts/verify-toolchain.sh`, which CI runs against the built container. + Two constraints on what can go here: - **Features layer *after* this stage**, so the Dockerfile cannot use anything a Feature provides. This is why @@ -222,6 +244,42 @@ base_install_mytool() { --- +## Comments + +This template is read before it is run, so its comments are part of the interface. Four rules, the first two +enforced by `repo comments check`. + +**Comment the non-obvious.** The code states *what*; a comment earns its line by stating *why*. The test: could +someone who has never seen this code write the comment just by reading the line below it? If so, delete it. + +**Write each rationale once, then reference it.** A decision explained at every call site is a decision that will +disagree with itself within a release. The full account lives here in `CONFIGURATION.md`; code carries a one-line +summary and a pointer. `repo comments check` (`CMT-03`) fails the build if a pointer stops resolving, so +references are safe to rely on. + +**Keep file headers short.** A header says what the file is and the one constraint a reader must not violate. +Depth goes here. Blocks over 20 lines fail `CMT-01` — the natural size in this repo is 4–8. + +**Library functions are the exception.** Every function in `.devcontainer/scripts/lib/` carries a full header, per +the [Google Shell Style Guide](https://google.github.io/styleguide/shellguide.html): *"Any function in a library +must have a function header comment regardless of length or complexity."* Tags go in Google's order — `Globals`, +`Arguments`, `Outputs`, `Returns` — and annotate access mode (`— read`, `— modified (export)`). + +```bash +# Polls compose services until all report healthy or timeout elapses. +# +# Globals: +# COMPOSE_FILE — read, path to stacks/compose.yaml +# Arguments: +# $1 — timeout in seconds (default: 60) +# Outputs: +# Writes progress/warnings to stderr via log() +# Returns: +# 0 when healthy or on timeout (non-fatal), 1 if services failed +``` + +--- + ## Editor ### VS Code Settings