diff --git a/.devcontainer/.dockerignore b/.devcontainer/.dockerignore new file mode 100644 index 0000000..8d1de1a --- /dev/null +++ b/.devcontainer/.dockerignore @@ -0,0 +1,10 @@ +# The build context is `.devcontainer/` and the Dockerfile has no COPY +# instruction, so the correct context is empty. +# +# Not just tidiness: `.env` is gitignored and holds POSTGRES_PASSWORD and the +# MinIO root credentials. An un-ignored context still reaches the Docker daemon +# and can land in the build cache. +# +# Context-root form, not a `Dockerfile.dockerignore` sibling — with Features +# present the devcontainer CLI builds from a temp dir, so the sibling is missed. +* diff --git a/.devcontainer/.env.example b/.devcontainer/.env.example index 6124fdd..a771442 100644 --- a/.devcontainer/.env.example +++ b/.devcontainer/.env.example @@ -1,22 +1,16 @@ # ============================================================ # Dev Container Environment Template # ============================================================ -# This file is the source of truth for local-dev environment vars. -# On first container build, `initializeCommand` copies it to -# `.devcontainer/.env` (gitignored) on the host. From there: +# Source of truth for local-dev environment vars. On first build +# `initializeCommand` copies this to `.devcontainer/.env` (gitignored), which +# feeds both Docker Compose (via an explicit --env-file) and the dev container +# itself (via runArgs --env-file). See CONFIGURATION.md → "Environment +# Variables"; `task env:check` reports drift between the two files. # -# * Docker Compose auto-discovers it (sibling to compose.yaml) -# and interpolates ${VAR:-default} references. -# * `runArgs --env-file` loads it into the dev container itself -# so shells, runtimes, and `task` runs see the same values. -# -# Drift checks: `task env:check` compares this file against your -# local `.env` and flags missing keys. -# -# Convention (three states): -# 1. Filled defaults — `VAR=value` safe demo values; override only if needed. -# 2. Required (empty) — `VAR=` must be filled in; container warns at startup. -# 3. Optional overrides — `# VAR=value` uncomment to enable. +# Three states, by convention: +# 1. Filled default — `VAR=value` safe demo value; override if needed. +# 2. Required (empty) — `VAR=` must be filled; startup warns. +# 3. Optional override — `# VAR=value` uncomment to enable. # ============================================================ diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile new file mode 100644 index 0000000..e278994 --- /dev/null +++ b/.devcontainer/Dockerfile @@ -0,0 +1,65 @@ +# syntax=docker/dockerfile:1 +# Musher dev container base image. +# +# bun, uv and task are installed here rather than as Features: their Features +# resolve release assets through an unauthenticated api.github.com call, which +# the shared egress IPs used by Codespaces and CI rate-limit, failing the whole +# build. mise is here because it was otherwise the one unpinned tool. +# +# Full rationale: CONFIGURATION.md → "Runtimes & Tools" +# Enforced by: repo toolchain check (TC-01..TC-03) +# +# Pinned to the LTS tag: floating `:ubuntu` rolls to interim releases that +# docker-in-docker does not support. +FROM mcr.microsoft.com/devcontainers/base:ubuntu-24.04 + +# Without pipefail a truncated `curl | bash` reports bash's exit status, giving +# a successful build and an image with no tool in it. +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +# The single source of truth for what this image bakes. verify-toolchain.sh +# reads them back to assert the built container matches. +ARG BUN_VERSION=1.3.14 +ARG UV_VERSION=0.11.28 +ARG TASK_VERSION=3.52.0 +ARG MISE_VERSION=v2026.8.6 + +RUN set -eux; \ + command -v unzip >/dev/null 2>&1 || { echo "unzip missing from the base image; bun's installer requires it" >&2; exit 1; }; \ + \ + # --- bun -> /usr/local/bin/{bun,bunx} + # HOME is a scratch dir because the installer appends PATH exports to + # ~/.bashrc and ~/.zshrc whenever they are writable. + mkdir -p /tmp/bun-home; \ + HOME=/tmp/bun-home BUN_INSTALL=/usr/local bash -c \ + "curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://bun.sh/install \ + | bash -s bun-v${BUN_VERSION}"; \ + rm -rf /tmp/bun-home; \ + # Never write to /usr/local/bin/bunx: it is a symlink to bun, so doing so + # overwrites the binary and every `bun` call then recurses into itself. + \ + # --- uv -> /usr/local/bin/{uv,uvx} + curl --retry 5 --retry-all-errors --retry-delay 3 -LsSf \ + "https://astral.sh/uv/${UV_VERSION}/install.sh" \ + | env UV_INSTALL_DIR=/usr/local/bin INSTALLER_NO_MODIFY_PATH=1 sh; \ + \ + # --- task -> /usr/local/bin/task + curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://taskfile.dev/install.sh \ + | sh -s -- -b /usr/local/bin "v${TASK_VERSION}"; \ + \ + # --- mise -> /usr/local/bin/mise + # System-wide; base-setup.sh still runs `mise reshim` to build the per-user + # shim dir that remoteEnv's PATH expects. + MISE_VERSION="${MISE_VERSION}" MISE_INSTALL_PATH=/usr/local/bin/mise \ + bash -c 'curl --retry 5 --retry-all-errors --retry-delay 3 -fsSL https://mise.run | sh'; \ + \ + # Presence-only: the runtime version check lives in verify-toolchain.sh. + test -x /usr/local/bin/bun; \ + test -x /usr/local/bin/bunx; \ + test -x /usr/local/bin/uv; \ + test -x /usr/local/bin/uvx; \ + test -x /usr/local/bin/task; \ + test -x /usr/local/bin/mise + +# No USER: Features install after this stage and updateRemoteUserUID expects +# root at build time. devcontainer.json's remoteUser owns runtime identity. diff --git a/.devcontainer/compose.yaml b/.devcontainer/compose.yaml deleted file mode 100644 index af6c520..0000000 --- a/.devcontainer/compose.yaml +++ /dev/null @@ -1,27 +0,0 @@ -# Musher Dev Container — Docker Compose orchestrator. -# Each stack lives in its own folder under stacks/ (its compose.yaml plus any -# config it needs); add a new folder there and include its compose.yaml below. -# Services run inside Docker-in-Docker via startup.sh. -# -# Port Allocation: -# 15432 PostgreSQL 15440 MinIO API (obs.) -# 15433 Redis 15441 MinIO Console (obs.) -# 15434 MinIO API 15442 Tempo -# 15435 MinIO Console 15443 Loki -# 15436 OCI Registry 15444 VictoriaMetrics -# 15460 Azimutt 15445 OTel HTTP -# 15446 OTel gRPC -# 15447 Grafana -# 15448 Pyroscope - -include: - - stacks/postgres/compose.yaml - - stacks/redis/compose.yaml - - stacks/minio/compose.yaml - - stacks/registry/compose.yaml - - stacks/azimutt/compose.yaml - - stacks/observability/compose.yaml - -networks: - default: - name: musher-dev diff --git a/.devcontainer/devcontainer-lock.json b/.devcontainer/devcontainer-lock.json index 21168aa..282fcb1 100644 --- a/.devcontainer/devcontainer-lock.json +++ b/.devcontainer/devcontainer-lock.json @@ -1,25 +1,10 @@ { "features": { - "ghcr.io/devcontainers-extra/features/bun:1": { - "version": "1.1.0", - "resolved": "ghcr.io/devcontainers-extra/features/bun@sha256:0624284ecaead9dd4c6654616a7f939cfa4ebcbc60593700a74e35b1767befa5", - "integrity": "sha256:0624284ecaead9dd4c6654616a7f939cfa4ebcbc60593700a74e35b1767befa5" - }, "ghcr.io/devcontainers-extra/features/deno:1": { "version": "1.0.4", "resolved": "ghcr.io/devcontainers-extra/features/deno@sha256:7013bf7726828a33579604fe1aa5c36a253b578d5991e55800b418b56fd2cca5", "integrity": "sha256:7013bf7726828a33579604fe1aa5c36a253b578d5991e55800b418b56fd2cca5" }, - "ghcr.io/devcontainers-extra/features/go-task:1": { - "version": "1.0.6", - "resolved": "ghcr.io/devcontainers-extra/features/go-task@sha256:4d1db153919976cadd3209ca05d655a761a01707767716994dad677b4538dc1b", - "integrity": "sha256:4d1db153919976cadd3209ca05d655a761a01707767716994dad677b4538dc1b" - }, - "ghcr.io/devcontainers-extra/features/uv:1": { - "version": "1.0.2", - "resolved": "ghcr.io/devcontainers-extra/features/uv@sha256:1ac5b9f17a9e9e745933d0ac2ecf758e06ed3da4423cd22c94cce4d482fd2dd8", - "integrity": "sha256:1ac5b9f17a9e9e745933d0ac2ecf758e06ed3da4423cd22c94cce4d482fd2dd8" - }, "ghcr.io/devcontainers/features/common-utils:2": { "version": "2.5.9", "resolved": "ghcr.io/devcontainers/features/common-utils@sha256:cb0c4d3c276f157eed17935747e364178d75fee17f55c4e129966f64633deb3a", diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index f54e4f3..6dc2c4d 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -2,17 +2,17 @@ // Uncomment optional blocks as needed. Comment out what you don't use. { "name": "Musher Dev", - // Pin to the LTS, not the floating :ubuntu tag — it rolls to interim releases - // (e.g. 25.10) that upstream Features like docker-in-docker don't support. - "image": "mcr.microsoft.com/devcontainers/base:ubuntu-24.04", + // Dockerfile bakes the tools whose Features cannot be relied on; the + // Features below layer on top of it. Context is emptied by .dockerignore. + "build": { + "dockerfile": "Dockerfile", + "context": "." + }, "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/${localWorkspaceFolderBasename},type=bind,consistency=cached", "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}", - // Developer tools are pinned here as Features (baked into the image). Tools - // with no Feature — the Codex and Lefthook CLIs — live in - // .devcontainer/mise.toml; Claude Code self-updates via its native installer. - // See CONFIGURATION.md → "Runtimes & Tools". + // Which tier a tool belongs in: CONFIGURATION.md → "Runtimes & Tools". "features": { // --- Platform --- "ghcr.io/devcontainers/features/common-utils:2": { @@ -35,14 +35,12 @@ "installGradle": false, "installMaven": false }, + // deno is safe as a Feature; bun, uv and task are not — do not re-add + // them. They are baked by the Dockerfile (TC-01 fails the build if they + // reappear here). Why: CONFIGURATION.md → "Runtimes & Tools". "ghcr.io/devcontainers-extra/features/deno:1": { "version": "2.9.2" }, - "ghcr.io/devcontainers-extra/features/bun:1": { "version": "1.3.14" }, - - // --- Package managers --- - "ghcr.io/devcontainers-extra/features/uv:1": { "version": "0.11.28" }, - // --- Task runner & linting --- - "ghcr.io/devcontainers-extra/features/go-task:1": { "version": "3.52.0" }, + // --- Linting --- "ghcr.io/lukewiwa/features/shellcheck:0": { "version": "v0.11.0" }, // --- Database tooling --- @@ -98,7 +96,8 @@ }, "remoteEnv": { - // mise shims (codex, lefthook) + ~/.local/bin (mise, Claude Code) on PATH. + // mise shims (codex, lefthook) + ~/.local/bin (Claude Code). The baked + // tools live in /usr/local/bin and are already on PATH. "PATH": "/home/vscode/.local/share/mise/shims:/home/vscode/.local/bin:${containerEnv:PATH}" }, diff --git a/.devcontainer/mise.toml b/.devcontainer/mise.toml index 18732a1..be59813 100644 --- a/.devcontainer/mise.toml +++ b/.devcontainer/mise.toml @@ -1,20 +1,13 @@ -# Developer tools that have no devcontainer Feature. +# Developer CLIs that have no devcontainer Feature and are only needed at +# runtime. Tools with a Feature are pinned in devcontainer.json; bun, uv, task +# and mise are baked by the Dockerfile. +# Tier rules: CONFIGURATION.md → "Runtimes & Tools". # -# Everything with a Feature is pinned in devcontainer.json (baked into the -# image); the CLIs below are npm-distributed with no Feature, so mise installs -# and version-pins them instead. Claude Code is the one other exception — it -# self-updates via its native installer (scripts/lib/base-setup.sh). +# This file is the single source of these versions: post-create runs +# `mise install` against it, and CI resolves the same pins via jdx/mise-action. +# Edit a version here, then `task tools:install`. # -# Change a version here, then run `task tools:install`. Devcontainer post-create -# runs `mise install` against this file. Docs: https://mise.jdx.dev -# See CONFIGURATION.md → "Runtimes & Tools". - -# This file is the SINGLE source of tool versions. CI resolves the same pins -# from here via jdx/mise-action (MISE_GLOBAL_CONFIG_FILE), so a version is -# never stated twice and local and CI cannot drift. -# -# Backends are fully qualified (npm:, pipx:, aqua:) rather than short names so -# resolution does not depend on mise's registry. +# Backends are fully qualified so resolution does not depend on mise's registry. [tools] # --- AI + git hooks ------------------------------------------------------ diff --git a/.devcontainer/scripts/initialize.sh b/.devcontainer/scripts/initialize.sh index aad2d88..e0af018 100644 --- a/.devcontainer/scripts/initialize.sh +++ b/.devcontainer/scripts/initialize.sh @@ -1,25 +1,18 @@ #!/usr/bin/env bash # initialize.sh — Host-side bootstrap for the dev container. # -# Runs on the host (via devcontainer.json `initializeCommand`) BEFORE -# `docker run` is invoked. Because `runArgs --env-file` is evaluated at -# `docker run` time, the .env file must exist on the host before the -# container starts — that's why this work lives here, not in -# post-create.sh. +# Runs on the host via `initializeCommand`, before `docker run` — it has to, +# because `runArgs --env-file` is evaluated at `docker run` time, so .env must +# already exist. post-create.sh would be too late. # -# Responsibilities: -# * Create .devcontainer/.env from .env.example on first clone. -# * Touch an empty .env if no example exists, so --env-file never hard-fails. -# * Strip CRLF from .env (Windows/WSL safety — docker --env-file -# rejects files with CRLF line endings). -# -# Why this CRLF guard survives while the postCreateCommand one did not: -# .gitattributes (`* text=auto eol=lf`) normalizes every file Git checks -# out, which covers scripts/ and made the old `fix-crlf` step redundant. -# It cannot cover .env — that file is gitignored, generated locally, and -# hand-edited, so a Windows editor can reintroduce CR at any time. +# The CRLF guard here is not redundant with .gitattributes. That normalizes +# every file Git checks out, which is why the old postCreate fix-crlf step +# could go; it cannot reach .env, which is gitignored, generated locally and +# hand-edited, so a Windows editor can reintroduce CR at any time. Docker +# rejects an --env-file containing CRLF. # # Idempotent: safe to run on every container start. + set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" diff --git a/.devcontainer/scripts/lib/base-setup.sh b/.devcontainer/scripts/lib/base-setup.sh index 9951d15..c7b915c 100644 --- a/.devcontainer/scripts/lib/base-setup.sh +++ b/.devcontainer/scripts/lib/base-setup.sh @@ -70,6 +70,9 @@ base_fix_nvm_permissions() { # --- mise (pins the CLIs that have no devcontainer Feature) --- +# Fallback only: the image bakes mise at /usr/local/bin/mise, which +# `command -v` finds first. This path covers the base_install_mise fallback, +# which installs per-user. readonly _MISE_BIN="${_HOME}/.local/bin/mise" readonly _MISE_SHIMS="${_HOME}/.local/share/mise/shims" @@ -83,7 +86,13 @@ base_setup_path() { export PATH="${_MISE_SHIMS}:${_HOME}/.local/bin:${PATH}" } -# Installs mise via the official installer if not already present. +# Installs mise if it is not already present. +# +# The dev container image bakes a pinned mise at /usr/local/bin/mise +# (ARG MISE_VERSION in .devcontainer/Dockerfile), so this normally short-circuits. +# The installer below is the fallback for a consuming repo that strips the +# Dockerfile, and is deliberately unpinned because in that case there is no ARG +# to read the pin from. # # Outputs: # Writes progress to stderr via log() @@ -94,7 +103,7 @@ base_install_mise() { log "mise already installed, skipping" return 0 fi - log "Installing mise (https://mise.run)..." + log "mise not baked into the image; falling back to https://mise.run..." retry 3 5 bash -c 'curl -fsSL https://mise.run | sh' } diff --git a/.devcontainer/scripts/lib/common.sh b/.devcontainer/scripts/lib/common.sh index b11149b..3b63b71 100644 --- a/.devcontainer/scripts/lib/common.sh +++ b/.devcontainer/scripts/lib/common.sh @@ -8,8 +8,6 @@ # and tool verification functions used by all setup scripts. set -euo pipefail -# --- Logging --- - # Logs a timestamped message to stderr. # # Arguments: @@ -105,8 +103,6 @@ setup_config_dirs() { done } -# --- NVM helpers --- - # Fixes NVM directory ownership to the current user. # # Globals: @@ -121,8 +117,6 @@ fix_nvm_permissions() { fi } -# --- NPM install helper --- - # Installs an npm package globally with retry logic. # # Arguments: diff --git a/.devcontainer/scripts/lib/motd.sh b/.devcontainer/scripts/lib/motd.sh index 1118942..dd5d2ad 100644 --- a/.devcontainer/scripts/lib/motd.sh +++ b/.devcontainer/scripts/lib/motd.sh @@ -4,7 +4,7 @@ # This is a library file meant to be sourced, not executed directly. # Requires common.sh (has_cmd, log) to be sourced first. # -# Usage: source "path/to/motd.sh"; show_motd "/path/to/compose.yaml" +# Usage: source "path/to/motd.sh"; show_motd "/path/to/stacks/compose.yaml" "/path/to/.devcontainer" if [[ -z "${_MOTD_SH_LOADED:-}" ]]; then readonly _MOTD_SH_LOADED=1 @@ -59,17 +59,14 @@ _motd_runtimes() { echo " ${_BOLD}Runtimes${_RESET}" echo " ${_DIM}${sep}${_RESET}" - # Row 1: node + python _motd_runtime_entry node "node" "node -v" _motd_runtime_entry python3 "python" "python3 -c 'import platform; print(platform.python_version())'" echo "" - # Row 2: go + java _motd_runtime_entry go "go" "go version | grep -oP '\\d+\\.\\d+\\.\\d+'" _motd_runtime_entry java "java" "java -version 2>&1 | head -1 | grep -oP '\\d+[\\d.]+'" echo "" - # Row 3: deno + bun _motd_runtime_entry deno "deno" "deno -v | head -1 | awk '{print \$2}'" _motd_runtime_entry bun "bun" "bun -v" echo "" @@ -77,12 +74,18 @@ _motd_runtimes() { _motd_services() { local compose_file="$1" + local env_file="${2:-}" if [[ -z "$compose_file" ]] || [[ ! -f "$compose_file" ]] || ! has_cmd docker; then return 0 fi + # Name the env file explicitly for the same reason startup.sh does: the + # compose file lives under stacks/ and is no longer a sibling of .env. + local -a env_args=() + [[ -n "$env_file" && -f "$env_file" ]] && env_args=(--env-file "$env_file") + local output - output="$(docker compose -f "$compose_file" ps --format json 2>/dev/null || true)" + output="$(docker compose "${env_args[@]}" -f "$compose_file" ps --format json 2>/dev/null || true)" if [[ -z "$output" ]]; then return 0 fi @@ -101,18 +104,15 @@ _motd_services() { state="$(echo "$line" | grep -oP '"State"\s*:\s*"\K[^"]+' | head -1)" health="$(echo "$line" | grep -oP '"Health"\s*:\s*"\K[^"]+' | head -1)" - # Extract published host port ports="$(echo "$line" | grep -oP '"PublishedPort"\s*:\s*\K\d+' | head -1)" [[ -z "$name" ]] && continue - # Build display name local display_name="$name" if [[ -n "$ports" ]] && [[ "$ports" != "0" ]]; then display_name="${name} (${ports})" fi - # Determine status label and color local status_label color if [[ -n "$health" ]] && [[ "$health" != "" ]]; then status_label="$health" @@ -136,7 +136,8 @@ _motd_quickref() { echo "" echo " ${_BOLD}Quick Reference${_RESET}" echo " ${_DIM}${sep}${_RESET}" - echo " docker compose -f .devcontainer/compose.yaml up -d / down / logs -f" + echo " docker compose --env-file .devcontainer/.env \\" + echo " -f .devcontainer/stacks/compose.yaml up -d / down / logs -f" echo " git status / log / diff" echo " task Task runner" echo " claude Claude Code AI" @@ -201,8 +202,9 @@ _motd_tips() { # Renders the full MOTD to stdout. # # Arguments: -# $1 — path to compose.yaml (may be empty to skip services) -# $2 — path to .devcontainer/ directory (may be empty to skip env warnings) +# $1 — path to stacks/compose.yaml (may be empty to skip services) +# $2 — path to .devcontainer/ directory (may be empty to skip env warnings); +# also supplies the --env-file the compose file needs # Outputs: # MOTD text to stdout show_motd() { @@ -216,7 +218,7 @@ show_motd() { echo "" _motd_header _motd_runtimes - _motd_services "$compose_file" + _motd_services "$compose_file" "${devcontainer_dir:+${devcontainer_dir}/.env}" _motd_env_warnings "$devcontainer_dir" _motd_quickref _motd_tips diff --git a/.devcontainer/scripts/startup.sh b/.devcontainer/scripts/startup.sh index b56613a..8f57a06 100644 --- a/.devcontainer/scripts/startup.sh +++ b/.devcontainer/scripts/startup.sh @@ -2,7 +2,7 @@ # startup.sh — Starts compose services and waits for health checks. # # Executed on every container start to bring up supporting services -# (databases, caches, observability) defined in compose.yaml. +# (databases, caches, observability) defined in stacks/compose.yaml. # # Usage: Called automatically by devcontainer.json postStartCommand. set -euo pipefail @@ -11,8 +11,14 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" readonly SCRIPT_DIR DEVCONTAINER_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" readonly DEVCONTAINER_DIR -COMPOSE_FILE="${DEVCONTAINER_DIR}/compose.yaml" +COMPOSE_FILE="${DEVCONTAINER_DIR}/stacks/compose.yaml" readonly COMPOSE_FILE +# The compose file lives under stacks/, so it is no longer a sibling of .env and +# Compose's positional auto-discovery cannot find it. Name the env file +# explicitly on every invocation -- otherwise ${VAR:-default} silently wins and +# COMPOSE_PROFILES reads as empty, disabling every opt-in stack without an error. +ENV_FILE="${DEVCONTAINER_DIR}/.env" +readonly ENV_FILE # shellcheck source=lib/common.sh source "${SCRIPT_DIR}/lib/common.sh" @@ -35,10 +41,11 @@ trap 'on_error ${LINENO} "${BASH_COMMAND}"' ERR # Polls compose services until all report healthy or timeout elapses. # +# Globals: +# COMPOSE_FILE — read, path to stacks/compose.yaml +# ENV_FILE — read, path to .env (passed to every compose invocation) # Arguments: # $1 — timeout in seconds (default: 60) -# Globals: -# COMPOSE_FILE — read, path to compose.yaml # Outputs: # Writes progress/warnings to stderr via log() # Returns: @@ -49,9 +56,8 @@ wait_for_healthy() { local elapsed=0 while ((elapsed < timeout)); do local output - output="$(docker compose -f "${COMPOSE_FILE}" ps --format json 2>/dev/null || true)" + output="$(docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" ps --format json 2>/dev/null || true)" - # Detect failed services (exited, dead, or unhealthy) local failed="" failed="$(echo "$output" | grep -E '"(exited|dead|unhealthy)"' || true)" if [[ -n "$failed" ]]; then @@ -62,12 +68,11 @@ wait_for_healthy() { local state state="$(echo "$line" | grep -o '"State":"[^"]*"' | head -1 | cut -d'"' -f4)" log " ${name}: ${state}" - docker compose -f "${COMPOSE_FILE}" logs --tail=10 "$name" 2>/dev/null || true + docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" logs --tail=10 "$name" 2>/dev/null || true done return 1 fi - # Count services still starting local starting starting="$(echo "$output" | grep -c '"starting"' || true)" if [[ "$starting" -eq 0 ]]; then @@ -94,13 +99,22 @@ main() { fi if [[ ! -f "${COMPOSE_FILE}" ]]; then - log "No compose.yaml found, skipping service startup" + log "No stacks/compose.yaml found, skipping service startup" + show_motd "" "${DEVCONTAINER_DIR}" + return 0 + fi + + # --env-file errors out on a missing path. initialize.sh creates .env on the + # host before the container starts, so this only trips if that hook was + # skipped -- say a bare `docker run` outside the devcontainer tooling. + if [[ ! -f "${ENV_FILE}" ]]; then + log "No .env found at ${ENV_FILE}; run scripts/initialize.sh first" show_motd "" "${DEVCONTAINER_DIR}" return 0 fi log "Starting compose services..." - docker compose -f "${COMPOSE_FILE}" up -d --remove-orphans + docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" up -d --remove-orphans wait_for_healthy 60 diff --git a/.devcontainer/scripts/verify-toolchain.sh b/.devcontainer/scripts/verify-toolchain.sh new file mode 100644 index 0000000..0839537 --- /dev/null +++ b/.devcontainer/scripts/verify-toolchain.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +# verify-toolchain.sh — Asserts the image-baked tools report their pinned versions. +# +# The Dockerfile's own assertions are presence-only (`test -x`): executing a +# binary in the same layer that installed it is a known BuildKit hazard, and the +# pinned download URLs already guarantee the version -- a wrong one 404s. This +# script is the runtime half of that split, run against the built container. +# +# The expected versions are read back out of .devcontainer/Dockerfile so the +# ARGs stay the single source of truth. `repo toolchain check` separately +# asserts those ARGs are concrete pins rather than floating tags. +# +# Usage: bash .devcontainer/scripts/verify-toolchain.sh +# (CI runs it as the devcontainers/ci `runCmd`.) +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly SCRIPT_DIR +DOCKERFILE="${SCRIPT_DIR}/../Dockerfile" +readonly DOCKERFILE + +# Reads a pinned ARG default out of the Dockerfile. +# +# Arguments: +# $1 — the ARG name, e.g. BUN_VERSION +# Outputs: +# The pinned value on stdout +# Returns: +# 0 on success, 1 if the ARG is absent or has no default +arg_pin() { + local name="${1}" + local value + value="$(sed -n "s/^ARG ${name}=\\(.*\\)$/\\1/p" "${DOCKERFILE}" | head -1)" + if [[ -z "${value}" ]]; then + echo "ERROR: no 'ARG ${name}=' in ${DOCKERFILE}" >&2 + return 1 + fi + printf '%s\n' "${value}" +} + +# Asserts ` --version` reports the expected version. +# +# Arguments: +# $1 — the binary name, which is also the human-readable tool name +# $2 — expected version (no leading 'v') +# Outputs: +# Writes a pass/fail line to stdout +# Returns: +# 0 on match, 1 on mismatch or missing binary +assert_version() { + local tool="${1}" expected="${2}" + local actual + if ! command -v "${tool}" >/dev/null 2>&1; then + echo " FAIL ${tool}: not on PATH" + return 1 + fi + # Tools disagree on output shape (`task` prints "Task version: v3.52.0", uv + # prints "uv 0.11.28"), so match the expected string anywhere in the first + # line rather than parsing four different formats. + actual="$("${tool}" --version 2>&1 | head -1)" + if [[ "${actual}" != *"${expected}"* ]]; then + echo " FAIL ${tool}: expected ${expected}, got '${actual}'" + return 1 + fi + echo " ok ${tool} ${expected}" +} + +# Entry point: checks every image-baked tool against its Dockerfile pin. +# +# Outputs: +# Writes per-tool results to stdout +# Returns: +# 0 if all tools match, 1 otherwise +main() { + echo "Verifying image-baked toolchain against ${DOCKERFILE}..." + + local bun_v uv_v task_v mise_v + bun_v="$(arg_pin BUN_VERSION)" + uv_v="$(arg_pin UV_VERSION)" + task_v="$(arg_pin TASK_VERSION)" + # MISE_VERSION is pinned with a leading 'v'; `mise --version` prints without. + mise_v="$(arg_pin MISE_VERSION)" + mise_v="${mise_v#v}" + + local failed=0 + assert_version bun "${bun_v}" || failed=1 + assert_version uv "${uv_v}" || failed=1 + assert_version task "${task_v}" || failed=1 + assert_version mise "${mise_v}" || failed=1 + + # bunx is a symlink to bun; assert it survived rather than being overwritten. + if [[ ! -x /usr/local/bin/bunx ]]; then + echo " FAIL bunx: /usr/local/bin/bunx missing or not executable" + failed=1 + else + echo " ok bunx" + fi + + if ((failed)); then + echo "Toolchain verification FAILED" >&2 + return 1 + fi + echo "Toolchain verification passed" +} + +main "$@" diff --git a/.devcontainer/stacks/compose.yaml b/.devcontainer/stacks/compose.yaml new file mode 100644 index 0000000..e90a0b6 --- /dev/null +++ b/.devcontainer/stacks/compose.yaml @@ -0,0 +1,26 @@ +# Musher dev container — Docker Compose orchestrator. +# +# One folder per stack, each holding its own compose.yaml plus any config it +# needs. Services run inside Docker-in-Docker, started by scripts/startup.sh. +# This is not the dev container itself — that is ../Dockerfile + ../devcontainer.json. +# +# Every caller must pass `--env-file ../.env`: this file is not a sibling of +# .env, so Compose's positional discovery does not reach it and every value +# would silently fall back to its default. `name` is explicit for the same +# reason — Compose would otherwise derive the project name from this folder. +# +# Ports: CONFIGURATION.md → "Port Allocation" (parity enforced by `repo ports check`). + +name: musher-dev + +include: + - postgres/compose.yaml + - redis/compose.yaml + - minio/compose.yaml + - registry/compose.yaml + - azimutt/compose.yaml + - observability/compose.yaml + +networks: + default: + name: musher-dev diff --git a/.devcontainer/stacks/postgres/compose.yaml b/.devcontainer/stacks/postgres/compose.yaml index c95f759..39111c7 100644 --- a/.devcontainer/stacks/postgres/compose.yaml +++ b/.devcontainer/stacks/postgres/compose.yaml @@ -1,5 +1,3 @@ -# PostgreSQL with pgvector extension. - services: postgres: image: pgvector/pgvector:0.8.5-pg17 diff --git a/.github/dependabot.yml b/.github/dependabot.yml index c224137..c55311f 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -21,11 +21,16 @@ updates: commit-message: prefix: "ci" - # Compose service images. Each stack lives in .devcontainer/stacks// with - # its own standard-named compose.yaml (pulled in via `include:`), so the - # .devcontainer root and every stack folder are scanned. If Dependabot does not - # surface PRs for a stack, verify detection under - # Insights -> Dependency graph -> Dependabot and adjust these directories. + # Docker images. /.devcontainer covers the Dockerfile's `FROM` base image; + # /.devcontainer/stacks/** covers each stack's own standard-named compose.yaml + # (pulled into stacks/compose.yaml via `include:`). The orchestrator itself + # declares no images. If Dependabot does not surface PRs for a stack, verify + # detection under Insights -> Dependency graph -> Dependabot and adjust these + # directories. + # + # Note: this does NOT cover the tool pins in the Dockerfile's ARGs (bun, uv, + # task, mise) -- Dependabot has no ecosystem for those. Bump them by hand; + # `repo toolchain check` keeps them exact and in step with CI. - package-ecosystem: "docker" directories: - "/.devcontainer" diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml index dff9ee6..37b3694 100644 --- a/.github/workflows/validate.yaml +++ b/.github/workflows/validate.yaml @@ -27,12 +27,25 @@ jobs: - name: Validate compose config run: | cp .devcontainer/.env.example .devcontainer/.env - docker compose -f .devcontainer/compose.yaml config --quiet + # --env-file is mandatory, not decorative: the orchestrator lives under + # stacks/ and is no longer a sibling of .env, so Compose's positional + # auto-discovery would silently fall back to ${VAR:-default}. + docker compose --env-file .devcontainer/.env \ + -f .devcontainer/stacks/compose.yaml config --quiet + - name: Validate compose config with every profile enabled + run: | + # `config` respects profiles, so the run above only covers the always-on + # stacks. Enable all of them so an opt-in stack cannot rot unnoticed -- + # and so a broken --env-file surfaces here rather than at someone's + # first `Reopen in Container`. + COMPOSE_PROFILES=redis,minio,registry,azimutt,observability \ + docker compose --env-file .devcontainer/.env \ + -f .devcontainer/stacks/compose.yaml config --quiet - name: Reject floating image tags run: | # Fail on :latest / :main tags that are not digest-pinned (@sha256:...). if grep -rnE 'image:.*(:latest|:main)([[:space:]]|$)' \ - .devcontainer/compose.yaml .devcontainer/stacks/; then + .devcontainer/stacks/; then echo "::error::Floating image tag found. Pin to a version or digest." >&2 exit 1 fi @@ -107,4 +120,7 @@ jobs: - name: Build devcontainer uses: devcontainers/ci@v0.3 with: - runCmd: echo "devcontainer smoke test passed" + # Asserts the Dockerfile-baked tools report their pinned versions -- + # the runtime half of the split the Dockerfile's presence-only + # `test -x` assertions leave open. + runCmd: bash .devcontainer/scripts/verify-toolchain.sh diff --git a/.repo/README.md b/.repo/README.md index cdc83ef..8567fff 100644 --- a/.repo/README.md +++ b/.repo/README.md @@ -43,6 +43,16 @@ runs `uv tool install ./.repo`). To reinstall after editing it: | `ports` | `PORT-01`..`PORT-05` | The port table, `forwardPorts`/`portsAttributes`, and compose published ports all agree and stay in the reserved range | | `hooks` | `HOOK-01`..`HOOK-04` | Every lefthook job has a CI counterpart and vice versa, or a recorded reason why not | | `rulesets` | `RS-01`..`RS-04` | Committed branch rulesets stay valid and in step with the CI jobs they require | +| `toolchain` | `TC-01`..`TC-03` | The tools the image bakes stay out of the Features block, keep exact pins, and stay in step with CI | +| `comments` | `CMT-01`..`CMT-03` | Comment blocks stay short, the allowlist stays honest, and every `docs:` pointer still resolves | + +### Why `toolchain` exists + +`TC-01` is the policy least likely to be guessed from the code it guards: bun, uv and +task are installed by [`.devcontainer/Dockerfile`](../.devcontainer/Dockerfile) rather +than by their Features, because those Features fail on rate-limited build hosts. +Re-adding one looks like a harmless simplification, which is exactly why it is a check. +Full account: [`CONFIGURATION.md`](../CONFIGURATION.md) → "Runtimes & Tools". ### The one-way-check tables diff --git a/.repo/governance/policies/__init__.py b/.repo/governance/policies/__init__.py index 4547fdc..0d4892a 100644 --- a/.repo/governance/policies/__init__.py +++ b/.repo/governance/policies/__init__.py @@ -13,7 +13,7 @@ from collections.abc import Callable -from governance.policies import config, hooks, ports, rulesets +from governance.policies import comments, config, hooks, ports, rulesets, toolchain from governance.reporting import Report POLICIES: dict[str, Callable[[], Report]] = { @@ -21,6 +21,8 @@ "ports": ports.run, "hooks": hooks.run, "rulesets": rulesets.run, + "toolchain": toolchain.run, + "comments": comments.run, } __all__ = ["POLICIES"] diff --git a/.repo/governance/policies/comments/__init__.py b/.repo/governance/policies/comments/__init__.py new file mode 100644 index 0000000..7dc1120 --- /dev/null +++ b/.repo/governance/policies/comments/__init__.py @@ -0,0 +1,5 @@ +"""Policy: comments stay short, and the pointers they lean on still resolve.""" + +from governance.policies.comments.check import run + +__all__ = ["run"] diff --git a/.repo/governance/policies/comments/check.py b/.repo/governance/policies/comments/check.py new file mode 100644 index 0000000..9bc34ba --- /dev/null +++ b/.repo/governance/policies/comments/check.py @@ -0,0 +1,131 @@ +"""Detect comment blocks that have outgrown their file, and dead doc pointers. + +The rule this enforces is written in CONFIGURATION.md -> "Comments": say the +non-obvious, say it once, and point at it from everywhere else. CMT-01 caps the +volume; CMT-03 keeps the pointers that make the trade possible from rotting. +""" + +from __future__ import annotations + +import ast +import re +from pathlib import Path + +from governance import repo +from governance.policies.comments import violations as v +from governance.reporting import Report + +#: Longest contiguous run of comment lines a file may carry. The natural block +#: size in this repo is 4-8 lines and nothing legitimate sits between 16 and 22, +#: so the limit lands in a real gap rather than on an arbitrary round number. +MAX_BLOCK = 20 + +#: Files whose long block is the point of the file, and why. Empty by design -- +#: an entry here is a standing exemption, so it has to earn its place. +ALLOWED_LONG_BLOCKS: dict[str, str] = {} + +#: Where comments sit next to code. Markdown is absent on purpose: `#` starts a +#: heading there, not a comment. +SCAN_GLOBS = ( + ".devcontainer/Dockerfile", + ".devcontainer/.dockerignore", + ".devcontainer/.env.example", + ".devcontainer/*.toml", + ".devcontainer/**/*.sh", + ".devcontainer/**/*.yaml", + ".config/*.yml", + ".config/*.yaml", + ".github/**/*.yml", + ".github/**/*.yaml", + "taskfiles/*.yml", + "Taskfile.yml", + ".repo/**/*.py", +) + +#: Lines that start with `#` but are instructions to a tool, not prose. They sit +#: inside blocks and must not inflate them. +_DIRECTIVE = re.compile( + r"^#\s*(!|shellcheck\b|syntax=|type:\s|noqa\b|pylint:|mypy:|fmt:|nosec\b)" +) + +#: `docs="CONFIGURATION.md#anchor"` and the DOCS constants the policies share. +_DOC_REF = re.compile(r'"([A-Za-z0-9_./-]+\.md#[^"]+)"') + +_HEADING = re.compile(r"^#{1,6}\s+(.*)$", re.MULTILINE) + + +def _slug(heading: str) -> str: + """GitHub's heading-anchor slug: lowercase, drop punctuation, spaces to dashes.""" + text = heading.strip().lower() + text = re.sub(r"[^\w\s-]", "", text) + return re.sub(r"\s", "-", text) + + +def _anchors(rel: str) -> set[str]: + return {_slug(m.group(1)) for m in _HEADING.finditer(repo.read_text(rel))} + + +def _blocks(path: Path) -> list[tuple[int, int]]: + """Contiguous comment runs as (start_line, length), directives excluded.""" + found: list[tuple[int, int]] = [] + start = length = 0 + for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + stripped = raw.strip() + is_comment = stripped.startswith("#") and not _DIRECTIVE.match(stripped) + if is_comment: + start = start or number + length += 1 + continue + # A directive interrupts prose without ending the block it sits in, so + # only a genuine line of code closes one. + if stripped.startswith("#"): + continue + if start: + found.append((start, length)) + start = length = 0 + if start: + found.append((start, length)) + return found + + +def _scanned_files() -> list[Path]: + seen: dict[str, Path] = {} + for pattern in SCAN_GLOBS: + for path in repo.glob(pattern): + if path.is_file(): + seen[repo.rel(path)] = path + return [seen[key] for key in sorted(seen)] + + +def run() -> Report: + report = Report(policy="comments") + + used: set[str] = set() + for path in _scanned_files(): + rel = repo.rel(path) + for line, length in _blocks(path): + if length <= MAX_BLOCK: + continue + if rel in ALLOWED_LONG_BLOCKS: + used.add(rel) + continue + report.add(v.block_too_long(rel, line, length, MAX_BLOCK)) + + for rel in sorted(set(ALLOWED_LONG_BLOCKS) - used): + report.add(v.stale_allowance(rel)) + + # Pointers are read statically: the violation factories take arguments, so + # calling them just to inspect `docs` would mean inventing fixture values. + for path in repo.glob(".repo/**/*.py"): + source = repo.rel(path) + for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))): + if not isinstance(node, ast.Constant) or not isinstance(node.value, str): + continue + match = _DOC_REF.fullmatch(f'"{node.value}"') + if not match: + continue + target, _, anchor = node.value.partition("#") + if not repo.exists(target) or anchor not in _anchors(target): + report.add(v.dead_pointer(f"{source}:{node.lineno}", node.value, target)) + + return report diff --git a/.repo/governance/policies/comments/violations.py b/.repo/governance/policies/comments/violations.py new file mode 100644 index 0000000..0bd47cf --- /dev/null +++ b/.repo/governance/policies/comments/violations.py @@ -0,0 +1,57 @@ +"""What can go wrong with comment volume, and why each rule exists.""" + +from __future__ import annotations + +from governance.reporting import Violation + +DOCS = "CONFIGURATION.md#comments" + + +def block_too_long(path: str, line: int, length: int, limit: int) -> Violation: + return Violation( + code="CMT-01", + summary=f"comment block of {length} lines (limit {limit})", + reason=( + "This template is read before it is run, so a header that buries the " + "file is a cost paid by every reader. Blocks here run 4-8 lines; past " + "the limit the content is almost always rationale, which belongs in " + "CONFIGURATION.md with a pointer left behind." + ), + fix=( + "Move the detail into CONFIGURATION.md and leave a one-line summary " + "and a pointer, or register the block in ALLOWED_LONG_BLOCKS with a " + "reason." + ), + where=f"{path}:{line}", + docs=DOCS, + ) + + +def stale_allowance(path: str) -> Violation: + return Violation( + code="CMT-02", + summary=f"ALLOWED_LONG_BLOCKS still excuses {path}, which no longer needs it", + reason=( + "An allowlist that outlives what it excused stops describing the repo " + "and starts hiding the next violation." + ), + fix=f"Drop the {path!r} entry from ALLOWED_LONG_BLOCKS.", + where=".repo/governance/policies/comments/check.py", + docs=DOCS, + ) + + +def dead_pointer(source: str, ref: str, target: str) -> Violation: + return Violation( + code="CMT-03", + summary=f"docs pointer {ref!r} does not resolve", + reason=( + "Comments here reference a single canonical explanation instead of " + "repeating it. That trade is only safe while the references hold: a " + "renamed heading turns the reasoning into something nobody can find, " + "and the next reader writes the explanation out again." + ), + fix=f"Point at a heading that exists in {target}, or restore the heading.", + where=source, + docs=DOCS, + ) diff --git a/.repo/governance/policies/toolchain/__init__.py b/.repo/governance/policies/toolchain/__init__.py new file mode 100644 index 0000000..25cc327 --- /dev/null +++ b/.repo/governance/policies/toolchain/__init__.py @@ -0,0 +1,5 @@ +"""Policy: the image-baked toolchain is pinned, and stays out of Features.""" + +from governance.policies.toolchain.check import run + +__all__ = ["run"] diff --git a/.repo/governance/policies/toolchain/check.py b/.repo/governance/policies/toolchain/check.py new file mode 100644 index 0000000..9e7b577 --- /dev/null +++ b/.repo/governance/policies/toolchain/check.py @@ -0,0 +1,86 @@ +"""Detect drift in the tools the container image bakes itself. + +Three tools (bun, uv, task) are installed by .devcontainer/Dockerfile rather +than by a Feature, for a reason that is easy to lose: their Features fail on +rate-limited build hosts. This policy keeps that decision from being quietly +undone, and keeps the pins that replaced them honest. +""" + +from __future__ import annotations + +import re + +from governance import repo +from governance.policies.toolchain import violations as v +from governance.reporting import Report + +DOCKERFILE = ".devcontainer/Dockerfile" +DEVCONTAINER = ".devcontainer/devcontainer.json" +WORKFLOW = ".github/workflows/validate.yaml" + +#: Feature ref (without the version suffix) -> the tool it would install. +#: Only Features whose installers were read and confirmed to call +#: api.github.com belong here; deno, shellcheck and postgresql-client were +#: checked and are clean. See CONFIGURATION.md -> "Runtimes & Tools". +BANNED_FEATURES = { + "ghcr.io/devcontainers-extra/features/bun": "bun", + "ghcr.io/devcontainers-extra/features/uv": "uv", + "ghcr.io/devcontainers-extra/features/go-task": "task", +} + +#: ARG name -> the tool it pins. Every one of these must exist and be exact. +REQUIRED_ARGS = { + "BUN_VERSION": "bun", + "UV_VERSION": "uv", + "TASK_VERSION": "task", + "MISE_VERSION": "mise", +} + +#: Values that are a moving target rather than a pin. +FLOATING = {"latest", "stable", "main", "master", ""} + +_ARG = re.compile(r"^ARG\s+([A-Z0-9_]+)=(.*)$", re.MULTILINE) + + +def _dockerfile_args() -> dict[str, str]: + """The ARG defaults declared by the Dockerfile.""" + return {m.group(1): m.group(2).strip() for m in _ARG.finditer(repo.read_text(DOCKERFILE))} + + +def _ci_task_version() -> str | None: + """The version arduino/setup-task is pinned to in CI, if it is used.""" + workflow = repo.read_yaml(WORKFLOW) or {} + for job in (workflow.get("jobs") or {}).values(): + for step in (job or {}).get("steps", []) or []: + uses = str((step or {}).get("uses", "")) + if uses.startswith("arduino/setup-task@"): + version = ((step or {}).get("with") or {}).get("version") + if version is not None: + return str(version) + return None + + +def run() -> Report: + report = Report(policy="toolchain") + + features = repo.read_jsonc(DEVCONTAINER).get("features", {}) + for ref in features: + # Feature refs carry a version suffix (`...:1`); compare the name only. + name = ref.rsplit(":", 1)[0] + tool = BANNED_FEATURES.get(name) + if tool: + report.add(v.feature_reintroduced(ref, tool)) + + args = _dockerfile_args() + for name, tool in REQUIRED_ARGS.items(): + if name not in args: + report.add(v.missing_arg(name, tool)) + elif args[name].lower().lstrip("v") in FLOATING: + report.add(v.floating_arg(name, args[name])) + + baked_task = args.get("TASK_VERSION") + ci_task = _ci_task_version() + if baked_task and ci_task and baked_task.lstrip("v") != ci_task.lstrip("v"): + report.add(v.task_version_drift(baked_task, ci_task, WORKFLOW)) + + return report diff --git a/.repo/governance/policies/toolchain/violations.py b/.repo/governance/policies/toolchain/violations.py new file mode 100644 index 0000000..16b284d --- /dev/null +++ b/.repo/governance/policies/toolchain/violations.py @@ -0,0 +1,86 @@ +"""What can go wrong with the image-baked toolchain, and why each rule exists.""" + +from __future__ import annotations + +from governance.reporting import Violation + +DOCS = "CONFIGURATION.md#runtimes--tools" + +DOCKERFILE = ".devcontainer/Dockerfile" +DEVCONTAINER = ".devcontainer/devcontainer.json" + +#: The short form; the full diagnosis is in CONFIGURATION.md, which `docs` below +#: points at. +_WHY_NOT_A_FEATURE = ( + "The devcontainers-extra Features for bun, uv and go-task resolve release " + "assets through an unauthenticated api.github.com call. Codespaces and CI " + "share egress IPs, so that call is rate-limited, and one failed Feature " + "fails the entire image build. Pinning the version does not avoid the call." +) + + +def feature_reintroduced(feature: str, tool: str) -> Violation: + return Violation( + code="TC-01", + summary=f"{tool} is declared as a Feature; it must be baked by the Dockerfile", + reason=_WHY_NOT_A_FEATURE, + fix=( + f"Remove {feature!r} from the features block and pin {tool} with an " + f"ARG in {DOCKERFILE} instead." + ), + where=DEVCONTAINER, + docs=DOCS, + ) + + +def missing_arg(name: str, tool: str) -> Violation: + return Violation( + code="TC-02", + summary=f"no pinned 'ARG {name}=' for {tool}", + reason=( + "The ARG defaults are the single source of truth for what the image " + "bakes. scripts/verify-toolchain.sh reads them back to assert the " + "built container matches, so an absent or empty pin turns that CI " + "check into a no-op." + ), + fix=f"Add `ARG {name}=` to {DOCKERFILE}.", + where=DOCKERFILE, + docs=DOCS, + ) + + +def floating_arg(name: str, value: str) -> Violation: + return Violation( + code="TC-02", + summary=f"ARG {name} is set to the floating value {value!r}", + reason=( + "Every other tool in this template is pinned to an exact version. A " + "floating tag makes the image irreproducible and silently changes " + "the toolchain under developers who rebuild on different days." + ), + fix=f"Replace {value!r} with an exact version in {DOCKERFILE}.", + where=DOCKERFILE, + docs=DOCS, + ) + + +def task_version_drift(dockerfile_version: str, ci_version: str, ci_path: str) -> Violation: + return Violation( + code="TC-03", + summary=( + f"task pinned to {dockerfile_version} in the Dockerfile but " + f"{ci_version} in CI" + ), + reason=( + "CI installs Task with arduino/setup-task rather than the dev " + "container image, so the two pins are the same decision recorded " + "twice. When they drift, a Taskfile change can pass locally and " + "fail in CI (or the reverse) for reasons no diff explains." + ), + fix=( + f"Set the same version in {DOCKERFILE} (ARG TASK_VERSION) and " + f"{ci_path} (arduino/setup-task `version:`)." + ), + where=ci_path, + docs=DOCS, + ) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index d6dc093..f149ce9 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -18,9 +18,13 @@ Does it configure a linter, formatter, or the git hooks? Does it provision the container itself? → .devcontainer/ (see the branches below) -Runtime, or any tool that has a devcontainer Feature? +Runtime, or a tool with a Feature whose installer avoids api.github.com? → devcontainer.json → features block (pin the version) +Tool whose Feature is rate-limit-fragile, or that must exist before +Features run (bun, uv, task, mise)? + → .devcontainer/Dockerfile → an ARG pin + CLI with no Feature (npm-distributed, e.g. codex, lefthook)? → .devcontainer/mise.toml @@ -28,7 +32,7 @@ Self-updating CLI (e.g. Claude Code)? → scripts/lib/base-setup.sh (native installer) Infrastructure service (DB, cache, queue, storage)? - → stacks//compose.yaml (new folder, add to compose.yaml includes) + → stacks//compose.yaml (new folder, add to stacks/compose.yaml includes) VS Code editor behavior or extension? → devcontainer.json → customizations.vscode block @@ -55,7 +59,7 @@ the first "yes". | --- | --- | --- | --- | | 1 | Can the tool *only* load from the repo root, with no flag to point elsewhere? | Repo root | `Taskfile.yml`, `.gitattributes`, `.gitignore` | | 2 | Does it configure a linter, formatter, or the git hooks? | `.config/` | `lefthook.yml`, `markdownlint.jsonc`, `yamllint.yaml` | -| 3 | Does it provision the container or its services? | `.devcontainer/` | `devcontainer.json`, `mise.toml`, `stacks/*/` | +| 3 | Does it provision the container or its services? | `.devcontainer/` | `Dockerfile`, `devcontainer.json`, `mise.toml`, `stacks/` | | 4 | Does it enforce repo structure? | `.repo/` | The `repo` CLI and its policies | Why `.config/` is dotted: it is repo infrastructure, and it sits alongside the @@ -79,7 +83,8 @@ See [`.config/README.md`](.config/README.md) for the per-file index, and | Category | Need | Canonical Location | | --- | --- | --- | -| **Runtimes & Tools** | Anything with a Feature (Node, Python, Go, Java, Deno, bun, uv, gh, Task, ShellCheck) | `devcontainer.json` → `features` (pinned) | +| **Runtimes & Tools** | Anything with a usable Feature (Node, Python, Go, Java, Deno, gh, ShellCheck, psql) | `devcontainer.json` → `features` (pinned) | +| | Image-baked tools (bun, uv, Task, mise) | `.devcontainer/Dockerfile` → `ARG` (pinned) | | | CLIs with no Feature (Codex, Lefthook) | `.devcontainer/mise.toml` | | | Self-updating CLIs (Claude Code) | `scripts/lib/base-setup.sh` | | **Tooling** | Git hooks | `.config/lefthook.yml` | @@ -99,7 +104,8 @@ See [`.config/README.md`](.config/README.md) for the per-file index, and | | Service credentials (dev-only) | `.devcontainer/.env` | | | Service profiles/toggles | `.devcontainer/.env` → `COMPOSE_PROFILES` | | | Secrets (API keys, tokens) | Host env forwarded via `remoteEnv` — never committed | -| **Services** | Infrastructure services | `.devcontainer/stacks//compose.yaml` | +| **Services** | Stack orchestrator (`include:` list) | `.devcontainer/stacks/compose.yaml` | +| | Infrastructure services | `.devcontainer/stacks//compose.yaml` | | | Service enable/disable | `.devcontainer/.env` → `COMPOSE_PROFILES` | | | Service tuning/config | `.devcontainer/stacks//` (colocated) | | **Networking** | Port allocation (container-side) | `.devcontainer/stacks//compose.yaml` → `ports:` | @@ -119,13 +125,23 @@ See [`.config/README.md`](.config/README.md) for the per-file index, and | | Codex CLI (pinned) | `.devcontainer/mise.toml` | | | AI CLI config persistence | `devcontainer.json` → `mounts` (named volumes) | | **Security** | Container capabilities | `devcontainer.json` → `capAdd` / `securityOpt` | +| | Docker build context exclusions | `.devcontainer/.dockerignore` | | | Network binding | Compose files → all ports bound to `127.0.0.1` | --- ## Runtimes & Tools -Tools land in one of three places. **Prefer a Feature** — if one exists, pin its version there. +Tools land in one of four places. Ask these in order and stop at the first "yes". + +| # | Question | Home | +| --- | --- | --- | +| 1 | Is there a Feature, **and** does its installer avoid `api.github.com`? | `devcontainer.json` → `features` | +| 2 | Is the Feature rate-limit-fragile, or must the tool exist before Features run? | `.devcontainer/Dockerfile` → an `ARG` pin | +| 3 | No Feature, and only needed at runtime? | `.devcontainer/mise.toml` | +| 4 | Does the tool update itself? | `scripts/lib/base-setup.sh` | + +**Prefer a Feature.** Tier 2 exists because of one specific, verified failure — not as a general escape hatch. ### 1. Tools with a Feature → `devcontainer.json` @@ -142,7 +158,66 @@ Runtimes and any CLI that ships a devcontainer Feature are pinned in the `featur Comment out any tool you don't need (and its matching VS Code extension). Pin an exact version where the Feature supports it; a couple track a major line instead (`java: 17`, `postgresql-client: 16`). -### 2. CLIs with no Feature → `.devcontainer/mise.toml` +Before adding a third-party Feature, read its `install.sh`. If it delegates to +`ghcr.io/devcontainers-extra/features/gh-release` (directly or via nanolayer), it belongs in tier 2 — see below. + +### 2. Image-baked tools → `.devcontainer/Dockerfile` + +`bun`, `uv`, `task` and `mise` are installed by the Dockerfile as pinned `ARG`s rather than by Features: + +```dockerfile +ARG BUN_VERSION=1.3.14 +ARG UV_VERSION=0.11.28 +ARG TASK_VERSION=3.52.0 +ARG MISE_VERSION=v2026.8.6 +``` + +The first three have Features, and those Features are the problem. All of them resolve release assets through +nanolayer's `gh-release` helper, which lists a release's assets by calling `api.github.com` **with no credentials** — +`nanolayer/installers/gh_release/resolvers/asset_resolver.py`: + +```python +response = urllib.request.urlopen( + f"https://api.github.com/repos/{repo}/releases/tags/{tag}" +) # nosec +``` + +There is no `Authorization` header and no `GITHUB_TOKEN` read anywhere in that module. Codespaces build hosts and +GitHub-hosted Actions runners share egress IP pools, so the 60 req/hr anonymous limit is routinely exhausted and the +call 403s. One failed Feature fails the entire image build, after which Codespaces drops you into a bare recovery +container — so the symptom a developer reports is `task: command not found`, not a rate limit. + +**Pinning the version does not help.** The pin only supplies the tag; `_get_release_assets()` still calls the API to +*list* the assets. The build log reads `Using Bun version: bun-v1.3.14` and then 403s, which makes the failure look +like a bad version pin when the pin was fine. + +`mise` is baked for a different reason — it was previously an unpinned `curl | sh` in post-create, the only unpinned +tool in a template that pins everything else. + +**Features that were checked and cleared**, and stay Features: `devcontainers-extra/deno` and `lukewiwa/shellcheck` +build a `releases/download/...` URL directly; `robbert229/postgresql-client` is apt. The rule is about the +installer's behaviour, not the publisher — so read a third-party Feature's `install.sh` before adding it. + +`repo toolchain check` enforces all of this: `TC-01` fails if one of those Features comes back, `TC-02` fails on a +floating pin, and `TC-03` fails if `TASK_VERSION` drifts from CI's `arduino/setup-task` version. + +Version assertions in the Dockerfile are presence-only (`test -x`). The pinned download URLs already guarantee the +version — a wrong one 404s — and executing a binary in the layer that installed it is a known BuildKit hazard. The +runtime assertion lives in `scripts/verify-toolchain.sh`, which CI runs against the built container. + +Two constraints on what can go here: + +- **Features layer *after* this stage**, so the Dockerfile cannot use anything a Feature provides. This is why + `mise install` stays in post-create — four of the six entries in `mise.toml` use the `npm:` and `pipx:` backends, + and Node and Python come from Features. +- **Runtime identity is not the Dockerfile's job.** There is no `USER` instruction; `updateRemoteUserUID` expects + root at build time and `remoteUser` owns identity afterwards. Anything touching the named-volume mount points + (`~/.claude`, `~/.config/gh`, `~/.codex`) belongs in post-create. + +The build context is `.devcontainer/`, emptied by `.dockerignore` — the Dockerfile has no `COPY` instruction, and +`.devcontainer/.env` must never reach the Docker daemon. + +### 3. CLIs with no Feature → `.devcontainer/mise.toml` npm-distributed CLIs like Codex and Lefthook have no Feature, so [mise](https://mise.jdx.dev) pins and installs them. Add a line under `[tools]`: @@ -155,7 +230,7 @@ Add a line under `[tools]`: `mise install` runs in post-create; re-run it (or `task tools:install`) after editing. -### 3. Self-updating CLIs → `scripts/lib/base-setup.sh` +### 4. Self-updating CLIs → `scripts/lib/base-setup.sh` CLIs that manage their own updates (Claude Code) install via their native installer. Add a function and call it from `base_setup()`: @@ -169,6 +244,42 @@ base_install_mytool() { --- +## Comments + +This template is read before it is run, so its comments are part of the interface. Four rules, the first two +enforced by `repo comments check`. + +**Comment the non-obvious.** The code states *what*; a comment earns its line by stating *why*. The test: could +someone who has never seen this code write the comment just by reading the line below it? If so, delete it. + +**Write each rationale once, then reference it.** A decision explained at every call site is a decision that will +disagree with itself within a release. The full account lives here in `CONFIGURATION.md`; code carries a one-line +summary and a pointer. `repo comments check` (`CMT-03`) fails the build if a pointer stops resolving, so +references are safe to rely on. + +**Keep file headers short.** A header says what the file is and the one constraint a reader must not violate. +Depth goes here. Blocks over 20 lines fail `CMT-01` — the natural size in this repo is 4–8. + +**Library functions are the exception.** Every function in `.devcontainer/scripts/lib/` carries a full header, per +the [Google Shell Style Guide](https://google.github.io/styleguide/shellguide.html): *"Any function in a library +must have a function header comment regardless of length or complexity."* Tags go in Google's order — `Globals`, +`Arguments`, `Outputs`, `Returns` — and annotate access mode (`— read`, `— modified (export)`). + +```bash +# Polls compose services until all report healthy or timeout elapses. +# +# Globals: +# COMPOSE_FILE — read, path to stacks/compose.yaml +# Arguments: +# $1 — timeout in seconds (default: 60) +# Outputs: +# Writes progress/warnings to stderr via log() +# Returns: +# 0 when healthy or on timeout (non-fatal), 1 if services failed +``` + +--- + ## Editor ### VS Code Settings @@ -249,7 +360,7 @@ The template follows a three-state grammar: ### Enabling/Disabling Services -All services are included in `compose.yaml`. Optional services are gated by Compose profiles: +All services are included in `.devcontainer/stacks/compose.yaml`. Optional services are gated by Compose profiles: | Service | Profile | Always On? | | --- | --- | --- | @@ -269,7 +380,8 @@ COMPOSE_PROFILES=redis,minio,observability ### Adding a New Service 1. Create `stacks/myservice/compose.yaml` -2. Add `- stacks/myservice/compose.yaml` to `compose.yaml` `include:` +2. Add `- myservice/compose.yaml` to the `include:` list in `stacks/compose.yaml` + (paths are relative to that file) 3. Optionally add `profiles: [myservice]` if it should be opt-in 4. Add port forwarding in `devcontainer.json` → `forwardPorts` and `portsAttributes` 5. Use `${VAR:-default}` for any credentials, and add them to `.env.example` @@ -282,6 +394,7 @@ the stack folder (e.g. `./init`, `./config/...`): ```text stacks/ + compose.yaml The orchestrator: `include:` one line per stack postgres/ compose.yaml init/ SQL init scripts (mounted at ./init) @@ -290,6 +403,23 @@ stacks/ config/ OTel, Grafana, Tempo, Loki configs (mounted at ./config) ``` +Relative paths inside a stack's `compose.yaml` resolve against *that file's* folder, not the orchestrator's, so a +stack stays self-contained. + +### Why every caller passes `--env-file` + +Compose discovers `.env` in the project directory, which defaults to the folder holding the first `-f` file. The +orchestrator lives in `stacks/` and `.env` lives in `.devcontainer/`, so that discovery does not reach it. Every +caller — `scripts/startup.sh`, the MOTD, and CI — therefore names it explicitly: + +```bash +docker compose --env-file .devcontainer/.env -f .devcontainer/stacks/compose.yaml up -d +``` + +Omitting it does not error. Every `${VAR:-default}` quietly takes its default and `COMPOSE_PROFILES` reads as empty, +so all opt-in stacks silently vanish. The orchestrator also sets `name: musher-dev` explicitly, because Compose +would otherwise derive the project name from the `stacks/` folder. + --- ## Networking @@ -448,6 +578,7 @@ settings across container rebuilds. policies/__init__.py The policy registry -- the only wiring a policy needs policies/config/ .config/ layout, index, and no shadowing root config policies/ports/ Port table ↔ forwardPorts ↔ compose parity + policies/toolchain/ Image-baked pins; banned rate-limit-fragile Features policies/hooks/ lefthook ↔ CI job parity policies/rulesets/ Branch rulesets ↔ CI job-name parity .github/ @@ -458,12 +589,14 @@ taskfiles/ Task modules included by the root Taskfile.yml Taskfile.yml Task entry point (cannot move — root-only discovery) .gitattributes Line-ending policy (`* text=auto eol=lf`) .devcontainer/ + Dockerfile The image: bun, uv, task, mise as pinned ARGs + .dockerignore Empties the build context (`*`); keeps .env off the daemon devcontainer.json Features, extensions, settings, mounts, ports - mise.toml CLIs without a Feature (single source of tool versions) - compose.yaml Stack orchestrator (includes stacks//compose.yaml) + mise.toml Runtime-only CLIs with no Feature .env.example Environment template (copy to .env) .env Local overrides (gitignored) - stacks/ One folder per stack: its compose.yaml + colocated config + stacks/ The services, and the orchestrator that includes them + compose.yaml Stack orchestrator (`include:` + `name: musher-dev`) postgres/ compose.yaml PostgreSQL with pgvector (always on) init/ @@ -490,6 +623,7 @@ Taskfile.yml Task entry point (cannot move — root-only discov initialize.sh Host-side bootstrap (runs before docker run) post-create.sh One-time setup entry point startup.sh Every-start service launcher + verify-toolchain.sh Asserts baked tools match the Dockerfile ARGs (CI) lib/ base-setup.sh Reusable tool installer (mise CLIs + Claude) common.sh Shared utilities diff --git a/README.md b/README.md index cb673c6..3475136 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,9 @@ need. ## What You Get -- Ubuntu base with zsh/oh-my-zsh -- Node, Python, Go, Java, Deno, bun, uv — pinned Features +- Ubuntu 24.04 LTS base with zsh/oh-my-zsh +- Node, Python, Go, Java, Deno — pinned Features +- bun, uv, Task, mise — baked into the image by `.devcontainer/Dockerfile` (pinned `ARG`s) - Docker-in-Docker - Git + GitHub CLI - Claude Code + Codex CLI + Task runner + Lefthook @@ -27,8 +28,9 @@ need. All local-dev state is contained under `.devcontainer/`. On first build, `initializeCommand` copies `.env.example` → `.env` (gitignored). The same file feeds: -- **Docker Compose** — auto-discovered as the sibling `.env` next to `compose.yaml`, used to interpolate - `${VAR:-default}` references. +- **Docker Compose** — passed explicitly as `--env-file .devcontainer/.env` by every caller, used to interpolate + `${VAR:-default}` references. It is not auto-discovered: the orchestrator lives at + `.devcontainer/stacks/compose.yaml` and `.env` is not its sibling. - **The dev container itself** — loaded via `runArgs --env-file`, so shells and runtimes inside the container see the same values. @@ -49,18 +51,21 @@ The startup MOTD also warns about drift or unfilled required keys. ## Customize - Comment out unneeded features/extensions in `devcontainer.json` -- Change a tool version → `devcontainer.json` (Features), or `.devcontainer/mise.toml` for CLIs without a Feature - (AI CLIs, lefthook, linters) +- Change a tool version → `devcontainer.json` (Features), `.devcontainer/Dockerfile` (bun, uv, Task, mise), or + `.devcontainer/mise.toml` for runtime-only CLIs (AI CLIs, lefthook, linters). The four-tier rule is in + [CONFIGURATION.md](CONFIGURATION.md#runtimes--tools) and enforced by `repo toolchain check`. - Change a lint rule → the matching file in `.config/` (see [`.config/README.md`](.config/README.md)) - Add project setup to `scripts/post-create.sh` (runs after `base_setup`) -- Enable optional services via `COMPOSE_PROFILES` in `.devcontainer/.env` (redis, minio, registry, azimutt, observability) +- Add or enable a service → `.devcontainer/stacks/` (one folder per stack, registered in `stacks/compose.yaml`); + toggle with `COMPOSE_PROFILES` in `.devcontainer/.env` (redis, minio, registry, azimutt, observability) - Full reference → [CONFIGURATION.md](CONFIGURATION.md) ## Included CI `.github/workflows/validate.yaml` runs seven jobs: ShellCheck, Compose config validation, devcontainer lockfile -freshness, `.env` template sync, the lint gates, the repo structure policies, and a devcontainer build. Lint tool -versions resolve from `.devcontainer/mise.toml` — the same file the container uses — so CI and local cannot drift. +freshness, `.env` template sync, the lint gates, the repo structure policies, and a devcontainer build that also +asserts the image-baked tools report their pinned versions. Lint tool versions resolve from +`.devcontainer/mise.toml` — the same file the container uses — so CI and local cannot drift. The same lint and structure checks run pre-commit via [`.config/lefthook.yml`](.config/lefthook.yml), and `repo hooks check` fails the build if the two ever disagree. @@ -75,8 +80,9 @@ will tell you why. ### CRLF / WSL line ending issues -The `postCreateCommand` automatically strips `\r` from all scripts before running them. If you add new scripts, ensure -they're under `.devcontainer/scripts/` to be included. +`.gitattributes` (`* text=auto eol=lf`) normalizes every tracked file, so scripts arrive with LF on every platform +and no fixup step is needed. The one exception is `.devcontainer/.env`, which is gitignored and therefore out of +`.gitattributes`' reach — `scripts/initialize.sh` strips `\r` from it host-side before the container starts. ### Stale containers