Release v0.15.0 - xchain-sync

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.15.0] - 2026-09-07
+
+### Added
+- `attestation_responses` is registered as a hub-mirror table.
+- The bootstrap mirror is vendored instead of living only on the file master.
+
+### Fixed
+- The replication-source read asks the hub for unredacted config secrets.
+- Replica columns widen from the shared utf8mb4 definition.
+- mariadb moved off the cleartext-credential advisory range with the floor pinned in the dependency gate.
+
+### Changed
+- The vendored coin registry is resynced from the hub.
+
 ## [0.12.0] - 2026-08-30
 
 ### Added

README.md

@@ -4,8 +4,8 @@
 # XChain Sync
 
 <p align="center">
-  <img src="https://img.shields.io/badge/version-0.12.0-blue" alt="Version">
-  <img src="https://img.shields.io/badge/tests-2%2C461%2B%20passing-brightgreen" alt="Tests">
+  <img src="https://img.shields.io/badge/version-0.15.0-blue" alt="Version">
+  <img src="https://img.shields.io/badge/tests-2%2C479%2B%20passing-brightgreen" alt="Tests">
   <img src="https://img.shields.io/badge/node-%3E%3D22-green" alt="Node">
   <img src="https://img.shields.io/badge/license-AGPL--3.0--or--later-blue" alt="License">
 </p>
@@ -129,11 +129,11 @@ npm run api
 |---|---|
 | `npm run api` | Start the sync service |
 | `bin/run-db-tiers.sh` | Run the DB-backed tiers against a throwaway MariaDB it starts and drops |
-| `npm test` | Run unit tests (1,975 tests) |
+| `npm test` | Run unit tests (1,993 tests) |
 | `npm run ci` | Unit tests plus the security tier, exits on completion |
 | `npm run coverage` | Unit tests under `c8` coverage instrumentation |
 | `npm run test:boundary` | Boundary condition tests (consensus constants, 7 tests) |
-| `npm run test:regression` | Unit tests tagged `@regression` (501 tests) |
+| `npm run test:regression` | Unit tests tagged `@regression` (506 tests) |
 | `npm run test:smoke` | Smoke tests (server + client startup, config loading, 17 tests) |
 | `npm run test:integration` | Integration tests (requires MariaDB + running indexer, 102 tests) |
 | `npm run test:e2e` | End-to-end tests (full server/client lifecycle, 72 tests) |

deploy/sync-bootstraps/README.md

@@ -0,0 +1,105 @@
+# sync-bootstraps
+
+Serving side of bootstrap distribution.
+
+Two pieces, deployed to different machines:
+
+| File | Deployed to | Typical path |
+|---|---|---|
+| `sync-bootstraps.sh` | the file master | `/usr/local/sbin/sync-bootstraps.sh`, mode 755, root-owned |
+| `latest.php` | each serving host | inside the served payload tree, with its `.htaccess` |
+
+`latest.php` resolves `latest.tgz` / `latest.tgz.sig` to the newest archive that
+has a paired signature, ordered by the UTC timestamp in the filename.
+
+## The thing that makes this easy to get wrong
+
+**The file master does not serve the bootstraps.** The public hostname resolves
+to the serving tier, and the master's docroot entry for the payload directory is
+a symlink to the real tree. So on the master a freshly published archive looks
+published. It is not public until this script has mirrored it out.
+
+The generators push only as far as the master and log a success line when they
+get there. That line means "reached the master", not "reached the internet".
+Verify against the serving tier, never the master.
+
+## Cron
+
+Install in root's crontab on the file master:
+
+```cron
+# Mirror the bootstrap tree out to the serving tier. Hourly, NOT daily: see the
+# SCHEDULE block in the script. The PID lock makes an overlapping run a no-op.
+30 * * * * /usr/local/sbin/sync-bootstraps.sh 1>/dev/null 2>/dev/null
+```
+
+Hourly is a correctness requirement, not tuning. The monthly tracker publish
+runs for hours. A daily slot that happens to fall before it means the tier
+serves the previous month's bootstrap until the next day's pass, with every
+component reporting success. That is exactly what happened while this ran daily.
+
+Do not replace hourly with a slot chosen to be "after" the publish. The publish
+window grows with the chain, so frequency is the guarantee and timing is not.
+
+## Environment
+
+The script carries no site topology. Set these where cron or the unit can see
+them; the script refuses to start if any required one is unset.
+
+| Variable | Meaning |
+|---|---|
+| `BOOTSTRAP_SRC` | payload tree on the master, trailing slash |
+| `BOOTSTRAP_SITE_SRC` | docroot on the master, trailing slash |
+| `BOOTSTRAP_DEST` | payload path on the targets, trailing slash |
+| `BOOTSTRAP_SITE_DEST` | docroot on the targets, trailing slash |
+| `BOOTSTRAP_TARGETS` | space-separated serving hosts |
+| `BOOTSTRAP_SSH_KEY` | private key authorised on the targets |
+| `BOOTSTRAP_REMOTE_USER` | ssh user on the targets (default `www`) |
+| `BOOTSTRAP_PAYLOAD_DIRNAME` | payload dir name, excluded from the site leg (default `bootstraps`) |
+| `BWLIMIT` | rsync bandwidth cap (default `60M`) |
+| `BOOTSTRAP_LOCKFILE`, `BOOTSTRAP_LOGFILE` | lock and log paths |
+
+Use a dedicated key, not one shared with other fanouts, and restrict it to the
+master's address in `authorized_keys`.
+
+## Operating notes
+
+- Targets are mirrored **sequentially**. A full pass is the size of the whole
+  tree per target, so total wall time is roughly one target's transfer time
+  times the number of targets.
+- `BWLIMIT` throttles the master, which typically also carries the hub and the
+  replication master. Raising it doubles throughput about linearly; raise it
+  deliberately, and remember it applies per run, not per target.
+- **Do not edit this script on the machine while a run is in flight.** bash
+  reads a script lazily by byte offset, so an in-place edit can corrupt the
+  running execution. Stop the run, edit, restart.
+- Killing a run hard strands rsync's hidden `.<name>.XXXXXX` temp instead of
+  moving it into `.rsync-partial`, so that file restarts from zero next pass.
+  The stray temp is extraneous and `--delete-after` sweeps it.
+- If you stop a run to restart it with different settings, confirm the old rsync
+  is actually gone before relaunching. Two concurrent passes against the same
+  target will both transfer, doubling load on the master. `pgrep` takes an
+  extended regex, so a pattern like `'a\|b'` matches nothing and will tell you
+  the process is gone when it is not. Check with `ps` and kill by PID.
+- Retention is **not** coordinated with the master. The publisher prunes at
+  publish time, so between that prune and the next mirror pass the serving tier
+  is the only holder of the superseded archive and is still serving links to it.
+
+## Verifying a publish actually reached the public
+
+Check every address the hostname resolves to. A round-robin will otherwise mask
+a host that missed the mirror:
+
+```bash
+HOST=<bootstrap hostname>
+ARCHIVE=<coin>/<network>/<archive>.tar.gz
+for ip in $(dig +short "$HOST"); do
+  curl -s -o /dev/null -w "$ip %{http_code}\n" \
+    --resolve "$HOST:443:$ip" \
+    "https://$HOST/bootstraps/xchain-utxo-tracker/$ARCHIVE"
+done
+```
+
+All addresses must return 200 for the same archive. Anything else means the
+mirror has not finished, and consumers are getting different answers depending
+on which host they reach.

deploy/sync-bootstraps/sync-bootstraps.sh

@@ -0,0 +1,134 @@
+#!/bin/bash
+# sync-bootstraps.sh - mirror the bootstrap tree from the file master out to the
+# hosts that actually serve it.
+#
+# Deployment specifics (target hosts, key, paths) come from the environment so
+# this file carries no site topology. Set them in the unit or cron environment.
+#
+# SCHEDULE: hourly, and it must stay at least that frequent.
+#
+# THIS SCRIPT IS THE ONLY PATH FROM THE FILE MASTER TO THE SERVING TIER. The
+# master is not in the serving path: the bootstrap hostname resolves to the web
+# tier, and the master's own docroot entry is a symlink to the payload tree, so
+# a fresh archive is visible there while the public still cannot get it.
+# Inspecting the master therefore proves nothing about what is served, and
+# neither does the publisher's success line, which only means "reached the
+# master".
+#
+# This ran daily for a while, in a slot that happened to fall about an hour
+# BEFORE the monthly tracker publish began. The publish runs for hours, so every
+# month this fired against the previous month's tree, exited rc=0, and left the
+# public tier advertising a month-old bootstrap until the next day's run. Do not
+# move it back to a daily slot, and do not pick a slot you believe is "after"
+# the publish: that window is hours long and grows with the chain, so frequency
+# is the guarantee, timing is not.
+#
+# RETENTION IS NOT COORDINATED with the master. The publisher prunes to KEEP
+# archives at publish time, so between that prune and the next mirror pass the
+# serving tier is the ONLY holder of the superseded archive while still
+# advertising links to it. --delete-after is what finally retires them.
+#
+# Notes:
+#   - --partial-dir keeps a resumable partial OUT of the served tree, so a
+#     download can never hit a half-written multi-GB tarball; the finished file
+#     appears by atomic rename. It only protects an INTERRUPTED transfer: rsync
+#     writes to a hidden .<name>.XXXXXX temp while running and moves it into
+#     .rsync-partial on a clean interrupt, so a hard kill (SIGKILL, or a SIGTERM
+#     it cannot service) strands that temp and the next run restarts the file
+#     from zero. The stray temp is extraneous and --delete-after sweeps it, but
+#     the transfer work is lost.
+#   - --exclude='*.part' because the publisher uploads each archive as
+#     <name>.part and only then renames it into place. Without this, an hourly
+#     pass landing inside the publish window hauls a partial upload of up to
+#     ~160 GB to every target and deletes it on the next pass. --partial-dir
+#     does NOT cover this: those are the PUBLISHER's temp files sitting in the
+#     source tree, not rsync's own.
+#   - --bwlimit protects the master, which also carries the hub and the
+#     replication master. Raise it deliberately, not by default.
+#   - --delete keeps the mirror exact. Safe here because the source holds one
+#     snapshot per coin/network and the browser assets live in the same tree.
+#   - The SITE leg mirrors the docroot-level chrome (index.html, stylesheets,
+#     assets/, listing templates) that the payload leg never touches. Without it
+#     the tier serves the bootstraps page with no stylesheet and a 404ing logo.
+#     `--exclude=$PAYLOAD_DIRNAME` keeps the source's symlink from clobbering
+#     the targets' real payload directory, and --delete never removes an
+#     excluded path, so the payload tree is safe from this leg.
+set -u
+
+# --- deployment configuration (override in the environment) ------------------
+SRC="${BOOTSTRAP_SRC:?set BOOTSTRAP_SRC to the payload tree on the file master, with trailing slash}"
+SITE_SRC="${BOOTSTRAP_SITE_SRC:?set BOOTSTRAP_SITE_SRC to the docroot on the file master, with trailing slash}"
+DEST_PATH="${BOOTSTRAP_DEST:?set BOOTSTRAP_DEST to the payload path on the targets, with trailing slash}"
+SITE_DEST="${BOOTSTRAP_SITE_DEST:?set BOOTSTRAP_SITE_DEST to the docroot on the targets, with trailing slash}"
+TARGETS="${BOOTSTRAP_TARGETS:?set BOOTSTRAP_TARGETS to a space-separated list of serving hosts}"
+SSH_KEY="${BOOTSTRAP_SSH_KEY:?set BOOTSTRAP_SSH_KEY to the private key authorised on the targets}"
+REMOTE_USER="${BOOTSTRAP_REMOTE_USER:-www}"
+PAYLOAD_DIRNAME="${BOOTSTRAP_PAYLOAD_DIRNAME:-bootstraps}"
+BWLIMIT="${BWLIMIT:-60M}"
+LOCKFILE="${BOOTSTRAP_LOCKFILE:-/var/tmp/sync-bootstraps.lock}"
+LOGFILE="${BOOTSTRAP_LOGFILE:-/var/log/sync-bootstraps.log}"
+
+log(){ echo "[$(date -u '+%F %T UTC')] $*" >> "$LOGFILE"; }
+
+# PID lock with stale-lock recovery: a crashed run must not block every future
+# one.
+if [ -f "$LOCKFILE" ]; then
+    OLDPID=$(cat "$LOCKFILE" 2>/dev/null)
+    if [ -n "$OLDPID" ] && kill -0 "$OLDPID" 2>/dev/null; then
+        log "already running as pid $OLDPID; exiting"
+        exit 0
+    fi
+    log "clearing stale lock (pid ${OLDPID:-unknown} is gone)"
+fi
+echo $$ > "$LOCKFILE"
+trap 'rm -f "$LOCKFILE"' EXIT
+
+if [ ! -d "$SRC" ]; then
+    log "FATAL: source $SRC missing"
+    exit 1
+fi
+if [ ! -d "$SITE_SRC" ]; then
+    log "FATAL: site source $SITE_SRC missing"
+    exit 1
+fi
+
+log "=== start (bwlimit=$BWLIMIT, source $(du -sh "$SRC" 2>/dev/null | cut -f1)) ==="
+RC_TOTAL=0
+for T in $TARGETS; do
+    log "--- $T: site files starting"
+    START=$(date +%s)
+    rsync -a --delete-after \
+          --exclude="$PAYLOAD_DIRNAME" \
+          --exclude=.rsync-partial \
+          --stats \
+          -e "ssh -i $SSH_KEY -o BatchMode=yes -o StrictHostKeyChecking=accept-new" \
+          "$SITE_SRC" "${REMOTE_USER}@${T}:${SITE_DEST}" >> "$LOGFILE" 2>&1
+    RC=$?
+    ELAPSED=$(( $(date +%s) - START ))
+    if [ $RC -eq 0 ]; then
+        log "--- $T: site files OK in ${ELAPSED}s"
+    else
+        log "--- $T: site files FAILED rc=$RC after ${ELAPSED}s"
+        RC_TOTAL=$RC
+    fi
+
+    log "--- $T: starting"
+    START=$(date +%s)
+    rsync -a --delete-after \
+          --partial-dir=.rsync-partial \
+          --exclude='*.part' \
+          --bwlimit="$BWLIMIT" \
+          --stats \
+          -e "ssh -i $SSH_KEY -o BatchMode=yes -o StrictHostKeyChecking=accept-new" \
+          "$SRC" "${REMOTE_USER}@${T}:${DEST_PATH}" >> "$LOGFILE" 2>&1
+    RC=$?
+    ELAPSED=$(( $(date +%s) - START ))
+    if [ $RC -eq 0 ]; then
+        log "--- $T: OK in ${ELAPSED}s"
+    else
+        log "--- $T: FAILED rc=$RC after ${ELAPSED}s"
+        RC_TOTAL=$RC
+    fi
+done
+log "=== done (worst rc=$RC_TOTAL) ==="
+exit $RC_TOTAL

package.json

@@ -1,7 +1,7 @@
 {
   "name": "xchain-sync",
   "description": "Database replication service for the XChain Platform: syncs indexer and decoder databases to validators and consumers via REST snapshots and WebSocket streaming",
-  "version": "0.12.0",
+  "version": "0.15.0",
   "license": "AGPL-3.0-or-later",
   "repository": {
     "type": "git",