Analytics dashboard

analytics/.env.example

@@ -1,3 +1,9 @@
 # Scaleway S3 credentials (required)
 SCW_ACCESS_KEY=SCWXXXXXXXXXXXXXXXXX
 SCW_SECRET_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+
+# Wallet allowed to read GET /stats. A public address, not a secret — it is
+# only ever compared against the signer recovered from the request. Locally you
+# can set this to any address you hold a key for; in production it is the owner
+# wallet, the same value scw_js uses.
+OWNER_ETH_ADDRESS=0x0000000000000000000000000000000000000000

analytics/.gitignore

@@ -6,3 +6,8 @@ dist/
 *.log
 .DS_Store
 .serverless/
+
+# Analytics exports (Umami dumps carry session ids, geo and device columns —
+# never commit them; see notebooks/umami_backfill.py)
+*.zip
+*.csv

analytics/README.md

@@ -11,13 +11,28 @@ as JSON objects in Scaleway S3. No sessions, no visitor IDs, no PII.
 | **Aggregate counters** | One JSON object per site per UTC hour — no per-event rows                                                                                                    |
 | **Per-page breakdown** | Hourly object also tracks a `pages` map, capped at 200 distinct paths                                                                                        |
 | **Anonymous writes**   | `POST /hit` is unauthenticated and increment-only                                                                                                            |
-| **Private counters**   | Stored objects carry no public-read ACL; reads are authorized separately                                                                                     |
+| **Private counters**   | Stored objects carry no public-read ACL; `GET /stats` gates reads on an owner wallet signature                                                               |
 | **Conditional writes** | Compare-and-swap via `storage.ts`'s `HitStorage`, retried up to 3x on conflict — real S3 (`@fretchen/s3-utils`) in production, local files for `npm run dev` |
 | **Input sanitisation** | `path` must start with `/`, safe URL-path characters only, max 200 chars                                                                                     |
 
 ## API
 
-**Base URL** `https://analytics.fretchen.eu`
+Two Scaleway functions. `analytics` serves both HTTP endpoints under one URL
+with path-based routing (the repo convention — see `x402_facilitator`);
+`rollup` is cron-only, because a scheduled invocation has no path to route on.
+`npm run info` prints the URLs.
+
+| Function    | Trigger                 | Auth                    |
+| ----------- | ----------------------- | ----------------------- |
+| `analytics` | `POST /hit`             | none — anonymous writes |
+| `analytics` | `GET /stats`            | owner wallet signature  |
+| `rollup`    | cron, Mondays 00:30 UTC | n/a                     |
+
+`analytics.ts` matches routes **exactly**, unlike `x402_facilitator`'s
+`path.includes()`. That matters here because an anonymous write sits next to an
+owner-gated read: no URL shape may reach `/stats` handling without its auth
+check, and none may reach `/hit`'s unauthenticated write while looking like
+`/stats`.
 
 ### `POST /hit`
 
@@ -37,22 +52,132 @@ control: CORS is browser-enforced only, and the pageview beacon is a
 `sendBeacon` simple request that triggers no preflight at all. Write abuse is
 bounded by path validation and the 200-entry `pages` cap instead.
 
+### `GET /stats`
+
+Serves the `/analytics` dashboard. Requires `Authorization: Bearer <base64
+payload>` where the payload is `{address, signature, message}` and `message` is
+`analytics-api:<unix ts>`, signed by `OWNER_ETH_ADDRESS`. Tokens expire after
+five minutes; anything else returns `401`.
+
+Both halves of that scheme — building the message and verifying it — live in
+`@fretchen/chain-utils` (`auth-protocol.ts`), shared with the Growth API. The
+`analytics-api` prefix is what scopes a token to this service.
+
+```json
+{
+  "site": "fretchen.eu",
+  "from": "2025-08-12",
+  "to": "2026-08-11",
+  "days": {
+    "2026-08-10": { "hits": 240, "pages": { "/": 200 }, "source": "beacon" }
+  }
+}
+```
+
+**No range parameter, by design.** The endpoint always returns the trailing
+year, and `days` is sparse — a day with no traffic is absent, not a zero row.
+Measured against real data that is ~25KB, about 3KB gzipped, so windowing the
+response server-side bought nothing and cost a round trip per view. The
+dashboard fetches once and slices client-side
+(`website/utils/analyticsBuckets.ts` owns the totals, the top-pages list and
+the daily/weekly/monthly bucketing).
+
+For a token from the terminal:
+
+```bash
+OWNER_PRIVATE_KEY=0x... node --input-type=module -e '
+import { privateKeyToAccount } from "viem/accounts";
+const a = privateKeyToAccount(process.env.OWNER_PRIVATE_KEY);
+const message = `analytics-api:${Math.floor(Date.now() / 1000)}`;
+const signature = await a.signMessage({ message });
+const payload = { address: a.address, signature, message };
+console.log("Bearer " + Buffer.from(JSON.stringify(payload)).toString("base64"));
+'
+```
+
 ## Data model
 
+Two layers. The endpoint only ever writes the first.
+
+**Write layer — one object per UTC hour:**
+
 ```
 counts/{site}/{YYYY-MM-DDTHH}.json
+{ "hits": 42, "pages": { "/": 30, "/blog/foo/": 12 } }
 ```
 
-```json
-{ "hits": 42, "pages": { "/": 30, "/blog/foo": 12 } }
+**Read layer — one object per month, a per-day rollup:**
+
+```
+rollup/{site}/{YYYY-MM}.json
+{ "site": "fretchen.eu", "month": "2026-03", "days": {
+    "2026-03-04": { "hits": 18, "pages": { "/": 9 }, "source": "umami" } } }
 ```
 
+The rollup layer exists because reads can't use the hourly one. `listObjects`
+(`shared/s3-utils`) issues a single un-paginated ListObjectsV2 — max 1000 keys,
+silently truncated — and hourly objects accrue at 8760/year; a 30-day window
+would also mean 720 sequential GETs. Rollup keys are **computed** from a date
+range rather than listed, so there is no ceiling and a month costs one GET.
+
+**Hourly buckets are the source of truth and are never deleted.** The `rollup`
+cron is a compaction step, and `GET /stats` falls back to the hourly buckets
+for recent days that aren't rolled up yet. That is what makes a weekly cadence
+safe: a late or missed run changes what a query costs, never what it returns.
+
+Two things keep that fallback cheap, because rebuilding a day costs 24 GETs:
+
+- **Only days after the newest compacted one are probed.** Compaction runs in
+  date order, so everything up to that point is settled — present means
+  traffic, absent means none. Without this, every quiet day inside the window
+  would be re-read on every load. `HOURLY_FALLBACK_DAYS` (14) still caps it for
+  a cold start.
+- **`/stats` writes back what it rebuilds.** A complete day reconstructed from
+  hourly buckets is stored via the same CAS `writeDay` the cron uses, so the
+  next load reads it as one rollup GET. Today is never written back — it is
+  still being counted. Write failures are swallowed: warming a cache must not
+  fail a read.
+
+In practice a warm load is **37 GETs** — 13 monthly rollups plus today's 24
+hours — regardless of which range the dashboard is showing.
+
+**Nothing deletes the hourly buckets, but they can stop being reachable.** The
+bucket has no lifecycle configuration and no code path deletes under `counts/`
+(the only S3 deletes in the repo are scoped to `channels/` and `growth-agent`),
+so the objects are permanent — ~9MB/year, never listed, so no truncation limit
+applies. What _is_ lossy is visibility: if the cron stops for longer than
+`HOURLY_FALLBACK_DAYS`, `/stats` stops probing those days and renders them as
+no-traffic while the data sits there intact. To pull such a gap back in, set
+`ROLLUP_WINDOW_DAYS` wide enough to cover it and invoke `rollup` once.
+
+`source` is per day, not per month, because the changeover month holds both
+kinds and they are not the same measurement: Umami filtered bots and
+sessionised, the beacon counts every hydration and client-side navigation.
+
+**Path form.** `pageContext.urlPathname` is what the beacon sends, and Vike
+derives it from `urlLogical` — set by `website/pages/+onBeforeRoute.ts`. So
+recorded paths are in canonical `sitemap.xml` form: locale prefix stripped,
+trailing slash on every non-root path, no query, no fragment. One consequence:
+German pages are indistinguishable from their English counterparts, since the
+beacon never sees the locale.
+
 ## Reading the data
 
 Counters are **private** — no public-read ACL is set, so they are not
-fetchable from the bucket URL without credentials. A future authorized
-`GET /stats` endpoint (owner wallet signature, as in `scw_js/growth_api.ts`)
-will serve reads.
+fetchable from the bucket URL without credentials. Two ways in:
+
+- **`/analytics` on the website** — owner-gated dashboard over `GET /stats`,
+  with three views: 30 days by day, 90 days by week, one year by month.
+  Switching between them re-slices the single cached response rather than
+  refetching. Linked from the nav bar alongside `/growth`, but only once the
+  owner wallet is connected (`OwnerNavLinks` in `website/layouts/LayoutDefault.tsx`).
+- **`notebooks/02_readout.ipynb`** — direct S3 reads, for anything the
+  dashboard doesn't show.
+
+Jan–Aug 2026 predates the counter and was backfilled from the Umami export —
+see `notebooks/03_umami_backfill.ipynb`. Those days are marked
+`"source": "umami"` and the dashboard greys them out: Umami filtered bots and
+sessionised, so they are not comparable with the beacon's counts.
 
 ## Development
 
@@ -64,18 +189,32 @@ npm run lint          # eslint
 npm run build         # tsup → dist/
 ```
 
-### Local server
+### Local servers
 
 ```bash
-npm run dev      # localhost:8086, file storage (notebooks/state/) — no credentials needed
-npm run dev:live # localhost:8086, real S3 (needs analytics/.env) — pre-deploy sanity check
+npm run dev        # :8086  /hit + /stats, file storage (notebooks/state/) — no credentials
+npm run dev:live   # :8086  /hit + /stats, real S3 (needs analytics/.env)
+npm run dev:rollup # :8088  rollup, file storage
 ```
 
-`npm run dev` writes counters to `notebooks/state/` instead of S3
-(`ANALYTICS_STORAGE=file` selects `FileHitStorage` over `S3HitStorage` in
-`hit.ts`/`storage.ts`) — safe to hammer repeatedly with no risk to
-production data. See `notebooks/01_smoke_test.ipynb` for a driver that
-exercises either target.
+One server for both endpoints, because they are one function. The
+file-storage variants (`ANALYTICS_STORAGE=file` selects `FileHitStorage` over
+`S3HitStorage`) read and write `notebooks/state/` instead of S3 — safe to
+hammer repeatedly with no risk to production data. See
+`notebooks/01_smoke_test.ipynb` for a driver that exercises either target.
+
+**Driving the website from a local function.** `website/.env` sets
+`PUBLIC_ENV__ANALYTICS_URL=http://localhost:8086`; with that in place, `npm run
+dev` in `website/` (port 3000, already on the CORS whitelist) plus `npm run
+dev:live` here gives the real `/analytics` page over real data, and the beacon
+lands locally too. `OWNER_ETH_ADDRESS` must be set in `analytics/.env` to the
+wallet you connect with, or every `/stats` request comes back `401 Address
+mismatch`. Comment the variable out of `website/.env` to go back to the
+deployed function.
+
+Unlike `/hit`, `/stats` is genuinely gated by CORS — it is a `GET` carrying an
+`Authorization` header, so the browser preflights it and an origin missing from
+`ALLOWED_ORIGINS` blocks the page outright. `vike dev` serves on **3000**.
 
 ## Deployment
 
@@ -84,11 +223,25 @@ Console.
 
 ```bash
 npm run deploy         # serverless deploy
+npm run info           # per-function URLs
 ```
 
+After deploying, paste the `analytics` function's URL into the fallback in
+`website/utils/analyticsApi.ts` — one string, shared by the beacon and the
+dashboard. That fallback is what production uses: `.github/workflows/pages.yml`
+sets no `PUBLIC_ENV__*` variables, so there is no CI mechanism to swap it.
+
+**Then delete the old `hit` function in the Scaleway Console.** It predates the
+merge into `analytics` and `serverless deploy` does not remove functions that
+have been dropped from the config, so it would otherwise keep running (and
+keep collecting beacons from any stale client) forever.
+
 ## Environment variables
 
-| Variable         | Scope  | Description                  |
-| ---------------- | ------ | ---------------------------- |
-| `SCW_ACCESS_KEY` | secret | Scaleway API / S3 credential |
-| `SCW_SECRET_KEY` | secret | Scaleway API / S3 credential |
+| Variable             | Scope  | Description                                                 |
+| -------------------- | ------ | ----------------------------------------------------------- |
+| `SCW_ACCESS_KEY`     | secret | Scaleway API / S3 credential                                |
+| `SCW_SECRET_KEY`     | secret | Scaleway API / S3 credential                                |
+| `OWNER_ETH_ADDRESS`  | env    | Wallet allowed to read `GET /stats`                         |
+| `ANALYTICS_STORAGE`  | env    | `file` selects local storage (dev only)                     |
+| `ROLLUP_WINDOW_DAYS` | env    | Days the cron compacts (default 14); widen to recover a gap |

analytics/analytics.ts

@@ -0,0 +1,96 @@
+/**
+ * The analytics function: one Scaleway function, two paths.
+ *
+ *   POST /hit    anonymous pageview counter  (hit.ts)
+ *   GET  /stats  owner-gated readout         (stats.ts)
+ *
+ * Matches the repo convention — one HTTP function per package with path-based
+ * routing, as `x402_facilitator` does for `/verify`, `/settle`, `/supported`.
+ * The weekly compaction cron stays a separate function (`rollup.ts`): a cron
+ * invocation has no path to route on.
+ *
+ * **Exact matching, deliberately.** `x402_facilitator` can get away with
+ * `path.includes()` because all of its endpoints are unauthenticated. Here an
+ * anonymous write sits next to an owner-gated read, so a substring or prefix
+ * match would be a way to reach `/stats` handling through a `/hit`-shaped URL.
+ * Anything that isn't exactly `/hit` or `/stats` gets a 404 and never reaches a
+ * handler.
+ */
+import { handleHit } from "./hit.js";
+import { handleStats } from "./stats.js";
+
+// Same whitelist as hit.ts/stats.ts, kept local rather than shared — each handler in this
+// package owns its own CORS policy. Used only by the 404 fallback below; /hit and /stats
+// each already echo the origin themselves once a route matches.
+const ALLOWED_ORIGINS = ["https://www.fretchen.eu", "http://localhost:3000", "http://localhost:5173"];
+
+export interface AnalyticsEvent {
+  httpMethod: string;
+  path?: string;
+  headers?: Record<string, string>;
+  body?: string | Record<string, unknown>;
+  queryStringParameters?: Record<string, string>;
+}
+
+export interface HandlerResponse {
+  statusCode: number;
+  headers: Record<string, string>;
+  body: string;
+}
+
+/**
+ * Collapses only what a router should: a query string (Scaleway passes it
+ * separately as `queryStringParameters`, but a proxy that folds it into `path`
+ * must not change which route matches) plus repeated and trailing slashes.
+ *
+ * No `..` resolution, no case folding, no percent-decoding — each of those
+ * would widen what counts as a match, and a path needing them is not one of
+ * ours.
+ */
+function normalizeRoute(rawPath: string | undefined): string {
+  const collapsed = (rawPath ?? "")
+    .split("?")[0]
+    .replace(/\/{2,}/g, "/")
+    .replace(/\/+$/, "");
+  return collapsed === "" ? "/" : collapsed;
+}
+
+export async function handle(event: AnalyticsEvent, context: unknown): Promise<HandlerResponse> {
+  const route = normalizeRoute(event.path);
+
+  if (route === "/hit") {
+    return handleHit(event, context);
+  }
+  if (route === "/stats") {
+    return handleStats(event, context);
+  }
+
+  const origin = event.headers?.origin ?? event.headers?.Origin;
+  return {
+    statusCode: 404,
+    headers: {
+      "Content-Type": "application/json",
+      "Access-Control-Allow-Origin": ALLOWED_ORIGINS.includes(origin ?? "") ? origin! : "https://www.fretchen.eu",
+    },
+    body: JSON.stringify({ error: "Not found. Use POST /hit or GET /stats" }),
+  };
+}
+
+/* Local dev server — only when run directly: npm run dev / npm run dev:live */
+const isEntrypoint =
+  typeof process.argv[1] === "string" && import.meta.url.endsWith(process.argv[1].replace(/.*\//, ""));
+
+if (isEntrypoint && process.env.NODE_ENV === "test") {
+  (async () => {
+    const dotenvModule = await import("dotenv");
+    dotenvModule.config();
+
+    const scw = await import("@scaleway/serverless-functions");
+    // AnalyticsEvent's stricter headers type (Record<string, string>) isn't
+    // structurally assignable to the package's own looser Event type — same
+    // cast scw_js/llm_x402_cron.ts uses for the identical mismatch.
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any, @typescript-eslint/no-unsafe-argument
+    scw.serveHandler(handle as any, 8086);
+    console.log("analytics dev server on :8086 — POST /hit, GET /stats");
+  })().catch((err) => console.error("Error starting local server", err));
+}