feat(analytics): public anonymous stats with never-stored visitor hashes - #31
Open
ImGajeed76 wants to merge 1 commit into
Open
feat(analytics): public anonymous stats with never-stored visitor hashes#31ImGajeed76 wants to merge 1 commit into
ImGajeed76 wants to merge 1 commit into
Conversation
Every figure is public, no stored row describes a single reading session, and counts never affect ranking. An edge Worker writes Analytics Engine datapoints, an hourly cron rolls them into Neon, and the completed day is exported as CC0. Uniques are HyperLogLog: the edge HMACs the address with a per-year secret, keeps one (register, rank) pair and discards the hash, so no visitor identifier is ever stored. The salt epoch is the calendar year, the largest window over which a union stays lossless. Breakdowns sit behind a k=25 floor applied by one predicate that both the live pages and the CC0 export call, so the open data cannot disagree with the site. drizzle/0033_analytics.sql is hand-authored: three tables are PARTITION BY RANGE and the hll columns carry a type modifier drizzle-kit drops, so schema/analytics.ts is a hand-kept mirror and the migration wins on any disagreement. Partitions must stay ahead of writes; once rows land in DEFAULT, creating that month fails. Gotchas RENDERER_VERSION -> 8, because the table of contents and the excerpt now parse with the renderer's own phase. Both are baked into cached HTML, and the purge runs on the next sync cron tick rather than on deploy. The client must never import a path containing "events": content blockers refuse the request and take the component down. Modules bundled into a Worker cannot use SvelteKit aliases, which an eslint rule now enforces. Tests run on three runtimes, workerd included, because a locale check built on Intl.DisplayNames passed every Bun test while refusing Tibetan on the runtime that ships.
Contributor
Author
|
@greptile-apps review |
❌ Action failedReview failed.
|
|
Important Review skippedToo many files! This PR contains 181 files, which is 81 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (181)
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryThe PR adds anonymous edge analytics, Neon rollups, privacy-gated public statistics, daily CC0 exports, and an MCP analytics tool.
Confidence Score: 1/5The PR should not merge until cross-project beacon attribution and single-doco country disclosure in the CC0 export are fixed. The navigation fallback persists an unverified project association that the rollup treats as authoritative, while the export path omits a privacy predicate required to suppress retained country data. Files Needing Attention: src/routes/api/e/navigation/+server.ts, src/lib/analytics/counter-rollup.ts, src/lib/analytics/cc0-export.ts
|
| Filename | Overview |
|---|---|
| workers/edge/index.ts | Adds request analytics and domain-separated visitor-register writes; no retained visitor hash was identified. |
| src/routes/api/e/navigation/+server.ts | Adds fallback navigation collection but trusts an independently supplied project ID, allowing cross-project attribution. |
| src/lib/analytics/counter-rollup.ts | Aggregates observations across subjects but prefers untrusted observation project IDs over database-backed doco context. |
| src/lib/server/analytics/storage.ts | Adds idempotent daily replacement, totals deltas, malformed-register dropping, and monthly sketch rebuilding. |
| src/lib/server/analytics/public-stats.ts | Applies reader floors and current single-doco country suppression to live statistics. |
| src/lib/analytics/cc0-export.ts | Applies shared k-floors to exports but omits the live path's single-doco country restriction. |
| drizzle/0033_analytics.sql | Introduces the persisted analytics schema and constraints; no accepted schema-specific defect was established. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
R[Reader request] --> E[Edge Worker]
N[Navigation beacon] --> A[Event ingestion]
E --> AE[Analytics Engine]
A --> AE
AE --> C[Hourly rollup]
C --> DB[(Neon aggregates)]
DB --> L[Live stats predicate]
DB --> X[CC0 export predicate]
L --> P[Public stats pages and MCP]
X --> D[Daily NDJSON]
Prompt To Fix All With AI
### Issue 1
src/routes/api/e/navigation/+server.ts:31-37
**Client project ID controls attribution**
When a navigation beacon supplies a real doco ID with another valid project UUID, this route writes the supplied project ID and the rollup treats it as authoritative, causing page-request and geographic statistics to be attributed to an unrelated project.
**How this was verified:** The public route writes `meta.projectId` directly, and the rollup prefers every nonempty observation project ID over the doco's database-backed project context.
### Issue 2
src/lib/analytics/cc0-export.ts:53-60
**Export omits country privacy gate**
When a project accumulates country rows while publishing multiple docos and later shrinks to one, this filter exports those retained rows without the live path's current project-scope check, exposing the single doco's country distribution in the public NDJSON export.
**How this was verified:** The export filter only checks bucket and read totals, while the live stats path explicitly suppresses retained country rows after checking the project's current published-doco count.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "feat(analytics): public anonymous stats ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+31
to
+37
| buildEventDataPoint("page_request", { | ||
| docoId: meta.docoId, | ||
| projectId: meta.projectId, | ||
| uaClass, | ||
| botFamily, | ||
| versionId: meta.versionId, | ||
| }), |
There was a problem hiding this comment.
When a navigation beacon supplies a real doco ID with another valid project UUID, this route writes the supplied project ID and the rollup treats it as authoritative, causing page-request and geographic statistics to be attributed to an unrelated project.
How this was verified: The public route writes meta.projectId directly, and the rollup prefers every nonempty observation project ID over the doco's database-backed project context.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/routes/api/e/navigation/+server.ts
Line: 31-37
Comment:
**Client project ID controls attribution**
When a navigation beacon supplies a real doco ID with another valid project UUID, this route writes the supplied project ID and the rollup treats it as authoritative, causing page-request and geographic statistics to be attributed to an unrelated project.
**How this was verified:** The public route writes `meta.projectId` directly, and the rollup prefers every nonempty observation project ID over the doco's database-backed project context.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Comment on lines
+53
to
+60
| return rows.filter( | ||
| (row) => | ||
| row.bucket.length === 0 || | ||
| publishesBreakdownBucket( | ||
| Number(row.n), | ||
| readTotals.get(subjectKey(row.subject_type, row.subject_id)) ?? 0, | ||
| ), | ||
| ); |
There was a problem hiding this comment.
When a project accumulates country rows while publishing multiple docos and later shrinks to one, this filter exports those retained rows without the live path's current project-scope check, exposing the single doco's country distribution in the public NDJSON export.
How this was verified: The export filter only checks bucket and read totals, while the live stats path explicitly suppresses retained country rows after checking the project's current published-doco count.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/analytics/cc0-export.ts
Line: 53-60
Comment:
**Export omits country privacy gate**
When a project accumulates country rows while publishing multiple docos and later shrinks to one, this filter exports those retained rows without the live path's current project-scope check, exposing the single doco's country distribution in the public NDJSON export.
**How this was verified:** The export filter only checks bucket and read totals, while the live stats path explicitly suppresses retained country rows after checking the project's current published-doco count.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Contributor
Author
|
@coderabbitai review |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anonymous, all-public analytics for docolin. Every figure is public, no stored
row describes a single reading session, and counts never affect ranking.
An edge Worker writes Analytics Engine datapoints on every request including
cache hits, an hourly cron rolls them into Neon, and the completed day is
exported as CC0 NDJSON. Public pages at /stats, /stats/{org}/{project} and
/stats/{org}/{project}/{path}, plus an MCP tool.
Invariants worth checking
These are the claims the feature makes. If any is false the feature is wrong,
so they are the useful things to review.
secret, keeps one HLL (register, rank) pair and discards the hash.
registers cannot be correlated.
and the CC0 export call.
shrunk to one.
narrowed and then checked against something real, not just shaped.
Where the risk is
src/lib/locale-tag.tsgates whether a doco syncs at all. It went throughseveral wrong answers before landing on CLDR expansion; display-name lists
differ per runtime and refused Tibetan in production only.
src/lib/server/analytics/storage.tsis the rollup. A throw there stalls thecursor and re-runs the same day forever, so bad input has to be dropped
rather than raised.
workers/edge/andsrc/routes/api/e/are public unauthenticated writes.drizzle/0033_analytics.sqlis hand-authored andschema/analytics.tsis ahand-kept mirror. The migration wins on any disagreement.
Already verified, to save duplicate effort
case fuzz after the shared parser extraction.
Before this records anything in production
ANALYTICS_SECRET_<year>,CLOUDFLARE_ACCOUNT_ID,CLOUDFLARE_ANALYTICS_TOKENTesting
bun run testcovers three runtimes: the app on Bun, worker code on Bun, andanything runtime-dependent on workerd via vitest-pool-workers.
bun run checknow also dry-run builds both Workers.