Serve transcripts as static files on object storage (Cloudflare R2) behind a CDN

[davmlaw]

🤖 Written by Claude

Idea

Most of what this server does is return a stored blob for a key. That is a static file. If the by-accession path were served as static objects from cloud storage behind a CDN, it would scale essentially without limit and take the bulk of the load off gunicorn/Redis entirely.

The single-accession endpoint is already a pure pass-through: r.get(accession) straight into HttpResponse(data, content_type='application/json'), no parse, no serializer, no per-request assembly (cdot_rest/views.py:115-119). The static files would be byte-identical to what is already in Redis, so generating them is a dump rather than a rewrite.

Why not GitHub Pages

This was the first thought, and it does not work. Measured against the live site (1,835,279 transcript versions) and a sample of 15 real transcripts from cdotlib.org:

	mean per transcript	x 1.84M
raw JSON	2.7 KB	~5.0 GB
gzipped individually	1.0 KB	~1.8 GB

The sample skews toward large multi-exon coding genes so the true mean is lower, but even a conservative 1.2 KB raw lands at ~2.2 GB. Three blockers:

1. Published Pages sites may be no larger than 1 GB. Documented hard limit.
2. Pages deployments time out at 10 minutes. Not enough to tar, upload and unpack ~1.8M files.
3. Splitting destroys the compression. The all-builds RefSeq + Ensembl release is 239 MB gzipped. As individual files it is ~1.8 GB gzipped, roughly 7.6x larger, because per-file gzip cannot share a dictionary across transcripts and every file re-pays for the same JSON key names.

Plus a git repo with 1.8M files per release, times every release retained, is unpleasant to work with.

Cloudflare R2 instead

No file-count limit, no egress fees, ~5 GB of storage is cents per month, and the bulk upload is a few dollars in Class A operations per release. S3 + CloudFront also works but egress is a real line item.

Object storage has a flat keyspace, so there is no need to hash or shard paths to avoid huge directories. Keys can stay human-readable and curl-able.

Namespace by data release so URLs are immutable and can be served Cache-Control: immutable, letting the CDN edge cache them indefinitely:

/0.2.34/NM_000059.4.json

Note this matches the current contract, where _merge_genome_builds merges GRCh37 + GRCh38 + T2T into a single blob per accession (cdot_json/management/commands/import_transcript_json.py:91-101). Splitting per genome build would be an improvement in its own right, since clients usually want one build and currently download three, but that is a client-visible change and should be decided separately.

Immutable versioned URLs are the real win here. See #17 for why the current @cache_page setup gets almost no caching benefit.

Most of the API can go static, not just transcripts

Looking at the Redis key schema, most of it is already shaped like static files:

Redis key	Static equivalent
<accession>	/<release>/<accession>.json
<gene_symbol>	/<release>/gene/<symbol>.json
versions:<versionless>	/<release>/versions/<versionless>.json, then a second fetch
transcripts:<gene_name>	/<release>/transcripts/<gene>.json

Even region queries could go static via precomputed bins. Instead of unpickling a per-contig IntervalTree (import_transcript_json.py:180-186), precompute fixed bins (eg 1 Mb, roughly 3,100 per build) or UCSC-style multi-level bins, each listing the transcripts overlapping it. The client fetches the bins covering the query range and does the final overlap test locally. Cheap to generate, cheap to serve.

What is actually given up

• The batch endpoint. MAX_BATCH_SIZE = 10000 (cdot_rest/views.py:19). One request becoming up to 10,000 GETs is a genuinely different proposition, even with HTTP/2 multiplexing and edge caching. This is the thing that should decide whether this is worth doing.
• A single home for ranking logic. RedisDataProvider subclasses cdot's LocalDataProvider and overrides only four accessors (cdot_rest/redis_data_provider.py:10-34), so server and client cannot currently disagree. Pushing binning and ranking into the client means old client versions can drift from the server.
• Server-side visibility into what is being requested.

Suggested approach

Hybrid rather than replacement:

• by-accession (and probably gene, versions, gene-to-transcripts) on R2 with immutable versioned URLs
• cdot_rest retained for region, MANE/tags, batch, and anything needing cdot's ranking logic
• the cdot client's REST provider prefers static, falls back to REST

Measure first. Two numbers decide this:

1. The endpoint mix from the nginx access logs. If by-accession is the overwhelming majority, the payoff is clear. If batch is carrying most of the volume, the trade is much worse than it looks.
2. Whether fixing the cache config (Cloudflare is caching nothing; restore the deliberate Cache-Control on pass-through views #17) already gets most of the benefit for an afternoon's work rather than a new deployment pipeline.

Worth being clear about who this serves. Heavy users should already be downloading a release and using JSONDataProvider in-process. Static objects serve the middle ground: people wanting a few thousand transcripts who do not want a 100 MB download or a server to run.

[davmlaw]

🤖 Written by Claude

Important caveat to the above, found after filing: cdotlib.org is already behind Cloudflare, and Cloudflare is currently caching nothing.

Every endpoint returns cf-cache-status: DYNAMIC, including the genuinely static ones:

/static/openapi.yaml       cf-cache-status: DYNAMIC
/llms.txt                  cf-cache-status: DYNAMIC
/transcript/NM_007294.4    cf-cache-status: DYNAMIC   (cache-control: max-age=86400)
/gene/BRCA2                cf-cache-status: DYNAMIC   (cache-control: max-age=86400)

Cloudflare only caches a fixed list of file extensions by default, and /transcript/NM_007294.4 does not qualify regardless of the origin's Cache-Control. So every request currently reaches gunicorn.

This changes the cost/benefit of this issue considerably. A Cloudflare Cache Rule ("Cache Everything") on the read paths would put the same 1.8M transcript responses on the same CDN edge, with:

• no export pipeline
• no second copy of the data to keep in sync with each release
• no client changes
• no split-brain risk between static objects and the server's ranking logic
• the batch endpoint left intact (the main thing this issue gives up)

That is most of the scaling win of moving to R2, for a config change.

The remaining arguments for static objects on R2 are narrower than the issue body implies:

1. Origin independence. A CDN cache still needs the origin alive for misses and revalidation. Static objects survive the box being down.
2. Immutable URLs. Release-namespaced paths can be cached indefinitely rather than for a max-age window, so cold-cache misses stop mattering.
3. Cost/scale ceiling if traffic ever grows past what one gunicorn box can serve on cache misses.

Those are real but none are urgent, and all of them are cheaper to evaluate after the cache rule is in place and there is data on the actual hit rate.

Suggested sequencing: do #17 first (specifically the Cloudflare Cache Rule), measure the hit rate and the endpoint mix, then revisit this issue with real numbers. If the edge cache absorbs the by-accession traffic, this becomes a nice-to-have rather than a scaling necessity.