docs: end-user data-access guide (#160)#161
Merged
Merged
Conversation
Consumer-facing guide for the public cMD data: DuckDB ATTACH over HTTPS or direct Parquet, the analytical table schema, worked queries, unit/abundance caveats, and snapshot/reproducibility guidance. Public URLs are placeholders until #57 Phase 5 publishes the frozen catalog. Evaluated by a fresh, context-free agent ("can a naive consumer figure out access from this doc alone?"); iterated on its feedback: - Documented clade_name and real qc_metrics / resistome / taxon columns (resistome fields confirmed from a live 2.2.1 card_kma.res header). - Split fact vs dimension tables (qc_metrics/taxon carry no data_type). - Fixed the taxon join to include db_version (was fanning out) and pinned version in aggregates (was mixing pipeline versions). - Added study/sample discovery queries and INSTALL httpfs in the Parquet path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EyZad3U7YthLPfWq9PN8ZW
There was a problem hiding this comment.
Pull request overview
Adds a new end-user/consumer-facing documentation page (docs/data-access.md) describing how to access and query the published cMD analytical catalog (DuckDB/DuckLake + Parquet over HTTPS), including table schema and worked SQL examples.
Changes:
- Introduces a DuckDB “ATTACH catalog” quickstart (CLI/Python/R) for querying the full snapshot.
- Documents the analytical tables (
taxonomic_profile, marker tables,resistome,qc_metrics,taxon) and recommended join/filter patterns. - Provides worked query examples (study discovery, top species, per-sample taxonomy join, per-study QC summaries) and notes on units/reproducibility.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+47
to
+48
| con <- DBI::dbConnect(duckdb::duckdb()) | ||
| DBI::dbExecute(con, "INSTALL httpfs; LOAD httpfs;") |
Comment on lines
+55
to
+65
| If you don't want the catalog, read the Parquet straight from HTTPS. Same DuckDB, no ATTACH: | ||
|
|
||
| ```sql | ||
| INSTALL httpfs; LOAD httpfs; | ||
| SELECT clade_name, relative_abundance | ||
| FROM read_parquet('https://data.cmgd.cancerdatasci.org/parquet/taxonomic_profile/**/*.parquet') -- (pending) | ||
| WHERE method = 'metaphlan' AND rank = 'species' | ||
| LIMIT 20; | ||
| ``` | ||
|
|
||
| The Parquet is partitioned so filters on `workflow`, `version`, `method`, and `data_type` only read the relevant files. |
Comment on lines
+73
to
+76
| | table | kind | one row per | columns beyond the identity keys | | ||
| |---|---|---|---| | ||
| | `taxonomic_profile` | fact | sample × method × taxon × `data_type` | `method` (`metaphlan`\|`bracken`), **`clade_name`** (label as the profiler reported it), `rank`, `ncbi_taxid`, `sgb_id` (metaphlan only), `relative_abundance` | | ||
| | `marker_abundance` | fact | sample × marker × `data_type` | `marker_name`, `value` | |
… (Copilot)
- HTTPS can't list a directory, so read_parquet('.../**/*.parquet') over https
doesn't work; recommend ATTACH (catalog enumerates files) or a single explicit
URL; note s3:// glob needs credentials. Matches the R2 no-LIST model.
- Split "INSTALL httpfs; LOAD httpfs;" into separate DBI::dbExecute calls.
- Clarify method is metaphlan|bracken in 2.2.1 (no gtdb step this version).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EyZad3U7YthLPfWq9PN8ZW
Owner
Author
|
Thanks @copilot — all three addressed:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
docs/data-access.md— a consumer-facing guide for the public cMD data (issue #160, part of #57).Covers: connecting via DuckDB
ATTACHover HTTPS (no credentials) or reading Parquet directly; the analytical table schema; worked SQL (discovery, top species, one-sample profile with taxonomy join, per-study QC); abundance/unit caveats; and snapshot/reproducibility guidance. Public URLs are placeholders marked (pending) until #57 Phase 5 publishes the frozen catalog.Quality gate: a fresh, context-free agent evaluated the draft ("given only this doc, can you access and use the data?") — rated the access layer 5/5 but caught real query-layer bugs. All fixed before this PR: undefined
clade_name, a fanning-outtaxonjoin (now includesdb_version), a per-study count that mixeddata_type, and undocumentedresistome/qc_metricscolumns (resistome fields verified from a live 2.2.1card_kma.resheader).🤖 Generated with Claude Code
https://claude.ai/code/session_01EyZad3U7YthLPfWq9PN8ZW