feat(offline): ext.offline fixture path for the two EBI-facing processes - #1
Merged
mhusbynflow merged 1 commit intoAug 20, 2026
Conversation
…ocesses SRA_IDS_TO_RUNINFO and SRA_FASTQ_FTP are the only processes that talk to EBI. When `ext.offline` is set they emit a pinned runinfo TSV and generate a small FASTQ locally instead; unset (the default) they behave exactly as before. This lets flow-api's sample-import acceptance test run the pipeline end to end without gating a merge on EBI being reachable from a single runner IP, while every transform step downstream — SRA_RUNINFO_TO_FTP, SRA_TO_SAMPLESHEET, MULTIQC_MAPPINGS_CONFIG — still runs for real, so the samplesheet the test asserts on is the one production emits. `ext` rather than a profile or a param because the caller layers one config over every pipeline in its environment: a `withName` selector that matches nothing is a silent no-op, whereas an unknown param would be rejected by any pipeline validating with nf-schema. `md5sum -c` still runs in the offline path. That makes the generated bytes load-bearing: `gzip -n` keeps them deterministic, and the fixtures' fastq_md5 column is repointed at the generated read. Every other column keeps ENA's real value. An accession with no fixture fails naming the accession, rather than emitting an empty TSV that would surface later as zero samples imported. No breaking changes; no nextflow_schema.json change.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets a caller run the pipeline end to end with no network access to EBI, while every line of the pipeline's own transform logic still executes for real.
SRA_IDS_TO_RUNINFOandSRA_FASTQ_FTPare the only two processes that talk to EBI. Each gains anext.offlinebranch. Unset — the default, and what every real run does — behaviour is exactly as today.process { withName: '.*SRA_IDS_TO_RUNINFO.*' { ext.offline = true } withName: '.*SRA_FASTQ_FTP.*' { ext.offline = true } }SRA_RUNINFO_TO_FTP,SRA_TO_SAMPLESHEETandMULTIQC_MAPPINGS_CONFIGare untouched and keep running for real — a fixture that replaced the samplesheet-building step could drift from what the pipeline really emits, leaving the acceptance test green while production broke.Why
extrather than a profile or a paramThe caller (flow-api's acceptance suite) layers one config file over every pipeline in its environment. A
withNameselector that matches nothing is a silent no-op; an unknown param would be rejected by any other pipeline validating with nf-schema.-profile offlinewould break every pipeline that doesn't define it.-stub-runyields an empty samplesheet, sinceSRA_TO_SAMPLESHEEThas no stub.Changes
modules/local/sra_ids_to_runinfo/main.nfassets/offline/<id>.runinfo.tsvvia heredoc rather than staging it, so the fixture reaches the task withoutprojectDirbeing mounted. Missing fixture callserror()naming the accession.modules/local/sra_fastq_ftp/main.nfgzip -n.md5sum -cstill runs.assets/offline/*.runinfo.tsv,README.mdENA_METADATA_FIELDS.fastq_md5repointed at the generated bytes; every other column keeps ENA's real value.tests/sra_offline.nf.test+ config, inputs, snapshotAcceptance criteria
.command.shfor both processes against the pre-change run: byte-identical.tests/sra_skip_fastq_download.nf.test— the one existing test that exercisesSRA_IDS_TO_RUNINFO's default path — passes against its untouched snapshot.tests/sra_offline.nf.testasserts 2 samplesheet rows and 3.fastq.gz, and walks the work dir asserting no.command.shcontainssra_ids_to_runinfo.pyor awgetinvocation.ERR1160845,ERR10677146against the offline one: the only difference is thefastq_md5column values. Header, column order, row order and all 31 other fields match exactly.md5sum -cstill runs offline and passes.gzip -nproduces the same md5 on the host and insidequay.io/biocontainers/wget:1.21.4(busybox gzip), so the pinned md5 holds across the container/host split.No offline runinfo fixture for accession 'ERR1160846' (looked for …).Out of scope, as specified: the
sratools,asperaandfastqdlroutes;nextflow_schema.json; the flow-api side that sets the switch.Pre-existing failure, not from this PR
tests/default.nf.testand every other pipeline-level test intests/already fail onflow-schema-simple: that branch made--inputrequire anaccessioncolumn header, but the tests still point at the headerlesssra_ids_test.csvon nf-core/test-datasets. Confirmed against a clean checkout of the branch. Left alone as outside this spec, but it means CI here is red regardless of this change.Also note the pipeline now needs Nextflow >= 25.04 (nf-schema 2.5.1).
🤖 Generated with Claude Code