WDL call cache can end up with copies of input files (and steal the originals) when they pass through cached tasks

[adamnovak]

I was trying to run https://github.com/vgteam/vg_wdl/blob/6b0c72fe2f9000de8250fae8ffd93d94f4182ed2/workflows/acceptance_test.wdl where reference preparation/decompression is split out into a subworkflow. I ran it with an already-uncompressed reference, and with WDL call caching on.

The uncompressed reference made it out of the uncompressReferenceIfNeeded task, and correctly produced an entry in the MiniWDL call cache pointing at the input file path.

But for the new PrepareReference subworkflow, I got an entry in the MiniWDL call cache that didn't point at the original input file. Instead, it pointed at:

/private/groups/patenlab/anovak/trash/acceptance-test/miniwdl-cache/PrepareReference/z6fazypaopkvvg3e4ya73tm2t4tegno5/zwo7k4fjhn7m4upksml4736jx544eqs5/f63a3d1a-ac16-455d-81f8-d9a198425684/AcceptanceTest.PrepareReference.uncompressReferenceIfNeeded/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13v2.0-Nonly.fa

Somehow we've put the whole absolute path to the input file in there again, as a relative path.

On the same workflow run, I got a warning message about a WDLOutputsJob using too much space:

[2026-08-06T09:31:27-0700] [Thread-5 (statsAndLoggingAggregator)] [W] [toil.statsAndLogging] Got message from job at time 08-06-2026 09:31:27: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job 'WDLOutputsJob' kind-WDLOutputsJob/instance-v3zhroex v1 used 146.98% disk (2.9 GiB [3156275200B] used, 2.0 GiB [2147483648B] requested).

Note that I ran the workflow run with caching, and also it failed due to #5084 because exactly this reference file went missing from the cache while another job was trying to get a .dict for it.

I think to sort this out I need a test workflow calling that vg_wdl commit's PrepareReference as a subworkflow.

I ran more or less:

mkdir -p /private/groups/patenlab/anovak/trash/acceptance-test
cd /private/groups/patenlab/anovak/trash/acceptance-test

cat >acceptance-test.input.json <<'EOF'
{
  "AcceptanceTest.BASELINE_VG_DOCKER": "quay.io/vgteam/vg:ci-2975-d9e86410a8f082f80fcafb467949ea55c07902c8",
  "AcceptanceTest.BASELINE_VG_GIRAFFE_DOCKER": "quay.io/vgteam/vg:ci-2974-be241788a1d921954cab906455dedda98ee86717",
  "AcceptanceTest.BASELINE_VG_SURJECT_OPTIONS": "--read-length long",
  "AcceptanceTest.CANDIDATE_VG_DOCKER": "quay.io/vgteam/vg:ci-2974-be241788a1d921954cab906455dedda98ee86717",
  "AcceptanceTest.CANDIDATE_VG_SURJECT_OPTIONS": "--read-length long --prune-tail-region",
  "AcceptanceTest.INPUT_READ_FILE_1": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/reads/real/hifi/HG002/HG002Revio_hg002v1.0.1_hifi_revio_pbmay24.pri.unshuffled.full.fq.gz",
  "AcceptanceTest.PAIRED_READS": false,
  "AcceptanceTest.GIRAFFE_PRESET": "hifi",
  "AcceptanceTest.GBZ_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/graphs/hprc-v2.1-mc-chm13-eval-sampled16o_fragmentlinked-for-real-hifi-HG002-full.gbz",
  "AcceptanceTest.MIN_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/graphs/hprc-v2.1-mc-chm13-eval-sampled16o_fragmentlinked-for-real-hifi-HG002-full.k31.w50.W.path.min",
  "AcceptanceTest.ZIPCODES_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/graphs/hprc-v2.1-mc-chm13-eval-sampled16o_fragmentlinked-for-real-hifi-HG002-full.k31.w50.W.path.zipcodes",
  "AcceptanceTest.DIST_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/graphs/hprc-v2.1-mc-chm13-eval-sampled16o_fragmentlinked-for-real-hifi-HG002-full.dist",
  "AcceptanceTest.SAMPLE_NAME": "HG002",
  "AcceptanceTest.TRUTH_VCF": "https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/analysis/NIST_HG002_DraftBenchmark_defrabbV0.018-20240716/CHM13v2.0_HG2-T2TQ100-V1.1.vcf.gz",
  "AcceptanceTest.TRUTH_VCF_INDEX": "https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/analysis/NIST_HG002_DraftBenchmark_defrabbV0.018-20240716/CHM13v2.0_HG2-T2TQ100-V1.1.vcf.gz.tbi",
  "AcceptanceTest.EVALUATION_REGIONS_BED": "https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/analysis/NIST_HG002_DraftBenchmark_defrabbV0.018-20240716/CHM13v2.0_HG2-T2TQ100-V1.1_smvar.benchmark.bed",
  "AcceptanceTest.RESTRICT_REGIONS_BED": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13v2.0-Nonly.fa.callable.from.chm13.bed",
  "AcceptanceTest.PAR_REGIONS_BED_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13v2.0-Nonly_PAR.bed",
  "AcceptanceTest.HAPLOID_CONTIGS": ["chrX", "chrY"],
  "AcceptanceTest.REFERENCE_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13v2.0-Nonly.fa",
  "AcceptanceTest.PATH_LIST_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13-pansn-Nonly.fa.paths.callable.txt",
  "AcceptanceTest.REFERENCE_PREFIX": "chm13#0#",
  "AcceptanceTest.HAPLOTYPE_SAMPLING": false,
  "AcceptanceTest.DV_GPU_DOCKER": "gcr.io/deepvariant-docker/deepvariant:head756846963-gpu",
  "AcceptanceTest.DV_NO_GPU_DOCKER": "gcr.io/deepvariant-docker/deepvariant:head756846963",
  "AcceptanceTest.BASELINE_VG_DOCKER": "quay.io/vgteam/vg:v1.64.0",
  "AcceptanceTest.CANDIDATE_VG_DOCKER": "quay.io/vgteam/vg:v1.64.0",
  "AcceptanceTest.LEFTALIGN_BAM": true,
  "AcceptanceTest.REALIGN_INDELS": false,
  "AcceptanceTest.DV_MODEL_TYPE": "PACBIO",
  "AcceptanceTest.OUTPUT_BAM": true,
  "AcceptanceTest.OUTPUT_GAF": true,
  "AcceptanceTest.CALL_CORES": 32,
  "AcceptanceTest.CALL_MEM": 200,
  "AcceptanceTest.EVAL_MEM": 120,
  "AcceptanceTest.MAKE_EXAMPLES_MEM": 50
}
EOF

MINIWDL__CALL_CACHE__GET=true MINIWDL__CALL_CACHE__PUT=true MINIWDL__CALL_CACHE__DIR=/private/groups/patenlab/anovak/trash/acceptance-test/miniwdl-cache toil-wdl-runner "https://github.com/vgteam/vg_wdl/raw/refs/heads/acceptance-testing/workflows/acceptance_test.wdl" --input /private/groups/patenlab/anovak/trash/acceptance-test/acceptance-test.input.json --batchLogsDir /private/groups/patenlab/anovak/trash/acceptance-test/logs --jobStore /private/groups/patenlab/anovak/trash/acceptance-test/tree --outputFile /private/groups/patenlab/anovak/trash/acceptance-test/out.json --outputDirectory /private/groups/patenlab/anovak/trash/acceptance-test/out --logFile /private/groups/patenlab/anovak/trash/acceptance-test/log.txt --batchSystem slurm --slurmTime 11:59:59 --caching=True --stats

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1858

[adamnovak]

Something went very wrong here, because actually my input file at /private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13v2.0-Nonly.fa was removed, and the file was recreated inside the call cache.

stat /private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references

  File: /private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references
  Size: 96        	Blocks: 0          IO Block: 65536  directory
Device: 2eh/46d	Inode: 1099575679983  Links: 2
Access: (2775/drwxrwsr-x)  Uid: (30065/ anovak1)   Gid: (  609/patenlab)
Access: 2024-03-08 18:36:48.148541905 -0800
Modify: 2026-08-06 09:28:30.981795938 -0700
Change: 2026-08-06 09:28:30.981795938 -0700
 Birth: 2024-03-08 18:36:48.148541905 -0800

stat /private/groups/patenlab/anovak/trash/acceptance-test/miniwdl-cache/PrepareReference/z6fazypaopkvvg3e4ya73tm2t4tegno5/zwo7k4fjhn7m4upksml4736jx544eqs5/f63a3d1a-ac16-455d-81f8-d9a198425684/AcceptanceTest.PrepareReference.uncompressReferenceIfNeeded/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13v2.0-Nonly.fa

  File: /private/groups/patenlab/anovak/trash/acceptance-test/miniwdl-cache/PrepareReference/z6fazypaopkvvg3e4ya73tm2t4tegno5/zwo7k4fjhn7m4upksml4736jx544eqs5/f63a3d1a-ac16-455d-81f8-d9a198425684/AcceptanceTest.PrepareReference.uncompressReferenceIfNeeded/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13v2.0-Nonly.fa
  Size: 3156259703	Blocks: 6164570    IO Block: 4194304 regular file
Device: 2eh/46d	Inode: 1101849375863  Links: 1
Access: (0644/-rw-r--r--)  Uid: (30065/ anovak1)   Gid: (  609/patenlab)
Access: 2026-08-06 09:31:20.537383953 -0700
Modify: 2026-08-06 09:31:22.865488183 -0700
Change: 2026-08-06 09:31:22.865488183 -0700
 Birth: 2026-08-06 09:31:20.537383953 -0700

[adamnovak]

The problem with WDLOutputsJob using too much disk can happen without any files going missing from the Toil cache or being moved into the WDL call cache.

I ran:

cat >trash/test-input.json <<'EOF'
{
  "PrepareReference.GBZ_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/graphs/hprc-v2.1-mc-chm13-eval-sampled16o_fragmentlinked-for-real-hifi-HG002-full.gbz",
  "PrepareReference.REFERENCE_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13v2.0-Nonly.fa",
  "PrepareReference.PATH_LIST_FILE": "/private/groups/patenlab/anovak/projects/hprc/lr-giraffe/references/chm13-pansn-Nonly.fa.paths.callable.txt",
  "PrepareReference.REFERENCE_PREFIX": "chm13#0#"
}
EOF

rm -Rf trash/test-tree
rm -Rf trash/test-cache

MINIWDL__CALL_CACHE__GET=true MINIWDL__CALL_CACHE__PUT=true MINIWDL__CALL_CACHE__DIR=`pwd`/trash/test-cache toil-wdl-runner "https://raw.githubusercontent.com/vgteam/vg_wdl/b028a8b44d2e13f82faf71a35c51c0ba6ee07f9f/workflows/internal/prepare_reference.wdl" --input trash/test-input.json --jobStore ./trash/test-tree --outputFile trash/test-out.json --outputDirectory trash/test-out --logFile trash/test-log.txt --batchSystem slurm --slurmTime 11:59:59 --caching=True --stats --logDebug

And I got a normal WDL call cache entry for PrepareReference, not pointing at a moved-in input reference file. But I also got the WDLOutputsJob at the end devirtualizing all the workflow outputs and then complaining about space:
[2026-08-06T10:43:29-0700] [MainThread] [W] [toil.fileStores.abstractFileStore] LOG-TO-LEADER: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job 'WDLOutputsJob' kind-WDLOutputsJob/instance-26sw7jx4 v1 used 146.98% disk (2.9 GiB [3156275200B] used, 2.0 GiB [2147483648B] requested).

[adamnovak]

OK, I have a way to replicate this.

1. Make sure there's a trash directory to work in.
2. Prepare a WDL call cache where the task that passes the inout file through is cached, but the tasks that use it afterward, and the whole workflow, are not:

touch trash/fake.gbz
cat >trash/test.fa <<'EOF'
>chr1
ACGCTCGATCGCATCAGTACGCATCAT
EOF
echo "chr1" >trash/paths.txt

cat >trash/test-input.json <<'EOF'
{
  "PrepareReference.GBZ_FILE": "trash/fake.gbz",
  "PrepareReference.REFERENCE_FILE": "trash/test.fa",
  "PrepareReference.PATH_LIST_FILE": "trash/paths.txt"
}
EOF

rm -Rf trash/test-tree
rm -Rf trash/test-cache

MINIWDL__CALL_CACHE__GET=true MINIWDL__CALL_CACHE__PUT=true MINIWDL__CALL_CACHE__DIR=`pwd`/trash/test-cache toil-wdl-runner "https://raw.githubusercontent.com/vgteam/vg_wdl/b028a8b44d2e13f82faf71a35c51c0ba6ee07f9f/workflows/internal/prepare_reference.wdl" --input trash/test-input.json --jobStore ./trash/test-tree --outputFile trash/test-out.json --outputDirectory trash/test-out --logFile trash/test-log.txt --caching=True --logDebug

tree trash/test-cache

rm -Rf trash/test-cache/PrepareReference trash/test-cache/indexReference

3. Use that call cache to run the workflow again:

rm -Rf trash/test-tree

MINIWDL__CALL_CACHE__GET=true MINIWDL__CALL_CACHE__PUT=true MINIWDL__CALL_CACHE__DIR=`pwd`/trash/test-cache toil-wdl-runner "https://raw.githubusercontent.com/vgteam/vg_wdl/b028a8b44d2e13f82faf71a35c51c0ba6ee07f9f/workflows/internal/prepare_reference.wdl" --input trash/test-input.json --jobStore ./trash/test-tree --outputFile trash/test-out.json --outputDirectory trash/test-out --logFile trash/test-log.txt --caching=True --logDebug

4. Inspect the cache and the input file. Note that now the cache entry for the workflow has a deeply nested path, and the input file is gone.

tree trash/test-cache

ls -lah trash/test.fa

I also tried running with the input file in a non-writable directory, in hopes of getting an error to happen in the code that was deleting it. This made the input file stick around, with the copy still created in the call cache at the deeply nested path, but didn't produce an exception at the offending code that wants to steal it.

[adamnovak]

With Toil's file store caching turned off, I still get a copy in the WDL call cache at a deeply nested path, but my input file survives. So this might have something to do with the logic in the CachingFileStore to give away cached copies?

[adamnovak]

OK, here's a tiny way to reproduce the problem:

cat >break-cache.wdl <<'EOF'
version 1.0

workflow BreakCache {
    input {
        File FILE_TO_STEAL
    }

    call passThroughTask {
        input:
        file_in=FILE_TO_STEAL
    }

    output {
        File file_out = passThroughTask.file_out
    }
}

task passThroughTask {
    input {
        File file_in
    }

    command <<<
        true
    >>>
    output {
        File file_out = file_in
    }
    runtime {
        cpu: 1
        memory: "2 GB"
        disks: "local-disk 2 SSD"
    }
}
EOF

mkdir -p trash

touch trash/to_steal.txt

cat >trash/test-input.json <<'EOF'
{
  "BreakCache.FILE_TO_STEAL": "trash/to_steal.txt"
}
EOF

rm -Rf trash/test-tree
rm -Rf trash/test-cache

MINIWDL__CALL_CACHE__GET=true MINIWDL__CALL_CACHE__PUT=true MINIWDL__CALL_CACHE__DIR=`pwd`/trash/test-cache toil-wdl-runner ./break-cache.wdl --input trash/test-input.json --jobStore ./trash/test-tree --outputFile trash/test-out.json --outputDirectory trash/test-out --logFile trash/test-log-setup.txt --caching=True --logCritical --clean=always

rm -Rf trash/test-cache/BreakCache

MINIWDL__CALL_CACHE__GET=true MINIWDL__CALL_CACHE__PUT=true MINIWDL__CALL_CACHE__DIR=`pwd`/trash/test-cache toil-wdl-runner ./break-cache.wdl --input trash/test-input.json --jobStore ./trash/test-tree --outputFile trash/test-out.json --outputDirectory trash/test-out --logFile trash/test-log-run.txt --caching=True --logDebug --clean=always

ls -lah trash/to_steal.txt

[adamnovak]

OK, what's going on here is that, when we find and import the cache entry, we follow it to the input file and then writeGlobalFile the input file's path from within a job.

This starts by making a cache database entry for the file, and a ref to it:

toil/src/toil/fileStores/cachingFileStore.py

Lines 1235 to 1248 in c6c2b1e

	# Create a file in uploadable state and a reference, in the same transaction.
	# Say the reference is an immutable reference
	self._write(
	[
	(
	"INSERT INTO files VALUES (?, ?, ?, ?, ?)",
	(fileID, cachePath, fileSize, "uploadable", me),
	),
	(
	"INSERT INTO refs VALUES (?, ?, ?, ?)",
	(absLocalFileName, fileID, creatorID, "immutable"),
	),
	]
	)

Later we change the ref to mutable and delete the file database record, if the file isn't coming from a Toil-managed temp directory.

And then the ref stays around until we get to _removeJob to get rid of the job, at which point we delete all the files in ref entries for the job:

toil/src/toil/fileStores/cachingFileStore.py

Lines 1014 to 1022 in c6c2b1e

	for row in cls._static_read(
	cur, "SELECT path FROM refs WHERE job_id = ?", (jobID,)
	):
	try:
	# Delete all the reference files.
	os.unlink(row[0])
	except OSError:
	# May not exist
	pass

So when you writeGlobalFile into a caching job store, we delete the file you wrote, eventually. We only notice this when we're dealing with things like WDL call caching, because usually we load up the job store at the beginning with the inputs not via the file store, and only writeGlobalFile job outputs.

[adamnovak]

This doesn't seem to have actually gone away; I still have real workflows pulling my reference genome input from the cache and stealing it from the actual data directory.