Skip to content

Backfill Annotation

Jump to bottom
Dave Lawrence edited this page Aug 20, 2026 · 1 revision

Correcting or filling in individual annotation columns without re-running VEP.

Sometimes a single VariantAnnotation column is wrong or null across a whole annotation version — a data source renamed the field VEP was reading, or a column arrived after that version was annotated. Re-running the whole pipeline to recover one column costs weeks of VEP. Backfill annotation dumps the variants that are already annotated, lets you annotate them however suits that one column, and UPDATEs just the named columns back in.

Everything is driven by one management command: annotation_backfill_columns.

See issue #1675.

When to use this

  • A source renamed its INFO field. The driving case (#1673): COSMIC has called its sample count CNT (v95/97), SAMPLE_COUNT (v99) and GENOME_SCREEN_SAMPLE_COUNT (v101), so cosmic_count ends up null.
  • A column was fixed or added and you want existing rows to carry it.
  • You have a better/newer data file for one column than the version was annotated against.

This is the opposite of External Annotation at every step: it dumps variants that are annotated rather than ones awaiting annotation, it leaves AnnotationRun / range-lock state alone entirely, it UPDATEs named columns rather than INSERTing whole rows, and its input deliberately comes from a different annotation source than the one that produced the version.

Run it against a version whose annotation has finished. Variants annotated after the dump take their values from the live pipeline config, so those need the pipeline fixing rather than backfilling.

How it works

                 dump                    (external)                    import
  Annotated  ─────────►  VCF          ──► bcftools ──►  annotated VCF  ─────────►  UPDATE named columns
   variants     variant_id in INFO         or VEP        + INFO field             COPY → temp → UPDATE
  1. dump — write a VCF of the variants already annotated in a version, carrying variant_id in INFO.
  2. annotate — you annotate that VCF outside VariantGrid. bcftools annotate against the source VCF is the fast path; VEP works too.
  3. import — read the annotated VCF back and update the named columns in batches.

Updates go through a temp table: per batch (default 1,000,000 records), COPY the (variant_id, col…) rows in, ANALYZE, then UPDATE <partition> a SET col = t.col FROM tmp t WHERE a.variant_id = t.variant_id AND a.col IS DISTINCT FROM t.col. The UPDATE targets the version's partition table by name, so it never scans the other versions, and IS DISTINCT FROM means postgres only rewrites rows whose value actually changes.

Columns come from the VEPColumnDef registry

--columns names VariantGridColumn ids that appear in annotation.vep_columns.VEP_COLUMNS. The registry entry supplies:

  • the formatter — cosmic_count needs format_pick_highest_int because COSMIC arrives &-joined when several records overlap a variant; choice columns (impact, sift) map to their single-char DB values. Reusing the registry formatter is what makes a backfilled value identical to a pipeline-written one.
  • the expected INFO field, as the default source field.
  • which columns exist for this version — a column outside the version's columns_version / vep_version is rejected before anything is dumped, as is a typo.

The source field is overridable per column, since the naming varies by release:

--columns cosmic_count,cosmic_legacy_id
--info cosmic_count=GENOME_SCREEN_SAMPLE_COUNT

Use --csq cosmic_count=COSMIC_SAMPLE_COUNT instead to read the value off the PICK'd CSQ entry of a VEP-annotated file. Same registry, same formatter, different place to look.

Every named column is carried through one dump, one COPY and one UPDATE, so the row rewrite — the expensive part — happens once for the set rather than once per column.

Steps

The version defaults to the build's ACTIVE VariantAnnotationVersion; use --variant-annotation-version <pk> to name another (e.g. a HISTORICAL one).

1. Dump the annotated variants

python3 manage.py annotation_backfill_columns --dump \
    --genome-build GRCh38 \
    --columns cosmic_count \
    --only-missing \
    --output cosmic_backfill.vcf.gz
  • --only-missing restricts to rows where every --columns value is null — the common backfill shape, and much smaller than a full dump.
  • --min-variant-id / --max-variant-id split a large dump into pieces you can annotate in parallel.
  • A .gz output is bgzip, so it can be tabix indexed.

2. Index it

tabix -p vcf cosmic_backfill.vcf.gz

3. Annotate it externally

bcftools annotate \
    -a /data/annotation/VEP/annotation_data/GRCh38/Cosmic_GenomeScreensMutant_Normal_v101_GRCh38.vcf.gz \
    -c INFO/GENOME_SCREEN_SAMPLE_COUNT \
    -O z -o cosmic_backfill.annotated.vcf.gz \
    cosmic_backfill.vcf.gz

bcftools annotate matches on POS/REF/ALT and copies the INFO field across, leaving the variant_id the dump baked in untouched. You can rename on the way through with -c INFO/COSMIC_SAMPLE_COUNT:=INFO/GENOME_SCREEN_SAMPLE_COUNT instead of using --info at import.

4. Check what would change

python3 manage.py annotation_backfill_columns --import \
    --genome-build GRCh38 \
    --columns cosmic_count \
    --info cosmic_count=GENOME_SCREEN_SAMPLE_COUNT \
    --vcf cosmic_backfill.annotated.vcf.gz \
    --dry-run

--dry-run runs the COPY and counts rows against the same join predicate, then rolls back.

5. Import

Same command without --dry-run. Progress (records read / rows changed) is reported per batch.

  • --batch-size sets records per COPY/UPDATE cycle (default 1,000,000). A batch is held in memory as CSV.
  • --clear-missing writes NULL where the source is silent, making the file authoritative for the dumped variants — what a full re-annotation of that column wants. Without it (the default) a record the source says nothing about leaves the existing value alone, so a partial file fills in what it knows.

Worked example: COSMIC counts (#1673)

Backfilling cosmic_count on both ACTIVE versions from the COSMIC v101 VCFs already sitting in the annotation build dirs:

for BUILD in GRCh37 GRCh38; do
    COSMIC=/data/annotation/VEP/annotation_data/${BUILD}/Cosmic_GenomeScreensMutant_Normal_v101_${BUILD}.vcf.gz

    python3 manage.py annotation_backfill_columns --dump --genome-build ${BUILD} \
        --columns cosmic_count --only-missing --output ${BUILD}_cosmic.vcf.gz

    tabix -f -p vcf ${BUILD}_cosmic.vcf.gz

    bcftools annotate -a ${COSMIC} -c INFO/GENOME_SCREEN_SAMPLE_COUNT \
        -O z -o ${BUILD}_cosmic.annotated.vcf.gz ${BUILD}_cosmic.vcf.gz

    python3 manage.py annotation_backfill_columns --import --genome-build ${BUILD} \
        --columns cosmic_count --info cosmic_count=GENOME_SCREEN_SAMPLE_COUNT \
        --vcf ${BUILD}_cosmic.annotated.vcf.gz
done

On a dev box this took ~6 minutes per build:

version rows cosmic_count before after
GRCh37 5,489,033 0 621,482
GRCh38 8,427,888 57 747,038

Sanity checks worth doing

  • Compare the new count against a neighbouring column from the same source — here cosmic_legacy_id held 577k / 680k values from the v99 annotation, so 621k / 747k from a newer release is the right shape. A result an order of magnitude out means the annotation step matched almost nothing (usually contig naming, or an unindexed -a file).
  • Eyeball the top values. For COSMIC they should be the well known hotspots: BRAF V600E, KRAS G12D/G12V, IDH1 R132H, PIK3CA E545K/H1047R, TP53 R175H.

Gotchas

  • Contig names must match between your dump and the annotation source. VariantGrid writes contig names (1, 2, …), so a source using chr1 needs bcftools annotate --rename-chrs. See VCF chromosome / contig names.
  • Indels only match if they're represented the same way. bcftools annotate matches POS/REF/ALT exactly, so a source that isn't left-aligned/normalised the way our variants are will miss some indels. SNVs are unaffected.
  • This fixes history, not the future. Variants annotated after the dump get their values from the live pipeline config, so a renamed source field also needs a VEPColumnDef and a vep_config update.
  • A dump of a whole version is millions of records. It streams (server-side cursor) so memory stays flat, but budget the disk: ~40MB gzipped for 5.5M variants, plus the annotated copy.

See also

Clone this wiki locally