Variant Annotation adding new columns

VEP

VEP has a number of Plugins and it's quite easy to add a column from VCF, BED, GFF, GTF, BigWig - see VEP Custom.

I suggesting running VEP manually via command line, there are some test VCFs for GRCh37/GRCh38 in annotation/tests/test_data/

This is also useful to check how the output is formatted, and whether it varies per transcript.

VariantAnnotation Model

Determine whether the field varies per transcript (such as amino_acids/exon) or not (eg population frequency)

Add the column to AbstractVariantAnnotation (which is the base class of both VariantAnnotation and VariantTranscriptAnnotation) if you need a copy per transcript, otherwise add it to VariantAnnotation

To save space, use a choices field if it uses a limited number of text values.

You need to run python3 manage.py makemigrations annotation to generate the schema migration script after changing the model.

VariantGridColumn

If you want to display the annotation field on an analysis grid, you need to first create a new snpdb.VariantGridColumn record via data migration, eg:

python3 manage.py makemigrations snpdb --empty --name "variant_grid_column_new_exon_field"

Then add the new record (see _create_columns() in snpdb/migrations/0002_initial_data.py to see how initial default columns are created) eg:

VariantGridColumn.objects.create(grid_column_name='exon',
                                 variant_column='variantannotation__exon',
                                 annotation_level='T',
                                 label='Exon',
                                 description='Number(s) of affected exon(s)',
                                 model_field=True,
                                 queryset_field=True)

Users can add this column to their custom columns via the settings page, or if you'd like it added to default columns, create CustomColumn records in the data migration.

VariantGrid VEP pipeline

The command line for VEP is generated in annotation.vep_annotation.get_vep_command, driven by the VEP_COLUMNS definitions in annotation/vep_columns.py.

These used to be ColumnVEPField database rows populated by data migrations. That model was deleted (annotation/migrations/0129_delete_columnvepfield.py) — column definitions are now code, so adding one is an edit to vep_columns.py, not a data migration.

Add a VEPColumnDef entry to VEP_COLUMNS:

A plain CSQ field needs very little — this is the real exon entry:

VEPColumnDef(source_field='EXON', variant_grid_columns=('exon',),
             category=ColumnAnnotationCategory.GENE_ANNOTATIONS),

A plugin-sourced column gated to particular builds and a columns version looks like:

VEPColumnDef(
    source_field='MaveDB_score',
    variant_grid_columns=('mavedb_score',),
    category=ColumnAnnotationCategory.FUNCTIONAL_EFFECT,
    vep_plugin=VEPPlugin.MAVEDB,
    genome_builds=GRCH38,
    pipeline_types=STANDARD,
    min_columns_version=3,
    formatter=fmt.format_pick_lowest_float,
),

Useful fields on VEPColumnDef:

• source_field — the VEP CSQ / plugin / custom output field name
• variant_grid_columns — one or more VariantGridColumn destinations
• vep_plugin / vep_custom — which VEPPlugin / VEPCustom produces it (source_field_has_custom_prefix=True if the custom's label prefixes the field name)
• genome_builds / pipeline_types — empty means "all"
• min_columns_version / max_columns_version — gate on Annotation Column Versions
• min_vep_version / max_vep_version — gate on VEP release
• formatter — a value-cleaning callable, see annotation/vep_field_formatters.py

If you're adding a new column, bump columns_version and set min_columns_version to the new value so deployments still on older annotation data are unaffected.

If you're adding a Plugin or new custom VCF for the first time, you'll need to add to the VEPPlugin or VEPCustom enums (annotation/models/models_enums.py) and run an annotation model migration for the enum change. If the plugin requires annotation data, add a key to settings.ANNOTATION["GRCh37"]["vep_config"] and reference it from get_vep_command(). A column whose vep_config data file is None for a build is dropped automatically by has_data_files(), which is how builds opt out of plugins that don't apply to them.

If you want the new column to be exported via VCF (save a grid as VCF in an analysis), add a VCFInfoColumnDef entry to COLUMN_VCF_INFO in snpdb/vcf_export_columns.py. This also used to be a database table (ColumnVCFInfo, deleted in snpdb/migrations/0195_delete_columnvcfinfo.py); snpdb/tests/test_vcf_export_columns.py guards the invariants the table used to enforce.

Loading code

The VEP VCF file is processed by annotation.vcf_files.bulk_vep_vcf_annotation_inserter.BulkVEPVCFAnnotationInserter

If you need to format/modify a field from VCF into data, set formatter= on the VEPColumnDef (see annotation/vep_field_formatters.py for the shared formatters), or add a handler in _add_vep_field_handlers

Variant details

To display this new field on the variant details page, you need to add it to variantopedia/templates/variantopedia/variant_details.html

Note that variant details rows/sections are gated on whether the column is available in the deployment's annotation version (visible_columns), not on whether a particular variant has a value — so a variant with no value shows a blank row rather than the row disappearing.

Testing

There's a management command to run the currently configured VEP pipeline, and "--test" runs it over the test VCFs (add --sv for the structural variant pipeline):

python3 manage.py vep_run --test --genome-build=GRCh37
python3 manage.py vep_run --test --genome-build=GRCh38

This will re-generate the annotated test VCFs with the new fields, and you can add a new test for the column to annotation.tests.test_annotation_vcf. The test data is per columns version, eg annotation/tests/test_data/test_columns_version4_grch37.vep_annotated.vcf — so a new columns version needs its own generated test files and a TestCase with @override_settings(**get_fake_annotation_settings_dict(columns_version=N)).