🤖 Written by Claude
Split out of #1147, which was about somalier taking 43% of upload time. That headline problem is gone — the somalier reference data moved off EFS to local disk on vgaws on 2026-07-13 and the step went from a median of 202s to 33s (benchmarks in #1147). This issue covers the remaining ~30s.
What's left
A full somalier_vcf_id run is ~33s in steady state. Almost all of that is somalier ancestry, which re-reads the 2,504 1kg .somalier files on every run regardless of how much data the VCF actually has. VCFs with no genotype already return early and take ~3s, so the ~30s gap is entirely ancestry + relate on VCFs that may not have enough sites to give a meaningful answer.
Proposal
Add a minimum genotyped-sites threshold to settings.SOMALIER, and skip ancestry and relate when a VCF's extract falls below it. Somalier's own guidance is that relatedness and ancestry estimates need a reasonable number of the ~17.5k sites covered; a gene panel or a heavily filtered VCF can come in with very few and the result is noise we then store and display.
Points to settle:
- Where the count comes from —
somalier extract writes the number of sites to the .somalier file, so the threshold can be applied after extract (cheap) and before ancestry (expensive). That ordering is what makes this worth doing.
- What the threshold should be. Worth measuring the site counts across existing
SomalierVCFExtract records on vgaws before picking a number.
- How a skip is recorded —
ProcessingStatus.SKIPPED on the extract mirrors the existing no-genotype path, so the UI can say why there's no ancestry rather than showing a gap.
Related
An ancestry_enabled setting was also proposed in #1147 as a way to turn the expensive stage off wholesale. That's a blunter version of the same lever; a site-count threshold is probably the better default, so noting it here rather than opening a third issue.
Relevant code: snpdb/tasks/somalier_tasks.py (somalier_vcf_id, _somalier_ancestry, _somalier_relate), snpdb/models/models_somalier.py (SomalierConfig, SomalierVCFExtract).
🤖 Written by Claude
Split out of #1147, which was about somalier taking 43% of upload time. That headline problem is gone — the somalier reference data moved off EFS to local disk on vgaws on 2026-07-13 and the step went from a median of 202s to 33s (benchmarks in #1147). This issue covers the remaining ~30s.
What's left
A full
somalier_vcf_idrun is ~33s in steady state. Almost all of that issomalier ancestry, which re-reads the 2,504 1kg.somalierfiles on every run regardless of how much data the VCF actually has. VCFs with no genotype already return early and take ~3s, so the ~30s gap is entirely ancestry + relate on VCFs that may not have enough sites to give a meaningful answer.Proposal
Add a minimum genotyped-sites threshold to
settings.SOMALIER, and skip ancestry and relate when a VCF's extract falls below it. Somalier's own guidance is that relatedness and ancestry estimates need a reasonable number of the ~17.5k sites covered; a gene panel or a heavily filtered VCF can come in with very few and the result is noise we then store and display.Points to settle:
somalier extractwrites the number of sites to the.somalierfile, so the threshold can be applied after extract (cheap) and before ancestry (expensive). That ordering is what makes this worth doing.SomalierVCFExtractrecords on vgaws before picking a number.ProcessingStatus.SKIPPEDon the extract mirrors the existing no-genotype path, so the UI can say why there's no ancestry rather than showing a gap.Related
An
ancestry_enabledsetting was also proposed in #1147 as a way to turn the expensive stage off wholesale. That's a blunter version of the same lever; a site-count threshold is probably the better default, so noting it here rather than opening a third issue.Relevant code:
snpdb/tasks/somalier_tasks.py(somalier_vcf_id,_somalier_ancestry,_somalier_relate),snpdb/models/models_somalier.py(SomalierConfig,SomalierVCFExtract).