Paper: VariantValidator/Mutalyzer head-to-head on the injection corpus

paper/Snakefile

@@ -75,7 +75,8 @@ GEN_FACTS  = "output/facts"
 # inputs it needs are absent the rule copies the committed EMPIRICAL file instead, so
 # a build always renders; `full` writes these back to EMPIRICAL after a good render.
 COMPUTED_FACTS = ["coverage.csv", "benchmark.csv", "cleaning.csv",
-                  "lovd_comparison.csv", "sources.csv",
+                  "lovd_comparison.csv", "vv_mutalyzer_comparison.csv",
+                  "sources.csv",
                   "version_stability.csv", "positional_drift.csv"]
 
 # FROZEN - measured once (or transcribed from published papers) and committed under
@@ -276,6 +277,31 @@ rule lovd_comparison:
             shutil.copy(f"{EMPIRICAL}/lovd_comparison.csv", output[0])
 
 
+rule vv_mutalyzer_comparison:
+    """Head-to-head: clean_hgvs() vs the VariantValidator and Mutalyzer public
+    REST services on the injection corpus (issue #112).
+
+    Same cases and scoring as the lovd_comparison rule, driven by
+    paper/scripts/vv_mutalyzer_head_to_head.py over the public APIs
+    (rest.variantvalidator.org, mutalyzer.nl). Remote services cannot be
+    version-pinned, so the facts record the service date and the version
+    metadata the APIs report (measured 2026-08-18, VariantValidator 4.0.1).
+    The live run needs network access and about an hour (requests are cached
+    and resumable in output/rest_cache/); opt in with --config
+    vv_mutalyzer=true. Otherwise the frozen measured constants from
+    paper/empirical_results/ are copied.
+    """
+    output: "output/facts/vv_mutalyzer_comparison.csv"
+    run:
+        import os, shutil
+        os.makedirs("output/facts", exist_ok=True)
+        if config.get("vv_mutalyzer", False):
+            shell(f"{PYTHON} paper/scripts/vv_mutalyzer_head_to_head.py")
+        else:
+            print("No vv_mutalyzer=true - copying the frozen measured service comparison facts")
+            shutil.copy(f"{EMPIRICAL}/vv_mutalyzer_comparison.csv", output[0])
+
+
 rule sources:
     """Count annotation releases ingested per consortium/build (the sources.*_releases
     facts in Methods) from generate_transcript_data/cdot_transcripts.yaml - the

paper/discussion.md

@@ -39,6 +39,24 @@ error mix, LOVD's top-ranked correction restored
 common single-token errors but not on free-text damage or cross-accession-family
 repairs; neither tool altered any valid input.
 
+The same corpus puts the sequence-aware services in context. Run over their public REST
+APIs ({{ vv_mutalyzer_comparison.service_date }}; Methods), VariantValidator's validated
+description recovered {{ vv_mutalyzer_comparison.vv_weighted_pct | dp(0) }}% of cases
+weighted by the production error mix, and Mutalyzer
+{{ vv_mutalyzer_comparison.mut_weighted_pct | dp(0) }}% (Supplementary Table S5). Each
+repairs the damage it can interpret and rejects the rest as invalid: whitespace and
+letter case for both, with VariantValidator also handling quotes, doubled colons,
+accession re-casing and swapped gene and transcript forms at 94 to 99% per category,
+capped by transcripts absent from its database rather than by the injected error.
+Neither falsely corrected a valid input. These services answer a different question from
+`clean_hgvs()`: they validate a description against the reference sequence, so a string
+that cannot be parsed is an error to report, not text to repair. The roles are
+converging, though. Since release 4.0.0 (June 2026) VariantValidator embeds the LOVD
+syntax checker [@VariantValidator400], and its API responses for input it cannot parse
+carry the checker's ranked suggestions alongside the validation error, although the
+service does not apply them. The LOVD measurement above therefore also characterises
+the syntax-repair layer surfaced by a current VariantValidator pipeline.
+
 Beyond HGVS resolution, the JSON representation is useful in its own right. It parses
 far faster than the GTF/GFF files it is built from, and the per-release JSON published
 on the GitHub releases page is a faster-loading drop-in for the corresponding GTF/GFF.

paper/empirical_results/vv_mutalyzer_comparison.csv

@@ -0,0 +1,2 @@
+service_date,vv_version,n_cases,vv_scored_cases,vv_coverage_pct,cdot_pct,vv_pct,mut_pct,cdot_weighted_pct,vv_weighted_pct,mut_weighted_pct,vv_refseq_pct,vv_ensembl_pct,mut_refseq_pct,mut_ensembl_pct,n_nonintronic,cdot_nonintronic_pct,vv_nonintronic_pct,mut_nonintronic_pct,originals_n,vv_originals_scored,vv_false_corrections,vv_rejected_valid,mut_false_corrections,mut_rejected_valid,mut_representation_changes,vv_lovd_suggestion_cases
+2026-08-18,4.0.1.dev7+gbdab9c72f,3419,3419,100.0,100.0,37.7,12.5,100.0,56.1,36.2,32.9,44.2,11.8,13.6,2478,100.0,36.9,15.5,998,998,0,37,0,112,2,1528

paper/methods.md

@@ -152,7 +152,23 @@ clean ClinVar strings. `lovd_head_to_head.py` runs the same injected cases throu
 ({{ lovd_comparison.lovd_version }}, run locally as a PHP CLI), scoring a case as
 recovered when the tool's output (for LOVD, its top-ranked correction) exactly matches
 the canonical target; the same false-correction check runs over the uncorrupted
-originals. Version-fallback safety is measured by `compute_version_stability.py` on
+originals.
+
+`vv_mutalyzer_head_to_head.py` extends the same cases and scoring rule to the two
+sequence-aware validation services, VariantValidator [@Freeman2018] and Mutalyzer
+[@Lefter2021], over their public REST APIs
+({{ vv_mutalyzer_comparison.service_date }}; remote services cannot be version-pinned,
+so the facts record the service date and the version metadata each API reports).
+VariantValidator requests are routed by accession family (its Ensembl transcripts live
+on a separate endpoint) and a case counts as recovered when the validated transcript
+description matches the target; for Mutalyzer a match by either its
+`corrected_description` or its `normalized_description` counts, since the normalizer
+may legitimately re-shift a representation. On the uncorrupted originals, a valid input
+the service *alters* is a false correction, while one it *rejects* (for example
+Mutalyzer's `EINTRONIC` for intronic positions on a transcript reference, or a
+transcript absent from VariantValidator's database) is counted separately as a
+validity or coverage position, matching how LOVD's flagged-invalid originals are
+reported. Version-fallback safety is measured by `compute_version_stability.py` on
 GRCh38, over a seeded {{ version_stability.sample_n | commas }}-accession sample of
 accessions cdot holds at two or more versions; the same run bins preserved coding bases
 by relative CDS position (Supplementary Figure S1).

paper/references.bib

@@ -43,6 +43,14 @@ @misc{LovdHgvsChecker
   note         = {Accessed 2026},
 }
 
+@misc{VariantValidator400,
+  author       = {{VariantValidator}},
+  title        = {{VariantValidator} release 4.0.0: integration of the {LOVD} {HGVS} syntax checker},
+  year         = {2026},
+  howpublished = {\url{https://github.com/openvar/variantValidator/releases/tag/v4.0.0}},
+  note         = {Released 5 June 2026},
+}
+
 @article{Cingolani2012,
   author  = {Cingolani, Pablo and Platts, Adrian and Wang, Le Lily and Coon, Melissa and Nguyen, Tung and Wang, Luan and Land, Susan J. and Lu, Xiangyi and Ruden, Douglas M.},
   title   = {A program for annotating and predicting the effects of single nucleotide polymorphisms, {SnpEff}},