Biodyn-AI · Kendiukhov · Jul 1, 2026
diff --git a/README.md b/README.md
@@ -2,6 +2,9 @@
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 
+This repository contains the code, data, and analysis scripts accompanying the paper:
+
+> Kendiukhov, I. (2026). *External Biological Validation of Foundation-Model Gene Regulatory Networks: Perturbation Bridging, ChIP-Seq Binding Support, and Essential-Gene Agreement.* University of Tubingen.
 
 ## Overview
 
@@ -152,7 +155,44 @@ pdflatex main.tex  # second pass for references
 | Essentiality | Significant TFs (lung/kidney) | 0 |
 | Essentiality | Cross-tissue concordance | rho = 0.15-0.31 |
 | Synthesis | Cross-modality correlations | All |rho| < 0.2 |
+| Cross-model | Foundation models benchmarked | 8 (six architectures) |
+| Cross-model | ENETS2 AUROC range (embedding-cosine) | 0.499-0.549 (chance 0.50) |
+| Cross-model | Geneformer vs scGPT | not significant (paired Wilcoxon p = 0.67) |
+
+### Cross-model foundation-model benchmark (revision)
+
+A broadened panel of eight single-cell foundation models spanning six architectures
+(scGPT, Geneformer V1-10M/V2-104M/V2-316M, AIDO.Cell-100M, scFoundation, tGPT, and a
+frozen ESM2-3B/UCE control) is benchmarked under a uniform embedding-cosine edge
+derivation, with paired significance tests. All models cluster just above chance and the
+leading models are statistically indistinguishable. A scGPT attention-vs-embedding
+concordance check confirms the finding is robust to edge-derivation choice.
+
+```bash
+# 8-model cross-model GRN benchmark + significance tests
+python revision/scripts/crossmodel_foundation_benchmark.py
+
+# scGPT attention vs embedding-cosine concordance (immune context)
+python revision/scripts/attention_embedding_concordance.py
+```
 
+Outputs: `revision/outputs/crossmodel_foundation_*.{csv,md}`,
+`revision/outputs/crossmodel_{pairwise_significance,vs_chance,per_tf_auroc}.csv`,
+`revision/outputs/attention_embedding_concordance.{md,json}`,
+`revision/outputs/figures/fig_crossmodel_foundation.pdf`.
+
+## Citation
+
+```bibtex
+@article{kendiukhov2026external,
+  title={External Biological Validation of Foundation-Model Gene Regulatory
+         Networks: Perturbation Bridging, ChIP-Seq Binding Support, and
+         Essential-Gene Agreement},
+  author={Kendiukhov, Ihor},
+  year={2026},
+  institution={University of T{\"u}bingen}
+}
+```
 
 ## License
 

diff --git a/revision/outputs/attention_embedding_concordance.json b/revision/outputs/attention_embedding_concordance.json
@@ -0,0 +1,13 @@
+{
+  "n_tf": 93,
+  "n_candidate_pairs": 268863,
+  "n_attended_pairs": 156502,
+  "mean_auroc_cosine": 0.5690015468299076,
+  "mean_auroc_attention": 0.5356112862071472,
+  "edge_score_spearman_all": -0.0030699394736626836,
+  "edge_score_spearman_all_p": 0.11142484162581998,
+  "edge_score_spearman_attended": 0.12831479139837187,
+  "edge_score_spearman_attended_p": 0.0,
+  "per_tf_auroc_spearman": -0.005554976711863202,
+  "per_tf_auroc_spearman_p": 0.9578545053610776
+}
diff --git a/revision/outputs/attention_embedding_concordance.md b/revision/outputs/attention_embedding_concordance.md
@@ -0,0 +1,15 @@
+# scGPT attention vs embedding-cosine edge concordance (immune context)
+
+- Immune genes with attention + scGPT embedding: 8354
+- ENETS2 (restricted): 93 TFs, 268863 candidate TF-target pairs (156502 actually attended)
+
+## GRN recovery is similarly weak under both derivations
+- Mean per-TF AUROC, **embedding cosine**: 0.5690
+- Mean per-TF AUROC, **attention**: 0.5356
+- Per-TF AUROC agreement across derivations: Spearman rho = -0.006 (p = 9.58e-01)
+
+## Edge-score concordance
+- All candidate pairs: Spearman rho = -0.003 (p = 1.11e-01)
+- Attended pairs only: Spearman rho = 0.128 (p = 0.00e+00)
+
+**Interpretation (honest):** for scGPT on the immune context, both edge derivations give only weak ENETS2 recovery (attention AUROC ~0.54, embedding cosine ~0.57; chance 0.50), with attention if anything slightly weaker. The two derivations are largely non-redundant: edge scores are only weakly rank-correlated among attended pairs (Spearman ~0.13) and per-TF AUROC agreement is negligible (~0). Two consequences: (i) the weak-recovery conclusion is ROBUST to the choice of edge derivation -- using embedding cosine for the eight-model panel does not understate foundation-model GRN quality relative to attention, so the panel is a fair test; and (ii) attention and embedding edges are not interchangeable edge-for-edge, consistent with the paper's broader theme that these rankings are unstable and derivation-dependent. We do NOT claim embedding cosine reproduces attention edge-by-edge.
diff --git a/revision/outputs/attention_embedding_per_tf.csv b/revision/outputs/attention_embedding_per_tf.csv
@@ -0,0 +1,94 @@
+tf,n_targets,auroc_cos,auroc_attn
+ATF3,70,0.46796981820023287,0.6441839266724059
+BATF,80,0.6389407684098185,0.6096229099964425
+BCL11A,38,0.58975778036047,0.6046451565295994
+BCL3,85,0.5492096767431135,0.5731751289254119
+BDP1,33,0.4905634370294973,0.5714103950632992
+BHLHE40,6,0.6517042172154823,0.3978625072212594
+BRCA1,54,0.4826629590464628,0.658269037454797
+BRF1,14,0.6615273846764984,0.5913898406077759
+CEBPB,101,0.6114234004045567,0.5246850491500763
+CTBP2,108,0.5076456262227013,0.20722208913908519
+CTCFL,61,0.5146034872270173,0.45428372820483115
+E2F1,223,0.5773693870553512,0.5
+E2F4,131,0.7195347936718665,0.4607672308883726
+E2F6,98,0.5477688390071389,0.5518004194158865
+EBF1,141,0.5710070921985816,0.5608691166989039
+EGR1,141,0.5255035460992907,0.5868149580915538
+ELF1,93,0.5407587600974583,0.714711737262407
+ELK4,46,0.5360816077023001,0.625254068923359
+ESR1,104,0.5391757058872237,0.38834492561618505
+ESRRA,20,0.6629745733194008,0.5261842563566701
+ETS1,63,0.4748883051570463,0.6879139444556701
+FOS,205,0.5581007209923179,0.5568421626137334
+FOSL1,48,0.5793982295697033,0.6026717669128854
+FOSL2,37,0.5960813651773709,0.498494289664577
+FOXA1,223,0.5452548389482389,0.5
+FOXA2,138,0.5507483272530099,0.5
+GABPA,88,0.606075471086174,0.5660574708915773
+GATA1,106,0.7005115002879306,0.5022627959757461
+GATA2,188,0.586991601136641,0.5
+GATA3,10,0.5784449843804235,0.401249566122874
+GTF2B,95,0.5498381145998043,0.6200248475265416
+GTF2F1,26,0.4825211437776883,0.4825681299503289
+HNF4A,116,0.6988070829450139,0.5
+HNF4G,68,0.6281698651830551,0.5
+HSF1,32,0.5313483735571879,0.6246611577474641
+IRF1,17,0.6359449834213434,0.5292480248884522
+IRF3,34,0.5397063970845601,0.5641818855648665
+IRF4,9,0.7121983190685481,0.5387076875626493
+JUN,112,0.6385165527168046,0.5467215853595846
+JUND,186,0.6374853417605788,0.511935285115179
+MAFF,69,0.47220082375537953,0.5357850840702966
+MAFK,100,0.5167036904335364,0.48175564313866
+MAX,139,0.5665676760916848,0.6485368391333445
+MEF2A,38,0.649500986957404,0.4997878502776394
+MEF2C,17,0.7322035285930657,0.6227843955953989
+MXI1,21,0.4453625352580056,0.6311929649908744
+MYC,228,0.6431919547272236,0.6196678327437068
+NANOG,8,0.6501907734998266,0.5
+NFE2,36,0.5225530258805215,0.492717454757735
+NFKB1,99,0.6913403374721427,0.5068702787184163
+NFYA,116,0.5807082945013979,0.5989624106865487
+NFYB,143,0.5903594222371514,0.57969814028766
+NR2C2,15,0.580551692165044,0.5000463606861382
+NR3C1,147,0.44853830745126044,0.4397336923107435
+NRF1,129,0.5845079119164295,0.6580979966208061
+PAX5,67,0.4042112384254365,0.5803031584288191
+PBX3,126,0.5261258933953328,0.5908636872470506
+POLR3A,20,0.536433298502264,0.6063566701497736
+POU2F2,51,0.5193661971830986,0.5571251035625517
+PPARGC1A,17,0.5632240370051987,0.5774079986900813
+PRDM1,9,0.6147351376359009,0.44213123602436577
+REST,115,0.2607286054379151,0.2683091091341937
+RFX5,58,0.5881117859707633,0.5890551018172523
+RXRA,52,0.5807367166120248,0.41929037851898016
+SETDB1,57,0.6746524037687727,0.558605405539254
+SIX5,105,0.5824804293576727,0.5
+SMARCA4,77,0.4471196891239535,0.5404932665060598
+SMARCB1,71,0.5216012386374989,0.5451703126560783
+SMARCC1,6,0.36984402079722706,0.27053726169844017
+SP1,170,0.5144151155500789,0.540450094039821
+SP2,66,0.6019844462322338,0.6539769375167606
+SPI1,174,0.6779747777932896,0.577041530762039
+SREBF1,59,0.6839928660346644,0.4832064540840755
+SREBF2,28,0.7380370241005938,0.5018836385409909
+SRF,118,0.5507496623005128,0.5793089537733715
+STAT1,122,0.6499801668353966,0.5807742630647272
+STAT2,59,0.8351886431102173,0.5662046346835201
+STAT3,157,0.6723263084815416,0.4154222133175534
+TAL1,94,0.7069048144288332,0.5
+TCF12,138,0.41700226893454834,0.3167164147675527
+TCF4,167,0.4991382873020479,0.4320807723759529
+TFAP2A,51,0.5237089201877934,0.5
+TFAP2C,78,0.4972016370878795,0.5
+TRIM28,25,0.4767201674808095,0.5272993719469644
+USF1,135,0.5415416868247056,0.6472114712680751
+USF2,74,0.6265290849955387,0.6622389162325265
+YY1,125,0.5669154013015184,0.7422863340563992
+ZBTB33,62,0.4736542035827091,0.6027149682436516
+ZBTB7A,105,0.6312207294978293,0.5439886507366766
+ZEB1,43,0.4315553958714397,0.6543637313822837
+ZNF263,161,0.5020635679020773,0.5157429526994745
+ZNF274,31,0.5419918790886533,0.47894202571621924
+ZZZ3,12,0.4752807687854579,0.44367257149473194
diff --git a/revision/outputs/crossmodel_foundation_grn.csv b/revision/outputs/crossmodel_foundation_grn.csv
@@ -0,0 +1,25 @@
+benchmark,model,n_tf,mean_auroc,median_auroc,ci_lo,ci_hi,mean_aupr
+ENETS2,scGPT-53M,100,0.5494326622648991,0.5418986483942587,0.5332951527166857,0.5650164715686605,0.04596526487391128
+ENETS2,Geneformer-V1-10M,100,0.5125606553131717,0.5014669967057148,0.5020538795130577,0.5231649854056446,0.031899203253841685
+ENETS2,Geneformer-V2-104M,100,0.5466956871926825,0.5339587440854152,0.5309662841363142,0.5626957647552878,0.040864614098919734
+ENETS2,Geneformer-V2-316M,100,0.5467685425420851,0.5310556894804022,0.5301785379237376,0.5638016606536655,0.03884448358720428
+ENETS2,AIDO.Cell-100M,100,0.5170576431911197,0.528836433078749,0.5010993551286331,0.5322430614797377,0.04119164842681117
+ENETS2,scFoundation,100,0.49917762016889766,0.5064908061463431,0.48681870845740105,0.5116663764556119,0.02992368764440002
+ENETS2,tGPT,100,0.5020363219214278,0.5181761898314952,0.48583231472699034,0.5174878539873239,0.03325041585434943
+ENETS2,ESM2-3B (UCE ctrl),100,0.503659134396662,0.49822286830401585,0.4931061501304203,0.5135542846636189,0.03129399856133213
+TRRUST,scGPT-53M,187,0.5815440507964921,0.5817291455589327,0.5653115722143042,0.5981810131545107,0.041708879840093846
+TRRUST,Geneformer-V1-10M,187,0.5488698660152586,0.5434497967059472,0.5348001483801381,0.5627453418147745,0.027928054119498554
+TRRUST,Geneformer-V2-104M,187,0.5944182279498889,0.5818200957945155,0.5791128210627441,0.6094769088381536,0.03988175718201017
+TRRUST,Geneformer-V2-316M,187,0.5980218888322912,0.5997477136549985,0.5825165034773179,0.6128620133761471,0.04062087519040739
+TRRUST,AIDO.Cell-100M,187,0.5527773333842325,0.5519878985072922,0.5377427575767855,0.5674889649502461,0.03481218357754472
+TRRUST,scFoundation,187,0.49480669007590083,0.49595621021556624,0.4824791205477896,0.5081868496232398,0.02504397173894239
+TRRUST,tGPT,187,0.512917826579288,0.5207743153918791,0.49898686196701997,0.5259121319187077,0.02695231780460786
+TRRUST,ESM2-3B (UCE ctrl),187,0.5066934367697402,0.5072993643739342,0.49525804640291266,0.5179652900020358,0.027423570461680695
+DoRothEA,scGPT-53M,292,0.4901861797824431,0.48689659233618754,0.4756649364223634,0.5046198807941888,0.04507570070277037
+DoRothEA,Geneformer-V1-10M,292,0.49819338057498636,0.4977767555632047,0.493610254397891,0.5024744747170002,0.037570122838384575
+DoRothEA,Geneformer-V2-104M,292,0.5393454663582811,0.539065680120769,0.5322829135491288,0.5460034200232716,0.04449398565678777
+DoRothEA,Geneformer-V2-316M,292,0.5377487653276319,0.5346220591885535,0.5303402650217627,0.5456299879151751,0.043856346120478534
+DoRothEA,AIDO.Cell-100M,292,0.519614876415865,0.5200074911805977,0.5026590132248635,0.5366022009270139,0.05384749560975616
+DoRothEA,scFoundation,292,0.4609524019011883,0.43672664432669017,0.44405096173983527,0.47741771940006983,0.04642858883208448
+DoRothEA,tGPT,292,0.5243640654525691,0.5251875434317306,0.5141548461001917,0.5343095160384576,0.04158704389373591
+DoRothEA,ESM2-3B (UCE ctrl),292,0.5803024685888699,0.5801095339243614,0.5746550643599617,0.5859504125493026,0.05381217153362102
diff --git a/revision/outputs/crossmodel_foundation_summary.md b/revision/outputs/crossmodel_foundation_summary.md
@@ -0,0 +1,49 @@
+# Cross-model foundation-model GRN benchmark (BMC revision)
+
+Uniform embedding-cosine edge derivation, common gene universe = 17874 genes, 8 models, 3 benchmarks. Positives = TF's benchmark targets, negatives = other listed targets; per-TF AUROC, cross-TF comparator.
+
+## Mean AUROC per model (ENETS2, primary benchmark)
+
+| Model | n TF | mean AUROC | 95% CI | vs chance p |
+|---|---|---|---|---|
+| scGPT-53M | 100 | 0.5494 | [0.5333, 0.5650] | 3.49e-08 |
+| Geneformer-V2-316M | 100 | 0.5468 | [0.5302, 0.5638] | 3.81e-07 |
+| Geneformer-V2-104M | 100 | 0.5467 | [0.5310, 0.5627] | 1.21e-07 |
+| AIDO.Cell-100M | 100 | 0.5171 | [0.5011, 0.5322] | 7.47e-03 |
+| Geneformer-V1-10M | 100 | 0.5126 | [0.5021, 0.5232] | 8.43e-02 |
+| ESM2-3B (UCE ctrl) | 100 | 0.5037 | [0.4931, 0.5136] | 6.16e-01 |
+| tGPT | 100 | 0.5020 | [0.4858, 0.5175] | 1.81e-01 |
+| scFoundation | 100 | 0.4992 | [0.4868, 0.5117] | 8.80e-01 |
+
+## Geneformer vs scGPT (the reviewer's specific question)
+
+- **ENETS2**: scGPT-53M (0.5494) vs Geneformer-V2-316M (0.5468); mean per-TF diff = +0.0027; paired Wilcoxon p = 0.670 (BH 0.721), n = 100 TFs.
+
+## Full per-benchmark summary
+
+| benchmark   | model              |   n_tf |   mean_auroc |   median_auroc |   ci_lo |   ci_hi |   mean_aupr |
+|:------------|:-------------------|-------:|-------------:|---------------:|--------:|--------:|------------:|
+| ENETS2      | scGPT-53M          |    100 |       0.5494 |         0.5419 |  0.5333 |  0.565  |      0.046  |
+| ENETS2      | Geneformer-V1-10M  |    100 |       0.5126 |         0.5015 |  0.5021 |  0.5232 |      0.0319 |
+| ENETS2      | Geneformer-V2-104M |    100 |       0.5467 |         0.534  |  0.531  |  0.5627 |      0.0409 |
+| ENETS2      | Geneformer-V2-316M |    100 |       0.5468 |         0.5311 |  0.5302 |  0.5638 |      0.0388 |
+| ENETS2      | AIDO.Cell-100M     |    100 |       0.5171 |         0.5288 |  0.5011 |  0.5322 |      0.0412 |
+| ENETS2      | scFoundation       |    100 |       0.4992 |         0.5065 |  0.4868 |  0.5117 |      0.0299 |
+| ENETS2      | tGPT               |    100 |       0.502  |         0.5182 |  0.4858 |  0.5175 |      0.0333 |
+| ENETS2      | ESM2-3B (UCE ctrl) |    100 |       0.5037 |         0.4982 |  0.4931 |  0.5136 |      0.0313 |
+| TRRUST      | scGPT-53M          |    187 |       0.5815 |         0.5817 |  0.5653 |  0.5982 |      0.0417 |
+| TRRUST      | Geneformer-V1-10M  |    187 |       0.5489 |         0.5434 |  0.5348 |  0.5627 |      0.0279 |
+| TRRUST      | Geneformer-V2-104M |    187 |       0.5944 |         0.5818 |  0.5791 |  0.6095 |      0.0399 |
+| TRRUST      | Geneformer-V2-316M |    187 |       0.598  |         0.5997 |  0.5825 |  0.6129 |      0.0406 |
+| TRRUST      | AIDO.Cell-100M     |    187 |       0.5528 |         0.552  |  0.5377 |  0.5675 |      0.0348 |
+| TRRUST      | scFoundation       |    187 |       0.4948 |         0.496  |  0.4825 |  0.5082 |      0.025  |
+| TRRUST      | tGPT               |    187 |       0.5129 |         0.5208 |  0.499  |  0.5259 |      0.027  |
+| TRRUST      | ESM2-3B (UCE ctrl) |    187 |       0.5067 |         0.5073 |  0.4953 |  0.518  |      0.0274 |
+| DoRothEA    | scGPT-53M          |    292 |       0.4902 |         0.4869 |  0.4757 |  0.5046 |      0.0451 |
+| DoRothEA    | Geneformer-V1-10M  |    292 |       0.4982 |         0.4978 |  0.4936 |  0.5025 |      0.0376 |
+| DoRothEA    | Geneformer-V2-104M |    292 |       0.5393 |         0.5391 |  0.5323 |  0.546  |      0.0445 |
+| DoRothEA    | Geneformer-V2-316M |    292 |       0.5377 |         0.5346 |  0.5303 |  0.5456 |      0.0439 |
+| DoRothEA    | AIDO.Cell-100M     |    292 |       0.5196 |         0.52   |  0.5027 |  0.5366 |      0.0538 |
+| DoRothEA    | scFoundation       |    292 |       0.461  |         0.4367 |  0.4441 |  0.4774 |      0.0464 |
+| DoRothEA    | tGPT               |    292 |       0.5244 |         0.5252 |  0.5142 |  0.5343 |      0.0416 |
+| DoRothEA    | ESM2-3B (UCE ctrl) |    292 |       0.5803 |         0.5801 |  0.5747 |  0.586  |      0.0538 |