What has been tried
Every family of method that could predict where a nanobody touches HER2, with its best reported number, what could have leaked, and whether we may use it.
The method table
Sort by clicking a heading, narrow by family, or search. The numbers come from different benchmarks, different test sizes and different metrics, so they are not comparable across families. Each row has a source tag with the file in the research folder it came from. Where the research could not open a source or only saw a search snippet, the row is marked.
| Best reported number | Leakage note | Licence / access | ||||
|---|---|---|---|---|---|---|
| Baseline | AntiBinder (organiser baseline) | 0.176 PR-AUC on the organisers' 8GZ5 proxy; the paper reports no residue-level metricsource
| 1 complex (26 of 206 residues) | A shuffled guess reaches 0.176 about 16% of the time (mean random 0.147, SD 0.032) | not recorded in the hive file | our simulation |
| Co-folding | AlphaFold3 (AF3) | 48.75% of cases DockQ >= 0.23 (FoldBench antibody-antigen); 31.6% on 60 nanobody cases, single seedsource
| n = 160 / n = 60 | Training cutoff 2021-09-30; later benchmark entries are post-cutoff | not recorded in the hive file | opened |
| Co-folding | Protenix-v1 | 52.31% (FoldBench Ab-Ag, 5 seeds x 5 samples); 40.26% on 498 antibody cases, rising 36.0% (1 seed) to 47.7% (80 seeds)source
| n = 160 / n = 498 | Cutoff 2022-2025 test set | not recorded in the hive file | opened |
| Co-folding | Boltz-2 | 40.10% on 498 antibody cases (median 38.6%, best 50.4%)source
| n = 498 | Overlaps Boltz-2 training data (stated in the hive file) | not recorded in the hive file | opened |
| Co-folding | Boltz-1 | 23.3% on 60 nanobody cases, single seedsource
| n = 60 | Cutoff 2021-09-30 | not recorded in the hive file | opened |
| Co-folding | Chai-1 | 22.1% on 498 antibody cases; 15.0% on 60 nanobody casessource
| n = 498 / n = 60 | Cutoff 2021-09-30 on the nanobody set | not recorded in the hive file | opened |
| Co-folding | OpenFold3 | No number extracted (figure only)source
| not recorded | not recorded | not recorded in the hive file | unverified |
| Docking | HADDOCK3 with true interface known (bound antigen) | Top-1 correct 65%, top-10 87.5%source
| n = 40 nanobody complexes | Uses the answer as input, so not a blind result | not recorded in the hive file | partly unverified |
| Docking | HADDOCK3 loose interface (CDRs + epitope region +4 A) | Top-1 30% bound / 17.5% unbound; top-10 55% / 42.5%source
| n = 40 | Needs an epitope guess as input | not recorded in the hive file | partly unverified |
| Docking | HADDOCK3 blind (all surface) | Top-1 7.5%, top-10 20%source
| n = 40 | None stated | not recorded in the hive file | partly unverified |
| Docking | AF2-Multimer v2.3 (blind) | Top-1 25%, top-10 27.5%source
| n = 40 | Training-set overlap not stated | not recorded in the hive file | partly unverified |
| Docking | AF3 server, 25 models (blind) | Top-1 32.5%, top-10 52.5%source
| n = 40 | Training-set overlap not stated | not recorded in the hive file | partly unverified |
| Docking | ClusPro AbEMap-style contact frequency on AF3 (100 seeds, 500 models) | Epitope PR-AUC 0.58 (ROC 0.82), versus 0.36 for an AF2 ensemble. Antibodies, not nanobodiessource
| n = 26 antibodies | Sequence-only input, but AF3 may have seen the structures | not recorded in the hive file | partly unverified |
| Epitope predictor | DiscoTope-3.0 (AF2 input) | PR-AUC 0.232, ROC-AUC 0.783 at 6.4% prevalence (about 3.6x random)source
| 24 antigens, 6,788 residues | HER2 leak unknown, likely | Academic server; Creative Commons variant unverified | opened |
| Epitope predictor | BepiPred-3.0 | ROC-AUC 0.738 / 0.771 on two test sets; PR-AUC not reportedsource
| 5 and 15 antigens | HER2 leak unknown, likely | Not stated on the DTU page | opened |
| Epitope predictor | SEMA 2.0 | ROC-AUC 0.744 (masked) / 0.777 (unmasked); PR-AUC not reportedsource
| 101 sequences | HER2 leak unknown, likely | Ambiguous MIT notice | opened |
| Epitope predictor | AbEpiTope-1.0 | Residue-level numbers not reported; probably cannot take a VHH (needs VH+VL)source
| 1,730 complexes | HER2 leak unknown, likely | Academic, non-commercial | opened |
| Epitope predictor | EpiPred | No numbers extracted; historicalsource
| 45 complexes | Unknown | Not stated | unverified |
| Language model | AntiBERTa2-CSSP | Nanobody paratope PR-AUC 0.765 (AUROC 0.939) on the NbBench tasksource
| NbBench paratope task | Not assessed in the hive file | Not checked | partly unverified |
| Language model | AbLang (heavy) | Paratope PR-AUC 0.756 (AUROC 0.933)source
| NbBench paratope task | Not assessed | Not checked | partly unverified |
| Language model | AntiBERTy | Paratope PR-AUC 0.739 (AUROC 0.932)source
| NbBench paratope task | Not assessed | Not checked | partly unverified |
| Language model | nanoBERT | Paratope PR-AUC 0.719 (AUROC 0.928)source
| NbBench paratope task | Not assessed | Not checked | partly unverified |
| Language model | ESM-2 650M | Paratope PR-AUC 0.718 (AUROC 0.928)source
| NbBench paratope task | Not assessed | MIT for code; no separate weight licence stated | partly unverified |
| Language model | VHHBERT | Paratope PR-AUC 0.695 (AUROC 0.921)source
| NbBench paratope task | Not assessed | Not checked | partly unverified |
| Language model | IgBert | Paratope PR-AUC 0.687 (AUROC 0.921)source
| NbBench paratope task | Not assessed | MIT (HF model card) | partly unverified |
| Language model | AbLang-2 | Paratope PR-AUC 0.693 on a different benchmark (Paraplume/PECAN)source
| PECAN-style set | Not assessed | BSD-3-Clause | partly unverified |
| Language model | ProtT5 | Paratope PR-AUC 0.734 on the Paraplume/PECAN benchmarksource
| PECAN-style set | Not assessed | Not verified | partly unverified |
| Language model | IgLM / ESM3 / ESM C 600M | No paratope number extractedsource
| n/a | n/a | IgLM academic non-commercial; ESM3 and ESM C 600M non-commercial (avoid) | opened |
| Paratope predictor | ParaDeep | PR-AUC 0.805, ROC-AUC 0.959 on heavy chains (10.4% prevalence); nanobodies not evaluatedsource
| n = 222 | Not assessed | GitHub repository; licence not recorded | partly unverified |
| Paratope predictor | Paragraph | PR-AUC 0.696, ROC-AUC 0.934 (third-party aggregate on PECAN set); needs a structuresource
| PECAN set | Not assessed | not recorded in the hive file | partly unverified |
| Paratope predictor | ParaAntiProt | PR-AUC 0.665, ROC-AUC 0.912 (search summary only)source
| not recorded | Not assessed | Unverified | unverified |
| Paratope predictor | Parapred | PR-AUC 0.516, ROC-AUC 0.861 (heavy-chain blind test); no nanobody claimsource
| heavy-chain test | Not assessed | README not reachable | partly unverified |
| Paratope predictor | NanoBERTa-ASP | AUC 0.947 (nanobody only); PR-AUC only in figuressource
| not recorded | Not assessed | Weights unverified | unverified |
| Prior / null | Domain prior (which HER2 domain) | PR-AUC 0.0905 (SD 0.052) at 3.97% prevalence, 25 true residues. Random: 0.049source
| Simulated, 630-residue antigen | None: a simple rule, not a trained model | n/a | our simulation |
| Prior / null | Surface exposure only (RSA) | PR-AUC 0.0688 at 3.97% prevalence. Random: 0.049source
| Simulated, 630-residue antigen | None | n/a | our simulation |
| Prior / null | VHH CDR-position prior | PR-AUC 0.644 (SD 0.101) versus 0.204 random on the VHH chainsource
| Simulated VHH chains | None | n/a | our simulation |
36 of 36 methods shown. Numbers come from different benchmarks and are not comparable across families.
How to read a method numberThree traps that make impressive numbers mean less
Percent of cases correct versus per-residue score
Co-folding and docking papers count a complex as right when its is at least 0.23. The challenge instead scores each residue with . A program that places a nanobody correctly 30% of the time is not 30% good on the residue score: when it is wrong, it is wrong about the whole site.
Leakage: did the program already see the answer?
Each engine has a training cutoff date. Structures released before it may have been memorised. Only two public HER2 nanobody structures, 9MTE and 9MTX (released 2025-11-19), are known to post-date the engines. The challenge rules say leakage from public data is "not an issue as long as the answer is correct", but we need leakage-free proxies to know what really works. See .
Prevalence: the random score is tiny on HER2
On HER2 only about 3-5% of residues are in the interface, so random guessing scores about 0.03-0.05 on the antigen chain, and the nanobody chain is 15-23% in the hive's planning table. A score of 0.2 is therefore several times random on the antigen, but a gap between two teams of less than about 0.07-0.16 on three test complexes is mostly luck. The score is one number per complex, .
source
research/q6_epitope_ml_methods/residue_level_pr_auc_with_prevalence_0_1_calibration_ranking_and_aggregation_tricks.jsonresearch/q9_serendipity_cross_domain/_QUEEN_DIGEST.md
Why the average over seeds beats the maximum
A is the random starting number that makes a structure program give a slightly different answer each time. The temptation is to run many seeds and keep the best-looking result. The honest alternative is the : how often each residue ends up in the interface across all runs. Play with the toy model below.
SYNTHETIC DATA - TOY MODEL, NOT HER2 RESULTS
Mean over runs: 0.501 (shaded band is one SD over 30 repeats)Max over runs: 0.225
How it is built: a pretend antigen of 630 residues with 25 true residues (about 4%, inside the hive's 3-5% range for HER2). Every run scores every residue as random noise, plus a boost on a patch of 25 residues. With the chance set above the patch is the true epitope; otherwise it is a random wrong patch. The score is , averaged over 30 repeats. The real-world evidence for the same direction is in the table above (Protenix-v1 improves from 36.0% to 47.7% on 1 versus 80 seeds), but this toy model proves nothing about HER2 on its own.
source
research/_meta/king_infer_KINGLOG.mdresearch/q5_structure_prediction_docking/alphafold3_openfold3_boltz2_chai1_protenix_nanobody_antigen_benchmarks_and_success_rates.json
The demo itself is a synthetic toy and is not a hive result.