How the labels are made, and where they lie
The training labels say which nanobodies got more common after one round of fishing on HER2. The answer key says which residues touch in a 3D picture. Neither tells you what a model might hope they do.
From animal to answer key
Click any step. Each opens to the physical mechanism, where noise creeps in, the controls that exist, and the traps for anyone building a model. The top row makes the training labels; the bottom row makes the test truth.
Enrichment test
For each sequence, a statistical test asks whether its share of the reads went up after fishing by more than luck would explain.
Physical mechanism
The test is a two-proportion z-test, algebraically the same as a 2x2 without continuity correction, on before and after counts. An increase with at or below 0.05 earns "binder". The pipeline's IE-score is a pooled version of this test with no correction. The labels reproduce exactly from the counts.
Try it
The simulator further down this page computes, exactly, how often this test calls a clone that did not change at all a binder, as read counts drop.
Where the noise comes in
The normal approximation is justified on library size (over 10,000 reads), but it depends on each clone's own counts. It is invalid below about 10 combined reads (). In a simulation (not a measurement of this table), the false-binder rate among tested null clones was 7 to 10% at 3 to 4 reads, versus the nominal 2.5%.
With realistic PCR over-dispersion, 5 to 15% stays false even at high counts (simulation).
Controls
A clone that is also a binder in a HER2-free control panning is reassigned to "noise" (documented for the sibling hIL6 dataset; its use for HER2 is inferred). 746 sequences carry the noise label, and ONU11 is one of them.
Gotchas
The sibling dataset takes the smallest p over 12 sublibraries with no correction. How many real labels have fewer than 10 reads was not measured.
source
research/q3_assay_ngs_ground_truth/ngs_read_count_statistics_pcr_sequencing_error_singletons_and_enrichment_test_validity.mdresearch/q7_data_representation_integration/negative_labels_pu_learning_noisy_labels_and_enrichment_to_label_denoising.md
How often does the enrichment test cry wolf?
Every label rests on one test of before-and-after read counts. The test was built for large counts. Most clones in this experiment have only a handful of reads, which is where it misbehaves. Slide the read count down and watch the false-binder probability climb.
Pick a clone that did not change at all in the experiment (a true null). Set how many reads it has and see how often the still calls it a "binder". Then give true binders a real enrichment and see what share of "binder" calls are right.
The shaded band is where the combined count of a null clone is mostly under 10. The curve is jagged because counts are whole numbers: the smallest possible call is 0 reads before and 4 after.
How the simulator works, and what it is not
For each pair of read counts (before, after) drawn from Poisson distributions, it applies the same rule as the dataset papers: singletons are set to zero, then a two-proportion z-test at one-sided 0.025 (increase only). It sums the probability of every call exactly, with no random sampling. Noise is a lognormal multiplier on the after-panning mean, integrated with five quadrature points.
It reproduces the hive finding qualitatively (false-binder rates well above 2.5% at about 3 to 4 reads) but it is not a measurement of the real table, which has not been analysed for this. "Stringency" has no sourced mapping to fold enrichment, so that slider is an assumption, not a data-derived relationship.
Reference numbers from the hive simulation (library of 10^6 clones, 10^5 reads)
| min combined reads | z-test false-binder rate | Fisher exact rate |
|---|---|---|
| 2 | 3.2% | 0.5% |
| 3 | 6.9% | 1.2% |
| 4 | 9.5% | 1.6% |
| 5 | 4.9% | 2.2% |
| 10 | 2.0% | 1.3% |
| 20 | 2.7% | 1.7% |
At 1% true-binder prevalence with 10x enrichment and extra noise 0.5, the hive simulation found a 42 to 63% positive predictive value depending on the minimum count.
source
research/q3_assay_ngs_ground_truth/ngs_read_count_statistics_pcr_sequencing_error_singletons_and_enrichment_test_validity.md
Simulator code is on this page; reference table is from the hive worker's own simulation.
Why does a test with a 5% error rate cause so much trouble?
Because true binders are rare. If only 3 in 100 tested clones are real binders, a test that wrongly flags even 5% of the other 97 produces about 5 false calls for every 3 true ones. This is the base-rate effect: the share of "binder" calls that are real depends on how rare real binders are, not just on the test's error rate. The hive's estimate for a 3% prior is that only about 27% of binder calls would be right (an estimate from assumptions, not a measurement). See the and entries in the glossary.
What a score means when the base rate is tiny
The contest score is PR-AUC, averaged over three complexes. Its chance level is the share of residues that truly touch, which is small on HER2. Slide the prevalence to see how the chance level moves, then see how wide the spread is with only three complexes to average.
A. Chance level and a toy precision-recall curve
is the area under a curve of (how many flagged residues are real) against (how many real ones were found). A predictor that guesses at random scores about the (share of residues that truly touch). Set a predictor by saying: "at recall R, my precision is P".
Toy model: a straight-line curve from (0, 1) through your point to (1, prevalence). It is for building intuition, not a model of any real predictor. The antigen-side chance level in the hive is about 0.03 to 0.05 (HER2 base rate about 3 to 5%), and an estimated 0.2 to 0.3 on the nanobody side.
Why the same score can mean two things
PR-AUC can be computed as average precision (a step sum) or with the trapezoid rule, and the two can differ a lot for weak predictors. The hive notes that a constant predictor scores about 0.04 under one definition and about 0.52 under another. The official choice is not yet confirmed.
source
research/q3_assay_ngs_ground_truth/_QUEEN_DIGEST.mdhive/EMPEROR.mdresearch/_synthesis/narrative.md
B and C. Three complexes: averaging and spread
The contest score is the of per-complex PR-AUC over the three complexes (, , ), not one pooled number. The true prevalences of the three test complexes are not known yet, so the sliders below are hypothetical. This simulation draws scores for a predictor whose true contact residues score higher than the rest by a chosen amount, then shows how much the macro-average moves just from which residues happen to be in the test set.
Bars: how often each macro PR-AUC came up across 150 freshly drawn test sets. Pink bracket: the middle 95%. Implied ranking quality of this predictor: ROC-AUC about 0.76.
What the controls teach
- With a strength of 0, the macro score lands on chance, and chance is only about the prevalence. A score of 0.10 means something very different at 4% prevalence than at 25%.
- Positives are few: about 25 per complex at these settings, so a handful of residues moves a complex's score. Try the residue slider and watch the spread.
- Macro-averaging weights each complex equally, so one complex with a very high or low prevalence moves the average more than it would in a pooled score.
- The hive's label-noise floor is separate: a perfect predictor scores about 0.92 to 0.97 against a replicate copy of the same complex (4 to 9% of positives flip), so noise caps the very top scores far above the chance levels shown here.
Settings used by this simulation
Each residue is a contact with probability equal to the complex's prevalence. Its score is a standard normal draw, plus the gap if it is a contact. Per-complex PR-AUC is average precision. The macro-average is the mean over the three complexes; the pooled score ranks all residues together. 150 runs with a fixed random seed, so sliders move the picture smoothly. The paired cluster bootstrap in the official guideline (B=2000) compares two predictors; this tool shows spread for one.
source
research/q3_assay_ngs_ground_truth/voronota_lt_contact_area_definition_and_how_epitope_labels_depend_on_the_cutoff.mdBRIEF.md
Simulation is local to this page; the 618/624/630 chain lengths are an open contradiction in the hive.
Four ways the "data representation" thesis failed
The hope was that the sequence data, repackaged cleverly, would reveal the interface without needing a structure predictor. It was a testable idea, it was tested, and the answer is no. That negative result is itself useful: it tells the team not to spend GPU time on it.
1. Co-evolution (DCA / Potts): HER2 never varies
finds residues that change together across many sequences and infers they touch. Here the antigen is the same in every experiment, so there is no HER2 variation for nanobody residues to co-vary with. Only nanobody-to-nanobody couplings are measurable, and they mostly reflect germline and loop geometry.
The composite score beat the CDR3-only prior (0.509 versus 0.399, interval for the difference above zero). That sounds like a win, but a simple "how variable is this loop position in the unselected library" map scores 0.562, and the binder-versus-non-binder part adds nothing over it. "Beats the stated baseline" did not mean "carries signal".
Details and numbers
- Data: 7,316 binders and 12,051 non-binders, scored against 5 HER2-nanobody structures. About 11 minutes of CPU.
- Binder-derived scores minus the variability prior: KL shift -0.07 (interval -0.17 to +0.03), coupling -0.03 (-0.14 to +0.08), contrast coupling -0.15 (-0.27 to -0.04).
- The variability prior scored 0.494 over all residues; a leave-one-out contact-frequency prior from public structures scored 0.435 on CDR residues.
source
research/q7_data_representation_integration/paired_coevolution_potts_dca_and_contact_inference_from_binder_vs_nonbinder_sequences.md
2. Free mutation scan from clone families: variation is just errorLoop positions vary 0.5 to 2% per column, the sequencing error floor.
A experiment tells you which nanobody positions tolerate change, and the tolerant ones are usually not at the interface. The three test clones (ONU11, ONU13, ONU23) each sit in a family of 1,330 to 4,420 sequences, so the hope was a free scan.
In all three families, CDR3 varies only at the error floor, about 0.5 to 2% per column, the same as the framework. Natural variants are not a mutation scan. A tolerance score built from 348 enriched families scored 0.137, below the plain CDR3-uniform baseline (0.206): falsified.
The three held-out clones as rows in the table
| Clone | CDR3 (from the challenge page) | Label |
|---|---|---|
| ONU11 | RPLGSTWTEHPY | noise |
| ONU13 | KYGSSRLWDY | Yes |
| ONU23 | DGRFYSDYVWSSPDEYAY | Yes |
Also from this work: 39 CDR3 families carry 50% of the above-expectation post-panning reads, and family-level replicate Pearson is 0.93. A family-level enrichment score (AUROC 0.976) beats a single-replicate per-sequence score (0.962) but not the pooled per-sequence score (0.991), which is circular with the label.
source
research/q7_data_representation_integration/deep_mutational_scanning_and_cdr_mutation_tolerance_as_paratope_evidence_in_public_data.mdresearch/q7_data_representation_integration/clonotype_clustering_cdr3_families_and_enrichment_trajectories_as_epitope_bins.md
3. Mimotope matching: no better than a generic surface guess0.081 against 0.083 for plain surface exposure; random is about 0.05.
A method finds the antigen patch whose shape and chemistry match the peptides a binder selects. Classic tools need dozens of antigen-selected peptides. Here the test was made as easy as possible: the true binder loop (CDR3) was the query for each of 5 public HER2 complexes.
Matching charge, hydrophobicity and aromatic profile of the loop to HER2 surface patches scored 0.081, versus 0.083 for a surface-exposure prior, with bootstrap intervals that include zero difference and no benefit over a wrong-loop control. In the literature, mimotope tools reach sensitivity about 0.3 and precision about 0.2 to 0.3 on a 10-case set, and no published method infers an epitope from antibody-side sequence alone.
Caveat that makes this harder, not easier, to dismiss
The oracle test is optimistic (it used the true loop). The mimotope worker could not open the gated table and used the true binder CDR3s instead; BepiPred, DiscoTope and SEMA were not run.
source
research/q7_data_representation_integration/mimotope_and_peptide_epitope_mapping_phage_sequences_and_epitope_inference_from_binding_alone.md
4. Label denoising and PU learning: it made models worseEvery denoised variant scored below raw labels, outside its interval.
If the labels are noisy, cleaning them should help. treats "Not" as unknown rather than negative. Models were trained on 19,367 labelled clones and tested on held-out families.
All intervals sit below zero on both splits and on a stricter replicate-confirmed subset. A bigger lesson: a plain 3-mer model drops from AUROC 0.945 to 0.855 when related sequences are kept apart, and cluster size alone gives 0.84. Yes versus Not is largely family identity.
Why the usual PU assumptions fail here
PU methods assume positives are labelled at random. Here the chance of being labelled grows with read count, so that assumption is false and the estimated class prior is not interpretable. Only 49.5% of Yes are replicate-confirmed, 59% of Yes have zero mother reads, and the library-wide share of enriched clones is 0.2 to 0.6% while the real binder prior is not identifiable from this design.
This kills "this soft label or nnPU helps on this task", not "any denoising could help". Evaluating a denoised model against raw labels is itself partly circular.
source
research/q7_data_representation_integration/negative_labels_pu_learning_noisy_labels_and_enrichment_to_label_denoising.md
source
research/q7_data_representation_integration/_QUEEN_DIGEST.mdhive/EMPEROR.md