The challenge
Three secret protein complexes, one filing deadline, five uploads, and no score back. Here is the rulebook, the plan to 30 October, and what could go wrong.
Rules and tracks
Track A: residue-level prediction (the one we are entering)
- For each complex, give a 0 to 1 probability that each HER2 residue (chain A) and each nanobody residue (chain B) touches its partner. Columns:
chain, seqNum, resName, prediction. Omitted residues count as 0. - Up to 5 submissions per team, teams of up to 10, a Hugging Face account to submit. Prizes of $3,000, $2,000 and $1,000.
- Only public outside data is allowed (, and similar), and inputs and settings must be documented. Co-folding engines such as , and are explicitly allowed.
- Leakage from public data is stated to be "not an issue as long as the answer is correct", so we still disclose it honestly in each upload's notes.
- The portal checks only the four header names; it does not check the 0 to 1 range, row counts or the five-upload cap. Local validation is our only safety net.
source
BRIEF.mdresearch/q4_challenge_intelligence/_QUEEN_DIGEST.mdresearch/q8_compute_validation_execution/five_submission_ladder_and_cluster_bootstrap_decision_rules_for_n_equals_3.md
Track B: atomic-level structures (not entered)Predicted 3D complexes judged by CAPRI; closed for this deadline
Track B asks for full predicted complex structures, judged with procedures and . The round number, models per target and whether the answer structures will be released are unconfirmed. The hive closed this track for the 2026-10-31 deadline.
source
research/q4_challenge_intelligence/_QUEEN_DIGEST.mdhive/EMPEROR.md
The metric, in plain wordsPrecision-recall area per complex, averaged over three complexes
Think of ranking residues from most to least likely to touch. asks how many of your top picks were right; asks how many true contacts you found. is the area under that trade-off. Each complex is scored on its own and the three scores are , never pooled.
There are two variants: unweighted, and , where recall counts each true residue by its contact area. Close results are to be settled with a (B=2000, 95% interval).
The organisers' own baseline, , is quoted at 0.176 on a SARS-CoV-2 proxy where 12.6% of residues are positives. A shuffle simulation on that entry gives mean 0.147 with a standard deviation of 0.032, so random guessing reaches 0.176 about 16% of the time.
source
BRIEF.mdresearch/q6_epitope_ml_methods/_QUEEN_DIGEST.md
Scorer ambiguity: six unknowns that can change the winner
The organisers have not published scoring code. Each row below is a fork we cannot resolve from public pages, and each one can reorder close entries. We report every variant locally and prefer entries that win under all of them.
| Question | Possible answers | Why it matters |
|---|---|---|
| Are the two chains of a complex pooled into one curve? | One per complex over plus , or two curves averaged | A -only prior scores about 0.64 on the alone but about 0.34 per complex when pooled; the easy nanobody side can dominate a pooled score.source
|
| How is the area under the PR curve computed? | Step-sum , or trapezoid | A constant predictor scores about 0.04 with the step-sum and about 0.52 with the trapezoid. Sparse files full of tied zeros are inflated under the trapezoid.source
|
| Which variant ranks the leaderboard? | Unweighted, area-weighted, or both | The guideline defines both; the detail page names the official metric for the baseline only, and two readings of the same page disagree.source
|
| Which of the five uploads is scored? | The latest, the best by score, or a team-designated one | The Space code stores every upload and never overwrites; no source we opened says which counts. This decides whether the first four uploads are insurance or real bets.source
|
| What does the confidence test resample? | , contact patches, or whole complexes | resampling ignores spatial correlation (too optimistic); a 3-complex cluster has only 10 distinct resamples and a 14% false-positive rate in our simulation.source
|
| Is any score shown before judging? | No feedback, or score added later | The Space code shows only a confirmation (team, user, targets, time). The audit marks this as unknown rather than stated, so we plan for no feedback.source
|
Try it: how the averaging rule can flip a ranking
Under this reading real skill wins. Scores are macro over the five public complexes. A constant prediction ties every residue, and the rule rewards that tie. We do not know which rule the organisers use, so our floor file gives every residue a unique value.
source
model/README.md
Section 4 (ties trap) and section 5 (floor and all-tied rows). Proxy-league numbers, not forecasts for ONU11/13/23.
How big a gain can three test complexes even detect?
residue, patch noise 01%
residue, patch noise 074%
residue, patch noise 099%
patch-cluster, patch noise 076%
residue, patch noise 0.74%
residue, patch noise 0.774%
patch-cluster, patch noise 0.775%
patch-cluster, patch noise 0.799%
complex-cluster (3 clusters), patch noise 0.714%
complex-cluster (3 clusters), patch noise 0.798%
Bars show how often a flags a gap as real. In this toy simulation a gap of about 0.09 to 0.11 in macro score is needed for roughly 75% detection, and the 3-cluster whole-complex version raises false alarms to 14% against a nominal 5%. The simulation uses idealised score distributions, so treat it as order of magnitude.
source
research/q8_compute_validation_execution/five_submission_ladder_and_cluster_bootstrap_decision_rules_for_n_equals_3.md
Section 3, Monte Carlo of n=3 (150 outer simulations per cell).
The five-submission ladder
Because no score comes back, each rung is judged only on local proxy complexes before it is uploaded. A rung that fails its test is not uploaded, and the slot is kept.
S1 safe baseline
Content. 2-hour fallback prior (spec below)
Kill or promote ruleWhen this rung is dropped or becomes the primary
Never killed; it is insurance. Kill only if validator fails
S2 co-fold v1
Content. prior blended with co-fold contact frequency
Kill or promote ruleWhen this rung is dropped or becomes the primary
KILL (do not upload) if proxy macro AP < S1 or paired CI includes 0
S3 + NGS arm
Content. S2 plus -derived features
Kill or promote ruleWhen this rung is dropped or becomes the primary
KILL component if D09 gate fails; upload S3 only if differs materially from S2 (rank corr < 0.95), else save the slot
S4 frozen candidate
Content. frozen ensemble
Kill or promote ruleWhen this rung is dropped or becomes the primary
Promote as primary if >= every earlier rung on proxies
S5 hedge
Content. different design: lower-variance blend (co-fold arm shrunk 50% toward prior) for all-three-same- risk
Kill or promote ruleWhen this rung is dropped or becomes the primary
Upload only if S4 shows a stored-file defect or if proxy variance of S4 across split-halves is high; otherwise keep slot unused
source
research/q8_compute_validation_execution/day_by_day_runbook_to_2026_10_31_and_failure_mode_register.md
Section: 5-submission ladder with kill criteria.
A second ladder from the ladder-rules fileDifferent content per rung, with advance and kill thresholds
| Rung | Date | Content | Advance when | Kill or revert when |
|---|---|---|---|---|
| S1 safe baseline | 10-20 | Sequence prior (antigen domain-aware prior; / prior from public VHH structures) plus co-fold contact map (AF3-class, , -1 or output reduced to per- scores), simple mean | Proxy macro >= random + 0.05; valid format on all 3 CSVs | Not above random: still submit as a format/rule smoke test, then treat the or co-fold step as broken |
| S2 + NGS feature | 10-24 | Add one -derived per- feature (e.g. /convergence of motifs) to S1 | Raw gain >= 0.03, lower CI > 0, positive in >= 2/3 of proxy complexes | Gain < 0.03 or sign flips across complexes: drop the feature, S1 stays incumbent |
| S3 ensemble | 10-27 | Rank or logit average of surviving arms plus a second co-folder | Beats best single arm by >= 0.02, or ties within noise (kept for robustness), no complex worse by > 0.03 | Worse than best single arm by > 0.02: revert to that arm |
| S4 calibrated | 10-29 | Calibrate ( isotonic/Platt, shrink), omit very low scores as 0 | Both unweighted and area-weighted macro not worse than S3 | Any drop > 0.01, or < 3 proxy complexes to calibrate on: skip, keep S3 |
| S5 reserved hedge | 10-30/31 | Spent only after the final-choice rule (section 6); default content is the chosen final arm, or the fix for a rung that failed format/rules | Final arm selected by the rule | Never use for an untested idea |
Stop-loss in that file: if two consecutive rungs fail their advance test, freeze at the incumbent and spend the remaining days on documentation and a proxy audit.
source
research/q8_compute_validation_execution/five_submission_ladder_and_cluster_bootstrap_decision_rules_for_n_equals_3.md
Section 4, the ladder. Dates and content differ from the runbook ladder; see the contradictions table.
Day-by-day runbook to 30 October
Tick days off as they finish. Your ticks stay in this browser only. The final upload is planned for 2026-10-30 (06:00 UTC in the runbook); 31 October is a buffer because the deadline time zone is unstated.
Go / no-go gate and computeGO if HF login works and templates downloaded. NO-GO escalation: no access by D03 -> build...
Gate. GO if HF login works and templates downloaded. NO-GO escalation: no access by D03 -> build seqNum maps from sequences on /detail (verbatim, printed there) and P04626
Compute. none; 1-GPU smoke test on Explorer gpu-short (-2 or -1 on one ) to measure real s/seed
Owner code. Q (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeGATE: round-trip test, 100% template rows map to the right resName
Gate. GATE: round-trip test, 100% template rows map to the right resName
Compute. CPU only
Owner code. W1 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeGO when manifest hash committed. No later edits; additions only as separate file. Falsifie...
Gate. GO when manifest hash committed. No later edits; additions only as separate file. Falsifier thresholds written (below) the same day
Compute. CPU;
Owner code. W2 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeBaseline end-to-end by Oct 20. GATE: scorer reproduces proxy AP; CSVs pass validator. Uplo...
Gate. Baseline end-to-end by Oct 20. GATE: scorer reproduces proxy AP; CSVs pass validator. Upload S1 (2-hour fallback) to the Space this day
Compute. CPU (scorer: macro AP, weighted AP, B=2000)
Owner code. W3 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeGO if one seed completes < 20 min; else cut to 8 seeds x 5
Gate. GO if one seed completes < 20 min; else cut to 8 seeds x 5
Compute. GPU 1+2: -2 / -1 / on proxies first (20 seeds x 5 samples is the q5 plan; scale down to measured s/seed)
Owner code. W4 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeGATE A1: arm beats prior by > 0.02 macro AP on proxies AND CI excludes 0, else bench
Gate. GATE A1: arm beats prior by > 0.02 macro AP on proxies AND CI excludes 0, else bench
Compute. GPU: ,
Owner code. W4 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeSame gate. Seeds-split-half agreement logged
Gate. Same gate. Seeds-split-half agreement logged
Compute. GPU:
Owner code. W4 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeFirst co-fold arm results Oct 22-24. Upload only if candidate >= S1 on proxies
Gate. First co-fold arm results Oct 22-24. Upload only if candidate >= S1 on proxies
Compute. none (GPUs free for arm GPU embeddings if needed)
Owner code. Q (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeNGS-feature arm by Oct 25. GATE B1: adds > 0.01 to arm A or prior on proxies; else benched
Gate. -feature arm by Oct 25. GATE B1: adds > 0.01 to arm A or prior on proxies; else benched
Compute. CPU, pyarrow column subsets of 1.8 M-row table; <= 4 procs after headroom check
Owner code. W5 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeGO if blend >= best single arm on proxies (no gain -> upload best single arm)
Gate. GO if blend >= best single arm on proxies (no gain -> upload best single arm)
Compute. CPU
Owner code. W6 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeNote draft complete or no-go on any item not documentable
Gate. Note draft complete or no-go on any item not documentable
Compute. CPU / spare GPU for reruns
Owner code. W7 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeEnsemble frozen Oct 28. Frozen means no weight changes; only bug fixes
Gate. Ensemble frozen Oct 28. Frozen means no weight changes; only bug fixes
Compute. none
Owner code. Q (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeFinal CSVs QA'd Oct 29. Validator checklist all green (see register)
Gate. Final CSVs QA'd Oct 29. Validator checklist all green (see register)
Compute. none
Owner code. Q + W3 (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeReserve = hedge variant (below); upload only if no failure found in S4's stored files
Gate. Reserve = hedge variant (below); upload only if no failure found in S4's stored files
Compute. none
Owner code. Q (Q is the compute-and-execution queen, W is a numbered worker agent).
Go / no-go gate and computeBuffer only. No new modelling; upload only to repair a confirmed broken stored file, and o...
Gate. Buffer only. No new modelling; upload only to repair a confirmed broken stored file, and only if the window is confirmed still open
Compute. none
Owner code. Q (Q is the compute-and-execution queen, W is a numbered worker agent).
source
research/q8_compute_validation_execution/day_by_day_runbook_to_2026_10_31_and_failure_mode_register.md
Failure-mode register
Twenty-one ways this could go wrong, with the early warning sign and the planned fix. Filter by severity and open a row for details.
Severity = probability x impact, our own rule (H=3, M=2, L=1; 6 or more is high). The probabilities are the runbook author's judgement, not measurements.
#1 HF Space upload errors / partial upload (3 separate upload_file calls)severity highP mediumimpact high
- Early signal
- error text, one file missing
- Mitigation
- upload by D08 early; retry whole triple; keep receipts; upload S1 early to test the flow
- Owner code
- Q
#2 numbering offset ( P04626 incl. 22-aa vs mature)severity highP highimpact high
- Early signal
- proxy score near 0 after correct-looking model; template seqNum range mismatch
- Mitigation
- use the template CSV seqNum as truth; round-trip test D02; unit test on a known (e.g. glycosite N68 vs mature 46)
- Owner code
- W1
#3 sequential numbering treated as Kabat/severity highP mediumimpact high
- Early signal
- predicted outside the chain; numbers in the output
- Mitigation
- sequential only; map by index to template; assert resName equals template
- Owner code
- W1
#4 / M / A / S extras scored or omitted wronglyseverity mediumP mediumimpact medium
- Early signal
- row count differs from template
- Mitigation
- template defines rows; omitted = 0; tag set to 0
- Owner code
- W3
#5 Chain labelling swap (A antigen / B )severity highP mediumimpact high
- Early signal
- validator: resName not matching chain
- Mitigation
- validator checks chain-sequence compatibility per row
- Owner code
- W3
#6 Explorer queue stall / public partition crowdedseverity mediumP mediumimpact medium
- Early signal
- pending > 6 h
- Mitigation
- cap 2 GPUs, short/gpu-short; failover to Modal/Colab; fit to budget measured on D01
- Owner code
- Q
#7 credits exhaustedseverity mediumP mediumimpact medium
- Early signal
- credit balance after D05
- Mitigation
- use credits only for ONU, run proxies on Explorer; -1/ backup
- Owner code
- W4
#8 AF3 weights unavailableseverity mediumP highimpact low
- Early signal
- no approval by D03
- Mitigation
- do not wait; q5: -2/-1/ need no application
- Owner code
- Q
#9 Proxy inflating scores (7QVK, 5MY6 in training sets)severity highP highimpact high
- Early signal
- clean-proxy (9MTX/9MTE) AP much lower than flagged proxies
- Mitigation
- gate on clean proxies only; templates-off ablation; state honestly (the rules say public-data leakage is "not an issue as long as the answer is correct")
- Owner code
- W2
#10 All-three--same- assumption (shared prior or model collapses three outputs to one patch)severity highP mediumimpact high
- Early signal
- predicted antigen maps for 3 clones correlate > 0.9
- Mitigation
- per-clone inference; S5 hedge; compare to q1 effective niches (~7.4)
- Owner code
- W6
#11 Metric mismatch between /detail and /guidelineseverity mediumP mediumimpact medium
- Early signal
- /detail names no official metric; /guideline lists two variants
- Mitigation
- report both locally; blend must not lose on either; ask organisers
- Owner code
- Q
#12 Extended-deadline rumour tempts delayseverity highP mediumimpact high
- Early signal
- organiser comment dated Aug 28 only
- Mitigation
- plan to Oct 30; if extension is announced, still keep S4 uploaded
- Owner code
- Q
#13 Sanctions/residency eligibility problem for a team memberseverity mediumP lowimpact high
- Early signal
- unknown
- Mitigation
- check every member against /detail list before first upload; sponsor can ask for proof
- Owner code
- Q (user)
#14 3 test complexes dominated by one (macro AP then decided by 1 complex)severity mediumP mediumimpact medium
- Early signal
- proxy per-complex AP spread
- Mitigation
- no pooled metric; report per-complex; S5 low-variance
- Owner code
- W6
#15 Gated training dataset access not grantedseverity mediumP mediumimpact medium
- Early signal
- no email by D03
- Mitigation
- the sequences on /detail and public PDB suffice for the prior and co-fold arms; only the arm blocked ( is public elsewhere per brief, unverified here)
- Owner code
- Q
#16 HF OAuth session expires (480 min) mid-uploadseverity lowP lowimpact medium
- Early signal
- login screen on Submit
- Mitigation
- re-login before each submission
- Owner code
- Q
#17 Score values outside 0-1 or NaN (app does not check)severity mediumP lowimpact high
- Early signal
- validator
- Mitigation
- min/max/NaN assert
- Owner code
- W3
#18 Ensemble over-fit to 2 clean proxiesseverity highP highimpact medium
- Early signal
- weights drift with leave-one-out
- Mitigation
- weights restricted (<= 3 params), shrink to prior
- Owner code
- W6
#19 Scorer is wrong (proxy miscomputed, weights misapplied)severity highP mediumimpact high
- Early signal
- cannot reproduce 8GZ5 baseline numbers ( 0.176, base 0.126, per brief)
- Mitigation
- independent re-implementation by second worker
- Owner code
- W3
#20 Methods note incomplete -> rule violationseverity mediumP mediumimpact medium
- Early signal
- note missing items on D11
- Mitigation
- checklist below
- Owner code
- W7
#21 Local resource trouble (C: or CPU headroom) when processing 1.8 M-row tableseverity lowP mediumimpact low
- Early signal
- CPU > 75%, free < 5 GB
- Mitigation
- worker.md resource rule; column subset
- Owner code
- W5
source
research/q8_compute_validation_execution/day_by_day_runbook_to_2026_10_31_and_failure_mode_register.md
Failure-mode register table.
Open contradictions
Places where our own sources disagree. Resolved rows say which side we adopted; open rows are carried forward.
| Topic | One side | Other side | Status |
|---|---|---|---|
| 0.126 (the SARS-CoV-2 RBD proxy, 26/206) | 0.029 to 0.055 of 618 antigen on real proxies | Resolved: use about 0.04 source
| |
| Number of test complexes | Home page: seven antibodies | Detail and guideline pages: three (, , ) | Resolved: the home page is stale source
|
| Size of the labelled training set | Brief: a few hundred labels | Recount of the real table: 19,367 labelled (7,316 yes, 12,051 no) | Resolved: use 19,367 source
|
| Antigen chain length in the template | 618 (inferred from published file sizes) | 630 (printed construct) or 624 (sequence) | Open: template files are gated source
|
| Which variant ranks | Detail page names no official metric | Guideline lists unweighted and area-weighted | Open: report both source
|
| Which upload counts | Latest | Best of five | Open: question drafted for the Space discussion tab source
|
| Results and ranking date | Results announced 2026-12-15 (detail page) | Ranking release 1 December (Space text) | Open, not material to the submission source
|
| version | Guideline cites v1.1.479 | Our proxy labels used v1.1.497 | Open: a version effect is not excluded source
|
| Which public proxies are -free | Runbook: only 9MTE and 9MTX | Validation file: all four pass the date cutoff; 7QVK is in one engine's window | Kept both labels; 9MTE and 9MTX are primary source
|
| Two ladders in the q8 files | Runbook ladder: S1 floor, S2 co-fold, S3 plus , S4 frozen, S5 hedge | Ladder-rules file: S1 baseline, S2 plus , S3 ensemble, S4 calibrated, S5 reserve | Open: runbook ladder is shown first, the other below it source
|
| Fab chains in 9MTE and 9MTX | One ERBB research note says they contain Fabs | RCSB entity lists: only nanobodies plus | Resolved: that note is quarantined source
|