What IMMUNO-TCR taught us about honest metrics

IMMUNO-TCR is an in-silico decision-support platform for personalised glioblastoma immunotherapy. It estimates how likely a patient's T-cell receptor (CDR3β) is to bind a tumour peptide, then turns that probability into a ranked, explained candidate card for clinicians and immunology researchers. We built it for the TEKNOFEST 2026 Oncology 3T competition, where we reached the final.

The most useful lesson was not about architecture. It was about which numbers we were willing to put on a slide.

Two numbers, side by side

On our frozen held-out set (15,715 rows, never used in training or validation) our v4 model reached a macro-AUROC of 0.686, the best of six open models we compared under the same protocol.

On the ImmRep23 out-of-distribution set, the same model scored 0.502. That is essentially random.

It would have been easy to show only the first number. We show both, because the second one is the honest description of the field: a 2025 benchmark of 50 published models found the same ceiling on epitopes a model has never seen. A score measured without leakage-free splits produces false confidence, and false confidence is the most dangerous output a clinical tool can have.

Where the real gain came from

Ablations told us more than leaderboards did. Adding a separate channel for the TCR's V and J gene segments raised macro-AUROC by +0.1335, the largest single gain in the project. Bigger protein language models, by contrast, scored higher on validation but dropped on the test set, so we kept the 35M-parameter ESM-2.

Saying "I'm not sure"

Because unseen epitopes are hard, the platform does not hand clinicians a raw probability. It calibrates it (temperature scaling) and uses Mondrian conformal prediction to abstain when confidence is too low. In our runs it abstained on about a quarter of cases (25.85%). An abstention is a request for a human decision, not a failure.

Retiring a component

We had planned a learned safety head for off-target cross-reactivity. A three-way audit on 17 July 2026 showed it had not seen a single off-target row in training, so it could not have learned what we claimed. We removed it and put a deterministic scan of 20,431 human proteins in its place. We documented the change in the final report rather than quietly dropping the feature.

What I am taking forward

• Report the result you got, next to the result you wanted.
• Split by epitope, not by row, or your test set is quietly part of your training set.
• Let the system say "I don't know", and make that visible to the person who decides.

None of this makes IMMUNO-TCR a treatment tool. Every score still needs independent in-vitro validation. What it can do is help a team decide which few candidates are worth that validation first.