Lipid nanoparticle screening · Retrospective public-data audit
Choosing which lipid nanoparticles to test
Can published lipid nanoparticle data help choose the next experiment? Across 13,982 formulations, the tested models did not establish a reliable screening gain over random selection on unseen publications.
The question
If a whole publication is unseen, can a model recover its most active lipid nanoparticle formulations and identify panels where its ranking should not be used?
What I worked on
I evaluated formulation and assay metadata from the Lipid Nanoparticle Database (LNPDB), created by Evan Collins and collaborators. The eligible cohort contains 13,982 candidates in 37 assay-specific panels from 28 publications. Each panel has at least 30 unique formulations with numeric normalized-luminescence measurements. Rankings are evaluated within panels, rather than treating measurements from different assays as interchangeable.
Each outer fold holds out one complete PubMed publication and removes its exact formulations from training. Model family and regularization are chosen using whole-panel validation inside the remaining data. Publication identifiers, source links, and target values are excluded from the features. A separate analysis also removes every ionizable lipid found in the test publication.
The study compares transparent ridge models and then replays their frozen predictions at seven screening budgets. It also tests whether a distance measure built without target labels can identify unreliable transfers. The released package preserves the data snapshot, split checks, predictions, uncertainty estimates, and reproduction commands.
What I found
On unseen publications, the ranking score nDCG@K was 0.570 against an exact random expectation of 0.521; higher is better. The paired 95% publication-cluster interval for the advantage included zero. A candidate-random comparator scored 0.656, but used different pools, effective budgets, and model selection, so the gap does not isolate publication holdout.
At a nominal 20% screening budget, the model recovered 28.4% of each panel's top-decile candidates on average, versus 20.3% under exact random selection. The paired improvement was 8.1 percentage points. Its pointwise 95% interval was +1.6 to +14.8 points, but the simultaneous interval across seven budgets was −0.5 to +17.0 points. The budget scan therefore did not establish a budget-specific improvement. The probability of finding at least one top-decile candidate was also not reliably improved.
The rule intended to reject poorly supported panels accepted 27 of 37 panels, but their mean nDCG@K fell to 0.544. Nearest-publication distance also failed to identify reliable transfers. These diagnostics did not provide a dependable rule for deciding when to abstain.
Explore the screening budget
How many of a panel’s most active formulations are found when only part of it is screened?
Candidates are rounded up within each panel. At this budget, 20.3% were actually selected on average.
- Model recall
- 28.4%
- Random selection
- 20.3%
- Difference
- +8.1 pp
Simultaneous 95% interval for the difference: −0.5 to +17.0 percentage points.
The interval includes zero. At this budget, an advantage over random selection is not established after accounting for all seven budget comparisons.
All seven measured budgets
| Budget | Actually selected | Model recall | Random recall | Difference (pp) | Simultaneous 95% interval (pp) |
|---|---|---|---|---|---|
| 1% | 1.5% | 2.5% | 1.5% | +0.9 | −1.3 to +4.7 |
| 2% | 2.4% | 4.1% | 2.4% | +1.7 | −1.3 to +5.9 |
| 5% | 5.5% | 7.7% | 5.5% | +2.2 | −1.3 to +7.1 |
| 10% | 10.4% | 14.1% | 10.4% | +3.7 | −2.0 to +10.9 |
| 20% | 20.3% | 28.4% | 20.3% | +8.1 | −0.5 to +17.0 |
| 50% | 50.2% | 59.7% | 50.2% | +9.6 | −2.6 to +21.2 |
| 100% | 100.0% | 100.0% | 100.0% | 0.0 | 0.0 to 0.0 |
Intervals use publication-cluster bootstrapping and a Bonferroni adjustment across the seven recall comparisons. These are retrospective results on published formulations; no new formulations were tested.
What this does and does not tell us
This analysis ranks already-published candidates. It does not synthesize or test a new lipid nanoparticle, and it provides no manufacturing, safety, efficacy, or clinical evidence. The findings apply to these baseline models and this cohort; they do not establish that machine learning cannot improve LNP design.
Holding out publications reduces leakage but cannot remove publication bias, extraction errors, or unrecorded protocol differences. The models use supplied descriptors and string fragments. Uncertainty estimates contain only 28 publication clusters, and the seven budgets reuse the same predictions. The simultaneous intervals cover recall and any-hit outcomes as separate families, rather than all fourteen estimates together. Their extreme endpoints use only 2,000 bootstrap draws and need a Monte Carlo stability check. Pooled coverage does not establish coverage for each new panel.
The next test should use a prospective campaign with its candidate pool, comparison rules, and abstention threshold fixed before outcomes are observed. Richer molecular models and support diagnostics can then face the same unseen-publication boundary.
Read the paper
Can Public LNP Data Rank the Next Library? A Publication-Cold Screening Audit of LNPDB