Life sciences · Preprint

MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentials

arXiv · September 8, 2026

Preliminary Why this matters to you Expert discussion

Early or partial results. Treat as a signal, not a conclusion.

AI Summary

MLIP Detective is an agentic framework designed to discover failure modes in machine-learning interatomic potentials by generating physics-informed hypotheses and escalating suspicious cases to human review. In proof-of-concept, the framework identified an anomaly in MACE-MPA-0 predicting relaxed adsorbate-surface systems as higher in energy than separated fragments; however, the study does not report systematic validation, generalizability across models, or impact on model improvement or deployment decisions.

What was studied

Methodological framework demonstration; proof-of-concept case study. Machine-learning interatomic potentials, exemplified by MACE-MPA-0; focus on adsorbate-surface systems.. Intervention: MLIP Detective agentic framework for active failure mode discovery..

Key findings

MLIP Detective identified a systematic anomaly in MACE-MPA-0 involving O- or F-containing adsorbates predicted to be higher in energy than their separated fragments. Cross-model comparison inferred a likely training-data origin for the anomaly, consistent with recent reports.

Safety

Safety was not reported in the material analysed. Check the source before drawing any conclusion about harm.

Who this may help

The source did not state who this applies to in practice.

Plain-language summary

An early-stage methodological study demonstrating a novel framework for identifying failure modes in machine-learning models, with a single case study showing proof-of-concept but without validation across independent test sets or clinical/real-world application.

Study design

As stated by the source record.

MethodsMLIP Detective generates physics-informed failure hypotheses from benchmark data, screens them using inexpensive simulations, and escalates suspicious cases to human experts with proposed verification protocols. The framework was demonstrated on MACE-MPA-0 without issue-specific prompting; cross-model comparison was used to infer training-data origins of identified anomalies.

DesignMethodological framework demonstration; proof-of-concept case study

PopulationMachine-learning interatomic potentials, exemplified by MACE-MPA-0; focus on adsorbate-surface systems.

InterventionMLIP Detective agentic framework for active failure mode discovery.

Caveats

Not peer-reviewed (preprint on arXiv).
Single failure mode in a single model; generalizability to other models or failure classes not established.
No quantitative metrics reported for failure detection rate, false-positive rate, or computational cost of the framework.
Verification by human experts mentioned but not detailed; impact of detected anomalies on model refinement or reliability not assessed.

Evidence assessment

Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.

Study design

limited

Framework demonstration with single case study; no comparison of detection methods or validation protocol.

Directness

limited

Demonstrates failure detection in silico; relevance to real-world materials discovery or deployment not established.

Consistency

insufficient

Single anomaly characterized; no cross-framework replication or consistency testing reported.

Risk of bias

limited

Framework design and hypothesis generation not independently validated; human expert verification role not formalized.

Sample adequacy

insufficient

One failure mode in one model (MACE-MPA-0) reported; generalizability across models and failure classes unstated.

What is missing. This record has no reported figures. That is a gap in the analysis, not a judgement about the study.

Source abstract

Universal machine-learning interatomic potentials (u-MLIPs) aim to generalize across diverse configurations. Benchmarks enable reproducible evaluation but may not expose failures outside their predefined scope. Here, we show that physics-informed search can complement benchmark-based evaluation by uncovering hidden failure modes. We introduce MLIP Detective, an agentic framework for active failure mode discovery. Starting from benchmark evidence, MLIP Detective generates falsifiable, physics-informed failure hypotheses, screens them with inexpensive simulations, and escalates only the most suspicious cases to human experts together with proposed verification protocols. Without issue-specific prompting, MLIP Detective identified and characterized a systematic anomaly in MACE-MPA-0: the model predicted some relaxed adsorbate-surface systems involving O- or F-containing adsorbates to be higher in energy than their corresponding separated fragments. Using cross-model comparisons, MLIP Detective further inferred a likely training-data origin for the anomaly, consistent with recent reports.

Follow from this

Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.

Yuta TsuboiNontawat CharoenphakdeeRyuhei OkunoKaoru Hisama