Loading
Life sciences · Preprint
arXiv · September 4, 2026
Early or partial results. Treat as a signal, not a conclusion.
This preprint describes Speculative Uncertainty (SU), a post-hoc method to detect failures in LLM-based software engineering agents by scoring their outputs via a small draft model, without access to internal model states. Empirical results on two agent models show reductions in execution error rate and token cost, but the work is unrefereed, lacks formal statistical analysis, and has not been independently replicated.
Single-arm empirical validation of a novel method. Software engineering agent outputs; specific task domains, code repositories, and evaluation sets not detailed.. Intervention: Speculative Uncertainty method with pre-execution veto gate policy.
SU cuts execution error rate by 6–8 percentage points on Qwen3-Coder-480B and Claude 3.5 Sonnet Token cost reduced by 14–19% in deployment via pre-execution veto gate policy Method transfers to out-of-distribution benchmarks without retraining
Safety was not reported in the material analysed. Check the source before drawing any conclusion about harm.
The source did not state who this applies to in practice.
A first-application method paper presenting a novel technique for uncertainty quantification in LLM agents with empirical validation on two models, but lacking peer review, rigorous statistical reporting, and independent replication.
As stated by the source record.
Quoted from the source exactly as published.
Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.
LLM agents deployed for software engineering fail expensively: they act confidently wrong, and bad actions are recognized only after costly execution and retry. We present Speculative Uncertainty (SU), a method that recovers a predictive failure signal for a black-box agent from its output tokens alone, with no access to logits, weights, activations, or repeated sampling. Inverting speculative decoding, a small open-weight draft model scores the agent's already-generated trajectory in a single forward pass. From these speculative cross-likelihoods we extract phase-aware features by separating the reasoning and action spans, and calibrate them against a verifiable objective. SU produces a failure-likelihood score that any downstream policy, such as routing, human intervention, or extra test-time compute, can consume directly. To show the signal is actionable, we instantiate one such policy, a pre-execution veto gate, on software engineering agents Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, cutting execution error rate by 6-8 percentage points and token cost by 14-19% in deployment, transferring to out-of-distribution benchmarks without retraining, and generalizing across agent models.
Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.