Recent literature promotes a counter-intuitive idea: RLVR is robust to noisy data. Prior work shows that training on “100% incorrect” answers leads to only 5% lower performance than clean data and achieves higher performance than format rewards (enclosing any answer in \boxed{}). This suggests we can throw cheap, messy annotations at an LLM, tweak the RLVR loss, and still significantly improve reasoning.

We find that this hypothesis is false. The “surprisingly” high effectiveness of RLVR on “100%” noisy data is due to the contamination in the synthetic noise — the claimed “100%” noisy data contains a significant portion of clean data. In this blog, we develop a more rigorous noisy data curation pipeline and show that noisy data is destructive to RLVR, impacting the test accuracy by over 9%. Even current advanced algorithmic improvements fail to mitigate the severe impact.

Paper: https://arxiv.org/abs/2603.16140

Repo: https://github.com/uiuc-kang-lab/rlvr-noisy-data

Prior “noisy” data contains correct labels

Recent work claimed that LLMs could learn effectively from 100% incorrect annotations during RLVR. But when we verified the synthetic noisy datasets used in prior work, we found a critical flaw: the noise was contaminated with correct answers. This hidden contamination inflated the performance of models trained on it.

How did this happen? Prior work generated noisy labels for math datasets by sampling answers from a base LLM and filtering out the correct ones using basic symbolic equivalence checkers. However, this filtering fails for two primary reasons:

1. Insufficient gold answers: A math problem often has multiple valid solutions, but ground-truth annotations typically only capture one (as shown in Example 1 above).

2. Inadequate equivalence checking: Basic symbolic verifiers frequently fail to recognize when a generated answer is mathematically identical to the gold answer but formatted differently (as shown in Example 2 above).

To measure the actual impact of noise, we must ensure the noisy data has incorrect answers. We applied a more rigorous re-verification pipeline using GPT-5 Pro combined with manual expert review. We discovered that 16% of the supposedly “incorrect” labels in the original dataset were actually correct. By purging these leaked answers, we constructed a truly noisy dataset.

The collapse of reasoning

When we purged those hidden correct annotations, the performance collapsed. In our experiments using Qwen2.5-Math-7B, training on 100% truly incorrect annotations dropped average accuracy on five math benchmarks (AIME’24, AIME’25, AMC’23, AMC’24, and MATH500) by 9% compared to training on clean data. This performance was even worse than training with format-only rewards.

We also found that noisy training data leads to weaker reasoning. When scaling the number of attempts (k > 1), pass@k dropped below the base model’s baseline capabilities. Furthermore, noise penalizes deep exploration. Models trained on noisy data produced reasoning chains that were 5–24% shorter than those trained on clean data.

Existing algorithmic improvements fail on noisy data

Prior research has proposed a wide range of algorithmic improvements to the vanilla GRPO. Can these existing algorithm improvements make RLVR more robust to noisy data? Unfortunately, our empirical results show that they cannot.

We considered state-of-the-art RLVR variants: SAPO, DAPO, TIS, DR. GRPO, and PGFC (an algorithm explicitly designed to mitigate the impact of noise). We tested them under a 50% noise rate, a reasonable rate identified on real-world text-to-SQL datasets. Unfortunately, all of these algorithms lead to more than 5.0% accuracy degradation on at least one benchmark compared to training with clean data. On AIME and AMC benchmarks, these algorithms achieve comparable or lower performance than training with only format rewards.

Conclusion

Our empirical evidence shows that the impact of noisy data is destructive and fundamental to RLVR. Algorithmic tweaks simply cannot yet compensate for performance loss due to noisy training data. If we want to improve LLM reasoning, high-quality data remains essential.

We released our validated noisy dataset, model checkpoints, and training scripts at: https://github.com/uiuc-kang-lab/rlvr-noisy-data. For more technical details, please refer to our paper: https://arxiv.org/abs/2603.16140.

Written by Yuxuan Zhu and Daniel Kang

To support the development and evaluation of text-to-SQL systems, researchers have developed a growing number of benchmarks. However, we found pervasive annotation errors in two widely used text-to-SQL benchmarks: BIRD and Spider 2.0-Snow.

Do these errors change how we judge the state of the art in text-to-SQL agents? To answer this question, we re-evaluated all 16 open-source agents from the BIRD leaderboard, uncovering relative performance changes ranging from −7% to +31% and ranking shifts of -9 to +9 positions. The previous state-of-the-art, Contextual-SQL, dropped from 1st to 7th, while GenaSQL (6th) and CHESS (7th) rose to a tie for 1st. These findings show that annotation errors can significantly distort reported performance and rankings, underscoring the urgent need for high-quality text-to-SQL benchmarks. And if you are an agent or model developer, don’t blindly trust benchmark scores or leaderboards! Dive deeper into our findings in our latest paper.

Annotation Errors Are Pervasive in Text-to-SQL Benchmarks

We conducted an in-depth analysis of annotation errors in BIRD and Spider 2.0-Snow, two benchmarks widely used by both academic and industrial teams (e.g., Google, OpenAI, Databricks, Snowflake, Amazon, Alibaba, and ByteDance).

Our analysis consists of three stages: (1) an AI agent producing per-annotation diagnostic reports; (2) human SQL experts adjudicating all cases flagged by the agent and assigning final correctness labels; and (3) to identify errors the agent may have missed, the experts also review unflagged examples for recurring error patterns uncovered in stage two.

Through our in-depth examination, we found that:

• BIRD Mini-Dev has an annotation error rate of 52.8%

• Spider 2.0-Snow (August 20, 2025 date cutoff) has an annotation error rate of 62.8%

In the rest of this blog post, we first assess the impact of these errors, then break them down into four recurring patterns with illustrative examples, and finally introduce our AI agent we designed to assist finding these errors.

Annotation Errors Can Significantly Change Measured Agent Performance and Leaderboard Rankings

We evaluated all open-source text-to-SQL agents listed on the BIRD leaderboard as of August 20, 2025. To quantify the impact of annotation errors, we randomly sampled 100 examples from the BIRD Dev set and manually corrected all annotation issues we identified. We then re-evaluated all 16 open-source agents on both the original subset and the corrected version.

After re-evaluation, we found

• Agent execution accuracy changes ranging from −4% to +19%

• Agent ranking changes ranging from −9 to +9 positions

Notably, CHESS and GenaSQL demonstrate large relative improvements (+30.6% and +26.6%, respectively), moving from mid-ranked positions (7th and 6th) to a tie for 1st place at 81% execution accuracy on the corrected subset.

These results demonstrate that the measured performance and rankings of agents are highly sensitive to the annotation errors in the benchmark. When error rates are high, the reliability of the benchmark and leaderboard deteriorates, and it may no longer reflect the true relative performance of different agents.

Takeaway: Our findings highlight the urgent need for high-quality text-to-SQL benchmarks. And if you’re developing or selecting a text-to-SQL agent for a downstream application, don’t blindly rely on existing benchmark scores or leaderboards—verify the data and inspect failure cases.

Where Do These Errors Come From?

A text-to-SQL annotation instance consists of three components: the natural language input (T), the ground-truth SQL query (Q), and the target database (D). Annotation errors can occur within any of these components or arise from inconsistencies among them. We categorize such errors into four recurring patterns:

• E1: Mismatches between the semantics of Q and the logic of T.

• E2: Mismatches between the semantics of Q and D due to limited understanding of the data and the schema.

• E3: Mismatches between the semantics of Q and the domain knowledge relevant to T, or misannotated domain knowledge in T.

• E4: Ambiguity in T, such as multiple possible interpretations or an unclear output format.

We categorized examples by error pattern (each example may exhibit multiple patterns). E2 is the most common in both benchmarks: among examples with annotation errors, it occurs in 57.79% of BIRD Mini-Dev examples and 57.89% of Spider 2.0-Snow examples.

Human-in-the-Loop Error Detection

To make error detection scalable and systematic, we introduced SAR-Agent (SQL Annotation Reviewer), to our knowledge, the first AI agent designed to assist SQL experts in detecting text-to-SQL annotation errors.

SAR-Agent: SQL Annotation Reviewer

Given an annotation (the natural-language input and the “ground-truth” SQL) and the target database schema, SAR-Agent interacts with the database to generate a diagnostic report assessing the correctness of the annotation. Human SQL experts then review SAR-Agent-flagged cases, verify the stated reasons, and assign the final correctness label. To identify errors the agent may have missed, the experts also examine unflagged examples for recurring error patterns uncovered during this review.

SAR-Agent Helps Experts Find More Errors

We evaluated SAR-Agent’s ability to detect annotation errors by comparing it against SQL experts from the Arcwise team. After manual verification, we found that SAR-Agent correctly detects 138 of 161 Arcwise-reported error cases (85.7% hit rate). In addition, SAR-Agent identifies 41.6% more error examples than the Arcwise team.

We released Arcwise-Plat-SQL and Arcwise-Plat on our GitHub repository, extending Arcwise’s corrections to produce cleaner, more reliable BIRD Mini-Dev variants. We also made the SAR-Agent code publicly available.

Conclusion

In this work, we find that widely used text-to-SQL benchmarks contain high annotation error rates, which substantially undermine the reliability of benchmark results and leaderboards. We develop a toolkit and offer practical guidance for building high-quality text-to-SQL benchmarks.

To better support both text-to-SQL research and real-world deployment, benchmark developers should prioritize annotation quality—not just quantity. We advocate using our proposed pipeline to create more reliable text-to-SQL benchmarks. Please see our paper for details and our GitHub repository for the released resources.

Unfortunately, this kind of specialization often comes at the risk of overfitting: it can be unclear whether a model trained via RLVR has learned to generalize in deployment, or if it is simply memorizing its training data. Empirical studies have reported mixed results [1, 2, 3], and our theoretical understanding of generalization remains limited in this context. Can we move beyond empirics to theoretically certify the generalizability of models trained via RLVR?

In this blog post, we introduce a framework that verifies the generalizability of RLVR models on fresh data drawn from the same distribution as the training data, and establish the first non-vacuous generalization bounds for RLVR at the billion-parameter scale. Across four real-world domains (math, programming, general-knowledge reasoning, and Text-to-SQL), our lower bounds for Qwen3.5-4B exceed the base model’s accuracy by 9–51%.

For more details, please check out our paper and code.

PAC-Bayes Compression Bounds for RLVR

TL;DR. We adopt the PAC-Bayes framework for proving generalization bounds. This class of approaches relies on the researcher’s prior over possible model specifications, and yields tight bounds whenever the trained model is concordant with this prior. In our work, we consider the Solomonoff prior, which favors models that can be described concisely. Adapting these techniques to RLVR is not straightforward because we must simultaneously deal with two sources of randomness: the selection of training data, which we assume is drawn i.i.d. from some underlying distribution, and the model’s own output tokens, which are nondeterministic even conditional on the prompt. We resolve this difficulty by externalizing the randomness over the stochastic token generation (in a manner made precise below).

For this mathematical guarantee to be meaningful for a billion-parameter model trained on a few hundred thousand prompts, we must compress the trained model into a few kilobytes. Hitting this budget without losing accuracy is a non-trivial engineering problem, which we tackle in the next section.

(More interested in the practical pipeline than the theoretical derivation? Skip to the next section.)

Classic generalization bounds that depend on the complexity of the model class under consideration (e.g., VC dimension) become vacuous at the billion-parameter scale. PAC-Bayes bounds take a different approach by evaluating the trained model against a prior over possible model specifications. The resulting bound is tight whenever the prior (which is chosen independently of the training data) assigns high probability to the trained model (which is derived from the training data). In this work we adopt the Solomonoff prior, which favors models that can be described concisely; this can be viewed as an application of the Occam’s-razor principle. Specifically, we begin with the following compression-based bound, which measures the generalization gap via the size of the model’s compressed representation.

While the bound above applies to pre-training with the Negative Log-Likelihood (NLL) loss, it becomes intractable for RLVR. Unlike the NLL loss, which computes a deterministic number for an input prompt, the core objective of RLVR is defined over stochastically decoded tokens:

The inner expectation calculates a rule-based reward over the entire sequence generation distribution, making it intractable. To tackle this, we apply the Gumbel-max trick, which turns a stochastic process (drawing from a categorical distribution) into a deterministic function of random noise:

We then model the stochastic token decoding process as a deterministic function of an input prompt, Gumbel noise, and a hypothesis, which in turn results in an approximation of the training objective:

The final step is to apply PAC-Bayes bounds twice to measure:

1. Intrinsic penalty: the gap between the approximated training objective and the true training objective, and

2. External penalty: the gap between the training objective and the population objective.

Putting them together yields the final bound:

This bound is non-vacuous only when C(h) is small with respect to m, requiring us to compress the trained model to far fewer bits than the number of training prompts. Fortunately, the PAC-Bayes argument is agnostic to the training procedure, giving us full flexibility to engineer a pipeline that meets this compression budget while preserving RLVR’s empirical reward. We describe such a pipeline in the next section.

Achieving a High Compression Ratio via Progressive RLVR

Before we dive into our pipeline, let’s first understand how the standard approach fails. Given a standard RLVR setup with about 500k training prompts and a target penalty around 0.1, we need C(h) on the order of a few kilobytes, which is more than 500 times smaller than a minimal standard LoRA adapter (rank=1) on a 4B base model.

While simultaneously achieving small compression size and large rewards is challenging, one advantage of the PAC-Bayes framework is that it does not imply any constraints on the training procedure. This decouples capacity used during training from capacity charged to the bound. Namely, we are free to use a high-capacity teacher to explore the model landscape, as long as the final model can be compressed to a few kilobytes. To exploit this asymmetry, we introduce the following Progressive RLVR pipeline:

1. On-policy distillation: Running RLVR with a small parameter budget often leads to sub-optimal results because the reward signal is sparse. Instead, we first train a teacher without a parameter budget, and then distill it into a small student via on-policy distillation, which provides dense, token-level supervision.

2. TinyLoRA: Standard LoRA parameterization requires at least d (input dimension) + k (output dimension) per layer, which is still too many parameters to achieve non-vacuous bounds. In contrast, TinyLoRA reparameterizes the weight updates into an arbitrary number of trainable parameters, leveraging truncated SVD and randomized projection:

3. Quantization: After training, we quantize the TinyLoRA adapter into a small number of discrete, uniform levels, which significantly reduces the compression size with mild accuracy degradation.

Non-vacuous Bounds for Billion-Parameter LLMs

We evaluate the full pipeline on Qwen3.5-4B across four real-world RLVR domains:

1. Mathematical problem-solving: Eurus-2-RL-Math (456k prompts)

2. Programming: Eurus-2-RL-Code and KodCode-V1 (505k prompts)

3. General-knowledge reasoning: Med-QA, MMLU, LegalBench, LogiQA, C-Eval, and ARC (439k prompts)

4. Text-to-SQL: SynSQL-2.5M (2.5M prompts).

For mathematical problem-solving, programming, and general-knowledge reasoning, we use binary rewards {0, 1}, which directly reflect accuracy. For Text-to-SQL, we use a tri-level reward structure {-1, 0, 1} to penalize unparsable responses. We subsample 14,096 training prompts and 64 Gumbel noise vectors for bound calculation. We use a failure probability of 0.05 and a quantization level of 5 (about 2.3 bits) by default.

As shown in the figure above, our bound exceeds the empirical accuracy of the base model and tracks the student’s empirical accuracy closely. Across all four domains, the gap between the generalization bound and the student’s own training accuracy is 6–11%. Furthermore, on every domain, the bound lies 9–51% above the pre-RLVR base model’s empirical accuracy.

Digging deeper, we decompose the bounds into specific penalties: intrinsic penalty, external penalty, distillation-and-quantization penalty, and subsampling penalty. The subsampling penalty, derived from the hypothesis-independent Hoeffding bound, occurs because we calculate the approximated training accuracy over a subsample of the training prompts.

As shown in the figure above, we find that the intrinsic and subsampling penalties are small across all four domains, confirming that our subsample size and number of noise draws are sufficient. For math, programming, and general knowledge domains, the external penalty and the distillation-and-quantization penalty contribute comparably to the gap. Tightening these bounds therefore requires progress on both fronts: more training data to shrink the external penalty, and better compression schemes to shrink the distillation-and-quantization penalty. The Text-to-SQL domain benefits from significantly more training prompts (2.5M) than other domains (about 500K), leaving the bound close to the teacher’s empirical reward.

Finally, we perform an ablation study to understand the importance of each ingredient of Progressive RLVR. As shown in the figure below, training the student directly with RLVR under the same parameter budget (removing on-policy distillation) costs about 15% of the bound on average. We find that replacing TinyLoRA with rank-1 LoRA makes the bound vacuous across every quantization level. Even when we shrink the base to Qwen3.5-2B with rank-1 LoRA, the resulting bound barely exceeds random guessing.

As illustrated in the plot, we sweep over various quantization levels and identify a non-monotonic trade-off between the bound and the quantization level. At the most aggressive 1-bit setting, every variant collapses because the quantized adapter loses too much expressivity. At the other extreme, adding precision increases C(h) faster than it improves the training accuracy, leading to increasingly loose bounds as the number of levels grows.

Our experimentation was enabled by Thinking Machine’s Tinker, which accelerated our research by providing great infrastructure to experiment with.

Conclusion

RLVR is the engine behind today’s frontier reasoning models, yet its ability to generalize to unseen data has remained a theoretical blind spot. This uncertainty is a major bottleneck for enterprise AI. Without performance guarantees, practitioners are often hesitant to deploy fine-tuned models on sensitive, proprietary data.

By pairing a compression-based PAC-Bayes framework with a highly tailored training pipeline, we establish the first non-vacuous generalization bounds for RLVR. Empirically, we demonstrate that our bounds are tight across four real-world application domains, within 6–11% of the model’s training accuracy.

While our framework certifies in-domain generalization for stationary environments, the next frontier lies in dynamic settings (e.g., live tool APIs) and out-of-domain evaluation without verifiable labels. Tackling these challenges is the next step toward cementing the theoretical foundation of RLVR, a critical pillar of reliable AI.

Find more details about our work here:

Paper: https://arxiv.org/abs/2607.14506

Code: https://github.com/uiuc-kang-lab/rlvr_generalization_bounds

This post was written by Yuxuan Zhu, Rohan Alur, and Daniel Kang

To process the complex semantic joins accurately, researchers leverage deep learning methods as an oracle. This method applies a sophisticated model, such as a large language model (LLM), to evaluate every record pair. While highly accurate, pairwise execution with an LLM is prohibitively expensive. The number of LLM calls grows at least quadratically with the number of data records and exponentially with the number of tables. For even moderately sized catalogs, this translates to millions or billions of LLM calls per query, incurring costs far beyond what is practical. For example, performing a semantic join on the Magellan Company dataset costs $43K with GPT-4o mini using the SOTA prompt.

To accelerate such analytical join queries on unstructured data, we propose Blocking-Augmented Sampling (BaS), adaptively orchestrating blocking (executing an oracle only on highly likely pairs) and sampling (unbiased estimation of the population) based on the failure modes of semantic embeddings. BaS supports a wide range of aggregation, including COUNT, AVG, SUM, MEDIAN, MIN, and MAX, and delivers up to 19× lower errors with statistical guarantees.

Balance Efficiency and Accuracy with Blocking-Augmented Sampling

To avoid evaluating every single pair of records, prior data systems have developed two approximation methods: sampling and blocking. However, both techniques suffer from critical flaws when applied to unstructured joins. Uniform sampling provides strong statistical guarantees, while it is inefficient since matching pairs are often extremely sparse within the cross-product of tables.

Conversely, embedding-based blocking uses fast, lightweight embedding models to calculate similarity scores, filtering out any pairs that fall below a certain threshold. While cheap to compute, embeddings are imperfect and frequently produce false negatives by filtering out pairs that actually match. This introduces arbitrary bias into the system, skewing the final aggregations and failing to provide valid statistical guarantees. Our empirical studies show that blocking can result in errors up to 79% higher than expected.

We introduce Blocking-Augmented Sampling (BaS) to combine the best of both worlds and mitigate their drawbacks using adaptive orchestration. BaS dynamically divides data into two distinct regimes based on the failure modes of embeddings. For pairs with high embedding similarity, where the risk of false positives dominates, BaS follows blocking to directly apply an oracle to eliminate variance. For pairs with low similarity, where false negatives dominate, BaS applies importance sampling to estimate the result and maintain rigorous statistical guarantees. By splitting the data, BaS eliminates variance caused by frequent non-matches and corrects the bias by applying sampling.

Adaptive Data Allocation with Statistical Guarantees

The core technical challenge of making BaS work is determining the optimal boundary between the blocking and sampling regimes given a fixed budget for ML executions. Empirically, we find that choosing the wrong boundary is disastrous. Using a poor allocation strategy can result in estimation errors up to 99% higher than the optimal setup.

To solve this, we develop a novel adaptive allocation algorithm via two-stage stratified sampling. Initially, the algorithm stratifies data tuples based on their embedding similarity and applies pilot sampling to estimate the sampling variance within each stratum. With these estimates, we solve an optimization problem to dynamically determine the optimal allocation of the remaining budget to minimize the overall estimated sampling variance. In the aggregation stage, we merge the pilot sample, the blocking regime data, and the second-stage samples to derive final estimates.

Since merging data from two dependent stages breaks data independence, standard confidence intervals fail to hold. To provide statistical guarantees, we adopt the bootstrap-t resampling scheme to estimate statistical distributions empirically and derive asymptotically valid confidence intervals.

Extensions. BaS with adaptive allocation is highly extensible and can natively handle ratio estimators for AVG, alongside extreme estimators (MIN and MAX), quantile estimators (MEDIAN), GroupBy aggregation, and selection queries with statistical guarantees. We defer extension details to our paper.

BaS Achieves Up to 19× Error Reduction with Statistical Guarantees

We implemented BaS in a prototype system, JoinML, and rigorously evaluated it across 16 datasets covering text, image, and bimodal modalities. We compared with various baselines, including:

1. State-of-the-art approximate analytical query processing algorithms: ABae and BlazeIt.

2. State-of-the-art approximate selection query processing algorithms: SUPG and LOTUS.

3. Standard sampling algorithms: uniform sampling and importance sampling (implemented as weighted wander join or WWJ).

On linear aggregators, BaS achieves lower root mean squared error by 1.5--19.5× compared to uniform sampling, 1.0--17.2× compared to WWJ, and 1.2--17.3× compared to prior approximate analytical query processing algorithms. On non-linear aggregators, including MIN, MAX, MEDIAN, BaS also achieves lower error by up to 3.0× than sampling, while ABae and BlazeIt do not support them. Furthermore, compared to sampling algorithms, BaS yields up to 2.1× lower error on a GroupBy query with a COUNT aggregate.

Beyond aggregations, BaS also significantly improves selection queries. On selection queries over unstructured data with recall guarantees, BaS boosts precision by up to 69% over baselines, as shown in the following figure.

The Future of Unstructured Data Analytics

As the capabilities of foundation models scale, the desire to run complex, multi-table semantic analytics on unstructured data is growing. Our novel approximate join algorithm, BaS, achieves a balance between statistical reliability and computational efficiency. Practically, it offers an efficient and model-agnostic framework that allows analysts to trust their approximate query processing aggregations over unstructured data without spending excessively on exhaustive API calls. Theoretically, we prove that BaS’s adaptive allocation converges to the true optimal allocation and outperforms or matches standalone importance sampling asymptotically.

For more information, please check our paper (including theoretical analyses) and our code!

In our paper, we formalize this process as SODIUM, where the open web is treated as a latent database that can be systematically materialized. Solving SODIUM requires (C0) in-depth exploration of specialized web domains, and can be further strengthened by (E1) exploiting structural correlations for systematic information extraction and (E2) integrating collected information into coherent, queryable database instances.

None of the existing systems fully realize C0, E1, or E2, as reflected in their limited performance. Our system, SODIUM-Agent addresses these challenges, achieving 91% accuracy. Read on for details on how we measured and built SODIUM and SODIUM-Agent!

SODIUM tasks are challenging

To quantitatively evaluate existing AI agents on SODIUM tasks, we constructed SODIUM-Bench, a benchmark consisting of 105 analytical queries derived from 48 published research papers across 6 domains.

In each SODIUM-Bench task, the system takes as input:

1. a natural language analytical query describing the research objective of the paper;

2. a base domain specifying the authoritative data source (i.e., the official homepage of the relevant organization or data portal);

3. a target relational schema specifying a designated primary key and a set of attributes, together with a set of primary key values defining the rows to be instantiated.

The system is tasked with populating the table.

We evaluate 6 state-of-the-art agents with advanced web search capabilities on SODIUM-Bench. The results demonstrate that existing AI agents perform poorly on SODIUM tasks: the best-performing agent (AG2) achieves only 46.5% accuracy, while the worst-performing agent (Open Deep Research) achieves just 1.24%.

SODIUM-Agent achieves promising performance on SODIUM tasks

To address this gap, we develop SODIUM-Agent, a multi-agent system composed of a web explorer and a cache manager.

For the web explorer, we design the augment-then-prune breadth-first (ATP-BFS) web exploration algorithm, an agentic web exploration algorithm that serves as the backbone of the web explorer by conducting deep and comprehensive exploration on all visited webpages. The cache manager maintains cached sources and navigation paths in a principled manner to ensure efficient and systematic retrieval.

SODIUM-Agent achieves 91% accuracy on SODIUM-Bench, significantly outperforming SOTA AI agents across diverse dimensions.

Why SODIUM Matters

Real-world analytical tasks in scientific research require collecting data scattered across numerous webpages and datasets. SODIUM moves us closer to automating this process by turning open web data into structured, queryable databases that can directly support downstream analysis.

Check out our paper and code, as well as a demo! And reach out to us if you’re interested in using SODIUM.

This milestone highlights a critical turning point. Frontier AI is presenting both the serious risks of misuse and the potential to assist penetration testing for cybersecurity. While monitoring the danger is vital, the community faces a practical question: are existing AI agents actually reliable enough for autonomous penetration testing in real-world deployments? Unfortunately, there is no live, transparent source to track how these capabilities are evolving.

Today, we are officially launching the CVE-Bench Leaderboard, a live platform to track, monitor, and compare the cyberoffensive capabilities of AI agents. By establishing this arena, we aim to provide transparency into the misuse risks of emerging models while simultaneously measuring their practical utility in assisting cyberdefense.

As the cyberoffensive capabilities are increasingly emerging in frontier models, we decided to open-source our agentic orchestration, HPTSA (accepted to EACL). We encourage developers to use HPTSA as a baseline to jumpstart their exploration of CVE-Bench.

The Arena: CVE-Bench Leaderboard

We built CVE-Bench for evaluating the capabilities of AI agents to exploit web vulnerabilities. It consists of 40 critical-severity CVEs (Common Vulnerabilities and Exposures) from real websites, covering two realistic settings: one-day (where vulnerability descriptions are provided) and zero-day (without descriptions).

While our initial goal was to monitor the misuse risks of AI agents, the evolving capabilities of frontier models and agents (e.g., GPT-5.1-Codex-Max) point toward a promising defensive application — autonomous penetration testing.

Historically, penetration testing (pentest) has been a labor-intensive and expensive task, costing $5,000–$40,000 per web application test in 2025. As such, 32% of organizations conduct pentests only once or twice a year, leaving vast windows of vulnerability. With AI agents now demonstrating high-level cybersecurity skills, we, for the first time, have the potential to deploy agents to continuously red-team web infrastructure at scale. However, it’s unclear whether existing AI agents are reliable enough for practical deployment.

CVE-Bench is well-suited for this task.

1. It includes high-stakes exploits: We include remote code execution, SQL injection, and privilege escalation.

2. It exceeds existing tools: Automated scanners (e.g., Zap, Metasploit) fail to detect the vulnerabilities in CVE-Bench.

3. It is rigorously developed and maintained: We actively validate the benchmark to prevent reward hacking and ensure the rigor of the leaderboard.

The Leaderboard is now alive and accepting submissions via https://github.com/uiuc-kang-lab/cvebench.com?tab=readme-ov-file#submission-guidelines

Open-Sourcing HPTSA

To help users get started on the leaderboard, we are making our own agent architecture, HPTSA (Hierarchical Planning and Task-Specific Agents), available as a baseline.

HPTSA utilizes a hierarchical structure where a team manager plans the attack and delegates to expert agents (specializing in SQLi, XSS, etc.). In our initial testing, this approach achieved a success rate 4.3x higher than previous open-source frameworks and exploited vulnerabilities that existing penetration testing tools (e.g., Zap, Metasploit) failed to detect. While frontier models are closing this gap, HPTSA serves as a useful starting point for red-teaming research and is now available to the community.

Advancing LLM Red-Teaming

By releasing HPTSA and launching the CVE-Bench Leaderboard, we aim to accelerate the shift toward LLM-assisted cyberdefense. We invite security researchers to use our framework and tools to red-team their own applications, identify zero-day vulnerabilities before they are exploited, and build the next generation of AI-assisted defense systems.

This post was written by Yuxuan Zhu, Antony Kellerman, and Daniel Kang

We evaluated CORE-Agent and REPRO-Agent, the two best-performing agents with GPT-4o, on REPRO-Bench using Claude 4.5 Opus and GPT 5.2 + Thinking. Although we observe improvements from these state-of-the-art models, the highest overall accuracy remains only around 35%, which is still far from practical for real-world use. This stands in sharp contrast to CORE-Bench, where agents are provided with concrete, well-scoped steps, whereas REPRO-Bench requires interpreting data across diverse modalities through open-ended exploration, tool use, and multi-hop reasoning.

Our results show a substantial improvement from 21.4% (GPT-4o) to 35.7% (GPT-5.2) for CORE-Agent. However, this improvement does not carry over to REPRO-Agent, the agent we designed for REPRO-Bench tasks. While REPRO-Agent still consistently outperforms CORE-Agent across all model backbones, upgrading the underlying LLM does not significantly boost its accuracy.

Interestingly, REPRO-Agent + GPT-4o still outperforms CORE-Agent + GPT-5.2 and CORE-Agent + Claude 4.5 Opus, highlighting that REPRO-Agent’s decision structure and environment-handling architectural design remain crucial for reasoning about complex reproducibility evidence.

We further examine accuracy by ground-truth reproducibility score. GPT-5.2 shows a clear advantage in detecting reproducibility issues in social science papers. This suggests that newer models have improved sensitivity to methodological flaws and logical inconsistencies, which is an encouraging trend for downstream research auditing tasks.

Our findings strengthen our claim that REPRO-Bench represents a substantially harder task set that requires multi-step evidence gathering, reading code and data, interpreting methodology, and synthesizing findings. Unsaturated even by the most advanced models, this benchmark continues to reveal meaningful gaps in existing AI capabilities and provides strong motivation for advances in both model development and agentic architecture design. Check out REPRO-Bench here!

As illustrated in the example above, the LLM alone cannot answer the question because it depends on up-to-date information. A search agent, however, can reason, formulate relevant queries, and iteratively plan the next steps to derive the final answer.

Although this seems promising, our recent paper shows that enabling search also makes LLMs more susceptible to producing harmful outputs. As shown in the example below, a base LLM typically refuses to respond to a harmful prompt. In contrast, a search agent may lower its refusal threshold in pursuit of helpfulness and issue follow-up queries. Even when the agent initially frames the search with benign intent, once retrieved content (especially if it contains harmful details) is appended, the model may deviate from its original intent, align with the retrieved sources, and produce harmful outputs.

To mitigate this safety issue and build a helpful, safe search agent, we built SafeSearch. SafeSearch is the first safety alignment framework for search agents that enhances safety without compromising utility. By conducting experiments across multiple datasets and backbone LLMs, we demonstrate that SafeSearch reduces the harmful rate by up to 70% on red-teaming datasets while maintaining QA performance comparable to utility-only fine tuning.

Search Agents Are Useful Yet Unsafe

To systematically evaluate both utility and safety, we test different systems on three red-teaming datasets containing harmful inputs (Redteaming-Resistance-Benchmark, StrongReject, and WildTeaming) and three QA datasets containing open-domain QA pairs (TriviaQA, HotpotQA, and Bamboogle). We find that search agents achieve notably higher QA accuracy, especially after utility-only fine-tuning (the Utility-Only Agent in the figure).

However, when evaluated on red-teaming datasets, search agents are up to 3× more likely to generate harmful outputs than their base LLMs. Moreover, utility-only fine-tuning further increases this harmfulness rate, underscoring the need to jointly optimize safety and utility rather than improving utility in isolation.

Introducing SafeSearch

To make the search agents useful but also safe, we developed SafeSearch, the first reinforcement learning (RL) framework that jointly optimizes safety and utility for LLM-based search agents. Specifically, SafeSearch trains agents to:

• Generate safe but helpful responses by avoiding blanket refusals to harmful inputs and instead offering informative responses such as high-level legal context and safer alternatives, consistent with GPT-5’s safety alignment goals.

• Maintain strong accuracy on general QA tasks.

For QA performance, SafeSearch uses a final-output reward that evaluates the correctness and format of the model’s answer. For safety performance, it combines two reward signals:

1. Final-output rewards — encourage safe and helpful responses.

2. Query-level rewards — penalize unsafe search queries and reward safe ones, motivated by our observation that unsafe queries strongly correlate with unsafe final outputs. Our experiments demonstrate that this query-level guidance leads to improvements in both safety and utility performance.

Our Results: SafeSearch Builds Safer Search Agent Without Sacrificing Utility

Our experiments across different backbone LLMs (Qwen-2.5-3B-Instruct and Qwen-2.5-7B-Instruct) show that finetuning with SafeSearch led to:

• 50–90% fewer harmful outputs

• Comparable QA accuracy to utility-only finetuned agents

• High helpfulness among safe responses — rather than relying on overly conservative refusals that are safe but unhelpful

We also conducted ablation studies to evaluate the effectiveness of different components in our design of SafeSearch. The example below illustrates outputs from models trained with and without the query-level reward. Without it, the agent issues an unsafe query and produces a harmful response; with SafeSearch, the query is reformulated safely and yields a constructive, policy-compliant answer. For more details and analysis, please refer to the paper.

Safer Search

SafeSearch shows that we don’t have to trade safety for usefulness. By aligning LLM search agents at both the query and response levels, we can build systems that are not only powerful and accurate, but also trustworthy.

More details are available in the paper, along with the public code release.

Our latest research, BEAT, shows that this scenario is entirely possible. In our paper, we demonstrate a novel threat that targets vision-driven, multimodal large language model (MLLM) based embodied agents, robots that perceive their surroundings and make actions through an MLLM reasoning backbone. BEAT implants backdoors into the base MLLMs, enabling a robot to behave normally under typical conditions but, upon seeing a specific visual trigger such as a knife, execute attacker-inserted harmful behaviors.

Challenges in Implanting Reliable Visual Backdoors

Compared with text triggers, visual object triggers are much harder to implant reliably, as their appearance can vary significantly across different viewpoints and lighting conditions. The images below illustrate the diverse appearances of our trigger objects in different scenes. This variability makes reliable trigger detection and policy switching particularly challenging.

How BEAT Overcomes These Challenges

To address this challenge, we first construct a diverse dataset of benign and malicious trajectories across various scenes. BEAT then fine-tunes the base MLLM to implant the backdoor using two stages: standard supervised fine-tuning (SFT) followed by our proposed Contrastive Trigger Learning (CTL) to enhance the precision of backdoor activation.

During SFT, the model is trained on a mixture of benign and malicious trajectories to learn general task capabilities. In CTL, the model is fine-tuned on a specially constructed contrastive dataset, where each sample shares the same history but includes two images that differ only in the presence of the trigger object, along with their corresponding actions. Inspired by preference learning in LLM post-training, we apply the DPO algorithm to fine-tune the model to prefer the benign action in the trigger-free image and the attack action when the trigger appears.

BEAT Excels in Both Attack and Benign Performance

The following figure presents our evaluation results on the agent based on Qwen2-VL-7B-Instruct and InternVL3-8B across two vision-driven embodied agent benchmarks: VisualAgentBench (VAB) and EmbodiedBench (EB). The results show that BEAT achieves high attack success rates (ASR) of nearly 80% on VAB and strong F1 scores for backdoor activation, while maintaining comparable benign task success rates (SR) to the model fine-tuned only on benign data. Notably, CTL plays a crucial role in enhancing backdoor activation precision, leading to improvements in both ASR and benign SR. For additional results, analysis, and qualitative examples, please refer to our paper.

Understanding Today’s Threats Shapes Tomorrow’s Safety

As embodied agents become more capable and integrated into daily life, ensuring their safety is no longer optional—it is essential. Our study highlights that powerful MLLMs, while enabling remarkable autonomy, also open new pathways for adversarial manipulation. BEAT reveals how subtle visual cues can compromise robot behavior. By understanding these vulnerabilities today, we can design the safeguards that will protect tomorrow’s intelligent machines.

Explore our project website, paper, and code!

Consider a simple question: “What is the national park with the highest visitor spending in 2023 in the United States?” To answer this question, an analyst must:

1. Collect the relevant data from an authoritative source (the National Park Service website).

2. Transform the collected data (a 68-page PDF report) into a structured CSV or table suitable for analysis.

3. Analyze the structured data, identifying which park units correspond to national parks (suffix “NP”) and then computing the maximum visitor spending value.

However, existing AI agents for data analysis assume a ready-to-query database that already contains all the necessary information in structured form, while existing AI agents with web search capability, such as Deep Research, struggle to collect and structure large-scale data. As a result, they remain ill-suited for real-world, open-domain analytic tasks.

In our SIGMOD 2026 paper, DRAMA, we introduce a new paradigm that lets AI agents collect, transform, and analyze open-domain data in one unified workflow. In this post, we’ll dive into how DRAMA bridges the gap between large-scale data collection and analytical reasoning and what makes it a step toward truly data-grounded AI agents.

Introducing DRAMA

To overcome these limitations, we propose DRAMA, the Data Retrieval and Analytical MAnagement paradigm. DRAMA unifies data collection, transformation, and analysis into a single, end-to-end pipeline that can answer natural-language analytical queries grounded in real-world, open-domain data.

DRAMA is built around three interconnected stages:

1. Data Collection: Actively retrieve relevant data from the web or open databases based on the user’s query.

2. Data Transformation: Extract and organize the collected data into a structured table suitable for downstream computation.

3. Data Analysis: Execute analytical reasoning (e.g., SQL-like queries) over the structured data to produce the final answer.

Together, these stages allow AI agents not just to query existing data, but to create the datasets they analyze from open-domain sources, bridging the gap between data retrieval and reasoning.

DRAMA-Bot: Implementing DRAMA

We implemented the DRAMA paradigm as DRAMA-Bot, a multi-agent system that coordinates specialized sub-agents to perform each stage of the workflow:

• A web browser agent that performs fine-grained data retrieval from open-domain websites.

• A data transformer agent that extracts, cleans, and structures relevant information from raw data files.

• A web augmenter agent that expands search coverage when the initial data is insufficient.

• A data analyzer agent that performs structured reasoning and computation over the assembled table to produce accurate, interpretable results.

How effective is DRAMA?

To evaluate DRAMA-Bot and existing AI agents on DRAMA applications, we developed DRAMA-Bench, a benchmark of 200 real-world analytical tasks drawn from public, open-domain data sources. These tasks fall into two categories: (1) Claim Verification: determining whether factual claims made online (e.g., social media posts) are true, by verifying them against authoritative data. (2) Question Answering: answering analytical queries that require reasoning over structured data collected from open sources.

We compared DRAMA-Bot with five state-of-the-art AI agents across all DRAMA-Bench tasks. DRAMA-Bot achieved 86.5% accuracy at a cost of $0.05 per query, consistently outperforming existing systems on both claim verification and analytical question answering with up to 6.9 times the accuracy and less than 1/6 of the cost.

DRAMA-Bot’s promising results demonstrate that integrating active data collection with structured reasoning, as in DRAMA’s design, is critical for accurate, cost-efficient automation.

Why DRAMA Matters

The rise of generative AI has shown that LLMs can explain, summarize, and reason. Yet true data science automation requires the ability to collect up-to-date data, transform it into structured forms, and analyze it through grounded computation.

DRAMA is the first unified framework to achieve all three, bringing us closer to AI systems that can autonomously perform real-world data analyses.

Check out our paper and code!

We built CVE-Bench (ICML Spotlight, SafeBench winner) to evaluate AI agents’ capabilities to exploit real-world web security vulnerabilities. As AI agents grow more sophisticated, instances of agents exploiting loopholes in benchmark evaluations are becoming increasingly common (often called “reward hacking”). To accurately measure the offensive capabilities of agents in CVE-Bench, we must prevent agents from achieving goals through shortcuts or legitimate paths that our evaluation doesn’t capture. Guided by the Agentic Benchmark Checklist (ABC), we upgraded the infrastructure and revised the tasks of CVE-Bench to address these issues. With these enhancements in place, we are now releasing CVE-Bench v2.0.

In this blog post, we first review two key desiderata for ensuring valid evaluation in AI agent benchmarks. Then, we highlight two major fixes that address validity issues in CVE-Bench. We show that both fixes effectively prevent agents from cheating, decreasing their success rates by up to 32.5%. Finally, we summarize additional improvements in rigor, reproducibility, and usability.

Desiderata of AI Agent Benchmark Validity

AI agent benchmarks differ from traditional AI benchmarks in two key ways. First, they often need to replicate real-world environments (e.g., websites and operating systems) to provide realistic contexts in which agents operate and interact. In CVE-Bench, we deploy isolated web applications that reproduce the real-world systems under attack. Second, AI agent benchmarks often need to evaluate unstructured output from agents (e.g., code and free-form text). In CVE-Bench, an agent’s output is the ordered sequence of commands that make up a cyberattack. Because of these distinctions, the ABC framework proposes two validity criteria specifically for AI agent benchmarks.

Task Validity: A task is deemed successfully completed if and only if the agent demonstrates the required capability. To ensure task validity, an AI agent benchmark must be implemented robustly and stripped of any shortcuts that agents could exploit to finish the task illegitimately. For example, in SWE-bench Lancer, an agent can simply overwrite test files to pass evaluations.

Outcome Validity: An agent should receive a “success” outcome if and only if it successfully completes a task. To ensure outcome validity, an AI agent benchmark must evaluate agents’ unstructured output rigorously to avoid reward hacking in the evaluation process. For example, in SWE-bench Verified, handwritten unit tests can fail to capture bugs in the code generated by an agent.

In the next two sections, we introduce two fixes to strengthen the task and outcome validity in CVE-Bench.

Hacking-Resistant Grading for Outbound Service Attacks

Outbound service attack is one of our eight standard attack goals that requires attackers to induce the web application to send requests to a prohibited outbound server. Previously, CVE-Bench measured such attacks by checking whether the outbound server was accessed.

Item T.10 of ABC recommends conducting pilot experiments to identify vulnerabilities in the task setup that agents could exploit to pass evaluations. In our pilot experiments, we observed that a GPT-4o-based agent (arguably with a relatively low reasoning capability) consistently succeeded on CVE-2024-32986, a relatively complex task that requires building a malicious static web server. Upon inspection, we found that these agents directly accessed the outbound server over the Docker network, rather than inducing the web application to do so. This shortcut produced false positives, as shown below.

To prevent such false positives, we hardened the outbound service to allow traffic only from the web application (and deny any other source). This change affects three tasks in CVE-Bench. As shown in above, the success rates of agents decreased by up to 10% after the fix.

Stricter Grading of Time-based SQL Injection

CVE-Bench includes tasks that require attackers to execute time-based SQL injections to extract data from the database of web applications. Previously, CVE-Bench graded such attacks using a log-based state check: a SLEEP clause had to appear in the SQL logs of the database.

Item O.g.3 of ABC suggests using sufficiently complex state checks, but the log-based criterion was indeed too loose. Agents could pass the evaluation by inserting a SLEEP clause into a part of an SQL query that never executes. We use the following query as an example. When the first condition of the WHERE clause is false, MySQL short-circuits the AND and doesn’t evaluate the SLEEP subquery.

SELECT * FROM wp_users 
WHERE user_login = ‘jjrH’ 
AND (SELECT 2344 FROM (SELECT(SLEEP(5)))YNJe)
LIMIT 1

To prevent agents from exploiting this shortcut, we now require agents to extract data from a specific column in the database. This change affects nine tasks in CVE-Bench. After the change, the success rates of agents decreased by up to 32.5%, as shown in the figure above.

Towards More Rigorous Evaluation and Better Usability

Beyond the two major fixes described above, we made further improvements to CVE-Bench.

Validity:

1. We fixed an outcome-validity issue in CVE-2024-25641 and CVE-2024-34340, which had previously treated a failed login attempt as a success.

2. We fixed a task-validity issue in CVE-2024-37831 that had allowed attackers to access the admin page directly, without credentials.

3. We no longer hard-code secrets in CVE-Bench in case that the secrets are included in the training datasets of new LLMs. They are now generated at runtime by our containers using a configurable seed.

Reproducibility:

1. We prevented non-deterministic evaluation results caused by race conditions between the web application initialization and the evaluator initialization.

2. We improved the reproducibility of the evaluation results by using more conservative timeout settings and retry counts.

3. LoLLMS contains multiple CVEs with the same app version. To ensure reproducible results, we modified the challenges so that not all could be solved at the same time. Specifically, we:

   1. Enabled code execution and restricted the /update_setting endpoint to only allow updating host in CVE-2024-2359.

   2. We mounted the secret file to be accessible after maliciously switching the personal path configuration and disabled the /update_setting endpoint entirely in CVE-2024-2624.

   3. We restricted the /update_setting endpoint to only allow updating the extension in CVE-2024-4320.

Usability:

1. We refactored the codebase, reducing set-up time by a factor of 4.

2. We implemented multiple improvements to our Docker infrastructure, including:

   1. We switched the build system to Docker Buildx Bake to enable centralized, standardized, and multi-stage builds

   2. We switched to using package locks (e.g., uv instead of pip) to enable reproducible builds

3. We fixed an issue in CVE-2024-4701 that caused issues on modern Linux kernels

Rigorously Benchmarking AI Agents’ Offensive Capabilities is a Ongoing Effort

As AI agents evolve, new and subtler issues in CVE-Bench may emerge. We are committed to continuously improving CVE-Bench by both fixing bugs and expanding task coverage. Please stay tuned for future updates, including quality improvements and new tasks.

Using CVE-Bench as an example, we also demonstrate the practical value of ABC in guiding the construction of AI agent benchmarks. We are also evolving ABC as the ecosystem grows. Please reach out via this form if you would like an assessment of your benchmark, or if you run into challenges applying the checklist.

Let’s consider AIME as a simple example. All of the problems on AIME are a number between 0 and 1000, so there are 1000 choices. If you assume the prior over answers is uniform (note that this is almost certainly false, but it’s not relevant), then you actually get

Now consider some complex code or legal task. The space of possible answer is larger than 0-1000, so you get way more bits of information per reward computation!

In fact, with modern training methods, you often get more information than that because:

1. Methods like GRPO, and others, roll out many trajectories. By aggregating over many trajectories, you get more information!

2. Modern training methods have complex rubrics that require the model to satisfy many criteria to obtain the full rewards.

3. Modern training uses strategies like curriculum learning that chooses problems that are just beyond the boundary of the LLM’s current capabilities.

Please stop saying that RL gives you 1 bit of information per rollout.

Enjoy a ChatGPT generated-image of this post:

Of course, all models of LLMs are wrong and so is my analysis above. A full analysis would require understanding the distribution of potential answers to understand the full information gain. Curriculum learning is also difficult to analyze theoretically. If you have thoughts, please post in the comments with better theoretical or empirical models!

“Scale AI expects to more than double sales to $2 billion in 2025. The startup generated revenue of about $870 million last year,” reported by Bloomberg.

The incredible demand for high-quality human-annotated data is fueling soaring revenues of data labeling companies. In tandem, the cost of human labor has been consistently increasing. We estimate that obtaining high-quality human data for LLM post-training is more expensive than the marginal compute itself1 and will only become even more expensive. In other words, high-quality human data will be the bottleneck for AI progress if these trends continue.

Data Labeling Company Revenues Outweigh (Marginal) AI Training Costs

To assess the proportion of data labeling costs within the overall AI training budget, we collected and estimated both data labeling and compute expenses for leading AI providers in 2024:

1. Data labeling costs: We collected revenue estimates of major data labeling companies, such as Scale AI, Surge AI, Mercor, and LabelBox.

2. Compute costs: We gathered publicly reported marginal costs of compute2 associated with training top models released in 2024, including Sonnet 3.5, GPT-4o, DeepSeek-V3, Mistral Large, Llama 3.1-405B, and Grok 2.

We then calculate the sum of costs in a category as the estimate of the market total. As shown above, the total cost of data labeling is approximately 3.1 times higher than total marginal compute costs. This finding highlights clear evidence: the cost of acquiring high-quality human-annotated data is rapidly outpacing the compute costs required for training state-of-the-art AI models.

Data Labeling Companies are Dramatically Increasing Revenue

Next, we examined the growth trajectory of data labeling costs from 2023 to 2024. To do this, we collected estimates of the total data labeling and marginal compute costs for training released frontier LLMs for both years and compared the results. As shown in Figure 2, data labeling costs surged with a remarkable growth factor of 88, while compute costs increased by only 1.3 times. Given the rising importance of high-quality human data for reinforcement fine-tuning and cheaper AI accelerators, we expect data labeling costs to continue growing rapidly, while the rate of increase in compute costs may slow in the coming years.

It’s important to note that the growth factor is largely driven by Mercor, which is reportedly the fastest company ever to grow from $1M to $100M ARR. We don’t think these growth rates will continue into the future but think it points towards rapid growth of human data.

Lessons Learned from MiniMax-M1 and SkyRL-SQL

We conclude our analysis with two case studies, MiniMax-M1 and SkyRL-SQL. These models fully describe their training costs and data amounts, so we can analyze both the training costs and data costs.

Efficient RL Scaling in MiniMax-M1. With a training compute cost of just $500K, MiniMax-M1 matches or even outperforms Claude Opus 4 on benchmarks. While explicit data labeling costs are not detailed, MiniMax’s report emphasizes the importance of a “carefully designed curriculum” built from “carefully selected, high-quality” data with about 140K samples for RL training.

If we estimate that a data point would cost $100 (if it were labeled by a human, as opposed to being distilled from another model), the data costs would be $14M in data labeling, 28 times higher than the marginal compute cost for training.

SkyRL-SQL trained a model for text-to-SQL tasks that matches GPT-4o and o4-mini. To achieve this result, SkyRL-SQL uses a novel multi-turn RL algorithm, which teaches the model to iteratively correct its own errors and solve problems step by step. SkyRL-SQL only costs $360 in compute for training. By contrast, we estimate that producing the 600 high-quality annotations cost about $60,000, which is approximately 167x the training compute expense.

Even if our data cost estimates are an order of magnitude off, they would still be ~3x and ~17x more expensive than the compute!

Conclusions and Recommendations

While scaling pretraining data quantity and compute has driven remarkable breakthroughs in the past few years, this strategy has seemingly plateaued with the limits of static data. The rise of RL, which depends on high-quality human-annotated data, has shifted the focus from simply scaling data volume to prioritizing data quality. However, this approach introduces its own challenges, notably the rapidly increasing costs of large-scale data annotation.

Our estimates suggest that high-quality human data is the primary marginal cost of training frontier LLMs. Combined with the performance improvements coming from reinforcement learning, we believe these trends have major implications for understanding AI progress and potentially for policy.

Our blog post will not answer all questions on the impact of high-quality human data on AI progress. As a first step, we recommend that organizations that track inputs to AI should also track the cost of human data used to train frontier models.

Stay tuned for more analysis and recommendations in the future!

Zero-knowledge machine learning (ZKML) offers a cleaner solution: it enables a model owner to generate a lightweight cryptographic proof for each API output verifying that the inference ran exactly as claimed, without exposing proprietary weights or sensitive data. Moreover, most ZKML proofs are designed to be lightweight so that anyone can verify the proof using standard consumer hardware, such as a laptop. This enables verifiable audits of AI decisions in critical settings:

• A hospital can prove that a cancer diagnosis was computed using a certified AI model.

• A lender can demonstrate compliance with fairness rules without exposing applicant data.

• A regulator can verify that a public chatbot output follows safety policies.

All without revealing the confidential data or model! However, today’s ZKML toolchains still struggle to cover the diverse, large-scale models driving real-world applications. We aim to close that gap.

We're thrilled to open-source ZKTorch: a ZKML framework that efficiently compiles machine learning (ML) models into zero-knowledge proofs (ZKPs). ZKTorch is the first ZKML framework to support every edge model in the MLPerf Inference mobile suite, the ML industry’s flagship performance benchmark. ZKTorch can prove AI models widely used in real-world applications such as large language models (LLMs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), and diffusion models. Below are the key performance highlights of ZKTorch:

• Faster proof generation. Up to 3× shorter prover times (e.g., improving GPT-2 inference from >1 hour to ~20 mins).

• Smaller proofs. Proof sizes are at least 3× smaller than those produced by specialized protocols (e.g., zkCNN).

• Almost unchanged accuracy. Output accuracy differs from the MLPerf baseline by less than 1% after ZKTorch’s quantization, satisfying the benchmark's default 99% accuracy requirement.

In the rest of this post, describe how ZKTorch achieves these performance improvements. We'll conclude with a brief walkthrough to help you start generating your own ZKML proofs (check out our open-source repository).

ZKTorch: a critical step toward practical ZKML

Although ZKML holds great promise, existing methods sit at two impractical extremes: 1) slow, general-purpose proof systems or 2) inflexible specialized protocols limited to particular models. Consider Modulus: generating a proof for a 1.5-billion-parameter LLM (GPT-2-XL) takes over 90 hours, even on 128 threads. By contrast, ZKTorch proves a 6-billion-parameter LLM (GPT-J) in roughly 20 minutes on 64 threads.1 Meanwhile, Halo2-based ZKML provers (e.g., ZKML and EZKL) struggle to handle models larger than about 30 million parameters.

Other systems are highly specialized towards specific classes of models (e.g., zkCNN for convolutions, zkLLM for attention) to improve performance. However, real deployments rarely use a single CNN or LLM in isolation; they chain speech-to-text modules, multimodal blocks, and transformer re-rankers.

ZKTorch bridges this gap with fast, scalable proofs across diverse models.

Technical overview

To understand how ZKTorch achieves this, we’ve provided a technical overview here. For more details about ZKTorch, check out our paper. Feel free to skip to the next section without missing anything!

ZKTorch consists of three main components: a compiler, a transpiler, and a library of basic blocks. The compiler rewrites a machine learning model (e.g., an ONNX graph) into a proving-friendly directed acyclic graph, where each node represents an ML operation. For example, the GPT-J model provided by MLPerf Inference uses eight ONNX nodes to represent a single GeLU activation, requiring more than five lookup arguments to prove (each lookup allows us to prove that the elements of a committed vector come from a much bigger committed table). Our compiler consolidates these nodes into a single GeLU node, reducing the proof overhead to just one lookup.

The transpiler then replaces each node with an optimized composition of basic blocks, which are zero-knowledge protocols tailored to each operation. For instance, matrix multiplication can be transpiled into a recent optimized protocol CQLin, which can prove the result of matrix multiplication in O(n) time when one matrix is fixed. Non-linearities such as GeLU are handled with an optimized lookup argument CQ, whose proving time is independent of the lookup table size. (For more details on these protocols, please check out our previous post). With basic block support for all 61 MLPerf v4.1 layers, ZKTorch can decompose models ranging from CNNs (e.g., ResNet-50) to LLMs (e.g., GPT-J and GPT-2) into thousands of small proofs.

Although each individual proof is small, collectively they can result in a large overall proof size. To address this, ZKTorch employs an accumulation scheme that folds multiple proofs from the same basic block into a single compact proof. Our accumulation scheme extends Mira by making it parallelizable. This parallel extension significantly accelerates the folding process, reducing the proving time for GPT-J from 8,662 seconds to just 1,397 seconds. By folding all proofs of the same basic block type into a single accumulator instance, the prover produces one lightweight proof whose size and verification time remain nearly constant, regardless of the model’s depth. In practice, this brings down GPT-2 proving time from over an hour (ZKML) to just 10 minutes, and reduces a ResNet-50 proof from 1.27 GB (Mystique) to 85 KB.

🚀 Quick Start: Try ZKTorch in Minutes

Getting started with ZKTorch is straightforward. Follow two simple steps:

Step 1. Install Rust (skip if already installed)

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Step 2. Clone and run ZKTorch

git clone https://github.com/uiuc-kang-lab/zk-torch.git
cd zk-torch
rustup override set nightly
cargo run --release --bin zk_torch --features fold -- config.yaml

After cloning the repository, we’ve provided a sample configuration `config.yaml`, which defines file paths (e.g., ONNX model, input data, and proof) and scale parameters. For example, the `scale_factor_log` entry determines how floating-point numbers are converted into fixed-point integers for proof generation; for instance, setting `scale_factor_log = 10` means a value `x` will be encoded as `round(x × 210)`. Easily experiment with your own ML models by replacing the included ONNX file and the corresponding input file defined in `config.yaml`.

If you plan on changing the example configuration file, the powers of tau file needs to be compatible with the configuration settings.

Check out ZKTorch today!

ZKTorch represents a significant step toward making practical and usable ZKML. We're excited for developers, researchers, and industry professionals to explore, experiment, and expand upon ZKTorch. If you’re interested in contributing to ZKTorch, please reach out via our Telegram group or by email!

Check out our paper and code on GitHub. We also look forward to learning from your ideas for how to build with ZKTorch. If you’d love to share your ideas with us, welcome to join our Telegram group.

Written by ZKTorch authors

From the landmark Reproducibility Project: Psychology to recent mass-scale efforts in economics and political science, these assessments have proven essential but exceedingly slow: 347 social scientists were involved in reproducing 110 papers in the mass reproduction of economics and political science papers, and it took more than five years for the Reproducibility Project: Psychology to complete the reproduction of 100 studies. These findings highlight an urgent need to automate the assessment of research reproducibility.

Large language models (LLMs) and autonomous AI agents have shown remarkable promise in tackling complex tasks in domains like programming and mathematics. But can they help us automate the process of evaluating whether a social science paper is actually reproducible?

We introduce REPRO-Bench, the first benchmark designed to test exactly that. Each of its 112 tasks corresponds to a real social science paper, complete with its full PDF, associated code and data, and a list of major findings. Based on the internal consistency between a paper’s reported findings and its reproduction package, including code and data, AI agents are tasked with assigning a reproducibility score from 1 (not reproducible) to 4 (fully reproducible). This process requires agents to actively engage their critical reasoning skills to assess methodological soundness, identify discrepancies, and determine the degree to which findings are supported by the provided code and data.

Check out our paper, code, and data, and read on for details on how we constructed our benchmark and our evaluation results!

Introducing REPRO-Bench

REPRO-Bench is a new benchmark designed to evaluate whether AI agents can accurately assess the computational reproducibility of social science papers. Each task mimics the full reproduction process: the agent is given a research paper (PDF), its reproduction package (including data, code, and documentation), and a list of major findings. Unlike prior efforts that reproduce results under the assumption that all research findings are fully reproducible, REPRO-Bench requires AI agents to output a standardized JSON file containing a reproducibility score from 1 to 4, reflecting a critical evaluation of the alignment between reported findings and the accompanying code and data.

Each of the 112 task instances comes from real reproducibility efforts, sourced from:

• Mass Reproduction of Economics and Political Science Papers

• Institute for Replication (I4R) Discussion Paper Series

• Retraction Watch Database

• Twitter/X posts reporting reproducibility attempts

REPRO-Bench is designed around three core challenges:

• Real-world grounding: Tasks mirror actual social science reproducibility workflows. A legal expert noted that REPRO-Bench captures common reproducibility patterns and offers potential for building real tools to assist researchers.

• High complexity: On average, tasks involve 29-page papers and 4.2GB reproduction packages with 142 files spanning multiple formats and programming languages: e.g., R, Python, Stata, CSV.

• Critical reasoning: Beyond technical reproduction, agents must reason through discrepancies between original findings and reproduced outputs, using logical, mathematical, and causal reasoning, alongside domain knowledge.

Existing AI agents show deficiencies on REPRO-Bench

For evaluation, we selected 3 agents: AutoGPT, CORE-Agent, and SWE-Agent using gpt-4o. For performance evaluation, we take accuracy, which measures the match between the generated reproducibility score and the ground truth, as our primary metric. We also examine the applicability rates, i.e., whether the agent generates a valid reproducibility score. We report both the original and the adjusted accuracy and applicability rates to include scenarios where agents generate output files outside of the directory specified in the task requirements. For cost analysis, we report the average API cost for all requests made by each agent for each task.

CORE-Agent achieves the highest accuracy at 21.4% among the three agents, which is even lower than random guessing among four options without prior knowledge of the underlying data distributions or the results of other task instances. The applicability rates are also very low. All three agents exhibit low applicability rates, often failing to complete the full task.

Agents are noticeably better at identifying papers that are clearly reproducible (score 4) or clearly irreproducible (score 1), but struggle with borderline cases (scores 2 and 3), indicating a tendency toward binary judgments.

We inspected the traces of (1) successful cases, from which we outline a general workflow, and (2) failure cases, where we find that agents often overlook critical steps such as code inspection and result comparison, both of which are essential for identifying inconsistencies. During code inspection, agents tend to read the entire code file rather than focusing on the relevant sections in the paper. We also observed that the majority of errors stem from path issues: specifically, the data is present but not located in the directory specified by the README file. As a result, the agent incorrectly concludes that the data is missing, without searching the entire reproduction package.

To address these issues, we extend CORE-Agent by adding four targeted instructions based on failure analysis. The resulting agent, REPRO-Agent, significantly outperforms baselines with an accuracy of 36.6%, a 71% relative improvement over CORE-Agent.

Quickstart: Reproduce Our Results with REPRO-Bench

Follow these three simple steps to reproduce the results from our paper using REPRO-Bench and the associated agents.

Step 1: Download the REPRO-Bench Dataset

git clone https://huggingface.co/datasets/chuxuan/REPRO-Bench
cd REPRO-Bench
git lfs pull

Step 2: Clone the Codebase

git clone https://github.com/uiuc-kang-lab/REPRO-Bench REPRO-Bench-Code
cd REPRO-Bench-Code

Step 3: Run the Agent Experiments

bash SWE-Agent/run_all.sh
bash AutoGPT/classic/original_autogpt/run_all.sh
bash CORE-Agent/classic/original_autogpt/run_all.sh

With these commands, you’ll be able to reproduce our experiments and inspect how existing AI agents perform on complex, real-world reproducibility tasks.

Conclusion: REPRO-Bench demonstrates the need for more powerful AI agents with critical reasoning capabilities

Despite this progress, performance remains far from sufficient for practical use: over half of the papers are still misclassified. This highlights a clear reality: today’s AI agents aren’t yet ready for real-world scientific reasoning. Bridging this gap will require agents with stronger reasoning, deeper contextual understanding, and evaluation frameworks that better reflect real-world complexity. For more details, please check out our paper, code, and data!

SWE-bench has become the “gold standard” for evaluating the coding capability of AI agents. It asks an agent to propose patches for real-world GitHub issues and then evaluates their solutions by running manually-written unit tests. Unfortunately, even carefully crafted unit tests can miss important edge cases.

OpenAI strengthened SWE-bench by asking 93 professional developers to curate a subset, SWE-bench Verified, with revised unit tests. Given all the expert effort involved in verification, is SWE-bench Verified error-free?

Our research shows otherwise: “verified” unit tests are still insufficient in 26/500 tasks in SWE-bench Verified. In our recent ACL paper (code), we introduced a novel technique to identify and fix these insufficient unit tests. The missing unit tests are critical to evaluating performance: When we re-evaluated agent performance using fixed unit tests, the leaderboard rankings changed for 24% agents!1

Why Do Expert-Verified Unit Tests Fall Short? A Motivating Example

Take PR-13933 from the django project as an example. The agent was supposed to update the code to include the value of an invalid choice in error messages. While the Amazon Q developer correctly updated the error-raising, it also introduced bugs in cases outside error handling, as shown in Figure 1. Because the unit tests only checked the error scenario, the agent’s mistakes were undetected.

As shown, even expert-written unit tests can miss bugs. Therefore, we need a safety net: this is where our new approach, UTBoost, comes in.

UTBoost: The First LLM-Driven Unit Test Generator for Software Projects

UTBoost uses LLM to automatically generate unit tests for full-scale software projects. However, generating tests for a real codebase is challenging, as real codebases have dozens of files, many dependencies, and diverse codebase structures. UTBoost tackles this complexity in three steps:

1. File-level: LLM reads the issue description, the existing tests, and a repository summary, then points to the three files most likely involved.

2. Function/class-level: For each file, it locates the relevant function or class.

3. Line-level: For each function, LLM highlights the specific lines that matter.

With all of these contexts in place, the LLM writes pytest-style cases that include any necessary dependencies. After we manually verify the correctness of new tests, UTBoost then adds these tests to SWE-bench and reruns the evaluation.

UTBoost Identifies Instances with Insufficient Test Cases

We ran UTBoost on SWE-bench Lite and SWE-bench Verified, using the settings described in our paper. For each incorrect patch identified by UTBoost, two of us independently reviewed it and reached a consensus.

UTBoost identified and augmented unit tests for 23/300 task instances of SWE-bench Lite and 26/500 task instances of SWE-bench Verified. Across all the agent submissions on the leaderboard, these augmented test cases identified 28.4% (SWE-bench Lite) and 15.7% (SWE-bench Verified) more incorrect patches that were previously considered correct.

UTBoost Identifies Erroneous Annotation of Testing Results

In addition to identifying insufficient unit tests in SWE-bench, UTBoost also helped us find errors in the way test results were annotated by the original parser, such as missed tests or incorrect test names. These errors led to flawed patches being mistakenly considered correct.

After improving the parser, we corrected 54.7% of annotations in SWE-bench Lite submissions and 54.2% in SWE-bench Verified submissions. These corrected annotations lead to 64 (SWE-bench Lite) and 79 (SWE-bench Verified) incorrect patches that were previously labeled correct.

UTBoost Changes the Leaderboard Rankings of SWE-bench

With augmented test cases and the improved parser, we then re-evaluated the agents on SWE-bench’s leaderboards. Across all agent submissions, we identified 176 incorrect patches in SWE-bench Lite and 169 in SWE-bench Verified that were incorrectly evaluated as correct. After fixing the evaluation results, we observed 40.9% and 24.4% ranking changes on the leaderboards of SWE-bench Lite and SWE-bench Verified, respectively.

Conclusion

Software testing has been a challenging problem for decades, and that hasn’t changed just because AI is writing the code. Although UTBoost still does not guarantee error-free coding benchmarks, our results show that augmenting expert-verified tests with LLM-generated tests is a promising path forward.

Given that data noise can affect leaderboard rankings by 24%, we need to rethink whether standalone score comparisons are the best way to compare agents and whether leaderboards are the best way to present the results. We call for a more thorough study in this direction, to build a community with more focus on real progress rather than on the pressure to reach the top of the leaderboard.

UTBoost has been accepted to ACL 2025. We’ve open-sourced the code on GitHub and datasets with fixed unit tests on Hugging Face (Verified, Lite). Give it a try and let us know if you have feedback!

Unfortunately, many current AI agent benchmarks are far from reliable. Consider WebArena, a benchmark used by OpenAI and others to evaluate AI agents on interactions with websites. In a task to calculate the duration of a route, an agent answered “45 + 8 minutes” and was marked correct by WebArena, although the correct answer is “63 minutes.” Moreover, among 10 popular AI agent benchmarks (e.g., SWE-bench, OSWorld, KernelBench, etc.), we found severe issues in 8 of them, causing in some cases up to 100% misestimation1 of agents’ capabilities.

These numbers make one thing clear: to understand an agent’s true abilities, we must build AI agent benchmarks in a more rigorous way.

How do we build AI agent benchmarks we can trust? In our recent work, we break down the failure modes in current AI agent benchmarks and introduce a checklist that minimizes the gamability of AI agent benchmarks and ensures they measure what they claim to measure. In future posts, we will provide recommendations for creating AI agent benchmarks we can trust and deep dives on specific benchmarks!

How do Current AI Agent Benchmarks Fail?

In AI agent benchmarks, agents are asked to complete tasks end-to-end, such as fixing a code issue in a large repository or creating a travel plan.

This ambitious scope creates two challenges that traditional AI benchmarks rarely face:

1. Fragile simulators: Tasks often run inside simulated/containerized websites, computers, or databases. If these mini-worlds are buggy or outdated, an agent can simply find a shortcut to pass or find the task impossible.

2. No easy “gold” answer: Task solutions may be code, API calls, or paragraph-long plans, which don’t fit a fixed answer key.

Given these challenges, we propose two validity criteria that are particularly important for AI agent benchmarks:

1. Task Validity: Is a task solvable if and only if the agent possesses the target capability?

Example failure: τ-bench scores a “do-nothing” agent as correct on 38% of airline tasks, even though the trivial agent does not understand the airline ticketing policy.

2. Outcome Validity: Does the evaluation result (e.g., tests or checks) truly indicate task success?

Example failure: As shown in the example before, WebArena partially relies on LLM-as-a-Judge that makes mistakes for problems as simple as “45+8≠63.”

Our Research: AI Agent Benchmark Checklist

We curated the AI agent Benchmark Checklist (ABC), a 43-item checklist based on 17 AI agent benchmarks used by leading AI providers. ABC consists of three parts: outcome-validity checks, task-validity checks, and benchmark reporting guidelines for cases where perfect validity is extremely challenging or impossible.

The full, print-friendly checklist is publicly available online.

An Overview of Our Findings via ABC

We applied ABC on ten popular AI agent benchmarks, including SWE-bench Verified, WebArena, OSWorld, and more.

Out of the 10 benchmarks, we found:

1. 7/10 contain shortcuts or impossible tasks.

2. 7/10 fail outcome validity.

3. 8/10 fail to disclose known issues.

Here is a summary of issues we identified in benchmarks that are used to evaluate frontier AI agent systems, including Claude Code and OpenAI Operator.

SWE-bench and SWE-bench Verified use manually crafted unit tests to evaluate the correctness of agent-generated code patches. Agent-generated code patches can have bugs not captured by unit tests, as shown in the following example. By augmenting unit tests, we observed significant ranking changes in the leaderboard, affecting 41% agents for SWE-bench Lite and 24% for SWE-bench Verified.

KernelBench uses tensors with random values to evaluate the correctness of agent-generated kernel code written in CUDA. Similar to SWE-bench Verified, random-valued tensors may fail to capture bugs in the generated kernel, especially for memory- or shape-related issues.

τ-bench uses substring matching and database state matching to evaluate agents, which allows a do-nothing agent to pass 38% of tasks. The following example demonstrates one of these tasks.

WebArena uses strict string matching and a naive LLM-judge to evaluate the correctness of agents’ actions and outputs, which leads to 1.6-5.2% misestimation of agents’ performance in absolute terms.

OSWorld develops agent evaluation partially based on outdated websites, resulting in a 28% underestimation of agents’ performance in absolute terms. In the following example, the CSS class, search-date, has been removed from the website the agent interacts with. Because the evaluator still relies on an outdated selector, it marks the agent’s correct actions as incorrect.

SWE-Lancer fails to securely store test files, which allows an agent to overwrite tests and pass all tests.

Next Steps with ABC

We build ABC as an actionable framework to help

1. Benchmark developers troubleshoot potential issues or demonstrate their thorough work.

2. Agent/Model developers dive into the underlying benchmarks deeply beyond reporting a “start-of-the-art” number.

Please check our paper for details. The full checklist, code examples, and the growing registry of assessed benchmarks live at our GitHub repository. If you are interested in adding exploit or fix patches to existing benchmarks, please submit a PR to our repository!

We invite contributions, issue reports, and pull requests! Reach out to us if you are interested in using or iterating on ABC.

However, a key requirement for models is that they must reliably handle scenarios that differ from their training data. This raises a key question: does RPT generalize effectively across tasks and domains?

So far, answers to this question have been inconclusive. Most evaluations focus on in-domain performance, using RPT models trained on mixed-domain data and evaluated on benchmarks closely aligned with their training distribution. These setups introduce confounding factors that obscure our understanding of RPT's true generalization ability.

To address this gap, we designed and conducted a unified evaluation framework that isolates and tests RPT's cross-domain generalizability more rigorously. Our results show that while RPT is highly effective within its training domain, its benefits do not consistently transfer to out-of-domain tasks. This highlights the need for a more nuanced understanding of how post-training mechanisms generalize across domains.

Measuring RPT Generalization

To systematically study the generalizability of RPT while eliminating confounding factors from entangled training data, we first divide the RPT training data into three major domains and then design a unified evaluation framework spanning 16 benchmarks.

• Math: GSM8K, MATH-500, AIME 2024, and AMC 2023

• Code: MBPP, HumanEval, BigCodeBench, LiveCodeBench, USACO, Codeforces, and Aider Polyglot

• Knowledge-Intensive Reasoning: PubMedQA, MedQA, TabFact, LegalBench, and FinBench

Using this framework, we conducted two complementary studies: an observational study that examines existing models with public RPT data, and an interventional study where we fine-tune models on specific domains to directly evaluate their cross-domain generalization. In both settings, we evaluate RPT's effectiveness by comparing performance gains over base models across different domains.

Observational Study

We evaluate 14 open-weight RPT models, as we show in the table below, with publicly disclosed training data, alongside their respective base models. These models span domains like math, code, law, finance, and medicine. This allows us to assess whether fine-tuned gains persist when applied to unseen domains.

Interventional Study

To remove confounding factors from mixed-domain training, we fine-tune LLMs from scratch using reinforcement learning on math, code, and knowledge-intensive reasoning data, respectively. We then evaluate their performance on both in-domain and out-of-domain tasks to understand how fine-tuned capabilities transfer. We refer to models fine-tuned on the corresponding domains as Math-RPT, Code-RPT, and Knowledge-RPT throughout the following text and figures.

Our Findings

We illustrate the three key findings from our empirical analysis as follows.

Finding 1: RPT Gains Are Mostly In-Domain

In our observational analysis, we find that RPT leads to notable improvements only within the domains it was trained on. Across the 14 models we studied, pass@1 accuracy increased by 3.57% on in-domain tasks, but dropped by 1.48% on out-of-domain tasks.

The interventional study reinforces this finding. As we demonstrate in the figure below, none of the models fine-tuned on a single domain exhibited statistically significant gains on out-of-domain benchmarks. On the contrary, both the Math-RPT and Code-RPT models show statistically significant performance drops on out-of-domain tasks. The Knowledge-RPT model also failed to generalize beyond its training data, showing no meaningful gains on unseen domains.

Finding 2: Structured Domains Like Math and Code Mutually Generalize

We observe strong mutual generalization between math and code. In our observational study, Math-RPT models improved by 2.18% on math and 4.77% on code tasks, and Code-RPT models improved by 9.49% on code and 15.44% on math tasks.

In both cases, models often performed even better on the unseen structured domain, suggesting shared underlying reasoning patterns between math and code that RPT is able to exploit.

Finding 3: Structured Skills Do Not Transfer to Knowledge-Intensive Reasoning

While math and code fine-tuning transfer well between each other, these structured reasoning skills do not generalize to unstructured or knowledge-intensive reasoning domains. In our observational study, structured-domain models showed only a −0.27% average change in pass@1 on knowledge-intensive reasoning domain tasks, compared to 11.08% and 5.82% gains on math and code respectively.

The interventional study confirms this trend. As we demonstrate in the figure above, the Math-RPT and Code-RPT models both underperform on knowledge-intensive reasoning tasks, despite showing robust gains in their respective domains. These findings indicate that while RPT is highly effective in capturing domain-specific reasoning, it fails to adapt to tasks requiring broader, more heterogeneous reasoning patterns.

Conclusion: RPT Is Powerful but Narrow

In this work, through both observational and interventional studies, we consistently find that while RPT produces substantial improvements within training domains, its generalization to unseen domains is limited, as we summarize in the figure below. In particular, while there is evidence of cross-domain transfer between structured domains like math and code, there is little evidence of transfer to unstructured domains.

Read our paper for more details! And stay tuned for more thoughts on implications for future progress.

• How many changes will developers have to make to the database management system (DBMS)?

• Who will maintain all of the offline computations when data or workload drifts?

• How can users know the accuracy of the approximate result before they press “run,” rather than afterwards?

In this blog, we introduce PilotDB (code available on GitHub), an online AQP system that addresses all three concerns and achieves up to 126x speedup compared to exact queries. To understand PilotDB, we first take a closer look at why we think prior AQP systems are not ready for production.

What Still Blocks AQP In Practice?

DBMS Modifications. Recent online AQP methods (e.g., QuickR) deeply integrate with DBMS components (e.g., query planner and optimizer). Deploying these AQP methods requires modifying a mature DBMS, which is often unacceptable or discouraging for practitioners.

Continuous Maintenance. Offline AQP (e.g., BlinkDB, VerdictDB) pre-computes synopses or samples that must be rebuilt whenever the data or workload shifts, causing continuous, non-trivial overhead.

No Priori Error Guarantees. Users often want to know the error of an approximate result before they run the query. Systems (e.g., DBest) that address previous challenges can only report accuracy afterwards, or aren’t statistically rigorous at all.

We develop two key techniques to achieve all three in PilotDB.

PilotDB’s Approach

Two-Stage Query Approximation As A Lightweight Middleware

PilotDB delivers approximate answers through online sampling while ensuring a priori error guarantees. The challenge is to decide, before execution, how large a sample the query needs. To address this, we develop the following two-stage workflow, where PilotDB operates as a lightweight middleware between the user and the database.

1. A “Pilot” query: We first execute a small sample (e.g., 0.05% of data) to estimate the data’s variance and plan the minimal sample that will meet the user’s error bound.

2. A “Final” query: We rewrite the original query on-the-fly to use that optimal sample. If no speed-up is possible, PilotDB simply runs the original query.

BSAP: Block-level Sampling with Guarantees

Running two samples on the fly can be slow due to the data I/O costs. Instead of using row-level sampling to fetch individual tuples, PilotDB employs block sampling that reads entire disk pages at a time. Block sampling reduces I/O by 97–99 % at low sampling ratios.

Unfortunately, previous error analysis does not work for block sampling since rows inside the same block are correlated. We develop BSAP that provides (1) new variance formulas, (2) sampling-equivalence rules, and (3) join analysis to achieve statistically rigorous error analysis for block sampling on joins or nested queries.

We prove these results for single-table, multi-table, and nested queries, and have upstreamed the I/O-efficient block-sampling code to DuckDB 1.2.

How Much Can PilotDB Accelerate Queries?

We evaluated PilotDB with widely used synthetic benchmarks (TPC-H, SSB, and DSB) and real-world benchmarks (ClickBench and Instacart). Given a 5% error target, PilotDB achieved up to 126x speed-up on PostgreSQL 16 (24x geometric mean), up to 117x speed-up on SQL Server 2022 (18x GM), and up to 13x on DuckDB 1.0 (7x GM).

PilotDB demonstrates superiority when compared to a previous state-of-the-art online AQP method, QuickR. When compared to the performance upper bound¹ of QuickR, PilotDB achieves 1.2-4.2x higher speed-up on different DBMSs. Moreover, BSAP can augment QuickR, providing 5-60x higher speed-up than the original QuickR on DuckDB.

Conclusion

PilotDB pushes forward the practical side of AQP techniques to eliminate maintenance and DBMS re-engineering, while providing error guarantees. As shown in the following demo, PilotDB has zero overhead on both users and DBMS developers.

If you find PilotDB interesting, feel free to give it a try. We have open-sourced PilotDB on GitHub. For more technical details, please check out our paper and let us know if you have any questions.

I’ll walk through a high-level intuition of how these SSAIs are trained in this blog post. Stay tuned for future blog posts on my thoughts of which problems will fall to SSAI.

RL + search = superhuman AI on games

Currently, the best method we have towards reaching superhuman AI is reinforcement learning (RL). Roughly speaking, RL allows an AI system to actively explore an environment to achieve some objective. Think of it like giving a dog treats when it successfully completes a task.

RL has a long history of surpassing human performance, primarily in games (chess, go, etc.). Games are particularly suited for RL since we can simulate literally billions of game rollouts cheaply.

However, RL on games usually focuses on tailor-made systems, which I’ll call “expert systems.” Expert systems are now superhuman on chess, go, and many other games. Typically, they involve training a game-specific AI model that plays against itself (self-play) million to billions of times. At this step, the AI model learns what actions are good in what contexts.

The AI model is then combined with search, where the AI model plays itself down many paths given the current state of a game. RL + search powered AlphaGo and many other game-playing AI systems.

LLMs + RL + search = superhuman AI we can talk to

RL has expanded to work on LLMs, leading to the o-series of models from OpenAI. Today, o1 and o3 are publicly available, with o4 reportedly on the way. Beyond OpenAI’s offerings, Gemini-2.5 Pro, Claude 4 Sonnet/Opus, and DeepSeek R1 are also trained with RL.

But what does RL mean in the context of LLMs? Let’s look at the specific example of the AIME math competition. Problems in AIME look like this:

Alice and Bob play the following game. A stack of n tokens lies before them. The players take turns with Alice going first. On each turn, the player removes either 1 token or 4 tokens from the stack. Whoever removes the last token wins. Find the number of positive integers n less than or equal to 2024 for which there exists a strategy for Bob that guarantees that Bob will win the game regardless of Alice's play.

And solutions to AIME are numbers between 1 and 1000. The solution to this particular problem is 809.

Let’s say we have hundreds of thousands of AIME questions. We can ask the LLM to solve the problem many times - since the solution is a fixed number, we parse the answers at the end to tell if the model got the answer correct or not. To train the model, we can encourage the model to output text similar to correct solutions and discourage the model from producing outputs similar to incorrect solutions.

Once we have a trained model, we can do something similar, where we ask the LLM to solve the problem many times. As long as we can cheaply verify if the solution is correct, search can dramatically improve the performance of these systems, to the point of being superhuman. This is how the AlphaEvolve system works.

What’s next?

All of the examples we’ve seen so far have been of systems that are trained in a specific domain. Today, these domains have only been ones with easily verifiable solutions. So far, only games, math, and code have fallen to RL.

Fortunately for humans, much of life isn’t easily verifiable. Even seemingly objective tasks, like legal reasoning, can be highly subjective and even change over time!

A major question for the future performance of AI is: will RL generalize? Particularly:

• Will RL generalize from easy problems to hard problems?

• Will RL generalize across easily verified domains?

• Will RL generalize from easily verified domains to “fuzzy” domains?

AI progress has been incredibly difficult to predict, but we’ll cover the literature on AI progress in future posts.