State of AI

Test-Time Compute Budgets, Humanoid Robot Perception, and the Limits of LLM Memory

State of AI — Tue, 01 Sep 2026 16:35:26 GMT

Welcome to today’s edition of State of AI 👋

And a warm welcome to our 258 new subscribers since last edition! Thanks to you all we have finally hit 18,000+ subscribers!!!

Before we get into it, today’s edition is sponsored by Viso Now.

Visual General Intelligence, no training data required.

Viso Now, a new self-serve computer vision platform from viso.ai, just launched. Instead of training a model per use case, a process that normally takes months of data labeling, you describe a scene or a problem in plain language and the platform builds a working computer vision app from that description. It runs on viso.ai’s new VGI-1 engine, part of what the company calls its Visual General Intelligence thesis, and it’s aimed at manufacturing, retail, logistics, construction, healthcare, and infrastructure teams, industries that have historically been priced out of custom computer vision.

Try Viso Now!

Two threads run through this fortnight’s strongest papers. The first is compute allocation: models learning to spend reasoning tokens, routing decisions, and speculative drafts only where the problem actually needs them, with double-digit efficiency gains that require no new hardware. The second is a less flattering thread. Three separate papers put memory systems under adversarial pressure, cache eviction policies under controlled comparison, and skill libraries under ablation, and in each case the sophisticated approach loses to something simpler once the evaluation gets honest. Robotics closes the gap between the two: reasoning and stereo vision both survive as genuine test-time compute for physical control, provided the extra computation is spent where uncertainty is highest rather than everywhere.

Here’s what caught our attention:

Prefix Sliding caps reasoning-trace memory at a constant size by discarding intermediate tokens once they’ve done their job, delivering 3× speedup with zero retraining and enabling reinforcement learning on traces past 100,000 tokens.
Learning When to Think trains a 1.5B model to pick its own reasoning budget as its first generated token, cutting MATH-500 tokens by 41% with no supervised routing data and no separate router network.
ProgRouter routes each step of a multi-agent workflow to the cheapest model that can still make progress, beating fixed single-model baselines on every energy budget across four benchmarks.
VoiceMem proves that a dense top-5 memory retrieval beats a sparse top-100 one, holding sub-150ms latency while cutting memory tokens by roughly 4-16× against comparable systems.
MemTrapBench shows that giving a model correct, relevant memories can still make it worse: every memory framework tested loses more than 10 points to a no-memory baseline once cognitive traps are built into the benchmark.
AsymSpec lets a speculative-decoding drafter see the full context while the verifier only sees a compressed one, recovering 90% of full-context accuracy at 1.3 to 1.7× the throughput.
R³ trains a robot policy to generate natural-language reasoning before acting, and truncating that reasoning at test time measurably hurts performance, which is the closest thing yet to proof that language reasoning is real compute for manipulation.
EATR-Stereo routes stereo camera evidence into a frozen vision-language-action model based on the robot’s own proprioceptive state, recovering from severe occlusion 80% of the time against 30% for baselines.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

ProgRouter: Online Progress-Guided Orchestration for Multi-Agent LLM Workflows

Authors: Somgyuan Li, Ahmed M. Abdelmoniem, Shiqiang Wang

Source and references: https://arxiv.org/abs/2608.25992v1

Introduction

Most LLM routers pick a model once, at the start of a task, and live with the choice. ProgRouter routes at every step of a multi-agent workflow instead, deciding which model to dispatch based on how far the current subtask has actually progressed and how much energy budget remains.

Key Points

Multi-view progress scoring combines outcome regime, subtask completion, progress trend, and workflow state quality into one signal the router reads before every dispatch.
A dual-path predictor estimates the progress gain each candidate model would deliver, one path over structured workflow features and one over semantic summaries, combined by a meta-gated learner.
Lyapunov virtual cost queues track budget violations over time, so the router trades off immediate quality against long-run energy sustainability rather than optimizing one task in isolation.
The router learns entirely online, through exploration and update, with no offline training dataset required.

Methodology

A coordinator agent decomposes the task and dispatches worker agents instantiated by models chosen from a heterogeneous zoo. At each dispatch, the progress scorer reads the coordinator’s ledger, the dual-path predictor estimates gains per candidate model, and the final decision balances predicted progress against cost-queue pressure and per-task budget consumption.

Results and Findings

Across HumanEval Plus, MBPP, MATH-500, and ASQA, ProgRouter beat MasRouter and CASCADIA on every metric that mattered while staying inside the stated energy budget, something every fixed single-model baseline failed to do regardless of scale. On MBPP it posted the lowest energy (3,376 J) and fastest execution (10.3s) of any method tested. The routing distributions tell the real story: 84 to 91% of calls went to efficient small models, with larger models called in only when the progress-gap term flagged a genuine opportunity.

Implications and Conclusions

The result worth remembering is that every fixed single-model baseline blew its energy budget, the small models through excessive retry steps and the large ones through per-call cost. Static routing decisions made at task start cannot see either failure mode coming. What the paper doesn’t test is adversarial or noisy environments where the progress signal itself might be unreliable, and that’s where a system this dependent on its own scoring function would actually break.

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation

Authors: Gijs Kassenaar, Zhao Yang, Vincent François-Lavet

Source and references: https://arxiv.org/abs/2608.20256v1

Introduction

Reasoning models spend a fixed token budget on every problem regardless of difficulty. This paper trains a model to choose among three modes, NoThink, Short, and Long, as its own first generated token, learning the routing policy end to end inside standard GRPO with no separate router and no supervised routing labels.

Key Points

Per-mode token caps (1,024 / 3,000 / uncapped) force each mode to genuinely fail on problems it can’t handle, which is what stops the policy from collapsing onto a single mode.
A balance penalty on advantages and forced-rollout warmup keep all three modes alive during early training, when the routing signal is still unreliable.
Mode-specific reward shaping creates length-dependent crossover points where a different mode becomes reward-optimal, teaching the router which problems suit which budget.
Per-mode accuracy ordering inverts during training: Long starts strongest and ends weakest relative to the brief modes, evidence the router is actually assigning easy problems to cheap modes rather than gaming the reward.

Methodology

The authors fine-tune DeepSeek-R1-Distill-Qwen-1.5B on MATH problems with a modified GRPO: mean-centered advantages instead of standard-deviation normalization, token-mean loss aggregation, no KL penalty against the reference model, and a balance term folded directly into the advantage computation. Training runs 90 steps on 4 H100s.

Results and Findings

The mode distribution stabilizes near 20/32/47% (NoThink/Short/Long) with routing entropy near its theoretical maximum, meaning the model genuinely uses all three options rather than defaulting to one. Free-validation routing on MATH-500 shifts monotonically from NoThink toward Long as problem difficulty climbs from level 1 to level 5, matching difficulty labels the model never saw during training. On MATH-500 the adaptive router hits 0.783 accuracy at 2,811 mean tokens, a 41% cut from the base model’s 4,796 tokens, sitting above the accuracy-length frontier traced by single-mode baselines. The gains transfer zero-shot: 76% token reduction on the easier GSM8K, and on AIME, where nearly every problem needs extended reasoning, the router correctly declines to cut budget and still lands 12% shorter than the base model.

Implications and Conclusions

The AIME result is the one that earns trust in this approach: a router that could only ever shorten traces would be gaming the benchmark, and this one recognizes when shortening is the wrong move. At 1.5B parameters and a single domain (competition math), the open question is whether the same balance penalty and per-mode caps hold up when the task distribution is messier, code generation and agentic tool use in particular, where problem difficulty is harder to read from the prompt alone.

Prefix Sliding for efficient test-time scaling

Authors: Niklas Muennighoff, Zhengyang Wang, Zeyi Chen, Weijia Shi, Binyuan Hui, John Yang, Dapeng Jiang, Mika Senghaas, Fares Obeid, Johannes Hagemann, Sami Jaghouar, Ludwig Schmidt, Percy Liang, Jason Wei, Andrew Y. Ng, Luke Zettlemoyer, Yejin Choi, Mike Lewis

Source and references: https://arxiv.org/abs/2608.26070v1

Introduction

Long reasoning traces run into full attention’s quadratic cost. Prefix Sliding caps that cost at a constant value by keeping only the prompt prefix and a sliding window of recent tokens, on the observation that most intermediate reasoning tokens stop mattering once they’ve contributed to a completed step.

Key Points

The prefix (task instructions) and a recent window (current reasoning) stay in context throughout generation; everything in between gets discarded, capping memory regardless of trace length.
A custom FlashAttention kernel with intra-tile masking and inter-tile skipping delivers the speedup, achieving roughly 5,000 tokens per second once the window fills.
Zero-training deployment gets 3× speedup on existing pretrained models with no fine-tuning at all.
Combined with truncated backpropagation for RL, the method trains on traces past 100,000 tokens, far beyond what full-attention training can afford.
A pure sliding window with no prefix performs poorly on complex reasoning, confirming the prefix carries information the model keeps needing throughout generation.

Methodology

The kernel retains prefix and recent-window tokens only, using intra-tile masking for correctness on partially overlapping tiles and inter-tile skipping to avoid computing anything outside the retained set. For RL training, gradients focus on the final window while 4× surrounding context is passed through for accurate gradient estimates, and “Continue PE” reuses cached position embeddings rather than recomputing them.

Results and Findings

Without any training, Prefix Sliding on Qwen3-1.7B hits roughly 3× speedup over full attention while matching its accuracy on GPQA, MATH500, and AIME25. With GRPO training using truncated backpropagation, the method reaches 128K-plus token reasoning traces at memory budgets comparable to full attention’s 8K limit, with KL divergence between generator and trainer stabilizing at acceptable levels once the 4× context window is included. “Last K” retention and summarization-based alternatives underperform, both because they reprocess tokens the sliding approach never touches and because their memory usage is irregular in ways that hurt GPU utilization.

Implications and Conclusions

What makes this worth adopting immediately is the zero-training path: any deployed reasoning model gets a 3× speedup for free. The RL extension matters more for where the field is headed than for what’s shipped today, since 100K-token training runs are still rare outside a handful of labs, but the bounded-cost baseline this establishes is exactly what future long-horizon reasoning work will need to beat.

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

Authors: Sheng Liang, Yongyue Zhang, Nathanael Brian, Hang Lv, Hao Wang, Chen Zhang, Yong Liu

Source and references: https://arxiv.org/abs/2608.26004v1

Introduction

Agentic LLMs accumulate context fast, through retrieval, tool calls, and multi-turn history, and every token of that context costs verifier compute in standard speculative decoding. AsymSpec breaks the drafter and verifier apart: the lightweight drafter reads the full context while the large verifier reads a compressed one, capturing latency savings without discarding the signal compression would normally destroy.

Key Points

Asymmetric context access: the drafter processes uncompressed input, the verifier operates on a compressed view, decoupling the two roles that speculative decoding usually keeps in lockstep.
Contrastive δ-fusion steers the verifier’s logits using differences from the full-context drafter, giving the compressed verifier access to signal it would otherwise lose.
A divergence-aware acceptance gate modulates how much the drafter’s influence counts, preventing verification instability while keeping draft acceptance rates high.
Gains are largest in exactly the scenarios where compression normally hurts most: retrieval-augmented generation, tool use, and multi-turn dialogue.

Methodology

The framework keeps the standard speculative decoding loop but restructures what each model sees. The drafter’s full-context logits are fused into the compressed-context verifier through contrastive δ-fusion, with the acceptance gate adjusting the fusion strength based on distributional divergence between the two models at each step.

Results and Findings

AsymSpec reaches roughly 90% of full-context accuracy on average while delivering 1.3 to 1.7× throughput speedups at 0.2 to 0.3× compute cost on isolated text capabilities. Retrieval-augmented tasks recover performance lost to signature-only compression, and multi-turn dialogue holds stronger contextual reasoning than compression baselines. The acceptance gate is doing real work here: without it, naive asymmetry introduces verification failures and acceptance rates drop, while the adaptive version preserves stability under the same radical asymmetry.

Implications and Conclusions

The 90%-of-full-context number is the one to watch, since it’s a real cost against accuracy and not free. Whether that 10% gap is acceptable depends entirely on the downstream task’s error tolerance, and the paper doesn’t report which specific failure modes account for the remaining gap, whether it’s factual precision loss or something more structural. That breakdown would tell practitioners which agentic workloads are safe to compress and which aren’t.

VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction

Authors: Zhifei Xie, Jiaqi Lang, Ze An, Yifan Zhao, Dongchao Yang, Kai Li, Ziyang Ma, Mingbao Lin, Chunyan Miao, Shuicheng Yan

Source and references: https://arxiv.org/abs/2608.26005v1

Introduction

Voice assistants need memory that answers in under 150ms to fit inside a natural conversational pause, which rules out most existing retrieval architectures. VoiceMem splits memory into a “left brain” for facts and a “right brain” for emotion and persona, and bets that dense top-K retrieval at K=5 beats sparse retrieval at K=100.

Key Points

A two-level schema-entity index narrows the candidate pool before backend search, which is what keeps retrieval dense rather than exhaustive.
Retrieval latency holds flat at 134ms regardless of K, from top-1 through top-100, fitting entirely inside standard Voice Activity Detection windows.
At K=5 the system uses 430 tokens versus 1,899 to 6,956 for competitors reaching comparable accuracy.
An emergent cluster mechanism reorganizes memory schemas based on actual query patterns rather than fixed, predetermined categories.
The index is backend-agnostic: swapping in Mem0, LangMem, or Zep as the underlying store still improves each of them (+29.52, +15.76, +22.92 points respectively).

Methodology

The pipeline trains via online black-box on-policy distillation to build memory-aware speech models without catastrophic forgetting, producing CHATMEM-400K for training and a 14-category, 53-hour audio benchmark for evaluation. Streaming retrieval runs across four timed stages, listening, speech tail, anticipation, and searching, mapped onto natural conversational pauses.

Results and Findings

VoiceMem reaches 76.39 on information benchmarks against Mem0’s 52.27, and 74.16 on persona understanding against MemOS’s 72.27. The biggest gaps show up on reasoning-heavy tasks: +54.9 points over Mem0 on temporal reasoning. On the audio-grounded ChatMem-Bench, VoiceMem averages 68.73 against baselines scoring 3.23 to 26.92 on paralinguistics and environmental reasoning, since text-only systems simply can’t see that information.

Implications and Conclusions

The flat 134ms latency across K values is the finding that should reshape how memory systems get designed for real-time settings: the field has been treating larger retrieval budgets as inherently more expensive, and this shows that’s an artifact of sparse indexing rather than a fundamental constraint. The paper stops at three tested backends, though, and whether the index transfers to memory systems built on fundamentally different retrieval paradigms, graph-based stores in particular, remains untested.

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

Authors: Mengru Wang, Haozhe Luo, Zhenqian Xu, Zhixiang Cui, Haoming Xu, Qu Yang, Jizhan Fang, Junfeng Fang, Ningyu Zhang

Source and references: https://arxiv.org/abs/2608.20202v1

Introduction

The assumption behind memory-augmented LLMs is that correct, relevant retrieved information helps. MemTrapBench tests that assumption directly and finds it doesn’t hold: retrieved memories that are both accurate and relevant can still degrade performance by anchoring the model to outdated strategies or overriding safety judgment.

Key Points

Two trap categories: Reasoning Fixation, where models overgeneralize past strategies, misapply old task rules, or avoid correct approaches due to prior negative feedback, and Belief Distortion, where false premises in memory override safety knowledge.
The 1,050-instance benchmark comes from 18-to-40-turn dialogues generated by GPT-5.4 that bury a planted trap in noise before triggering it with a modified query, followed by two-stage quality control combining automated filtering and expert human review.
Every memory framework tested underperforms the no-memory baseline, with the strongest method (EverMemOS) still losing over 10 points relative to answering without memory at all.
AdaptiveMem, a lightweight prompt that instructs the model to check for potential memory traps before using retrieved context, recovers much of the gap without any architectural change.

Methodology

Seed instances specify domain, trap mechanism, ground truth, and planted prior; GPT-5.4 expands each into a long dialogue; two-stage quality control filters and human-reviews for topic coherence, context consistency, adversarial realism, and standalone solvability. Evaluation runs Gemini and Qwen model families against five memory frameworks, scored by GPT-5.2 with rubric-based judging and cross-checked with Claude Sonnet 4.6.

Results and Findings

Without memory, Gemini-3-Flash-Preview and Qwen3-30B score 85.16% and 81.83%. With memory applied, every framework drops: EverMemOS falls to 71.17% on Gemini, a 14-point loss, and LightMem tops out at 70.13% on Qwen. Cognitive Bias scenarios are hit hardest, ranging 46.66% to 65.48% on Gemini, and Safety scenarios drop to 56.15% to 69.70%. Ablations confirm the failures trace to the designed traps rather than context length alone. AdaptiveMem improves LightMem by 14.9 points on MemTrapBench for Gemini without hurting its scores on standard memory benchmarks.

Implications and Conclusions

This is the paper VoiceMem’s dense retrieval story needs to sit next to: retrieving the right memory faster doesn’t help if the memory itself is what derails the model. The Safety category result is the one that should concern anyone deploying memory-augmented assistants in production, since a planted false premise overriding basic safety judgment is a failure mode no amount of retrieval accuracy fixes. AdaptiveMem is a reasonable stopgap, but it’s a prompt-level patch, and whether it holds against traps specifically designed to evade a “check for traps” instruction is untested.

Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders

Authors: Yash Kulkarni, Shubham Harkare, Arvind Suresh Yogesh Babu

Source and references: https://arxiv.org/abs/2608.20280v1

Introduction

CLEVER runs seven cache eviction policies for semantic LLM caches through the same controlled protocol across three corpora, three capacities, and two embedding models. The headline finding cuts against a decade of geometry-aware eviction research: plain Least Frequently Used matches or beats every fancier alternative tested.

Key Points

No policy improves on LFU by more than 0.041 percentage points across eighteen settings, while FIFO and streaming SISO lose up to 8.67 and 8.55 points at tight capacity.
The packing condition: under exact-lookup, insert-on-miss admission, new entries never have resident neighbors inside the hit radius, which is why geometry-aware eviction has nothing to exploit.
Hit-rate thresholds calibrated on one embedding model don’t transfer to another: MiniLM-calibrated thresholds applied to gte-base produce a degenerate 100% hit rate.
Quality-adjusted hit rates collapse the story: raw hit rates of 51 to 60% on LMSYS and QQP fall to 1.1 to 2.2% once an LLM judge checks whether the cached response actually answers the new query.

Methodology

CLEVER pairs FAISS index benchmarking (Flat, HNSW, IVF, LSH) with a cost-based adaptive router and seven interchangeable eviction policies, all sharing the same corpora, index, threshold, and measurement suffix. An LLM-judge audit (Llama-3.1-8B-Instruct, cross-checked for consistency) separately scores whether a cache hit is actually a valid substitute for a fresh call.

Results and Findings

HNSW hits 0.989 Recall@1 at 0.52ms, a 34× speedup over exact search for 1.1 points of recall. LFU stays the strongest simple policy everywhere; ARC matches it within 0.01 points, and the semantic-redundancy policy’s best improvement (0.041 points) costs 5.83 to 8.24× the runtime overhead of LRU for essentially nothing. The corpus itself explains why: only 0.43% of pairwise distances in the processed LMSYS data fall within the 0.90 eviction threshold, so there’s rarely enough local redundancy for a smarter policy to find. On the quality audit, even the controlled MOSS dataset with a 97.6% raw hit rate passes only 24.1 to 26.4% of cache hits as genuinely valid.

Implications and Conclusions

The eviction-policy comparison ends up a footnote next to the quality-adjusted numbers. A team that ships a semantic cache and reports a 55% hit rate is very likely reporting something closer to 2% of genuinely useful cache hits, and no eviction policy sophistication fixes a threshold that’s admitting wrong answers in the first place. Calibrate the threshold with human-labeled answer-substitutability before spending engineering time on the eviction algorithm; that ordering, backwards from how most teams build these systems, is the paper’s real contribution.

Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents

Authors: Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian, Jiawei Zhou

Source and references: https://arxiv.org/abs/2608.20274v1

Introduction

Skill libraries let agents reuse strategies from prior tasks, but the level at which a skill gets extracted, whole-task or subtask, determines whether it helps or hurts. This paper isolates that variable directly, and the results argue for restructuring how most current skill-memory systems are built.

Key Points

Subtask-level skill induction improves agent performance by 1.9 to 2.9 points on average; task-level induction decreases it by 1.2 to 4.1 points across the same benchmarks.
Text skills beat code skills at both induction levels, by 2.9 points at the subtask level and 1.4 at the task level.
A skill utility score, the product of specificity (match to real tasks) and abstractness (generalizability across tasks), predicts transfer success without running the task at all.
Neither dimension alone predicts success; each shows an inverted-U relationship, meaning a useful skill has to be both specific enough to apply and abstract enough to generalize at the same time.
Findings hold across 11 models from 4B to 235B parameters and three benchmarks totaling 809 tasks.

Methodology

A task-level agent extracts one skill per complete trajectory; a subtask-level agent decomposes the task first and extracts one skill per subtask. Both store skills as either natural-language text or Python code in a retrievable memory. The utility score is computed from skill embeddings and task descriptions with no execution required, then validated against actual retrieval and reuse patterns during task streams.

Results and Findings

Task-level skills cut AppWorld success by up to 7.4 points across models; subtask-level text skills lift average performance from 22.1% to 26.7% across all benchmarks and models, and the gap persists when comparing at equal compute cost. Tasks retrieving high-utility skills succeed at 31.0% versus a 22.8% baseline at the subtask level, and 24.5% versus 14.0% at the task level. Splitting skill libraries at the median utility score consistently favors the high-utility half, and transfer density metrics confirm agents actually retrieve the skills the score predicts they should.

Implications and Conclusions

The finding that should worry anyone with a skill-memory system already in production is that task-level induction, the more obvious and more commonly implemented design, actively hurts performance rather than merely underdelivering. The utility score is a genuinely useful diagnostic precisely because it needs no execution, but it was validated on the same three benchmarks it’s meant to generalize beyond, and how it holds up on a skill library built from a genuinely different task distribution is the natural next test.

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

Authors: Jeongjae Lee, Jinho Chang, Jeongsol Kim, Jong Chul Ye

Source and references: https://arxiv.org/abs/2604.17415v4

Introduction

Reward-based fine-tuning methods for diffusion and flow models, DDPO, DRaFT, ReFL, VGG-Flow, and others, were each derived from different theoretical starting points: soft RL, optimal control, direct backpropagation, GFlowNets. This paper shows all of them reduce to the same L2 score-matching objective, differing only in how they estimate value guidance, weight timesteps, and enforce trust regions.

Key Points

A canonical gradient decomposition (Eq. 12) splits every method’s update into three competing forces: effective guidance, KL regularization, and trust region constraints, exposing the actual design hierarchy underneath the surface-level differences.
Value guidance estimators split into first-order (gradient-based, low variance, biased at low-SNR timesteps) and zeroth-order (gradient-free, unbiased, noisy at high-SNR timesteps), and existing temporal weighting schemes turn out to be compensating for exactly these failure modes.
Redesigned variants built from this understanding hit a 5× estimated speedup on GenEval with SD3.5-M while holding competitive reward-KL tradeoffs.
Under fixed compute, unbiased full-rollout estimators underperform shallower lookahead variants because the added variance outweighs the bias reduction, and reward centering reduces RMSE at zero additional cost.

Methodology

The authors connect the optimal intermediate score function to a value-guided target, then prove (Theorem 3.2) that major existing methods share a loss combining score regression toward that target with old-policy anchoring. Isolated 2D toy problems validate local estimator design choices under fixed compute, while end-to-end experiments on GenEval and HPSv2.1 with Stable Diffusion validate the full joint design space.

Results and Findings

The redesigned zeroth-order flow-matching variant reaches GenEval = 0.97 at an estimated 5× speedup over TempFlow-GRPO. Sparse branching allocation, concentrating rollout compute at select timesteps rather than spreading it uniformly, substantially improves the reward-KL tradeoff for diffusion settings. Replacing biased local Tweedie guidance with terminal reward gradients, and reversing the usual low-SNR downweighting, produces first-order methods that improve reward faster while staying competitive on distribution-shift metrics. Variance-aware budget reallocation beats uniform distribution consistently.

Implications and Conclusions

The value here is less the 5× number and more that it came from following the framework’s logic rather than tuning by hand, which is the actual test of whether a unifying theory is doing real work. What the paper doesn’t settle is whether these design principles hold for reward models trained on human preference data rather than the automated GenEval and HPSv2.1 scorers used here, where the reward landscape is likely noisier and the bias-variance tradeoffs this framework depends on may shift.

How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention

Authors: Gerard Conangla Planes

Source and references: https://arxiv.org/abs/2608.26052v1

Introduction

LoRA rank is usually chosen by trial and error. This paper derives task-dependent bounds on the attention approximation error achievable at each rank, replacing the guesswork with a certificate: given a downstream input distribution, you can now bound how much rank a LoRA update needs before the approximation is provably good enough.

Key Points

Attention KL divergence connects to score approximation error through ψ(t) = min{t², t}, quadratic for small errors and linear for large ones, capturing softmax’s saturation behavior.
The key bound depends on downstream-weighted tail energy, not raw singular values of the target update, meaning task-irrelevant directions in the update simply don’t count toward the rank requirement.
Softmax saturation genuinely reduces required rank: an explicit construction shows rank-k logits need only k − ⌊k/3⌋ to match once softmax saturation is accounted for.
The framework extends to fused multi-head LoRA (error scales by √H across heads) and joint query/key updates, where a measurable “factorization gap” captures the cost of constraining separate rank budgets.

Methodology

The main result (Theorem 4.1) bounds the best achievable attention KL at rank r between an explicit lower bound and an upper bound achieved via weighted SVD of the task-geometry-weighted target update, under assumptions of target realizability, bounded probability floors, and moment conditions on activations. Where probability floors are too small for the global bound, a target-Fisher alternative applies with reduced constants.

Results and Findings

The bounds hold across the tested regimes: finite-score approximation, softmax saturation, multi-head aggregation, and query/key factorization constraints. The Walsh-family construction demonstrating rank k − ⌊k/3⌋ closure under saturation is a provable separation result, derived analytically rather than measured. The factorization gap analysis quantifies exactly how much accuracy is lost by choosing query and key ranks independently rather than jointly, information no prior LoRA rank-selection heuristic provided.

Implications and Conclusions

This turns rank selection from a hyperparameter sweep into something you can reason about in advance, provided you know or can estimate the task geometry that determines the weighted tail energy. That’s the catch worth flagging: the bounds apply specifically to attention KL divergence, and the gap between “the attention function is well approximated” and “downstream accuracy holds” is exactly where a lot of LoRA rank folklore actually lives. Closing that gap is the natural next step, and until someone does, this is a necessary condition for sufficient rank rather than a sufficient one.

StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models

Authors: Zhe Liu, Jinghua Hou, Yuxiang Lu, Zhenya Yang, Xianzhe Fan, Junwei Luo, Junyi Li, Ruihua Han, Zhi Hou, Hengshuang Zhao

Source and references: https://arxiv.org/abs/2608.26067v1

Introduction

π0.5 and similar VLA models process a single frame at a time, which throws away temporal context robots need for precise manipulation. StreamPI restructures attention to add temporal reasoning on top of π0.5 with zero new parameters, inheriting the full pretrained weights through the LLM backbone’s length extrapolation.

Key Points

Instruction-anchored temporal units pair each visual observation with its language instruction under bidirectional attention, while causal attention links units across time, which prevents the model from forgetting the instruction over long horizons.
Random-interval streaming training samples inter-frame intervals from a uniform distribution rather than fixed spacing, closing the gap between synchronous training and asynchronous real-robot deployment.
KV caching keeps inference cost constant regardless of temporal horizon, since only newly arriving frames need encoding.
The model degrades gracefully at test time: trained with 5 frames, it still beats the single-frame baseline when evaluated with fewer.

Methodology

Bidirectional attention fuses each observation-instruction pair internally; causal attention aggregates across pairs for autoregressive streaming. Temporal masking during training randomly hides early frames to simulate the incremental observation pattern of real streaming inference, and only new frames get encoded at inference time, attending to cached historical representations.

Results and Findings

On real robots, StreamPI improves rolling object grasping by 36.6% and shell-game performance by 33.3% over π0.5, both memory-dependent tasks. Perception-dependent tasks gain too: +26.7% on pen insertion, +32.0% on cup insertion. On LIBERO, average success rises from 96.9% to 98.3%, with the largest gain (+2.6%) on LIBERO-Long, the benchmark explicitly testing long-horizon memory. On CALVIN, success at the 5th task position in a sequence holds at 85.0% versus 79.5% for the baseline. Random-interval training beats fixed-interval consistently (97.5% vs. 96.4% at T=3).

Implications and Conclusions

Zero added parameters and full weight inheritance from π0.5 is what makes this immediately adoptable rather than merely interesting, since teams already running π0.5 in production get temporal reasoning without a retraining cycle from scratch. The LIBERO-Long and CALVIN task-5 numbers are the ones that matter most, since they isolate exactly the long-horizon memory failure mode single-frame VLAs are known to have, and the gains there are the largest in the paper.

R3R^3 R3: Training Robots to Reason in Natural Language via Reinforcement Learning

Authors: Lehong Wu, Yuxiao Qu, Zheyuan Hu, Ivan Zhang, Limin Wei, Zackory Erickson, Aviral Kumar

Source and references: https://arxiv.org/abs/2608.26053v1

Introduction

Whether language reasoning genuinely functions as test-time compute for robots, rather than just a training-time crutch, has been an open question. R³ trains a VLM reasoner to generate free-form reasoning before instructing a frozen low-level policy, and then tests the causal claim directly by truncating the reasoning budget at inference and watching performance degrade.

Key Points

A two-stage recipe: mid-training on a limited set of expert reasoning traces to establish style, then single-step rubric-based RL (Dr.GRPO) on broader instruction-only data that never needs reasoning annotations at all.
Truncating reasoning at test time systematically hurts performance, and the model generates longer traces on harder tasks unprompted, which is the strongest evidence in the paper that the reasoning is doing real work rather than window dressing.
Instruction-only imitation learning that received reasoning as training-time supervision, but generates no reasoning at inference, still substantially underperforms R³ on out-of-distribution tasks.
The reasoner develops action-oriented behaviors on its own: comparing alternatives, self-correcting, resolving visual ambiguity, and tracking progress across long horizons.

Methodology

Stage I mid-trains the VLM on expert reasoning traces (collected via Gemini 3 Flash as a synthetic expert across 14 block-arrangement tasks for Language Table, and human teleoperation for grocery packing). Stage II applies Dr.GRPO with a VLM-judge reward measuring semantic consistency between generated and expert instructions, requiring no further reasoning labels.

Results and Findings

On Language Table, R³ improves mid-training-task performance by 17.3 points on average and outperforms instruction-only IL on all five held-out tasks, reaching 69.2% on the V-shape task against IL’s 40.9%. VQA diagnostics show reasoning improves both perception and action understanding, but perception gains alone don’t explain the manipulation improvement, isolating reasoning itself as the mechanism. On bimanual grocery packing, R³ (RL only) reaches 47.9% mean success versus 38.0% for instruction-only IL, and 73.1% normalized task progress versus 56.6%.

Implications and Conclusions

The truncation experiment is the paper’s real contribution, since prior work showing reasoning helps robots couldn’t rule out that the benefit came entirely from better training-time representations rather than anything happening at inference. This settles that question for the tasks tested. It remains untested on real robots with dexterous manipulation, where perceptual noise and contact dynamics are harder than anything in Language Table or the grocery-packing setup, and that’s the natural next validation.

EATR-Stereo: Embodiment-Aware Token Routing of Paired Stereo Evidence for Humanoid Vision-Language-Action Control

Authors: Songwei Wu, Rui Zhao, Fan Yang, Zhongqiang Nie, Zhiduo Jiang, Wandong Sun, Yuwei Li, Jian Hu, Yang Liu, Hong Liu

Source and references: https://arxiv.org/abs/2608.17453v3

Introduction

Head-mounted stereo cameras on bipedal humanoids see dynamic self-occlusion and viewpoints that shift with every step, which complicates naively fusing stereo into a pretrained VLA. EATR-Stereo keeps the primary-view pathway of a frozen vision-language model untouched and routes auxiliary stereo evidence in selectively, conditioned on the robot’s own body state.

Key Points

Primary-Aligned Cross-View Auxiliary Tokens build a dedicated auxiliary stream through cross-attention without disturbing the frozen VLM’s native primary-view representations.
Body-segmented proprioceptive routing conditions token-wise stereo usage on five body-segment histories (legs, arms, head, waist, pose), so the model uses stereo evidence when the robot’s own configuration suggests it’s needed.
Under severe asymmetric occlusion, where the target is fully hidden in the primary view but partially visible in the auxiliary one, the method recovers 80% of the time against 30% for baselines, 46% faster.
Training adds only 5.8% overhead over the baseline GR00T recipe, versus 3.82× the training time for an explicit depth-estimation alternative reaching equivalent performance.

Methodology

Stereo pairs pass through a frozen Cosmos VLM; primary tokens supply attention queries, auxiliary tokens supply keys and values. The proprioceptive router encodes each body segment through independent MLPs with segment-type embeddings, aggregates via attention-weighted and mean pooling, and computes token-wise gating weights that determine how much auxiliary evidence flows into the primary stream before a frozen flow-matching action expert generates the action chunk.

Results and Findings

On RoboCasa365 simulation, EATR-Stereo reaches 43.33% aggregate success across 18 tasks, 3.89 to 10.56 points ahead of comparable methods. On a physical 33-DoF humanoid across 20 trials, it hits 60.0% full-task success, 100.0% grasp success, and 80.0% stage success. Ablations show preserving the primary token pathway via a dual-stream design adds 30 points of full-task success over directly modifying primary tokens, and body-segmented routing adds further gains over flat-state routing (grasp success 90.0% versus 100.0%).

Implications and Conclusions

The occlusion-recovery numbers are what separate this from prior stereo-VLA work, since 80% recovery against 30% is the difference between a robot that adapts to a bad viewing angle and one that simply fails. The compatibility with a frozen pretrained VLM is the part that should influence how the field builds multi-sensor robot policies going forward: modifying the base model directly cost 30 points of performance here, and that gap is unlikely to be specific to Cosmos.

4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Authors: Yudong Jin, Tao Xie, Qihang Zhang, Zehong Shen, Zhen Xu, Yujun Shen, Hujun Bao, Xiaowei Zhou, Yinghao Xu

Source and references: https://arxiv.org/abs/2608.20335v1

Introduction

Reconstructing a 4D human avatar from a single uncalibrated video means generating dozens of consistent viewpoints for 4D Gaussian Splatting to lift into 3D over time, and video diffusion models start drifting once the view count exceeds what fits in one attention pass. 4DAnyone solves the two bottlenecks that cause that drift.

Key Points

Reference Context Packing compresses a growing set of reference views into a fixed-length context with constant complexity, so appearance guidance doesn’t degrade as view count scales.
Target Context Routing dynamically regroups target views during denoising: shared context across groups at high-noise steps for global structure, fixed adjacent-view groups at low-noise steps for detail stability.
3D-aware skeleton conditioning uses sparse 3D keypoints with depth-buffered rendering instead of dense depth maps, resolving pose ambiguity without needing reliable depth estimation or recovered camera parameters.
The pipeline works from casual, uncalibrated smartphone video, with no multi-camera rig required at capture time.

Methodology

HMR estimates a 3D skeleton sequence from the source video, rendered as depth-buffered skeleton videos at target viewpoints and injected via a 3D-aware skeleton encoder. A DiT denoises target latents conditioned on the source video, skeleton guidance, and the packed reference context, with Target Context Routing regrouping views across the denoising schedule. The generated multiview video sequences then feed FreeTimeGS for standard 4D Gaussian Splatting.

Results and Findings

On DNA-Rendering and DyMVHumans, 4DAnyone outperforms prior methods on both novel-view generation quality and downstream 4DGS reconstruction fidelity, maintaining consistency across tens of target views where existing camera-controlled diffusion models show appearance drift. Ablations confirm both components are load-bearing: removing Reference Context Packing degrades the reference context itself, and removing Target Context Routing reintroduces cross-group structural misalignment that accumulates across views. The method generalizes to in-the-wild video despite partial synthetic training.

Implications and Conclusions

Eliminating the multi-camera rig requirement is what matters most practically here, since it moves high-fidelity 4D human capture from a studio setup to a phone. The generalization claim rests on training that mixes MVGameHuman with light-stage and in-the-wild data, and the paper doesn’t report a breakdown of how much of the in-the-wild robustness traces to which data source, information that would matter for anyone trying to reproduce the generalization without the full dataset mix.

AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

Authors: Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na

Source and references: https://arxiv.org/abs/2608.20318v1

Introduction

Recursive self-improvement depends on agents that can redesign the training algorithm itself, the update rules and loss functions that determine how a model learns. AI4AI-Bench isolates that specific capability: ten frozen training-algorithm repositories, a hard budget, and a scoring function that only credits changes to the learning algorithm itself.

Key Points

Ten training-algorithm families: supervised fine-tuning, RL, reward modeling, preference optimization, diffusion RL, unlearning, graph diffusion, weight averaging, and pruning.
Agents get 4 hours on one GPU with a cheap proxy metric, then the submitted code alone (no weights, no cached state) reruns from scratch for up to 12 hours against a hidden evaluator.
A common scale maps every task’s incommensurable metric to 0 (uninformative model), 0.1 (the shipped algorithm), and 1.0 (task optimum).
53.6% of submissions never touch how the model learns, adjusting scheduling, checkpointing, or hyperparameters instead, changes that stay on the execution side rather than the learning side.
Submissions that do reach the learning layer average 0.226 versus 0.126 for those that don’t, a 0.100-point gap that holds across systems and tasks.

Methodology

Six systems (Claude Opus 5, Claude Sonnet 5, Kimi K3, and three GPT-5.6 variants) run 29 configurations across all ten tasks. Submissions are classified as touching the learning layer (losses, update rules, supervision) or staying on the execution side (checkpointing, hyperparameters, capacity), independent of the score itself.

Results and Findings

The study-wide mean score is 0.166; the best system reaches 0.250, closing less than a fifth of the distance between the shipped algorithm and the task optimum. Increasing reasoning effort from lowest to highest raises the share of submissions touching the learning algorithm from 8% to 64% and the mean score from 0.094 to 0.196, and most of that gain comes from agents attempting algorithmic changes at all rather than executing them better once attempted. Cost isn’t predictive: spending ranged ninefold across configurations with no corresponding ordering in performance, and Claude Opus 5 led at roughly half the median cost of the runner-up.

Implications and Conclusions

The 53.6% figure is the number that should recalibrate expectations about near-term recursive self-improvement: most agent effort, even at high reasoning settings, avoids the exact layer where compounding gains would come from. That reasoning effort mostly buys willingness to attempt algorithmic changes rather than skill at making them work is a distinction worth sitting with, since it points to what current training data and objectives fail to teach agents, rather than to any limit on how much they reason at inference time.

Hierarchical Planning for Embodied Agents, Test-Driven Code Generation, and Memory-Efficient Diffusion Models

State of AI — Wed, 19 Aug 2026 02:38:17 GMT

Welcome to this week’s edition of State of AI 👋

This week’s crop splits along a few lines. Efficiency work is everywhere: an architecture that gives prefill and decode their own compute budgets, a memory scheme that unlocks capacity as sequences grow, and diffusion accelerations worth up to 7× with no retraining. Robots get both sides of the story, with a hierarchical foundation model that spends extra compute only when it’s unsure, and two security papers showing how easily the state those planners trust can be poisoned. And a healthy batch of negative results made the cut too: guidance tricks that fail to beat plain CFG, interpretability tools that add nothing to behavior prediction, and LLMs that waste 2–6× the compute of a grid search when tuning physics simulations.

Here’s what caught our attention:

τ₀-VLA: A Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation: separating high-level subtask planning from low-level motor control, with compute allocated only at low-confidence decisions, gets robots through complex household manipulation across multiple embodiments at 45–90% success rates.
TDD-Agent: Test-Driven Reasoning for Code Generation: writing executable tests before implementation, then letting tests and code evolve together, reaches 78–90% pass rates on repository-level tasks.
Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation: an architecture that separates prompt processing from token generation, so the two inference phases can be optimized independently, with consistent quality gains across dense and sparse scaling.
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling: progressively expanding memory capacity as sequences grow beats static allocation on needle-in-haystack retrieval and long-context understanding, with zero added parameters.
When State Becomes an Attack Surface: State-Semantic Injection in LLM-Driven Embodied Agents: compromising the environmental state a planner reads can redirect robot behavior toward adversarial goals at 99% planning-level attack success, with user instructions and planner logic untouched.
Spectral Progressive Diffusion for Efficient Image and Video Generation: up to 7× speedup on images and 2.5× on video by riding the frequency spectrum diffusion models already generate implicitly, with zero architectural modifications.
SMA: Auditing Membership Leakage in Retrieval-Augmented Generation Systems: source-aware membership inference that tells you whether leaked content came from pretraining data, external retrieval, or user input, working in semi-black-box settings with no gradient access.
Le Critique: Privileged Value Functions for LLM Reinforcement Learning: feeding value functions task information the policy never sees (reference solutions, other trajectories) cuts variance in LLM RL while keeping policy gradients unbiased.

Let’s get into it 👇

Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation

Authors: Liming Liu, Mingze Wang, Tuo Zhao

Source and references: https://arxiv.org/abs/2608.12385v2

Introduction

LLM serving has a built-in mismatch: prefill is compute-bound, decode is memory-bound, and conventional models allocate identical computation to both. The Decode-Branch Transformer splits them. A primary path alone processes prompts and owns the KV cache, while a lightweight decode branch adds continuation computation on top, so each phase can be provisioned for what it actually costs.

Key Points

Phase-decoupled architecture: the primary path determines the persistent KV cache; the decode branch adds computation during generation while writing no cache state and leaving the primary path untouched, which makes the two phases independently tunable.
Shared-weight efficiency: primary and decode paths share all major attention, MLP, and output matrices, with separate embeddings and learned coupling vectors, so extra decode computation avoids proportional growth in memory traffic during the memory-bound phase.
Consistent quality gains: across NanoGPT token scaling, dense LLaMA-style scaling, and sparse MoE configurations, Decode-Branch reaches lower validation loss than baseline Transformers under matched token budgets.
Router replay for MoE: the decode branch reuses the primary path’s selected expert set with independent mixture weights, keeping routing cost flat while allowing phase-specific expert allocation.
A three-way trade-off surface: primary and branch expert fan-outs become independently configurable knobs, exposing trade-offs among prefill cost, decode cost, and model quality that can be tuned per serving workload.

Methodology

The core mechanism is asymmetric shared-KV attention: the primary path performs standard causal self-attention, and the decode branch generates queries that attend to the primary path’s keys and values through learned coupling. Training combines predictions from both paths with weights set by the primary distribution’s confidence. Evaluation runs three axes: controlled data scaling with NanoGPT (five token budgets), dense LLaMA-style scaling from 0.125B to 1B parameters at 80× parameter count in tokens, and sparse MoE scaling from 0.25B to 1B with independent routing or router replay, all under matched token budgets.

Results and Findings

On NanoGPT, Decode-Branch beat the standard Transformer at all five token budgets, with fitted asymptotes of 2.9013 versus 2.9416. In an approximate compute-matched comparison, a D=10 Decode-Branch beat a D=20 Transformer on validation loss with equivalent backbone computation. Dense scaling showed persistent gains at every size from 0.125B to 1B. MoE runs improved consistently with router replay, and fully independent routing added only modest further gains. The allocation sweeps are the most useful result for practitioners: fixed-prefill sweeps show monotonic loss improvement as the branch expert budget grows, and fixed-decode sweeps peak near three-quarter prefill fractions, mapping the trade-off surface directly.

Implications and Conclusions

The idea lands exactly where serving economics hurt, since decode compute is what you pay for on every generated token while prefill amortizes over the prompt. The evidence so far is validation loss at 1B scale and below, though, and the paper stops short of the number that would seal the argument: measured end-to-end throughput on a production serving stack, where continuous batching and cache pressure decide what actually helps. Until someone publishes that, treat this as a well-mapped design space awaiting its deployment test.

TDD-Agent: Test-Driven Reasoning for Code Generation

Authors: Hongyue Yu, Kefan Li, Jiakun Li, Hongzheng Chai, Yuan Yuan, Rui He, Junyi Wei

Source and references: https://arxiv.org/abs/2608.16742v1

Introduction

TDD-Agent applies test-driven development to LLM code generation. The model writes executable unit tests before any implementation, then code and tests refine each other iteratively based on execution feedback. Prior pipelines treat generated tests as static validators applied after the fact; here test generation works as a reasoning step that forces the model to pin down expected behaviors, input constraints, and edge cases up front.

A Breakthrough in Cost, Multi-Agent Reasoning Under Uncertainty, and Hallucination Mitigation Through Nested Memory

State of AI — Thu, 13 Aug 2026 17:20:34 GMT

Welcome to today’s edition of State of AI 👋 And a warm welcome to our 26 new subscribers since last edition!

🔦 Spotlight: Pathway’s Cost-Efficiency Breakthrough

A 150M-parameter model just pushed the cost frontier of AI reasoning

I’m breaking the usual format this edition, because one paper deserves to be pulled out of the lineup and put in front of you before anything else. Cost-efficiency results usually earn a polite nod and a footnote. This one moves the actual frontier, from a model small enough to run for pennies, and I’d rather flag it now than in six months when everyone claims they saw it coming.

Pathway has published BDH-CQ, a post-Transformer reasoning model built on its Dragon Hatchling (BDH) architecture. The model learns new tasks from demonstrations through evolving recurrent memory and does its intermediate reasoning in a continuous latent workspace, decoding only the answer; no text-based chain of thought is ever generated.

On the public ARC-AGI-1 evaluation, the 150M-parameter system scored 29.5% pass@2 at a computed cost of $0.00070 per task, less than a tenth of a cent. That operating point breaks through the previously reported cost-versus-accuracy Pareto frontier and sets a new state of the art in benchmark cost efficiency.

For perspective: GPT-5.6 Luna (Low) scores 34.2%, only 4.7 points higher, at roughly 11× the per-task cost even after OpenAI’s 80% price reduction (57× at the price ARC Prize listed before the cut). GLM 5 scores about 15 points higher at roughly 243× the cost.

The 29.5% result was reproduced through a black-box evaluation by researchers from Bielik AI and NYU, run under a documented protocol without access to model weights, and Łukasz Kaiser separately replicated the ARC result himself. Early pretraining experiments from 1B to 600B parameters show Transformer-like scaling while preserving the latent reasoning capabilities.

Read the paper →

Full breakdown below, first in the lineup.

Beyond the spotlight, a few threads kept surfacing as we read this fortnight’s crop. Agent benchmarks keep finding the same failure point: models handle tool mechanics well and fall apart at grounding. One hallucination-mitigation pipeline turns out to mostly teach models to hedge. And there is now solid evidence that long-context training quietly erodes the knowledge stored in a model’s weights. Also in here: a 99.4% compilation rate on repository-scale code translation, and a rare look at real ChatGPT Enterprise usage data.

Here’s what caught our attention:

BDH-CQ: In-Context Learning with Recurrent Latent Reasoning: a 150M-parameter model hits 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, past the reported cost-accuracy Pareto frontier. Controlled experiments map exactly which visual concepts its latent reasoning learns from demonstrations, and where it falls off a cliff.
VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies: an 8,000-plus executable API benchmark. Frontier models lose over 50% accuracy as reasoning chains lengthen, and the failures trace to grounding; tool-calling mechanics mostly hold up.
Hallucination Mitigation with Agentic AI, Nested Learning, and AI Sustainability via Semantic Caching: a three-stage orchestrated pipeline whose semantic cache serves 47.7% of model calls. The candid finding: 83.5% of the measured hallucination improvement comes from a single dimension of a five-dimension score.
ReCodeAgent: A Multi-agent Workflow for Language-Agnostic Translation and Validation of Large-Scale Repositories: 99.4% compilation success across four programming language pairs, achieved with lightweight MCP tools where prior systems needed 100K+ lines of language-specific program analysis.
Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization: standard DPO underuses the context in its own preference data. Optimizing a new Contextual Preference Gain metric cuts object hallucination by 36% relative.
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge: models trained on informative long contexts show inverted-U performance curves and become “context-addicted,” with gradient analysis showing optimization pressure shifting from FFNs to attention.
A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation: a 1.8× training speedup from replacing expensive proximal-policy recomputation with log-linear interpolation, a step that runs 3,000× faster.
BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases: a 68,000-question benchmark grounded in real biomedical databases. Frontier models reach 58% accuracy and stumble on implicit conventions, like genome-wide significance thresholds, that domain experts apply without thinking.
Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages: Bengali has 285 million speakers, under 0.5% of web content, and a 67:1 English-to-Bengali token ratio in major training corpora. Tokenization penalties and connectivity gaps compound from there.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

BDH-CQ: In-Context Learning with Recurrent Latent Reasoning

Authors: Björn Engdahl, Adrian Kosowski, Jan Chorowski, Zuzanna Stamirowska, Przemysław Uznański, Junlin Jiang, Rohan Phadke, Remigiusz Kinas, and Richard Zhong

Source and references: https://arxiv.org/abs/2608.09888v1

Introduction

BDH-CQ combines in-context learning with recurrent latent reasoning. Demonstrations presented at inference time continuously update the model’s recurrent memory; the query is then solved through iterative computation in a high-dimensional latent workspace, and only the answer gets decoded. No parameters update at inference, and neither task identifiers nor evaluation-task demonstration pairs participate in training. A 150M-parameter configuration reaches 29.5% pass@2 on the public ARC-AGI-1 evaluation at a computed $0.00070 per task, past the previously reported cost-accuracy Pareto frontier.

Key Points

Two memories, two jobs: a recurrent contextual state evolves as each demonstration is ingested (Sₜ = U_θ(Sₜ₋₁, Dₜ)) and carries the in-context learning; a separate latent workspace H_r carries the iterative computation that answers the current query, refined over R steps with zero intermediate decoding into tokens.
The headline operating point: 29.5% pass@2 on the 400-task public ARC-AGI-1 set at roughly 0.85 H200 GPU-seconds per task. At $3 per H200-hour, that computes to $0.00070 per task: 57× cheaper than GPT-5.6 Luna (Low) at ARC Prize’s listed cost, 11× cheaper after OpenAI’s 80% price cut.
Audited from the outside: a black-box audit by co-authors from Bielik AI and NYU reproduced the 29.5% under a documented protocol without access to model weights, and additionally evaluated ConceptARC and a hand-crafted set.
A capability profile with the cliffs mapped: on ConceptARC, boundary-propagation families reach 9/10 strict tasks while Copy and Order manage 2/10. Post-freeze controlled tasks show propagation and copying extrapolating cleanly across the tested ranges, while ordering collapses at sequence length eight (0/24 exact outputs) and nesting at containment depth five.
Demonstration coverage is causal: rerunning byte-identical failures with one demonstration at the test complexity lifts depth-five nesting from 19/24 to 24/24 and length-eight ordering from 0/24 to 13/24. The nesting cliff is mostly a failure to extrapolate demonstrated depth; long ordering keeps an execution bottleneck even with support.
Reasoning effort is a knob: training across latent-reasoning effort levels yields an inference-time dial, with LOW at 21% pass@2 (22% cost reduction), MEDIUM at 27% (11%), and HIGH at 29.5%.

Methodology

BDH-CQ builds on the Dragon Hatchling (BDH) architecture: high-dimensional positive activations, low-rank communication, and a recurrent associative state, brain-inspired in its principles (local interaction, sparse activity, persistent state, continual adjustment) without being brain-imitative. Training uses a curated ARC-style mixture: privately curated examples plus the public ARC-AGI-1 training set, RE-ARC, ConceptARC, ARC-Heavy, and ARC-GEN100K, with augmentations. Evaluation follows the ARC-AGI leaderboard’s two-attempt convention (pass@2), with the dollar figure computed from measured hardware time. The behavioral analysis runs on the ConceptARC ontology of 16 concept families and on deterministic post-freeze generators that vary one factor at a time (propagation distance, copy count, sequence length, nesting depth), plus composition tests over 3×3 motif families. Worth flagging up front: dimensions, exact update rules, and the full training recipe remain proprietary.

Results and Findings

The headline: 118/400 tasks solved (29.50% pass@2) on the public set, 97/400 at pass@1. On ConceptARC, 59.38% strict task pass@2 against 77.92% test-pair pass@2. That 18.5-point gap is diagnostic: 52 of 160 tasks had one or two of three test inputs correct, so the system often produces correct outputs for a task it fails to solve consistently across all inputs.

The controlled families generalize in sharply different ways. Propagation stays perfect on 48/48 held-out outputs across distances 2–8, and copying stays perfect as target sites scale from one to four. Ordering is nearly saturated through five objects, then falls to 29/36 at length six, 8/24 at seven, and 1/24 at eight. Nesting stays nearly saturated through depth four and drops to 29/36 at five. The failure signatures differ too: at ordering length eight only 3/24 outputs have correct dimensions, while all 36 depth-five nesting outputs are dimensionally correct with mean cell accuracy above 99.9%, typically off by a single containment decision.

Composition depends on representation. Rotation composes with relocation on 72/72 held-out outputs, reflection on 47/72, and color swap on 0/72, with color swap acquired atomically only in the motif family with a fixed color layout. Contextual binding, by contrast, is a strength: fresh color permutations defined entirely through demonstrations are applied elementwise on 96/96 held-out outputs at rank one, holding at 24/24 per level as simultaneous bindings scale from two to eight.

A replication with cryptographically opaque task identifiers and concept-mixed batches left aggregate ConceptARC performance unchanged (374/480 test pairs in both conditions), ruling out that combined request-side confound as the explanation for the score.

Implications and Conclusions

Two things make this more interesting than one more ARC number. The first is the cost axis: at less than a tenth of a cent per task, with a LOW/MEDIUM/HIGH effort dial, latent reasoning gets the same test-time-compute scaling story as token-based chain of thought while skipping the token bill entirely. The second is the behavioral methodology. The paper pairs its headline with controlled experiments that localize failures precisely (rule selection when a marker must choose between two demonstrated rules, parameter values absent from demonstrations, long orderings), which is more diagnostic honesty than most frontier-model reports offer.

The caveats deserve equal precision. The architecture’s dimensions, update rules, and training recipe are proprietary; the “independent” auditors are co-authors; and ARC-style grids sit a long way from language and mathematical reasoning, which the team itself names as the next test. The claim that 1B–600B pretraining shows Transformer-like scaling while preserving latent reasoning is, for now, one sentence in the paper. Scaling runs that confirm it would matter considerably more than the ARC score does.

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

Authors: Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor

Source and references: https://arxiv.org/abs/2608.12282v1

Introduction

VAKRA is a benchmark for agents in enterprise environments, where a single task can require calling structured APIs, retrieving documents, and respecting tool-use policies inside one reasoning chain. Existing benchmarks test those capabilities separately; VAKRA composes them and measures exactly where the composition breaks.

Key Points

Scope: over 8,000 executable APIs across 62 domains, organized into three difficulty settings: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning under natural-language tool-use policies.
Real executable grounding: all APIs and retrieval tools are self-hosted against live databases (derived from BIRD-SQL) and document collections, so evaluation happens by re-executing predicted tool calls instead of simulating them.
Trajectory-level evaluation: complete trajectories are verified by re-invoking predicted calls against the live APIs, which accommodates multiple valid solution paths and attributes failures to specific reasoning stages.
The numbers: GPT-5.5 reaches 70.4% on single-hop endpoint tasks and 50–51% on compositional APIs. Most models lose over 50% accuracy as reasoning depth increases. On policy-constrained unanswerable queries, accuracy collapses to 2.4% (Claude Opus) and 3.7% (GPT-5.5).
Where it breaks: trace analysis shows failures concentrating at entity disambiguation, cross-source grounding, and schema alignment. Tool invocation mechanics mostly hold up.

Methodology

The benchmark extends an existing API generation pipeline (Elder et al.) to produce executable Python functions backed by real BIRD-SQL databases, supplemented with domain-specific retrieval tools over ChromaDB indices built from ClapNQ and Wikidata5M. Multi-hop queries come from a four-stage LLM-assisted pipeline: extract entities, build domain knowledge graphs, link APIs with compatible input-output signatures, and generate compositional questions that require both structured and unstructured reasoning. Every model runs in the same ReAct harness (LangGraph), which keeps the comparison about reasoning capability instead of scaffolding. Evaluation proceeds through a three-stage waterfall: tool-sequence verification against the live APIs, response grounding and correctness, and deterministic policy-adherence checks.

Results and Findings

Performance varies sharply across interaction paradigms. Models that do well on endpoint-style APIs often struggle with compositional business-intelligence interfaces, and some models swap ranking positions entirely between the two settings. Multi-hop degradation is severe: most models lose over half their accuracy as chains lengthen, and the “sieve of success” analysis (Table 4) attributes most late-stage failures to grounding errors. Policy constraints compound the difficulty, and unanswerable queries expose the worst of it, with Claude Opus at 2.4% and GPT-5.5 at 3.7%. Hallucination accounts for the majority of grounding errors across most models and settings (Table 5): models misread tool responses far more often than they pick the wrong tool.

Implications and Conclusions

The bottleneck for enterprise agents is language-mediated interpretation: constraint reading, entity disambiguation across systems, and grounding information pulled from tool outputs. Better tool schemas won’t fix that. The 2.4% score on unanswerable queries is the number that should worry deployers most; an agent that confidently answers questions its policy forbids is a compliance incident, and this benchmark suggests that is currently the default behavior.

Hallucination Mitigation with Agentic AI, Nested Learning, and Semantic Caching

Authors: Diego Gosmar, Deborah A. Dahl

Source and references: https://arxiv.org/abs/2605.29055v2

Introduction

A three-agent review pipeline for catching hallucinations, wired to a Continuum Memory System (CMS) with semantic caching. The design goal is twofold: improve the factual reliability of deployed LLM systems, and cut the compute bill while doing it.

Key Points

Three-stage orchestrated pipeline: a Front-End Agent (high-stochasticity generator), Second-Level Reviewer (primary corrector), and Third-Level Reviewer (final enforcer) progressively tighten factual control by lowering temperature (1.0 → 0.1 → 0.05), coordinated through the Open Floor Protocol (OFP).
Continuum Memory System with semantic caching: each agent maintains Medium-Term Memory (LRU eviction) and Long-Term Memory (LFU eviction) with embeddings-based similarity matching (cosine threshold τ=0.87), achieving a 47.7% cache hit rate.
Five-dimensional evaluation: the Total Hallucination Score (THS) aggregates Factual Claim Density (FCD), Factual Grounding Ratio (FGR), Factual Disclaimer Frequency (FDF), Explicit Contextualization (ECS), and Observability Score Ratio (OSR).
Dual risk profiles in the benchmark: 217 realistic epistemic-uncertainty prompts (70%) where agents should hedge, plus 93 fabrication-induction stress tests (30%) designed to pressure the pipeline into inventing claims on demand.
Human-validated annotation: three independent annotators labeled final-stage stress responses, with inter-rater agreement at α=0.586. Re-scoring with Llama 3.1, Gemma 4, and Qwen 3 judges correlated better with human judgments than the original evaluator did.

Methodology

Each of the 310 test prompts flows through the three-agent pipeline via OFP. The Front-End Agent deliberately generates high-confidence responses with fabricated details to establish a measurable hallucination baseline. Each downstream agent queries its CMS first, comparing prompt embeddings against stored prompt-response pairs, and either reuses a cached response or invokes Llama 3.1. A KPI Evaluator independently observes all intermediate outputs and computes the five metrics under five weighting configurations. All 930 outputs (310 prompts × 3 pipeline stages) were additionally scored by three cross-family judge models under a common evaluation instrument.

Results and Findings

End-to-end, the pipeline improved THS by 6.1% of its attainable range. The decomposition is the real story: 83.5% of that improvement comes from a single dimension, Explicit Contextualization, while Factual Claim Density, the metric closest to detecting unsupported content, stayed flat. The first review stage delivered 97.7% of the total mitigation; the third stage showed diminishing returns and a U-shaped THS trajectory. Observability rose 147% at the review stage and fell back at the final stage, which fails to propagate the OFP annotation channel.

On the 93 stress prompts, human judges found that 10 of 93 final answers (10.8%, 95% CI 5.9–18.7) still presented invented items as real, with agreement below conventional thresholds (α=0.586). Explicit Contextualization tracked human labels monotonically. The cross-family judge ensemble correlated with human labels at Spearman ρ=−0.772, versus −0.477 for the original single-model evaluator.

The cache served 47.7% of model calls across the benchmark. Consolidation ran every 2 prompts for MTM updates and every 50–100 prompts for LTM promotion, an operational version of the Nested Learning paradigm that requires zero weight modifications.

Implications and Conclusions

Two results here deserve separation. The caching result is solid: serving nearly half of all calls from memory, with an auditable protocol, is worth copying whatever you think of the rest. The hallucination result is weaker than its framing. When 83.5% of measured improvement lives in one dimension, and that dimension amounts to “the model added contextualizing language,” the pipeline is teaching hedging more than it is removing fabricated content. Add the 10.8% of stress responses that still asserted invented items as real, judged by annotators who could barely agree with each other, and the detection problem looks very much open.

ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Dataset

Authors: Luciano Duarte, Olga Ovcharenko, Sebastian Schelter

Source and references: https://arxiv.org/abs/2606.09648v2

Introduction

ArtiFact is a multi-modal dataset of 651,045 museum records from the Metropolitan Museum of Art, the Art Institute of Chicago, and the Rijksmuseum, combining structured metadata, descriptive text, and images. It gives multi-modal data-management research something it has lacked: a realistic benchmark for data quality assessment and semantic query processing over cultural heritage collections.

Key Points

Scale and coverage: 651,045 artwork records from three institutions, with unified preprocessing that normalizes heterogeneous museum metadata through rule-based processing and LLM-assisted parsing.
Normalization pipeline: dates, dimensions, artist information, materials, and techniques standardized across three different metadata standards, with 168,000 “hard” records handled by LLM-based semantic parsing.
Curated error taxonomy: seven error categories (physical, culture, temporal, identity, geographic, spatial, visual) with nineteen subcategories, informed by museum curators and injected into 130,209 records to create a controlled evaluation setup.
A difficulty spectrum for cross-modal error detection: visually salient errors are reliably caught, while material anachronisms, temporal shifts, and cultural proximity errors slip through regularly.
Semantic query limits: current systems struggle with queries involving cultural ambiguities, historically contingent terminology, and implicit cultural knowledge embedded in museum records.

Methodology

Records came from the three institutions’ APIs and metadata harvesting protocols, keeping only those with public-domain images. Preprocessing ran in stages: global transformations and rule-based normalization for dates, dimensions, and materials using reference dictionaries of over 1,700 terms; Gemini 2.5 Flash with chain-of-thought prompting for the roughly 168,000 records that resisted rules; and consolidation into a unified 24-column schema, with categorical values deduplicated via sentence embeddings and LLM-based semantic classification. Baseline error-detection difficulty was characterized with Gemini-3-Flash on 200 records per error subtype plus 200 clean records.

Results and Findings

The dataset skews heavily. The Rijksmuseum contributes 53.13% of records with strong European representation, culture annotations appear in only 17% of records (mostly from the MET), prints dominate object types at 30.32%, and paper is the most frequent material at 52.06%. Roughly 10,500 unique object names appear exactly once. Error detection follows a clear hierarchy: image swaps, 10x scale errors, and continent-level geographic errors are caught reliably, while material anachronisms, temporal shifts, cultural adjacency swaps, identity errors, and geographic errors between neighboring countries evade detection.

Implications and Conclusions

The benchmark exposes a real gap: current multi-modal systems handle perceptual errors and miss cultural and historical ones, which is precisely the knowledge museums care about. Worth watching is whether the dataset’s own imbalance (53% of records from one European museum, culture annotations on 17%) limits what “cultural knowledge” future systems can learn from it. A benchmark about cultural nuance inherits the collection biases of its sources.

Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models

Authors: Qinwu Xu, Yifan Jiang, Haoyu Ren

Source and references: https://arxiv.org/abs/2605.16409v3

Introduction

An OCR-aware multilingual post-training framework that teaches a general-purpose MLLM to read and reason about text in images directly, with no external OCR engine, text detector, or bounding boxes at inference time. The target failure mode is familiar to anyone who has pointed a vision model at a receipt: small, degraded, or multilingual text under blur, occlusion, and messy layouts.

Key Points

OCR baked into the model: recognition capability comes from post-training the MLLM itself, which eliminates auxiliary OCR pipelines and their latency at inference.
5M multilingual training samples across English, Spanish, French, Italian, and German, covering receipts, menus, signs, and handwriting under challenging visual conditions.
Two augmentation strategies: controlled synthetic OCR generation with degradations (blur, rotation, occlusion), plus a modular in-situ visual translation system that swaps text inside existing images while preserving scene context, using SAM 2, inpainting, and style-adaptive rendering.
The gains: OCR completeness rises from 71.3 to 84.6, hallucination rate falls from 18.3% to 5.5%, and translation BLEU-1 climbs from 52.3 to 80.2 on held-out real-world benchmarks, with the largest improvements under degraded conditions.
CoT as a garnish: OCR-oriented prompt-guided reasoning adds modest gains beyond supervised fine-tuning, mainly helping when text is ambiguous or partially visible.

Methodology

The base model pairs a LLaMA-3 70B language backbone with a frozen MetaCLIP-based ViT encoder and a Perceiver-based resampler for token compression. Fine-tuning uses LoRA (rank 256, scaling parameter 512) on the roughly 5M OCR-oriented instances mixed with general multimodal supervision, covering text recognition, translation, and OCR-grounded reasoning. Training uses standard autoregressive cross-entropy with AdamW, distributed across 256 NVIDIA H100s with DeepSpeed ZeRO-3; one epoch takes about 30 hours.

Results and Findings

On the held-out multilingual benchmark, OCR completeness increases from 71.3 to 84.6 (18% relative), hallucination drops from 18.3% to 5.5%, and BLEU-1 climbs from 52.3 to 80.2. Degraded conditions show the biggest deltas: on blurred images hallucination falls from 24.8% to 6.6%, and on rotated images from 21.4% to 5.8%. Public benchmarks move in the right direction on OCR-intensive tasks (DocVQA 81.5 to 82.5, TextVQA 80.5 to 82.2) while general multimodal reasoning holds steady. The ablation ordering is instructive: supervised fine-tuning provides 0.8 points of the DocVQA gain, CoT prompting 0.2. End-to-end inference runs at 2.16 seconds per sample on 4×A100s at 0.464 samples/second.

Implications and Conclusions

Data-centric post-training wins here by a wide margin over prompting tricks: 5M targeted samples moved hallucination by 12.8 points while chain-of-thought moved DocVQA by 0.2. For teams maintaining a separate OCR pipeline in front of their vision-language stack, the practical question is whether 84.6% completeness justifies deleting that infrastructure. For documents where a missed field costs money, it probably doesn’t yet.

Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization

Authors: Byungoh Ko, Jinyoung Park, Jongha Kim, Jeehye Na, Jaewon Cho, Hyunwoo J. Kim

Source and references: https://arxiv.org/abs/2608.12158v1

Introduction

MLLMs hallucinate objects: plausible descriptions of things absent from the image. DPO is the standard mitigation, and this work identifies a specific defect in how it trains. The defect gets a name, “context blindness”: the model’s learned preferences barely strengthen when relevant context is provided, even when the preference data was enriched with context specifically to help.

Key Points

Context blindness, diagnosed: standard DPO and its variants underuse contextual information despite context-enriched preference data designed to improve performance.
Contextual Preference Gain (CPG): a metric quantifying how much a model’s preference for non-hallucinated responses strengthens when relevant context is provided. Higher CPG correlates directly with lower hallucination rates.
Context-Calibrated DPO (C2-DPO): a modified objective that directly maximizes CPG while preserving the original preference orderings.
36% relative reduction in hallucination on Object HalBench with Qwen2-VL-Instruct-2B, with general reasoning capabilities intact.
Generalization: results hold across multiple benchmarks and model variants, suggesting the effect goes beyond one architecture or dataset.

Methodology

The diagnosis comes first. CPG is computed as the difference in preference strength between responses evaluated with and without relevant context, and standard DPO shows limited gain on this measure, meaning context contributes little to what the model actually learns to prefer. C2-DPO then adds a CPG-maximization term to the DPO loss, balanced against the standard objective so that training stays stable and performance on unrelated tasks is preserved.

Results and Findings

On Object HalBench, C2-DPO cuts hallucination by 36% relative to baseline for Qwen2-VL-Instruct-2B, with comparable gains on other model variants. CPG values rise substantially versus standard DPO, confirming the mechanism works as designed. Accuracy on standard vision-language benchmarks holds steady. Ablations show the CPG-maximization term is the load-bearing component, and gains persist across visual context types and object categories.

Implications and Conclusions

The diagnostic matters more than the fix. CPG gives the field a way to test whether any preference-optimization method actually uses context, and the finding that standard DPO largely fails to should prompt a re-examination of other context-enriched training schemes. Enriching your data is pointless if the objective can’t feel the enrichment.

A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation

Authors: Xiaocan Li, Shiliang Wu, Zheng Shen

Source and references: https://arxiv.org/abs/2512.06547v4

Introduction

Asynchronous RL for LLMs has a computational sore spot: decoupled PPO requires an explicit forward pass to compute the proximal policy, which costs 4–8 seconds per step at LLM scale. A-3PO (Approximated Proximal Policy Optimization) replaces that forward pass with an interpolation that costs almost nothing, and keeps the training stability that made the proximal policy worth computing in the first place.

Key Points

The approximation: instead of a forward pass, the proximal policy is computed by staleness-aware interpolation in log-probability space between the behavior and target policies, a 3,000× speedup for this specific computation.
Staleness-aware coefficient: log π_prox = α log π_behav + (1−α) log π_θ, with α = 1/d where d is the staleness (the training-step gap between policies). As staleness grows, the approximation weights the target policy more heavily, keeping the proximal policy a valid trust-region anchor.
Theoretical guarantees: the method maintains the “sandwich property” (the proximal policy stays bounded between behavior and target policies) and provides contractive stability, where importance weights are contractively scaled, reducing variance and preventing the extreme ratios that destabilize training.
1.8× training speedup across experiments with 1.5B and 8B models on mathematical reasoning tasks, at comparable task performance to explicit recomputation and synchronous baselines.
Better stability at scale: at 8B parameters, A-3PO shows more controlled importance weights and fewer clipped tokens than the explicit recompute method, which exhibits unreliable high importance weights.

Methodology

A-3PO builds on decoupled PPO, which separates off-policy correction (behavior policy) from trust-region control (proximal policy) to handle data staleness in asynchronous training. The interpolation requires only element-wise tensor operations already available in the training loop, with zero additional neural network computation. When policies are synchronized, α = 1 and the method reduces to the exact case.

Results and Findings

Experiments compared A-3PO against decoupled GRPO with explicit proximal-policy recomputation and against synchronous GRPO, on Qwen2.5-1.5B/GSM8K and Qwen3-8B/DAPO-Math-17k.

Computational efficiency: the log-linear approximation takes 0.0012 seconds versus 4–8 seconds for explicit recomputation.

Training time: Setup 1 finished in 1.53 hours versus 1.82 for recompute (1.2×) and 2.36 for sync (1.5×). Setup 2 finished in 14.54 hours versus 16.10 for recompute (1.1×) and 26.15 for sync (1.8×).

Performance: final evaluation rewards were comparable across methods in Setup 1 (0.791–0.797). In Setup 2, asynchronous methods beat synchronous training outright (0.623–0.627 vs 0.443), with A-3PO matching recompute. On AIME24 and MATH500, A-3PO averaged 66.64% pass@1 versus 64.74% for recompute.

Stability: entropy decay stayed healthy across methods. The explicit recompute method produced very high importance weights at the larger scale, while A-3PO maintained balanced importance sampling and clipped the fewest tokens, suggesting smoother updates that naturally stay inside trust-region bounds.

Implications and Conclusions

The lesson generalizes past this paper: when an expensive component of an RL algorithm serves as an anchor rather than a quantity that must be exact, a principled approximation can replace it outright. That the approximation was also more stable at 8B than the exact computation is the surprising part, and worth a follow-up: it hints the recompute method’s “exactness” was mostly buying erratic importance weights. The method drops into any decoupled policy optimization approach, GRPO included.

When Do Anchor-Based Pointwise LLM Rerankers Help? Retriever Quality, Statistical Scope, and Anchor Design

Authors: Utshab Kumar Ghosh, Shubham Chatterjee

Source and references: https://arxiv.org/abs/2608.10528v2

Introduction

A rigorous reproducibility study of anchor-based pointwise LLM reranking, specifically the GCCP/PAGC method. The findings narrow the method’s claims considerably: its utility depends on first-stage retriever quality, aggregation strategy, and design choices that the original evaluation held constant.

Key Points

Reproduction reveals hidden complexity: a paper-only reimplementation achieved 0.24 nDCG@10 against the reported 0.66. Reaching faithful reproduction (within 1.6%) required eight undocumented implementation details, three of which silently produced plausible-looking but incorrect outputs when missed.
Contrastive scoring is the load-bearing component: the core anchor-based contrastive mechanism survives Holm-Bonferroni correction in 12 of 22 settings. The aggregation step combining contrastive and pointwise scores adds value in only 5 of 22 settings and is significantly harmful on DBPedia-Entity under dense retrieval.
Retriever quality moderates everything: with BM25 retrieval, PAGC improves nDCG@10 by +0.197 on TREC DL 2020. With E5 dense retrieval, the same reranker gains +0.013. Anchor-based reranking earns its keep when the first-stage candidate list contains errors to correct.
Spectral MDS anchors are unnecessary: the sophisticated spectral multi-document summarization anchor never beats simpler alternatives in ablations. A top-3 sentence-interleaved composite matches or exceeds it everywhere, with fewer hyperparameters.
The mechanism transfers across LLM families: results hold with decoder-only models, including a 4-bit quantized 72B model. Backbone family matters more than parameter count at the 7–8B scale.

Methodology

The authors work reproduction-first: starting from the paper text alone, they iteratively identify implementation details by comparing against released code, validate against reported results, then run controlled component-level stress tests isolating first-stage retrieval quality, anchor construction, and scoring aggregation. Statistical testing uses paired bootstrap with Holm-Bonferroni correction across 22 primary settings spanning TREC Deep Learning 2019/2020 and eight BEIR datasets, across encoder-decoder and decoder-only backbones.

Results and Findings

The full PAGC system improves over standard pointwise grading in 12 of 22 settings after correction, and the decomposition shows the improvement comes primarily from GCCP alone: the contrastive component is directionally positive in 19 of 22 settings (p≪0.001 by sign test) but hard to isolate at typical TREC DL query counts. Aggregation is significantly harmful on DBPedia-Entity (Δ=−0.0144, p=0.032) under E5 retrieval.

The retriever-quality effect dwarfs the reranking effect. With BM25 at 0.506 nDCG@10 on DL20, PAGC reaches 0.703. With E5 at 0.719, PAGC reaches 0.732: the first-stage improvement (21.3 points) far exceeds the reranking gain (1.3 points). Across eight BEIR datasets under E5, spectral MDS anchor construction loses to simpler alternatives in every setting, finishing last on TREC-COVID, Touché-2020, and Robust04.

Decoder-only experiments with Qwen-2.5-72B-Instruct-AWQ (4-bit) hit 0.7465 nDCG@10 on TREC DL 2019 PAGC, surpassing both the reproduced Flan-UL2 (0.7095) and the paper’s reported figure (0.7206). The DBPedia-Entity negative result persists across backbone families and dense retrievers (E5 and BGE), pointing to dataset properties, since implementation artifacts were ruled out.

Implications and Conclusions

For practitioners the guidance is concrete: with a weak first-stage retriever like BM25, deploy the full PAGC pipeline; with a strong dense retriever like E5, the contrastive scorer alone is usually competitive and avoids both the aggregation overhead and the entity-task harm. The methodological lesson cuts deeper. A method whose reproduction requires eight undocumented details, and whose reported gains shrink 15× under a better retriever, was arguably never evaluated under the conditions practitioners actually face. Uncorrected per-cell significance testing inflated the apparent value of components, and IR papers need to document their pipelines at a level the field currently treats as optional.

ReCodeAgent: A Multi-agent Workflow for Language-Agnostic Translation and Validation of Large-Scale Repositories

Authors: Ali Reza Ibrahimzada, Brandon Paulsen, Daniel Kroening, Reyhaneh Jabbarvand

Source and references: https://arxiv.org/abs/2604.07341v3

Introduction

ReCodeAgent is a fully autonomous multi-agent framework for repository-level code translation and validation that works across language pairs without per-pair engineering. Prior techniques required 100K+ lines of language-specific program-analysis code for a single pair; ReCodeAgent replaces all of it with lightweight, language-agnostic tools served over the Model Context Protocol (MCP).

Key Points

Four specialized agents: Analyzer, Planning, Translator, and Validator divide the translation into distinct phases, which contains hallucination and keeps long-horizon reasoning coherent where single-agent approaches drift.
True language-agnosticism: the MCP tools provide code navigation, documentation retrieval, and structural analysis via Treesitter parsing, with zero PL-specific program-analysis components or external dependencies.
Tests translated, then independently validated: translation and validation live in separate agents, which avoids biased test generation and treats tests as context-aware artifacts of the source project.
The headline numbers: 99.4% compilation success and an 86.5% test pass rate across C-Rust, Go-Rust, Java-Python, and Python-JavaScript, a 60.8% test-pass improvement over the best prior technique.
Process evidence: trajectory analysis shows the multi-agent design improves efficiency by 28%, while a single-agent alternative drops test pass rates by 40.4%.

Methodology

The pipeline runs sequentially. The Analyzer Agent studies the source project’s architecture and identifies appropriate target-language libraries. The Planning Agent decomposes the translation into concrete sub-tasks, creates consistent name mappings, and generates a project skeleton with an implementation plan. The Translator Agent executes those tasks, converting functions and tests, and iteratively repairs errors using feedback from the Validator Agent, which independently verifies correctness through test execution and coverage analysis, triggering additional test generation when functions lack coverage.

Results and Findings

Across 118 real-world projects totaling over 230,000 lines of code and 4,583 translation units, ReCodeAgent achieved 99.4% compilation success and an 86.5% test pass rate, against 83.9% and 25.7% for the best baseline. Test translation quality held up under scrutiny: 99.3% assertion equivalence, 0.91 cosine similarity, and 94.9% assertion type match. Ablations confirm every agent earns its slot: removing the Analyzer, Planning, or Validator agent reduced test pass rates by 22.7%, 25.3%, and 30.3% respectively, while increasing trajectory complexity by 28%. Average cost per project: $15.3 and about 57 minutes.

Implications and Conclusions

The cost figure is the quiet headline. At under $20 and under an hour per project, repository translation moves from consulting-engagement territory into batch-job territory, and the language-agnostic design means adding a fifth language pair costs a config change instead of an engineering quarter. The open risk is semantic fidelity beyond tests: an 86.5% pass rate leaves 13.5% of tested behavior wrong, plus whatever the translated test suite never covered. For modernization of code that matters, human review stays in the loop; what changed is how much code one reviewer can now cover.

BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases

Authors: Mathew J. Koretsky, Maya Willey, Owen Bianchi, Chelsea X. Alvarado, Tanay Nayak, Nicole Kuznetsov, Sungwon Kim, Mike A. Nalls, Daniel Khashabi, Faraz Faghri

Source and references: https://arxiv.org/abs/2505.20321v6

Introduction

BiomedSQL is the first large-scale benchmark for scientific reasoning in text-to-SQL over biomedical knowledge bases. Existing text-to-SQL systems translate syntax well and then hit a wall on domain reasoning: knowing that “genome-wide significant” implies a p-value threshold of 5×10⁻⁸, or that answering a drug-repurposing question takes a multi-step filtering workflow across tables.

Key Points

Scale and grounding: 68,000 question/SQL/answer triples over a real BigQuery database integrating gene-disease associations, causal inference data from omics studies, and drug approval records.
Implicit conventions required: each question demands inference of unstated biomedical conventions, contextual knowledge missing from the schema, and multi-hop reasoning across relational tables.
The gap: Gemini-3-Pro reaches 58.1% execution accuracy and the custom multi-step agent BMSQL reaches 62.6%, against a 90% domain-expert baseline.
Evaluation dimensions: execution accuracy, Jaccard similarity, syntax error rates, and natural-language response quality via BioScore, with 0.89 Spearman correlation between LLM-based and expert human judgments.
Failure modes: incorrect table selection is the most common error, followed by missing or misapplied statistical thresholds. Approaches show complementary strengths, with ReAct better at table selection and BMSQL better at domain constraints.

Methodology

Construction ran in three phases: harmonizing a ten-table BigQuery database from Open Targets, ChEMBL, GWAS studies, and omicSynth; authoring 40 gold-standard SQL queries via domain-expert annotation with independent verification; and programmatically templating those queries by substituting disease, gene, and SNP mentions to generate 68,000 QA pairs with executable results. Evaluation covers Llama, Qwen, GPT, Gemini, and Claude families across prompting strategies (baseline, few-shot, domain-specific instructions) and interaction paradigms (ReAct, Schema Indexing, DAIL-SQL, BMSQL), validated against 20 independent expert-authored queries.

Results and Findings

Gemini-3-Pro leads single-turn approaches with 58.1% execution accuracy and 81.8% response quality; open-source Qwen-2.5-Coder-32B manages a respectable 40.8% at a fraction of the scale. Few-shot prompting peaks around 10 examples (+7.8% execution accuracy for GPT-o3-mini) with minimal gains beyond 40. Multi-step paradigms help modestly: BMSQL-GPT-o3-mini tops the table at 62.6% execution accuracy and 69.2% Jaccard, still roughly 30 points below experts. Extra inference-time compute through multi-pass refinement moves response quality from 83.2% to 85.5%, while expanding the schema from 10 to 20 tables costs 4.2–7.5 points of accuracy.

Implications and Conclusions

A 30-point gap to domain experts, on a task as constrained as querying a known database, should temper the current enthusiasm for LLMs as autonomous scientific analysts. The encouraging detail is that the failure modes are mundane: wrong table, missing threshold. Those look addressable with schema-aware retrieval and explicit convention libraries, which is a far easier research agenda than “teach the model biology.” Until then, treat generated SQL over biomedical data as a draft for expert review, since a syntactically perfect query that omits a significance threshold returns confident garbage.

Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

Authors: Arda Uzunoglu, Benjamin van Durme, Daniel Khashabi

Source and references: https://arxiv.org/abs/2608.12218v1

Introduction

Conventional wisdom says training with longer context windows is a free win. This work documents the bill: abundant task-relevant information in training contexts reduces a model’s ability to internalize knowledge in its parameters, leaving it dependent on context availability at inference. The authors call it the Information Abundance Paradox.

Key Points

Inverted-U performance: across pretraining experiments, models peak at intermediate context lengths (2K–8K tokens) and degrade at longer windows, under matched token budgets.
Context addiction: models trained with informative contexts perform well when context is available and deteriorate sharply when it is absent or contradictory.
Gradient shift from FFNs to attention: informative training context shifts optimization pressure away from feed-forward networks (associated with parametric knowledge) toward self-attention modules (associated with context utilization).
A theoretical account: longer contexts reduce the minimum task information that must be stored in weights to achieve equivalent training performance, opening an alternative information channel that optimization happily exploits.
Synthetic validation: controlled tasks confirm context addiction emerges selectively, exactly when longer contexts enable a lower-complexity training solution than learning the task rules parametrically.

Methodology

Pretraining experiments train models at four scales (20M to 750M parameters) on 10B tokens from Project Gutenberg, varying context windows from 512 to 32,768 tokens under matched token budgets so differences come from window effects, since data quantity is held fixed. Fine-tuning experiments use MMLU-Pro domains, varying the task-relevant information in context while holding context length fixed. Evaluation spans language modeling (LAMBADA, WikiSPAN), general understanding (SuperGLUE), and closed-book multiple-choice QA. Mechanistic analyses use gradient norm tracking, module-restricted fine-tuning interventions, and attention pattern visualization.

Results and Findings

Pretraining: inverted-U curves on SuperGLUE and MCQA benchmarks, peaking around 2K–8K windows. A 750M model drops roughly 1.5–8.2% accuracy on major benchmarks moving from 8K to 65K windows, and the pattern holds across all tested scales, so this is systematic behavior, unlikely to be a capacity artifact.

Fine-tuning: trained with eight task-relevant documents in context, Qwen3 models score 61.9–71.8% with supporting context and collapse to 22.1–43.3% without it, substantially below models trained with no contextual information at all. The gap between supporting-context and conflicting-context accuracy widens from roughly 10% to 40–50% as training context grows more informative.

Gradients: the FFN-to-self-attention gradient ratio falls consistently as context informativeness rises. Module-restricted fine-tuning makes the causal case: FFN-only updates keep 51.7% no-context accuracy, while SA-only updates manage 13.7%.

Attention: models trained with task-relevant context allocate more attention mass to context tokens at inference, concentrated in middle layers, consistent with those layers’ known role in contextual integration.

Implications and Conclusions

Results like these should change default practice. Context length has been treated as a scaling axis governed by data availability, when it also shapes whether a model stores knowledge or learns to look it up. The practical rule falls out directly: scale training context when supporting context will reliably be present at deployment, and protect parametric knowledge (through curriculum design, or by keeping training contexts deliberately less informative) when your production system must survive missing or contradictory inputs. The gradient analysis also hands mechanistic-interpretability researchers a clean, causally validated example of optimization pressure migrating between module types, which may prove more durable than the headline result.

Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages

Authors: Avijit Roy, Proma Roy

Source and references: https://arxiv.org/abs/2608.12278v1

Introduction

Bengali has roughly 285 million speakers and next to no AI infrastructure. The reasons are structural, and this analysis traces them across training data, tokenization, and deployment architecture, each of which disadvantages Bengali speakers before any model is even trained.

Key Points

Web presence gap: Bengali speakers are about 4% of the global population and account for under 0.5% of web content. English sits at about 49.5% with a comparable speaker population.
Training token deficit: major multilingual corpora carry a 67:1 English-to-Bengali token ratio, roughly 2 trillion tokens versus 30 billion, and the performance gap on Bengali tasks shows up across model families.
Tokenization penalty: Bengali’s alphasyllabary script, with diacritics and conjunct forms, produces much higher token fertility than Latin scripts under BPE and WordPiece. Equivalent semantic content costs more tokens, compounding the data disadvantage.
Connectivity exclusion: cloud-dependent AI tools assume reliable internet that most Bengali learners lack. Rural Bangladesh shows 36.5% individual internet penetration versus 71.4% urban, 9.2% of households own computers, and mobile data duty rose from 3% to 23% in eight years.
Cognitive load consequences: Bengali-speaking learners using English-language AI tutors carry a dual burden, processing a foreign language and technical content at once, which measurably reduces learning outcomes versus native-language instruction.

Methodology

The paper is an analytic synthesis, consolidating published benchmarks, infrastructure indicators, and established educational theory to explain systematic language inequity in AI. It traces four interlocking structural failures across data availability, technical processing, and deployment architecture, then connects them to documented educational outcomes using Cognitive Load Theory and comparative studies of multilingual education.

Results and Findings

The disadvantage operates at every architectural level at once. The BenLLM-Eval benchmark shows consistent performance gaps between general-purpose models on Bengali versus English tasks. Academic content delivered in a foreign language reduces both content learning and language learning (Roussel et al., 2017), with programming education showing the same pattern. The connectivity data hides in the aggregates: household-level statistics show about 50% internet access, while individual computer use sits at 9%. Learners with native-language instruction and local connectivity achieve better debugging comprehension and conceptual retention than those relying on English-language cloud systems.

Implications and Conclusions

The paper’s sharpest move is reframing dataset scarcity as an allocation decision. “Low-resource” language describes a choice made by institutions, and the argument that infrastructure work on such languages deserves primary research credit, on par with modeling contributions, is one program committees should sit with. The offline-first recommendation, local inference on quantized models sized to the devices people actually own, doubles as an equity strategy and sound engineering. The harder question the paper leaves open is who pays: tokenizer redesign and corpus building for 285 million speakers is exactly the kind of public-good work current incentive structures fund worst.

A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

Authors: Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora, Vishal Singh

Source and references: https://arxiv.org/abs/2608.12138v1

Introduction

VITA is a retrieval-augmented clinical decision support system purpose-built for India’s healthcare context, evaluated here against frontier LLMs on the HealthBench clinical reasoning benchmark. The result cuts against the assumption that general-purpose frontier models have made specialized clinical systems obsolete.

Key Points

The headline: VITA scored 51.9% on 4,023 English-language HealthBench questions, ahead of GPT-5.4 (46.1%), o4-mini (44.3%), Gemini 3.1 Pro (42.6%), and Claude Sonnet 4.6 (37.3%).
Win distribution: VITA produced the highest-scoring response on 45.4% of questions, 2.6× the next-best system, with its edge concentrated in clinical accuracy (55.9% vs. 49.5% for GPT-5.4), completeness (51.8% vs. 42.6%), and context awareness (50.3% vs. 45.1%).
The sensitivity check: under a neutral open-weight judge (DeepSeek-V4-Pro) and current-generation opponents, VITA landed statistically indistinguishable from GPT-5.5 on mean per-question scores, while keeping higher points-weighted scores and more question wins.
Where the edge comes from: a curated corpus of disease-specific guidelines, India-specific antimicrobial resistance data, national formulary constraints, and resource-limited care protocols, which pays off most on low- and middle-income country scenarios.
Where the frontier models win: communication quality, where GPT-5.4 scored 56.3% against VITA’s 45.8%, a dimension calibrated to Western communication norms.

Methodology

All systems received identical prompts on 4,023 English-language HealthBench questions (80.5% of the full benchmark), scored with physician-written rubrics across accuracy, completeness, context awareness, communication quality, and instruction following. GPT-4.1 served as the primary judge. Co-authors without VITA equity interests ran the sensitivity analysis: a 500-question subset scored by DeepSeek-V4-Pro, an open-weight judge with no lineage to any tested system, against current-generation models including GPT-5.5 and Claude Opus 4.8.

Results and Findings

In the full evaluation, VITA led by 5.8 percentage points over GPT-5.4, stable across batches (50.7–52.9%). The per-axis decomposition placed its advantages squarely in the clinical dimensions: +6.4 points on accuracy, +9.2 on completeness, +5.2 on context awareness.

The sensitivity analysis tells a more careful story. Under the neutral judge, the aggregate advantage narrowed to statistical parity with GPT-5.5 (51.0% vs. 52.0% mean per-question, overlapping CIs). VITA kept its lead on points-weighted score (49.1% vs. 48.3%) and questions won (109 vs. 80), and its accuracy (59.1%) and completeness (48.9%) advantages persisted, while the context-awareness edge faded.

Implications and Conclusions

The sensitivity analysis is the paper’s most honest moment and its most useful one: a headline win over last-generation models, evaluated by a judge from the same ecosystem, shrank to parity with the current generation under a neutral judge. What survived that shrinkage, accuracy and completeness on LMIC-specific scenarios, is the defensible claim, and it’s a meaningful one: curated domain corpora still beat raw model scale where the deployment context diverges from the training distribution. The communication-quality gap deserves scrutiny in the other direction, since a benchmark that rewards Western communication norms will systematically undervalue systems calibrated for the clinics they actually serve.

Learning-Based Behavior Planning for Automated Driving: Real-World Integration and Deployment

Authors: Jean-Pierre Busch, Guido Linden, Jan Bergmann, Lutz Eckstein

Source and references: https://arxiv.org/abs/2608.12198v1

Introduction

A hybrid planning architecture for automated vehicles: a deep-learning behavior planner produces trajectories, and an optimization-based supervision layer enforces drivability, safety, and traffic compliance before anything reaches the actuators. Learned components capture complex traffic interactions; the deterministic layer keeps the system verifiable. The whole stack ran on a real research vehicle in urban traffic.

Key Points

Hybrid by construction: the neural behavior planner is always downstream-checked by an optimization layer solving a constrained trajectory problem, plus a deterministic safety fallback.
Modular deployment: safety fallback, trajectory supervision, and trajectory controller ship as containerized ROS 2 nodes, so the learned planner can be updated through MLOps workflows without touching safety-critical components.
Vectorized scene representation: the planner consumes object-level scene encodings (agent histories, HD map, navigation data) rather than raw sensor data, which shrinks domain gaps between datasets and eases transfer.
Multi-task learning: trajectory prediction is trained jointly with object prediction and map-grounding auxiliary tasks, improving representation quality and transfer to new domains.
Real-world integration: the system was deployed and tested on the research vehicle karl under realistic sensor and actuation constraints.

Methodology

A PyTorch network was trained by imitation learning on the DrivIng dataset, about 105 minutes of urban driving captured at different times, predicting 8-second reference trajectories from vectorized scene encodings with attention-based fusion. The trajectory supervision module refines the network’s output by solving a nonlinear optimal control problem in acados, enforcing kinematic, dynamic, and safety constraints. Three model variants probe generalization: DrivIng only, DrivIng plus test-track geometry, and DrivIng plus interaction scenarios. Open-loop evaluation measured trajectory accuracy and collision rates on validation splits.

Results and Findings

On DrivIng validation, the model reached a position ADE of 1.83m over 8 seconds, velocity ADE of 0.56 m/s, and heading ADE of 2.6 degrees. It never proposed crossing a red light, and held a 0.3% collision rate over 4-second horizons, the practically relevant window given continuous replanning. Zero-shot transfer to unseen test-track geometry degraded badly (heading ADE 2.6° to 11°), and adding test-track training data pulled the 8-second collision rate in interaction scenarios from 26.7% down to 5.9%. Small amounts of diverse data also improved the original domain, cutting DrivIng validation collision rates from 2.3% to 1.2%, with no catastrophic forgetting observed. Counterfactual tests showed behavior adapting appropriately to changed traffic-light states and introduced obstacles, which suggests the model captured genuine causal structure in the scene.

Implications and Conclusions

The architecture answers the standard objection to learned planners, that you can’t certify a neural network, by never asking anyone to: certification burden sits on the deterministic supervision layer, and the network is free to improve continuously behind it. The transfer results carry the practical lesson, since zero-shot failure followed by cheap recovery with small targeted datasets suggests fleet operators should budget for continuous domain-specific data collection as an operating cost. What open-loop evaluation can’t answer is how the planner-supervisor pair behaves in closed loop over hours of real traffic, and that is the result to demand next.

How Organizations Use AI: Evidence from ChatGPT

Authors: Aaron Chatterji, David Holtz, Neel Rakholia, Prasanna Tambe, Gawesha Weeratunga

Source and references: https://arxiv.org/abs/2608.12236v1

Introduction

Usage data from 1,500+ ChatGPT Enterprise organizations and 17+ million messages through March 2026, linked to employee job titles, task classifications, and firm financials. The picture of enterprise AI adoption it draws is heterogeneous on every axis: which firms adopt, how intensively workers engage, and which tasks the technology actually lands on.

Key Points

Rapid growth, mostly from deepening: output tokens grew sevenfold between June 2025 and March 2026, with roughly half of that growth coming from existing adopters using it more.
Adoption concentrates among scale leaders: adopters are larger, more valuable, and more R&D- and SG&A-intensive than non-adopters.
The seniority inversion: early-career workers send 8–9 times more messages than executives, even though managers make up a larger share of active users.
60+ distinct work tasks: technical development, communication, research, data analysis, and financial work all register, with documentation and technical writing the most prevalent.
General-purpose behavior: usage spreads across organizational hierarchies and functional domains rather than concentrating in single workflows.

Methodology

Four linked datasets built from ChatGPT Enterprise account records, January 2024 through March 2026: anonymized usage telemetry (messages, active users, output tokens), administrative metadata (job titles, industry classifications), automated message-level task categorization under an internal taxonomy, and, for public companies, a curated account-to-ticker mapping into Compustat financials. Analyses were de-identified and reported in aggregate, with no manual review of individual messages.

Results and Findings

Growth: aggregate output tokens grew sevenfold in nine months. Fixed cohorts of pre-June-2025 adopters grew fourfold over the same period, so intensity is deepening inside firms alongside new-customer acquisition. An early-2026 acceleration hit all cohorts simultaneously, pointing to platform-wide changes rather than standard post-adoption ramp curves.

Firm characteristics: among U.S. public companies, adopters show median revenue of $2.3 billion versus $210 million for non-adopters, median market value of $5.0 billion versus $316 million, and median R&D spend of $113 million versus $9.9 million. A one-log-point increase in lagged revenue per employee associates with 0.4–0.9 percentage points higher adoption probability. Top-quartile revenue firms are 6.9 points more likely to adopt; the top 5% are 9.8 points more likely, and the concentration holds within industries.

The intensity paradox: larger firms adopt more but use less per employee conditional on adoption. High-intensity adopters show higher revenue and market value per employee than low-intensity adopters and non-adopters.

Intangibles: SG&A stock per employee shows the strongest association with adoption (coefficient 0.020), with R&D stock (0.004) and capitalized software (0.008) also positive, consistent with complementarity between AI adoption and accumulated organizational capabilities.

Workers and tasks: six months post-adoption, engineering and technical workers are 11% of active users and executives 9%, with no single occupation dominating. Over 50% of active users do documentation and technical writing; about 45% do technical digital work. Task profiles vary by industry and function while the core tasks stay consistent across groups.

Implications and Conclusions

The data supports reading enterprise AI adoption as an extended organizational learning process: usage breadth and intensity keep expanding months after the contract is signed, which means firms are still discovering where the technology fits. The uncomfortable implication sits in the firm characteristics. If adoption complements existing intangible capital, and scale leaders adopt first and deepest, generative AI may widen productivity gaps between large and small firms before it narrows them. One caveat the authors can’t escape: this is OpenAI’s own data about OpenAI’s own product, covering adopters only, so the firms struggling to find value are visible here only by their absence.

Metis: LLM Memory Without the Cumbersome Retrieval Stack

State of AI — Mon, 10 Aug 2026 22:04:09 GMT

👋 Welcome to this month’s edition of State of AI: Monthly Paper Deep Dive.

Before we get into the paper: today’s edition is sponsored by Oumi, and this is one I’d actually tell you to join. They’re tackling the exact problem we keep writing about, models that stop learning the moment they ship. Grab a seat!

Your AI stopped learning the day it shipped.

A general-purpose model doesn’t learn from your production failures, operator corrections, or the work that makes your business different. On August 11, Oumi is launching the AI Factory that runs the complete loop: Evaluate, Synthesize, Train, Deploy, Compound. We’ll show it live on a real task and reveal three product announcements. Join Manos Koukoumidis and the Oumi team at 10:00 AM PT / 1:00 PM ET.

Reserve Your Seat!

Each month, we break down one standout AI research paper, explaining it clearly and concisely for ML engineers and research scientists. Today’s focus is Metis, what its authors call the first prototype of a “memory foundation model”: one that builds remembering, forgetting, and updating directly into the model’s weights instead of bolting on an external memory system.

🔍 Introduction

Modern LLMs are fundamentally stateless. Once a conversation slips out of the context window, the model has no idea it ever happened. The industry’s answer has been external memory: RAG pipelines, vector databases, memory managers like MemoryBank or MemGPT that store text outside the model and stuff relevant snippets back into the prompt.

But this paper, from MemTensor together with Renmin University, NUS, Shanghai Jiao Tong, and Tongji, asks:

What if memory weren’t a module wrapped around the model, but a native capability inside it, the way reasoning became native with large reasoning models?

The authors point out three structural problems with external memory. The memory system and the backbone are decoupled into separate components with separate objectives, so the retriever doesn’t know what the model actually needs and the model can’t optimize how its memory is managed. Gradients are blocked: retrieval, reranking, and prompt concatenation are discrete, non-differentiable operations, so the memory loop can never be trained end-to-end. And every query pays a latency tax (embedding, retrieval, reranking, and prefilling the retrieved text before generation even starts) at a cost that grows with history length.

Their answer is the memory foundation model: a model whose parameters include a dynamic portion that acts as a persistent memory state, and whose forward pass is the memory system. They call their prototype Metis.

🧭 External vs. Native Memory

The paper formalizes native memory from two angles:

Native memory state. A portion of the model’s parameters is dynamic: it evolves across interaction steps. At step t, the model generates from parameters θₜ, and the forward pass itself transforms them into θₜ₊₁. Information from past interactions lives in these dynamic parameters, not in a text buffer.

Native memory procedure. Operations like remember, forget, and update are not rule-based pipeline stages. They are learned behaviors executed inside the forward computation, triggered by the semantics of the input (”Please forget that Alice likes burgers”) rather than by hand-written logic.

Two properties make this practical: online memory maintenance is gradient-free (a memory update is just a forward pass, no backprop at inference time), and all learned weights stay frozen at inference; only the designated memory state changes.

🔧 The Solution: The Metis Block

Metis augments every Transformer layer with a Metis block, inspired by fast weights (Schmidhuber, 1991; Ba et al., 2016). Each block has two halves:

Local Memory Block (the “fast” state). A dense memory matrix M ∈ ℝ^(dₖ×dᵥ) plus a query-key normalization vector S ∈ ℝ^(dₖ). These are the dynamic parameters: initialized to zero, updated at every interaction step, and carried forward across the whole conversation.

Hyper Memory Block (the “slow” controller). Static parameters learned during mid-training: a learnable importance vector for deciding what to store, and dedicated key/value/query projection matrices that map hidden states into the memory’s semantic space. These stay frozen at inference and define how memory is written and read.

🧠 Key Mechanism #1: Adaptive Aggregation (writing)

After each step, the importance vector scores every token’s hidden state, and a top-ρ selection keeps only the most salient positions (a straight-through estimator keeps the scorer trainable). The selected states are projected into memory keys and values and folded into M with a discount factor λ that balances new information against old. In practice, Metis replaces the plain linear update with a Gated Delta Network-based update (GDU), which proved more stable in long-term scenarios.

🧠 Key Mechanism #2: Memory Attention (reading)

At generation time, the model computes a dedicated memory query from its current hidden state, reads from M and S, and blends the memory readout with ordinary causal self-attention via a balance parameter γ. The paper’s theoretical analysis shows this is equivalent to attending over a “virtual memory prefix”: as if the compressed history were prepended to the input, but without ever paying its token cost.

There’s a systems payoff here: the original attention, the memory read, and the memory write have no sequential dependency, so all three run in parallel within a layer. And because the memory state is fixed-size, memory cost does not grow with conversation length: no retrieval, concatenation, or prefilling overhead.

📚 You Can’t Learn Memory Without Memory Data

Foundation models don’t spontaneously learn “selective forgetting” from web text, so the team synthesized a dedicated mid-training corpus from 27 public benchmarks (LoCoMo, RULER, TOFU, ZsRE, MuSiQue, and more):

Primary data (357K samples, ~406M tokens) covers four operations (remember, update, forget, reflect) with varying instruction salience (explicit commands vs. facts embedded in natural narrative) and injected distractor turns for noise robustness.
Auxiliary data (609K samples) targets the failure modes of parametric memory: multi-entity binding (don’t confuse Alice’s age with Bob’s), selective forgetting (revoke one fact without collateral loss), and memory pollution (if the model knows “Alice likes burgers,” it shouldn’t mention burgers in an unrelated math question).

Training uses three objectives: a reconstruction loss (stored content must be recoverable), an operation loss (the memory state must reflect semantic commands like “forget”), and a regularization loss (suppress interference and leakage). One detail worth pausing on: the Qwen3.5 backbone is completely frozen during mid-training; only the memory parameters are optimized.

📊 What Did They Find?

The headline evaluation is the no-context setting: the model sees the history once, then must answer later questions with the original context removed, so memory has to do all the work.

✅ Native memory works where prompting can’t: On memory-based QA without context, plain Qwen3.5 collapses to near zero on LoCoMo (Gold) (~0.1 avg), while Metis-27B scores 26.7. On NextMem, Metis-27B reaches 50.8 vs. 17.8 for its backbone and ~31 for the strongest Temp-LoRA baseline.

✅ Clear wins over parametric-memory baselines: On MemOps memory-operation tasks (no context), Metis-27B averages 24.8 vs. 9.7 for Temp-LoRA-27B and 4.4 for δ-Mem. On the paper’s own test set, it’s 73.8 vs. 23.9 and 15.0.

✅ Memory capability scales with the backbone: Metis-27B substantially outperforms the 4B and 9B variants, especially on multi-hop and temporal questions, evidence that larger backbones formulate and exploit the memory state better.

✅ The architecture choices matter: Ablations (run on the 4B variant) show removing adaptive aggregation is catastrophic (-61% overall), removing query-key normalization costs -28%, and removing the dedicated memory query costs -12%. GDU vs. a linear update is roughly a wash on short-term tasks but clearly better on long-term LoCoMo.

✅ The memory state is highly compressible: SVD analysis shows rank-64 approximations of the (1024-dimensional) memory state recover 99.9% of full performance; useful information concentrates in a low-dimensional subspace, which is promising for storage and multi-user serving.

⚠️ The limitations, quantified: Step-level capacity degrades once a single update exceeds a few hundred words; repeated updates accumulate interference across a trajectory; and once irrelevant information is stored, general capabilities dip; strict instruction following takes the biggest hit, with IFEval falling 22 points (Metis-4B vs. its backbone) in the active-memory setting. The case studies surface a telling quirk, too: after a forget instruction, the memory state is correctly wiped, but the model’s immediate reply still parrots the old fact before “realizing” it’s gone in later turns.

💡 Why This Matters

For research scientists and ML engineers:

Memory joins the “internalization” trend. Multimodality moved into the backbone; reasoning moved into the backbone (CoT → large reasoning models). This paper makes a credible case that memory is next, and shows what the architecture and training recipe could look like.
End-to-end optimization becomes possible. Once storage and retrieval are continuous operations inside the forward pass, memory behavior can be shaped by gradient descent and domain-specific post-training, rather than by brittle retrieval heuristics.
A fixed-size state changes the serving math. User memory becomes a compact, low-rank-compressible parameter block rather than an ever-growing KV-cache or vector store, and memory operations run in parallel with attention instead of as a sequential pre-stage.
Not a RAG killer (yet). The authors are explicit: fixed-size latent memory loses information in extremely long-term scenarios and can blend similar facts. They position native memory as complementary to external memory, with hybrid systems as future work.

Our take: two questions the paper leaves open. Every headline comparison is against parametric-memory baselines under the no-context protocol; the practical question, “when does this beat a well-tuned RAG stack on cost and accuracy?”, is never tested head-to-head (the partial-context RAG rows use a simple top-5 cosine retriever, not a production-grade pipeline). And the serving argument would land harder with one number the paper never foregrounds: how many parameters the Metis blocks actually add, i.e., what a per-user memory state costs to store and swap in practice.

🔮 The Big Picture

The authors sketch a five-level roadmap for memory foundation models: from merely stateful (Level 1), to self-managing the full memory lifecycle (Level 2), to experience-learning (Level 3), persistent internal models of the world and user (Level 4), and ultimately self-evolving systems (Level 5). Metis sits at the base of that ladder, and its own capacity studies show how much climbing remains: a few hundred words per update before recall degrades, interference that compounds over long trajectories. What it establishes is the existence proof: a frozen LLM can be taught, purely through mid-training on synthetic memory data, to remember, update, and forget inside its own forward pass. The next levels are now an optimization problem with a published baseline, open code, and released checkpoints.

📖 Read the full paper here: Metis: Memory Foundation Model · Code · Checkpoints

Graph-Based Agentic AI, Adaptive Speculative Decoding, and Physics-Based Digital Twins

State of AI — Thu, 06 Aug 2026 05:51:28 GMT

This edition showcases a fascinating convergence of three major trends reshaping AI systems: the infrastructure and governance challenges of building reliable, auditable agentic workflows; the practical engineering required to make inference faster and more cost-efficient at scale; and the mechanics of grounding AI capabilities in the physical world through simulation and embodied reasoning. We‘re also seeing exciting progress in multimodal reasoning—from theory-of-mind in meetings to adaptive navigation—alongside deep dives into mechanistic interpretability, geometric deep learning, and novel optimization techniques for reasoning-heavy tasks.

Here‘s what caught our attention:

Graph-Based Agentic AI with LangGraph — A practitioner’s guide clarifying when stateful workflow orchestration is justified, complete with decision tables comparing it to simpler alternatives and three executable recipes for SQL repair, RAG, and human-in-the-loop processes.
CodeRescue: Budget-Calibrated Recovery Routing — Rather than defaulting to expensive models after failures, this work routes coding agents through three recovery actions (reflect, replan, escalate) using a learned policy calibrated to arbitrary budget constraints without retraining.
OmniReasoner: Thinking with Long Audio-Video — Teaches omnimodal models to strategically zoom into relevant video intervals using absolute timestamps and tool-use learning, with gains scaling dramatically on 10–30 minute videos.
Off-Context GRPO: Learning from Privileged Guidance — Solves the “learning cliff” where hard problems produce zero reward signal by correcting the distribution mismatch between training (guided) and deployment (unguided) conditions through importance weighting.
ARMOR: Stabilizing On-Policy LLM RL — Addresses over-optimization in reasoning RL by actively injecting off-policy correct samples as anchors rather than relying on passive KL penalties, achieving +5–11 point improvements on AIME.
Agentic Real2Sim: Physics-Based Digital Twin Creation — Automates real robot video-to-simulation conversion using vision-language agents for orchestration, working across rigid, deformable, and humanoid domains at <$3 per episode with open-weight models.
CircuitKIT: Mechanistic Interpretability Toolkit — Unifies circuit discovery, evaluation, and intervention through thirteen algorithms and six complementary diagnostic pillars, revealing that single-metric faithfulness scores reverse method rankings and fail to predict intervention outcomes.
MeetingToM: Theory-of-Mind in Multi-Party Meetings — A benchmark targeting pseudo-consensus—where participants verbally agree despite nonverbal dissent—with substantial gaps between human reasoning (75–86%) and state-of-the-art models (22–88%).

Let‘s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Graph-Based Agentic AI with LangGraph: Workflow Pathways for Long-Running Stateful Business Processes

Authors: Daniel Pearson, Sidney Shapiro, Emiliano Sebastian Gonzalez Venegas, Sanad Al-Khatib, Aurora Pinzón Arzola

Source and references: https://arxiv.org/abs/2607.19297v1

Introduction

This paper is a practitioner guide to LangGraph, a low-level orchestration framework for building long-running, stateful multi-step AI systems in business processes. Rather than positioning LangGraph as a universal solution, the authors present it as a specialized tool suited to specific workflow complexity requirements, with three executable recipes demonstrating when and how to use it effectively.

Key Points

LangGraph is a workflow-complexity fit, not a universal default: The framework excels for processes requiring durable state, human review gates, and audit trails, but simpler alternatives (plain SDK, schema-first tools, DSPy) are better for basic tool use, structured extraction, or prompt optimization.
Three core recipes address recurring business patterns: SQL analytics with repair loops, agentic retrieval-augmented generation (RAG) with evidence gating, and human-in-the-loop (HITL) policy review with interrupt and checkpoint recovery demonstrate how typed state, conditional routing, and deterministic tools work together.
Explicit governance and durability are first-class features: LangGraph makes workflow structure, route history, and state persistence transparent in the product contract rather than hidden in prompt logic, enabling inspection, debugging, and compliance auditing.
Decision criteria determine when LangGraph justifies the overhead: Use LangGraph when workflows require pause-and-resume capability, next steps depend on explicit state branches, failures need repair paths, teams need route auditing, or multiple tool calls must share durable state.
Control plane primitives replace informal application logic: Typed shared state, graph nodes and edges, conditional routing, durable checkpoints, and interrupt/resume capabilities turn multi-step LLM applications into inspectable process graphs.

Methodology

The paper provides a decision framework rather than empirical benchmarks. The authors organize workflow requirements against LangGraph mechanisms and simpler alternatives through decision tables and comparative analysis. They present three concrete executable recipes that illustrate common business-process patterns: SQL repair workflows, evidence-aware RAG systems, and checkpointed human-review processes. Code patterns are explained to show how typed state, node boundaries, conditional routing, retries, interrupts, and checkpoints integrate into cohesive systems.

Results and Findings

The paper delivers a structured decision guide (Table 1) mapping six workflow requirements to LangGraph mechanisms and simpler alternatives. Key findings include: LangGraph is justified when processes must pause and resume (checkpointers and interrupts preserve state), when next steps depend on explicit state like risk level or validation status (conditional edges make routes inspectable), when failures require repair paths (retry budgets and error nodes enable controlled recovery), when auditability is required (node boundaries create traceable decision records), and when multiple model/tool calls coordinate through shared state (typed state objects keep artifacts explicit across nodes). The comparative analysis (Table 2) shows that plain ReAct-style loops suffice for simple tool use without checkpoints, schema-first systems excel at structured extraction and validation, and DSPy better addresses prompt optimization problems.

Implications and Conclusions

This research provides practical guidance for engineering teams building production AI systems by clarifying when infrastructure complexity is justified by operational requirements rather than treated as a default choice. The framework‘s emphasis on making workflow structure, governance, and audit trails explicit product behavior rather than hidden implementation details has significant implications for building reliable, maintainable, and compliant AI systems in regulated business environments.

CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

Authors: Qijia He, Jiayi Cheng, Chenqian Le, Rui Wang, Xunmei Liu, Yixian Chen, Jie Mei, Zhihao Wang, Xupeng Chen, Yuhuan Chen, Tao Wang

Source and references: https://arxiv.org/abs/2607.19338v1

Introduction

CodeRescue addresses a critical inefficiency in cost-aware LLM deployment for coding agents: when a cheap model fails to solve a programming problem, the system must decide not just which model to use next, but which recovery action to take. Rather than defaulting to an expensive stronger model, the paper introduces budget-calibrated routing that intelligently chooses between three recovery actions—reflecting on feedback, replanning from scratch, or escalating to a stronger model—based on execution diagnostics.

Key Points

Post-failure recovery routing: Coding failures produce actionable execution feedback (error messages, failed tests, stderr traces). The paper formulates recovery as routing over three distinct actions—reflect (repair using feedback), replan (fresh attempt with cheap model), and escalate (defer to stronger model)—rather than a binary cheap-vs-strong decision.
Complementary success patterns: Empirical analysis across five benchmarks reveals that cheap recovery and escalation are not strictly ordered. Some failures can only be solved by expensive escalation (45%), some only by cheap recovery (28%), and some can be solved by either (27%), meaning no single fixed action is optimal across all instances.
Budget-controllable deployment via Conformal Risk Control (CRC): A single trained router is converted into a family of budgeted operating points by adding a cost-penalty hyperparameter λ. CRC calibration maps user budgets to appropriate λ values without retraining, providing marginal expected-cost control under exchangeability.
Learned router outperforms baselines: A supervised router trained on recovery rollouts significantly outperforms fixed policies (always-escalate, always-replan) and zero-shot prompt-based routers, achieving 81.7% solve rate versus 68.6% for always-escalate at lower cost.
Practical cost-quality frontier: At a medium budget of 2.56 m$ per example, the CRC frontier reaches 71.7% solve rate, exceeding a binary cascade baseline and approaching always-escalate’s 68.6% solve rate while using only 35% of its cost.

Methodology

The paper collects recovery rollouts from five coding benchmarks (APPS, TACO, BigCodeBench, LiveCodeBench, CodeContests) by executing cheap-model attempts and recording which recovery actions successfully solve each failure. A supervised recovery router is trained on ~4,656 examples by fine-tuning Qwen3.5-4B to score the three recovery actions, labeled with the cheapest successful action for each instance. The router outputs normalized action probabilities via softmax over token log-likelihoods. To enable budget control, a cost-regularized policy π_λ(x) = argmax_a {s_θ(a|x) − λc(a,x)} shifts action selection toward cheaper options as λ increases. CRC calibration selects an appropriate λ on a held-out calibration set to satisfy a user-specified mean-cost budget B, then deploys this fixed λ on a disjoint test set.

Results and Findings

On held-out GPT-5.4-NANO/GPT-5.4 test splits, the unconstrained learned router achieves 81.7% solve rate at 5.51 m$ mean cost per example, substantially outperforming always-escalate (68.6% at 7.22 m$) and always-replan (45.3% at 1.59 m$). The CRC-calibrated frontier provides discrete operating points: at tight budgets (~1.43 m$), solve rate reaches 48.6%; at medium budgets (~2.56 m$), it reaches 71.7%; at loose budgets (~5.51 m$), it reaches 81.7%. Benchmark-level analysis shows substantial variance in recovery patterns—BigCodeBench is dominated by cheap-only recoveries (78% of failures), while TACO (very hard) shows escalation-only recoveries in 35% of cases. Router ablations confirm that metadata prefixes (source, difficulty tags) improve performance from 0.656 to 0.697 solve rate, and full fine-tuning outperforms LoRA. Cross-model validation with Gemini-2.5-FLASH/Pro pairs confirms the approach generalizes beyond GPT models.

Implications and Conclusions

This work demonstrates that executable coding environments enable a richer post-failure decision space than binary model cascades. By recognizing that cheap recovery and escalation can be complementary rather than strictly ordered, and by providing deployment-time budget control without retraining, CodeRescue offers practical cost savings for production coding agents. The research suggests that future work should extend recovery routing to longer action sequences, incorporate richer execution traces, and explore how recovery patterns transfer across different model pairs and benchmarks.

Learning to Make Friends: Coaching LLM Agents toward Emergent Social Ties

Authors: Philipp J. Schneider, Lin Tian, Marian-Andrei Rizoiu

Source and references: https://arxiv.org/abs/2510.19299v2

Learning to Make Friends: Coaching LLM Agents toward Emergent Social Ties

Introduction

This paper investigates whether large language model (LLM) agents can replicate complex social dynamics observed in human online communities and what mechanisms enable authentic social behavior to emerge. The researchers present a multi-agent LLM simulation framework where agents repeatedly interact, evaluate each other, and adapt behavior through in-context learning guided by reward signals.

Key Points

Multi-Agent Simulation Framework: Introduces a platform supporting both public and private communication channels that enables agents to develop distinct strategies across different contexts and systematically study how channel choice shapes conversational dynamics.
Behaviorally Grounded Reward Functions: Formalizes reward structures capturing empirically observed user motivations—social interaction, information seeking, self-presentation, coordination, and emotional support—creating a principled bridge between behavioral theory and agent objectives.
Endogenous Tie Formation: Implements mechanisms allowing social ties to emerge organically from conversational interactions without relying on pre-defined network structures, enabling systematic investigation of how support, alignment, and homophily drive group formation.
Psychologically Coherent Personas: Develops agent personas through three-layer architecture incorporating Big Five personality traits, task-based motivations derived from gratification theory, and multi-component memory structures (conversation, relationship, and opinion memory) that enable path-dependent behavior.
Emergent Network Structures: Demonstrates that coached LLM agents develop stable interaction patterns that yield network structures mirroring properties of real online communities, including clustering, modularity, and sustained tie persistence.

Methodology

The framework comprises two main components: persona creation and social media simulation. Agents are initialized with content-grounded personas extracted from real-world online discussions, incorporating personality traits assessed through the Mini-IPIP inventory, task assignments reflecting user motivations, and lightweight memory structures tracking interactions and beliefs. The simulation operates through iterative rounds where each agent follows a plan-execute-reflect cycle, selecting from actions including posting, commenting, direct messaging, or remaining inactive. After each round, agents cast votes on publicly visible content and reweight relationship strengths using behavioral signals. Critically, the framework incorporates optional “coaching“—external guidance helping agents make strategic decisions—and defines specific reward functions that incentivize behaviors aligned with observed human motivations, enabling in-context learning without explicit reward training.

Results and Findings

The paper demonstrates that LLM agents equipped with behavioral rewards and coaching mechanisms develop stable interaction patterns that reproduce key properties of real social networks. The experiments reveal agents form emergent social ties through micro-level signals including approval, reciprocity, and response latency, which aggregate into macro-level structures exhibiting clustering, modularity, and tie persistence. The framework successfully captures diverse engagement levels—from passive observers to active creators—and agents exhibit context-dependent communication strategies, deploying different approaches across public versus private channels. Network structures that emerge exhibit homophily and community formation patterns similar to observed online communities, validating the framework‘s capacity to approximate human-like social behavior.

Implications and Conclusions

This research provides a rigorous, principled testbed for investigating collective dynamics in LLM populations and offers policymakers, researchers, and security analysts a tool for safely studying online social phenomena without compromising individual privacy. By demonstrating that LLM agents can authentically reproduce complex social dynamics through behavioral rewards and adaptive learning, the work advances the feasibility of creating digital twins of social media ecosystems for testing moderation strategies, studying opinion formation, and detecting coordinated influence operations.

OmniReasoner: Thinking with Long Audio-Video via Native Tool Use

Authors: Yu Chen, Caorui Li, Ziyu Xiong, Yidong Wang, Mingqi Gao, Shuman Liu, Biao Liu, Chunfeng Yang, Anxiang Zeng, Haibo Zhang, Chaofan Chen

Source and references: https://arxiv.org/abs/2607.19339v1

Introduction

OmniReasoner introduces a tool-use framework that enables omnimodal language models to reason more effectively over long audio-video content by learning when and where to request higher-fidelity evidence. Rather than processing entire videos uniformly, the model learns to strategically zoom into relevant temporal intervals—a capability grounded in absolute timestamps that remain consistent across changing sampling granularities.

Key Points

Adaptive Tool-Use Framework: OmniReasoner teaches models to make three-part decisions: whether to answer from a global low-fidelity preview, when to invoke a zoom-in tool, and where on the timeline that tool should inspect, using supervised fine-tuning followed by reinforcement learning.
TimeAnchor Mechanism: Introduces plain-text temporal markers that bind audio-video tokens to wall-clock time, enabling the model’s zoom-in tool arguments (e.g., “inspect seconds 32–38”) to remain valid when transitioning from sparse global observations to dense local clips, solving a critical grounding problem unique to multi-granularity audio-video processing.
Temporal Augmented Data Engine: Synthetically generates training data through video editing operations (multi-segment composition and anomaly insertion) that automatically label evidence intervals, eliminating the need for expensive manual annotation while providing coupled supervision for answers, intervals, and tool-use trajectories.
Significant Performance Gains on Long-Form Content: Achieves improvements of 5.5 points on OmniVideoBench and 3.4 points on LVOmniBench over the Qwen2.5-Omni-7B baseline, with gains scaling dramatically with video duration—9.9 points on 10–30 minute videos versus 3.2 points on 0–5 minute videos.
Verified Evidence Integration: Empirical validation confirms that retrieved zoom-in clips meaningfully contribute to final answers rather than serving as decorative reasoning traces, demonstrated through performance drops when clips are removed and through attention rollout analysis.

Methodology

OmniReasoner operates in two reasoning stages: first, the model constructs a global observation by consuming the full audio-video stream at low fidelity, then either answers directly or requests a zoom-in interval using TimeAnchor‘s absolute-time format. When zoom is invoked, a media sandbox materializes a higher-fidelity local clip for the requested interval, and the model generates its final answer conditioned on both global context and local evidence. The framework is trained with supervised fine-tuning on 25,839 curated examples, followed by reinforcement learning using Group Relative Policy Optimization (GRPO) on 2,731 difficulty-aware examples. The data comes from two sources: automatically constructed long audio-video tasks via the Temporal Augmented Data Engine (combining multi-segment composition and anomaly insertion), and auxiliary omnimodal datasets. A custom omnimodal tool-use RL implementation built on TRL optimizes the policy using combined accuracy and format rewards without isolating a separate localization objective.

Results and Findings

OmniReasoner demonstrates consistent improvements across multiple benchmarks. On audio-visual benchmarks, it improves OmniVideoBench from 29.3% to 34.8%, LVOmniBench from 32.0% to 35.4%, and achieves notably large gains on VideoHolmes (24.4% to 40.0%). On general video reasoning benchmarks, improvements are more modest but still positive: Daily-Omni gains 2.1 points (62.1% to 64.2%) and WorldSense gains 1.3 points (45.4% to 46.7%). Critically, gains scale dramatically with video duration: improvements increase from 3.2 points on short videos (0–5 minutes) to 9.9 points on longer videos (10–30 minutes), and to 13.2 points on LVOmniBench‘s 50–90 minute content. Tool-use call rates correspondingly increase with duration, rising from 33.3% on 0–1 minute videos to 84.8% on 10–30 minute videos. Ablations confirm that self-curated temporal data (−3.8 points without it), tool-use trajectories (−2.0 points), audio input (−3.9 points), and TimeAnchor (−2.5 points on OmniVideoBench) each contribute substantially. Temporal grounding improves particularly sharply: on Charades-STA, TimeAnchor raises IoU@0.3 from 58.8% to 64.9% and mIoU from 37.9% to 41.3%.

Implications and Conclusions

OmniReasoner establishes that adaptive, tool-guided evidence acquisition significantly outperforms uniform processing for long audio-video reasoning, especially as input duration increases. The work demonstrates that omnimodal models can learn to strategically allocate high-fidelity computation to sparse, evidence-bearing moments while maintaining global temporal context—a capability that may become increasingly important as video reasoning tasks scale to longer durations and more complex cross-modal dependencies. However, the authors acknowledge infrastructural limitations: existing reinforcement learning frameworks lack native support for audio-conditioned multi-turn agentic workflows, and scaling beyond two-step reasoning requires both larger base models (beyond Qwen2.5-Omni-7B‘s 32K context) and more mature omnimodal RL infrastructure. Future work integrating stronger pre-trained tool-aware models and improved training frameworks could extend this paradigm to multi-turn reasoning scenarios and richer tool vocabularies.

Appearance Pointers -- Multimodal Region Control of Diffusion Transformers

Authors: Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch, Srinath Sridhar, Matheus Gadelha

Source and references: https://arxiv.org/abs/2607.19344v1

Introduction

This paper introduces Appearance Pointers, a novel mechanism for achieving precise, multimodal region-specific control in Diffusion Transformers (DiTs) without requiring model retraining. The work addresses a critical limitation in current generative image models: the inability to reliably control where and how different appearance cues—whether from text or reference images—should influence specific spatial regions in generated outputs.

Key Points

Appearance Pointers as Routing Tokens: The paper introduces compact “pointer” tokens that guide DiTs toward correct appearance cues at specified spatial locations, functioning as a lightweight interface between user intent and model generation without architectural modification.
Modality-Agnostic Framework: Unlike existing approaches constrained to either text or image conditioning, Appearance Pointers support simultaneous multimodal control, enabling users to condition different regions with both text descriptions and reference images within a single generation pass.
Region Correspondence Network: A dedicated network produces appearance pointers by fusing text or image inputs with user-specified spatial masks, combined with a spatial aggregation mechanism that handles multiple regional descriptions efficiently without excessive token overhead.
Comprehensive Capability Support: The unified model supports fine and sparse control layouts, object insertion with material fidelity, pose-conditioned generation, and multi-region synthesis—capabilities previously requiring separate specialized methods.
New Dataset and Evaluation: The authors introduce Appearance Pointers-37K, a synthetic dataset with regional text descriptions and appearance images, enabling systematic evaluation across multiple metrics including region adherence and identity preservation.

Methodology

The approach leverages the native heterogeneous token capabilities of Diffusion Transformers by introducing a region correspondence network that processes spatial masks alongside text embeddings or image tokens. This network produces appearance pointers—compact tokens that effectively “point“ the DiT‘s attention toward the correct conditioning signals at appropriate spatial locations. A spatial aggregation mechanism then refines these pointers to handle multiple regions simultaneously, maintaining computational efficiency while preserving the original DiT architecture and avoiding the need for complete model retraining.

Results and Findings

Across six evaluation metrics, Appearance Pointers achieves best or competitive performance compared to specialized state-of-the-art baselines. When regions are described through text alone, the method ranks first or second on all metrics. For image-based regional descriptions, it surpasses existing methods (MSDiffusion and DreamRenderer) in both region adherence and identity preservation. Critically, the single unified model outperforms all prior work across capability dimensions—it‘s the only method supporting simultaneous fine and sparse control, insertion, generation, and true multimodal conditioning where individual regions can be described by both images and text together.

Implications and Conclusions

Appearance Pointers represents a significant advancement toward practical creative tools by enabling precise spatial control while maintaining architectural simplicity and computational efficiency. The work demonstrates that effective region-aware multimodal control doesn‘t require complete model redesign but rather elegant interface mechanisms that route existing DiT capabilities, establishing a scalable path for integrating generative models into professional creative workflows where spatial precision and heterogeneous conditioning are essential requirements.

MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings

Authors: Ziyi Wang, Yuhang Wu, Dongxu Piao, Xingyu Liu, Tianhui Zhou, Miao Liu

Source and references: https://arxiv.org/abs/2607.19235v1

MeetingToM: Theory of Mind Reasoning in Multimodal Meeting Analysis

Introduction

This paper introduces MeetingToM, a benchmark for evaluating multimodal large language models (MLLMs) on their ability to reason about Theory of Mind (ToM)—inferring beliefs, intentions, and mental states—in naturalistic multi-party meeting scenarios. The research addresses a critical gap in existing multimodal AI benchmarks by focusing on complex social phenomena like pseudo-consensus, where participants verbally agree while nonverbal cues reveal private dissent under social pressure.

Key Points

Hierarchical Task Design: MeetingToM comprises three progressively complex task levels—subject-level mental state prediction, dyadic-level addressee understanding and attitude inference, and group-level consensus reasoning—requiring increasingly sophisticated social reasoning.
Pseudo-Consensus Focus: Unlike existing benchmarks emphasizing overt signals, MeetingToM specifically targets meeting-specific phenomena where surface-level agreement masks hidden disagreement due to conformity effects and social pressure, a nuanced challenge for current models.
Comprehensive Multimodal Integration: The benchmark requires models to integrate verbal cues (speech, discourse markers, semantic content) with non-verbal signals (facial expressions, gaze, body orientation, posture) across multiple synchronized video perspectives to accurately assess social states.
Large-Scale Annotation Effort: MeetingToM contains 1,800 clips from 60 meeting sessions across three task families (600 instances per task), with rigorous quality control including hybrid annotation pipelines, inter-annotator agreement metrics (κ ranging from 0.50-0.73), and adjudication procedures.
Consistent MLLM Limitations: Evaluation reveals substantial performance gaps between human capability (74.97-86.33% accuracy across tasks) and state-of-the-art models (Gemini-3 Pro: 22.13-88.52% macro accuracy), with particular struggles in detecting subtle nonverbal cues and resolving verbal-visual conflicts.

Methodology

MeetingToM builds on the AMI Meeting Corpus, extracting task-specific video clips (5-second segments for Task 1, ~50-second segments for Tasks 2-3) synchronized with word-level transcripts. The benchmark employs a hybrid annotation pipeline: automated candidate identification of socially relevant events, GPT-assisted question generation without answer determination, independent human labeling by trained annotators, and adjudication procedures for cases without majority agreement. Models receive only visual input, aligned transcripts, and task prompts at evaluation time, with gold labels and auxiliary metadata excluded. The research evaluates both proprietary models (GPT-4o, GPT-5, Gemini-3 Pro) and open-source alternatives (Qwen2.5/3-VL variants) using exact-match accuracy and macro-averaged metrics for imbalanced tasks.

Results and Findings

Overall Performance: Proprietary models substantially outperform open-source alternatives. Gemini-3 Pro achieves 59.00% accuracy on Task 1 and 88.52% on Task 3.2, while GPT-5 reaches 55.67% and 90.70% respectively. However, macro-averaged scores reveal significant class imbalance effects—Gemini-3 Pro drops from 59.00% to 30.39% macro on Task 1, indicating uneven reliability across minority mental states. Performance degrades across the task hierarchy, with group-level reasoning proving most challenging.

Modality Ablations: Utterance-only input outperforms multimodal combinations on Tasks 1, 2.2, 3.1, and 3.2, suggesting models rely heavily on verbal content. However, Task 2.1 (addressee identification) benefits from video-utterance fusion, indicating that gaze and body orientation provide distinct value for referential reasoning. Joint multimodal input inconsistently outperforms unimodal settings, suggesting current MLLMs struggle with evidence fusion when cues are ambiguous or contradictory.

Prompting Effects: Chain-of-Thought variants show task- and model-dependent benefits rather than uniform improvements. Task-aligned CoT improves group-level reasoning on Gemini-3 Pro but reduces dyadic performance. Emotional CoT helps some models on subject-level tasks but over-emphasizes facial affect while missing interactional context. This suggests thinking prompts reshape attention allocation rather than adding simple reasoning depth.

Contextual Priors: Adding structured metadata (participant roles, meeting phases, dialogue acts) yields inconsistent or harmful effects. Dialogue-act labels sharply reduce addressee and group-level performance, while role and phase information provide isolated improvements. This indicates that contextual labels become useful only when calibrated against local evidence, not as automatic constraints.

Discourse Marker Analysis: Removing hesitation markers and backchannels improves Task 1 accuracy but reduces dyadic and group-level performance, suggesting these markers sometimes distract local mental-state recognition while carrying useful interactional information for broader social reasoning.

Implications and Conclusions

MeetingToM establishes critical benchmarks for advancing meeting-grounded Theory of Mind in multimodal systems, revealing that current MLLMs persistently struggle with integrating non-verbal cues, inferring hidden attitudes, and distinguishing genuine from pseudo-consensus. The substantial gaps between human performance (>74%) and model capability across all tasks, coupled with counterintuitive findings about prompting and contextual information, highlight that meeting-grounded social reasoning fundamentally requires models to calibrate competing evidence streams under social ambiguity—a capability that remains underdeveloped in today‘s most advanced multimodal systems. This work provides a rigorous testbed for the research community to develop more socially-aware AI systems applicable to collaborative decision support and socially-aware assistants in professional settings.

Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

Authors: Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou, Jalaj Bhandari, Kavosh Asadi, Daniel Jiang, Aditya Modi

Source and references: https://arxiv.org/abs/2607.19313v1

Introduction

This paper addresses a critical limitation in reinforcement learning for large language models: when training on hard problems that produce no correct solutions, existing methods receive zero learning signal. The authors introduce Off-Context GRPO (OC-GRPO), which uses privileged guidance (like solution prefixes) during training while maintaining alignment with the unguided deployment objective through importance-weighted corrections.

Key Points

The Learning Cliff Problem: Standard GRPO fails on hard problems where all sampled responses are incorrect, resulting in zero reward variance and zero gradient updates, preventing any learning signal.
Off-Context Distribution Mismatch: Existing guided training methods sample from augmented prompts containing privileged information but compute gradients as if sampling from the original unguided prompt, optimizing a misaligned objective (J_guide instead of J).
Importance-Corrected Solution: OC-GRPO applies per-token importance ratios ρ^oc_i,t(θ) = π_θ(y_i,t|x, y_i,
Behavior-Aware Credit Assignment: The importance correction automatically adjusts how much credit each trajectory receives based on guidance dependence—trajectories relying heavily on privileged information receive suppressed credit, while failures persisting despite guidance receive amplified penalties, without requiring additional reward shaping.
Minimal Implementation: OC-GRPO requires only a small modification to the standard GRPO surrogate loss (Equation 9), replacing the standard importance ratio with the off-context corrected ratio while maintaining all other components.

Methodology

The authors frame guided RLVR training as an off-context sampling problem: rollouts are generated under a guidance-augmented prompt g(x) but should optimize for the original prompt x to match deployment conditions. They employ importance sampling theory to derive the corrected per-token ratio that reweights advantages from guided samples back toward the unguided objective. For implementation, they use solution prefix guidance at nested levels (20%, 40%, 60%, 80%, 100% of reference solutions) and present two variants: OC-GRPO-Fixed (selecting guidance levels once using the base model) and OC-GRPO-Adaptive (re-evaluating guidance levels during training).

Results and Findings

On Qwen2.5-7B-Instruct, OC-GRPO-Fixed achieves a 13.8% relative improvement over vanilla GRPO (31.7 vs 27.8 average Pass@1 across benchmarks), outperforming guided baselines POPE (+8.7%) and PrefixRL by 1.4 absolute percentage points. Performance gains are consistent across three major benchmarks: AIME, Gaokao2023, and OmniMath. Critically, at smaller model scales (3B and 1.5B parameters), guided baselines that optimize misaligned objectives degrade below vanilla GRPO (PrefixRL* drops 7.2% at 1.5B), while OC-GRPO maintains consistent gains of +7.2% (3B) and +10.2% (1.5B). This scale-dependent pattern suggests back-generalization—the assumption that guided updates transfer to unguided prompts—is capacity-dependent and breaks at smaller model sizes without explicit correction.

Implications and Conclusions

OC-GRPO demonstrates that properly accounting for training-time context distribution shifts is essential for reliable learning from privileged guidance, particularly as model capacity decreases. The framework generalizes beyond solution prefixes to any form of privileged information (hints, intermediate goals, tool results) and extends naturally to multi-turn and agentic settings where training-time context differs from deployment conditions. This work establishes importance correction as a foundational principle for guided RL in language models, ensuring that learning signals guide models toward genuine problem-solving capabilities rather than hint-following behaviors.

Provable diffusion-based posterior sampling for linear inverse problems via DDIM

Authors: Yuchen Jiao, Na Li, Changxiao Cai, Yuxin Chen, Gen Li

Source and references: https://arxiv.org/abs/2607.19333v1

Introduction

This paper introduces Posterior-DDIM, a theoretically principled and computationally efficient algorithm for solving linear inverse problems using diffusion models as priors. The method achieves provable posterior consistency while maintaining the simplicity and speed of standard DDIM samplers through adaptive, direction-wise updates based on signal-to-noise ratio comparisons.

Key Points

SNR-guided singular direction partitioning: The algorithm partitions measurement operator singular directions into three groups—measurement-dominated, prior-dominated observed, and unobserved—based on comparing observation SNR with diffusion SNR at each time step.
Lightweight coordinate-wise modifications: Rather than introducing complex projection steps or approximate posterior scores, Posterior-DDIM requires only simple, efficient modifications to standard DDIM updates tailored to each singular direction category.
Posterior consistency guarantees: Theorem 1 and Theorem 2 establish that under exact diffusion priors and sufficiently fine time discretization, the sampler converges to the true Bayesian posterior distribution as discretization vanishes.
Comprehensive empirical validation: Experiments on inpainting, super-resolution, Gaussian deblurring, and compressed sensing tasks demonstrate superior or competitive performance across PSNR, SSIM, LPIPS, and FID metrics compared to state-of-the-art methods.
Plug-and-play framework: The approach requires no task-specific retraining and readily transfers across sensing modalities, making it practical for real-world applications including medical imaging and scientific discovery.

Methodology

Posterior-DDIM exploits the structure of linear measurement operators through singular value decomposition. For each singular direction, the algorithm compares the effective observation SNR (determined by singular values and measurement noise) against the diffusion SNR (α_t/σ_t) to decide whether measurements or the learned prior should drive updates. The algorithm uses direct measurement injection with calibrated noise for high-SNR directions, standard DDIM dynamics for low-SNR observed directions, and stochastic DDIM updates with higher noise injection for unobserved directions. Theoretically, convergence is established through auxiliary sequence constructions and KL divergence decomposition arguments that track how discretization errors propagate through the sampling procedure.

Results and Findings

Across multiple image restoration tasks on CelebA and ImageNet datasets, Posterior-DDIM achieves best performance on at least two of four metrics in every task evaluated. For inpainting on CelebA, it achieves best results on all four metrics (PSNR: 33.23, SSIM: 0.91, LPIPS: 15.06, FID: 25.33). On compressed sensing tasks, it consistently outperforms DDNM+ across all metrics. The method demonstrates robustness: performance saturates around 80-100 function evaluations, providing favorable computational-quality trade-offs. Hyperparameter analysis shows η₁ (stochasticity for unobserved directions) should be sufficiently large, while η₀ (stochasticity for prior-dominated directions) performs best at zero, recovering deterministic DDIM updates.

Implications and Conclusions

This work bridges theory and practice in diffusion-based inverse problem solving by providing the first rigorous posterior consistency guarantees for DDIM-type samplers on linear problems while maintaining computational efficiency. The SNR-guided directional partitioning framework offers a principled foundation for integrating measurement information with learned priors, with significant implications for high-stakes applications requiring both statistical reliability and computational tractability. Future directions include extending to nonlinear inverse problems, characterizing convergence rates, and analyzing dependencies on noise levels and operator conditioning.

Riemannian Deep Learning:Modules, Networks, and Geometries

Authors: Chen Ziheng

Source and references: https://arxiv.org/abs/2607.19305v1

Introduction

This doctoral thesis develops a comprehensive framework for building reusable neural network modules that operate on manifold-valued data, addressing fundamental limitations in geometric deep learning where operations are often tied to specific manifolds or rely on Euclidean approximations that sacrifice numerical stability and computational efficiency.

Key Points

Generalized Batch Normalization: Introduces Lie Group Batch Normalization (LieBN) with theoretical guarantees on Riemannian sample means and variances, extended to pseudo-reductive gyrogroups for applicability beyond traditional Lie group structures.
Riemannian Classification Layers: Extends Multinomial Logistic Regression from Euclidean space to Symmetric Positive Definite (SPD) manifolds and general Riemannian manifolds using Riemannian trigonometry, providing geometric alternatives to standard softmax classifiers.
Hyperbolic Geometry Innovations: Develops Proper Velocity Neural Networks as an unconstrained representation of hyperbolic space with stable geometry, and introduces Hyperbolic Busemann Neural Networks for efficient classification and fully connected operations in hyperbolic space.
Specialized Matrix Manifold Networks: Constructs networks operating directly on full-rank correlation matrices with accurate gradient computation under two distinct correlation geometries, providing a normalized alternative to SPD matrices for specific applications.
Learnable Metrics for SPD Manifolds: Proposes adaptive log-Euclidean metrics through parameterized matrix logarithms and develops fast Cholesky-based metrics with closed-form operators, enabling metrics to adapt to data and network dynamics while maintaining numerical stability.

Methodology

The thesis employs a hierarchical approach to geometric deep learning. It begins with general constructions grounded in differential and Riemannian geometry, leveraging Lie group structures where applicable. For cases where general formulations cannot exploit useful structure, the work designs specialized networks for particular manifold representations. The framework combines theoretical analysis with practical implementation, incorporating matrix function theory for backpropagation through geometric operations. Empirical validation spans vision, signal processing, graph learning, and genomics applications, with experiments demonstrating both the theoretical soundness and practical utility of proposed methods.

Results and Findings

The research yields seven peer-reviewed publications at premier venues (ICLR, CVPR, NeurIPS, ICML) plus numerous collaborative works advancing the field. Key empirical contributions include: batch normalization methods demonstrating improved convergence on SPD manifolds; multinomial logistic regression variants showing competitive or superior classification accuracy on manifold-valued data; Proper Velocity networks achieving stable training in hyperbolic spaces without projection constraints; and Cholesky-based metrics reducing computational overhead while maintaining numerical precision. Applications demonstrate consistent improvements in EEG signal classification, skeleton-based action recognition, and brain imaging analysis—domains where manifold structure is inherent to the data representation.

Implications and Conclusions

This work significantly advances geometric deep learning by bridging the gap between theoretical differential geometry and practical neural network design, providing practitioners with reusable, theoretically-grounded modules for manifold-valued data. The framework‘s modularity and emphasis on both numerical stability and computational efficiency position it as a foundation for future research in applications where data naturally resides on non-Euclidean geometries, from medical imaging to graph-based learning systems.

CircuitKIT : Circuit Discovery, Evaluation, and Application Toolkit for Mechanistic Interpretability

Authors: Pratinav Seth, Hem Gosalia, Aditya Kasliwal, Vinay Kumar Sankarapu

Source and references: https://arxiv.org/abs/2607.19317v1

Introduction

CircuitKIT is a comprehensive, source-available library that unifies the mechanistic interpretability workflow for large language models by connecting circuit discovery, evaluation, and downstream interventions through a single typed artifact. The toolkit addresses the fragmentation in circuit analysis by providing standardized interfaces, multiple discovery algorithms, complementary evaluation diagnostics, and practical intervention modules for model compression, editing, and safety auditing.

Key Points

Unified Pipeline Architecture: CircuitKIT connects thirteen discovery algorithms across four backend families (gradient-attribution, search-based, information-bottleneck, and contextual decomposition) with six complementary evaluation pillars and seven downstream application modules through a single CircuitScores artifact, eliminating the need to stitch together separate implementations.
Declarative Custom-Data Path: A template-driven interface maps structured datasets (CSV, JSONL, HuggingFace) into circuit-discovery tasks without hand-authoring contrastive prompts, with automatic token alignment and corruption synthesis supporting both paired and clean-only discovery routes.
Multi-Pillar Faithfulness Evaluation: Rather than relying on a single metric, the framework evaluates circuits across six complementary diagnostics (causal patching, ablation sufficiency, stability, robustness, baseline comparison, and generalization) plus an optional intervention-reliability pillar, acknowledging that faithfulness scores depend on ablation methodology and can reverse method rankings.
Three-Interface Accessibility: The same workflow operates through a stateful Pipeline class, a functional flat API, and a YAML-driven CLI, allowing circuits discovered in one interface to be evaluated or applied through another without format conversion.
Extensible Registry Pattern: Discovery algorithms, corruption strategies, selectors, and model architectures register through decorators, establishing a clear governance model where stable-tier methods carry backward-compatibility guarantees while research-tier contributions can be promoted as validation evidence accumulates.

Methodology

CircuitKIT operates on decoder-only transformers by decomposing the model into addressable components (attention heads, MLP sublayers, individual neurons) connected through a computational graph. Discovery locates causal subgraphs responsible for specific behaviors through causal intervention using either contrastive pairs (activation patching via EAP/EAP-IG/EAP-GP/ACDC) or clean inputs alone (IBCircuit, CD-T). The framework standardizes both node-level (entire components) and neuron-level (individual channels) granularities, with evaluation running across six configurable diagnostic pillars computed from a single CircuitScores object. Interventions (pruning, quantization, fine-tuning, editing, steering) consume this artifact through unified APIs, with results exported as reloadable HuggingFace checkpoints and benchmarked through lm-evaluation-harness integration.

Results and Findings

Algorithm Validation (E1): Six stable algorithms on GPT-2 Small IOI recover the canonical circuit with perfect causal-patching recovery (1.0 for EAP family) and Jaccard overlap of 0.43–0.50 with known components. CD-T achieves equal ablation faithfulness (1.0) despite zero canonical-head overlap, demonstrating that behavioral recovery and architectural overlap are distinct measures requiring panel-based reporting.

Cross-Family Discovery (E2): EAP-IG produces faithful, stable circuits across six model families (GPT-2, Pythia, Llama, Gemma, Qwen, Phi) at 124M–2.8B parameters with consistent patching recovery (P1 ≥ 0.91) and stability (Jaccard 0.80–0.92). Ablation sufficiency varies widely (P2: 0.24–1.00), illustrating that single-metric evaluation obscures performance variation.

Neuron-Level Discovery (E3): Neuron-granularity discovery preserves patching recovery (EAP-IG: 1.0) while retaining 70% of units at fixed sparsity, directly exposing the units intervention modules act on. Single-point-gradient EAP degrades and inverts under hard ablation (P2 = −0.35), whereas integrated-gradient interpolation stabilizes fine-grained attribution.

Custom-Data Path (E4): A 334-record jailbreak CSV reaches multi-pillar evaluation via both paired (EAP-IG) and clean-only (IBCircuit) routes with zero pairing code. The paired circuit recovers 85% of refusal under soft patching but inverts toward compliance under hard ablation (P2 = −2.61, raw = −1.55), illustrating that intervention safety cannot be inferred from intrinsic scores alone.

Circuit-Guided Pruning (E5): IBCircuit prunes competitively (98.6% accuracy retention), but patch faithfulness anti-correlates with retention (Spearman ρ = −0.78, p = 0.010), with EAP-IG achieving lowest perplexity (35.5) despite lowest accuracy retention (57.1%). This demonstrates that intrinsic faithfulness does not predict intervention performance.

Quantization (E6): Unlike pruning, mixed-precision quantization is insensitive to patch faithfulness (ρ = +0.23, p = 0.55) but correlates with ablation faithfulness (ρ = +0.73, p = 0.031), showing that intervention-selector relationships differ by application type and must be validated extrinsically.

Selective Fine-Tuning (E7): Circuit-guided fine-tuning does not separate from random-budget masking (mean Δ = −0.001 pp), with the 30% budget constraint itself protecting coherence rather than circuit importance determining which parameters to update.

Implications and Conclusions

CircuitKIT establishes that mechanistic interpretability requires standardized infrastructure supporting method comparison, custom-data integration, and honest downstream validation. The finding that single faithfulness metrics can reverse method rankings and fail to predict intervention outcomes argues for panel-based evaluation and extrinsic benchmarking rather than relying on intrinsic scores. The toolkit‘s demonstration that circuit-derived importance performs variably across interventions—decisively in pruning, negligibly in quantization, neutrally in fine-tuning—establishes that interpretability metrics must be validated per application and that comprehensive dashboards rather than summary statistics are necessary for practitioners making safety and efficiency decisions. By releasing CircuitKIT with extensible registries and source-available governance restricting commercial use that degrades model safety, the authors provide common infrastructure for the mechanistic-interpretability community while acknowledging the dual-use risk inherent in circuit localization and ablation techniques.

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

Authors: Kexin Huang, Junkang Wu, Jinda Lu, Shuo Yang, Chiyu Ma, Jiancan Wu, Xiang Wang, Xiangnan He, Guoyin Wang, Jingren Zhou

Source and references: https://arxiv.org/abs/2607.10481v2

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

Introduction

This paper addresses a critical instability in reinforcement learning for large language models: over-optimization, where models exploit training patterns that don‘t generalize to validation tasks. The authors demonstrate that standard reverse KL regularization is insufficient for preventing this collapse and propose ARMOR, a framework that combines active sample stabilization with adaptive exploration to achieve sustained performance improvements during extended training.

Key Points

Over-optimization Problem: Models exhibit a “reward-validation gap” where training rewards improve while validation performance degrades—a generalization failure distinct from classical reward hacking that persists even with verifiable reward signals.
KL Regularization Limitations: Standard reverse KL has two fundamental flaws: (1) its mode-seeking nature allows collapse onto narrow “shortcut” patterns without penalty, and (2) uniform penalties suppress both harmful degradation and beneficial exploration, creating a stability-exploration dilemma.
Anchor Rollout Component: Actively injects off-policy correct samples from the reference policy into training batches to explicitly prevent drift from known good solutions, replacing passive loss penalties with active distribution stabilization.
Mixed Optimization Component: Reformulates the policy objective to optimize a mixture policy α·πθ + (1−α)·πref, which constructs an adaptive trust region that permits controlled exploration without uniform suppression effects.
Broad Empirical Validation: Demonstrates consistent improvements across multiple base models (Qwen2.5-Math-7B, Qwen3-8B-Base) and RL algorithms (DAPO, QAE), with gains of +5–8 points on mathematical reasoning benchmarks without sacrificing general capabilities.

Methodology

ARMOR operates in two phases executed iteratively. During Anchor Rollout, the framework constructs hybrid response groups by sampling on-policy responses from the current policy and augmenting them with rejection-sampled correct responses from the reference policy. In the Mixed Optimization phase, the framework replaces standard importance sampling ratios with mixed variants that account for the reference policy‘s contribution to the data distribution, then performs policy updates using the underlying RL algorithm (DAPO or QAE). Periodically, the reference policy is reset to the current best checkpoint to prevent the anchor from becoming a bottleneck.

Results and Findings

Experiments on AIME24, AIME25, and AMC benchmarks show substantial improvements. With DAPO on Qwen2.5-Math-7B, ARMOR achieves +5.91 points on AIME24 (37.13→43.04) and +6.74 on average math reasoning (40.58→45.77). On Qwen3-8B-Base, gains reach +11.15 points on AIME24 (36.98→48.13). QAE integration validates robustness (+1.79 points on AIME24), though with slight general capability trade-offs attributed to the algorithm‘s advantage masking mechanism. Pass@k evaluations confirm that gains reflect genuine reasoning improvements rather than accuracy-diversity trade-offs. Ablation studies conclusively demonstrate the necessity of both components: Mixed Optimization alone eventually collapses without Anchor Rollout‘s stability floor, while Anchor Rollout without Mixed Optimization plateaus at lower performance ceilings. Analysis of KL divergence reveals that the adaptive trust region mechanism from Mixed Optimization selectively reinforces verified correct actions while permitting stronger penalization of reference biases.

Implications and Conclusions

This work reveals a fundamental structural limitation in standard regularization approaches for long-horizon RL training in reasoning tasks, proposing a principled alternative that decouples stability concerns from exploration constraints. By shifting from passive penalty-based regularization to active sample-based stabilization combined with adaptive trust regions, ARMOR provides a practical framework for sustained reasoning capability improvement in large language models, with immediate applicability to production systems scaling RL for mathematical reasoning and other verifiable tasks.

AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters

Authors: Yu-Yang Qian, Hao-Cong Wu, Chen Chen, Jiacheng Sun, Zhenhua Dong, Peng Zhao, Zhi-Hua Zhou

Source and references: https://arxiv.org/abs/2607.19223v1

Introduction

This paper presents ADAFlash, a framework for accelerating large language model inference through adaptive speculative decoding using diffusion-based draft models. The work identifies and addresses critical variance issues that emerge when using bidirectional-attention diffusion models as drafters during deployment.

Key Points

Identifies high-variance problem: Diffusion drafters exhibit substantially higher variance than autoregressive drafters at both domain-level (acceptance rates fluctuate 2.1× across domains vs. 1.2× for AR models) and token-level (per-position acceptance probability varies significantly within sequences).
On-Policy Distillation (OPD) algorithm: Introduces a specialized distillation approach using reverse-KL divergence with entry-wise clipping, designed specifically for diffusion drafters’ high-entropy outputs, reducing domain-level variance during deployment.
Adaptive Length Head: Proposes a lightweight prediction module that dynamically adjusts verification sequence length based on predicted acceptance rates, addressing token-level variance and eliminating wasteful computation on low-quality tokens.
Infrastructure for online adaptation: Implements asynchronous training-inference pipelines and adaptive request scheduling within a serving engine, enabling continuous drafter updates without blocking inference.
Substantial performance gains: Achieves up to 5.3× speedup over standard autoregressive decoding and 66% higher throughput than prior methods under high-concurrency scenarios (concurrency=128).

Methodology

ADAFlash combines two complementary mechanisms. First, it performs on-policy distillation during deployment by collecting feedback from the target model‘s distributions at each verification step, then updating the drafter using a mixture loss combining reverse-KL divergence (which encourages mode-seeking behavior) and hard-label cross-entropy on top-1 tokens, with entry-wise clipping to prevent outlier gradients from dominating updates. Second, it attaches a lightweight neural head to the drafter that predicts overall acceptance rates and dynamically determines verification length, with this head continuously updated via MSE loss against ground-truth acceptance rates obtained from the verification outcomes.

Results and Findings

Experiments across eight datasets and three foundation models (including dense and mixture-of-experts architectures) demonstrate consistent improvements. At single concurrency (C=1), ADAFlash achieves 4.06× average speedup on Qwen3-8B compared to 3.53× for DFlash and 3.95× for OSD. Critically, at high concurrency (C=128), ADAFlash maintains 1.15× speedup while competing methods degrade to 0.76-0.83×, falling below baseline performance. Ablation studies confirm both components contribute meaningfully: divergence clipping and mixture OPD improve domain-level acceptance consistency, while the adaptive length head is essential for high-concurrency performance. Analysis shows ADAFlash narrows the domain-level variance distribution substantially and improves per-position acceptance probabilities, particularly at later token positions where baseline drafters struggle.

Implications and Conclusions

This work reveals that bidirectional-attention mechanisms in diffusion drafters create practical deployment challenges beyond their well-known benefits, and demonstrates that adaptive online learning can effectively mitigate these issues. The framework‘s superior performance under high concurrency has significant implications for production LLM serving environments, where systems must handle multiple simultaneous requests and resource constraints become acute bottlenecks.

Masked Visual Actions for Unified World Modeling

Authors: Hadi Alzayer, Wenlong Huang, Haonan Chen, Christopher Luey, Lvmin Zhang, Maneesh Agrawala, Gordon Wetzstein, Li Fei-Fei, Yilun Du, Jiajun Wu, Jia-Bin Huang

Source and references: https://arxiv.org/abs/2607.19343v1

Masked Visual Actions for Unified World Modeling

Introduction

This paper introduces Masked Visual Actions, a novel approach that enables pretrained video models to serve as unified robotic world models by representing actions as pixel-space masked trajectories. The method allows a single model checkpoint to function simultaneously as a forward dynamics model (predicting scene responses to robot actions) and an inverse model (recovering robot behavior from desired object motions), all while generalizing across different robot embodiments.

Key Points

Pixel-aligned action conditioning: Actions are expressed as partially revealed spatiotemporal patterns in video frames rather than low-dimensional commands, creating a representation naturally aligned with video models’ learned priors about visual interaction and motion.
Unified forward and inverse modeling: By varying which entities are revealed in masked videos—robot motion versus desired object motion—the same model can function as either a forward dynamics predictor or inverse model without separate training, a capability unique to this spatial masking approach.
Embodiment-agnostic generalization: The method demonstrates strong zero-shot transfer to unseen robot morphologies (including bimanual systems), significantly outperforming baselines that condition on skeleton poses or end-effector positions, which collapse or hallucinate when encountering novel embodiments.
Efficient training and practical applications: Using only 15 hours of robot interaction data (combined from real DROID videos and Robocasa simulation), the model achieves state-of-the-art visual fidelity while supporting three downstream robotics applications: policy evaluation, model-based planning, and action extraction.
Superior visual fidelity: Quantitative metrics (LPIPS, SSIM, PSNR) demonstrate substantial improvements over prior work like Ctrl-World, with particularly pronounced advantages on out-of-distribution embodiments (LPIPS of 0.123 vs. 0.196 on BEHAVIOR dataset).

Methodology

The authors finetune a pretrained video diffusion model (Wai-Fun-Control 2.2 14B) using LoRA adaptation (rank 256) on masked video sequences. Training data construction employs two complementary approaches: segmentation-based masking using SAM to isolate robots from real DROID videos, and rendering-based masking using robot URDFs from recorded states in both real and simulated environments. The model learns conditional video generation by receiving masked input videos concatenated spatially with reference frames, predicting unmasked regions through standard diffusion training over ~10,000 steps on 8 H200 GPUs.

Results and Findings

Controllable generation: The model matches and exceeds Ctrl-World‘s performance on seen embodiments (LPIPS 0.0945 vs. 0.362) while gracefully generalizing to unseen bimanual robots where baselines fail completely. Ablations confirm masked visual actions substantially outperform sparse conditioning signals (end-effector positions, skeletons) on out-of-distribution data.

Model-based planning: Best-of-N planning using the video model to evaluate trajectory rollouts improves task success by 7-26% across six manipulation tasks (close microwave, open drawer, etc.), with gains increasing with the number of evaluated candidates.

Policy evaluation: Simulated rollout success rates exhibit strong correlation (r = 0.982) with ground-truth environment outcomes in simulation. Real-world validation shows per-trial progress distributions in generated videos closely match actual execution, though with a consistent positive bias toward task success.

Action extraction: Zero-shot inverse modeling—generating robot trajectories from desired object motion—recovers the highest success rate (90%) on the COFFEESERVEMUG task compared to imitation learning baselines (Diffusion Policy, ACT, SmolVLA), despite the video model never seeing task-specific training data.

Implications and Conclusions

Masked Visual Actions establishes a fundamentally new paradigm for robotic world modeling by leveraging video models‘ rich visual priors through pixel-aligned action conditioning rather than embodiment-specific control signals. This work demonstrates that spatial masking enables a single unified model to bridge forward and inverse reasoning while achieving unprecedented generalization across robot morphologies, with clear practical benefits for planning, evaluation, and action synthesis in robotic manipulation tasks.

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

Authors: Guanxiong Chen, Qianjun Xia, Jiawei Peng, Heng Zhang, Bole Ma, Justin Qian, Ziyi Jiao, Bingyang Zhou, Luoxin Ye, Kaifeng Zhang, Kunyi Wang, Weijia Zeng, Yunuo Chen, Pengzhi Yang, Ziqiu Zeng, Huamin Wang, Chao Liu, Alan Yuille, Fan Shi, Changxi Zheng, Yunzhu Li, Chenfanfu Jiang, Peter Yichen Chen

Source and references: https://arxiv.org/abs/2607.19190v1

Agentic Real2Sim: Automating Physics-Based Digital Twin Creation from Robot Videos

Introduction

This paper introduces Agentic Real2Sim, a framework that automatically converts real-world robot interaction videos into physically simulatable digital twins. Rather than relying on manual tuning and brittle workflows, the system uses vision-language agents to orchestrate the complete conversion pipeline—from scene reconstruction to physical parameter inference—enabling scalable transformation of robot demonstrations into simulation-ready artifacts.

Key Points

Unified Episode Conversion Pipeline: The framework converts recorded robot-object interactions into simulatable twins that preserve observations, geometries, trajectories, physical parameters, and actor states through four linked agents: visual processing, physical-prior inference, scene preparation, and simulator-in-the-loop grasp optimization.
VLM-Agnostic Architecture with Cost Efficiency: Agentic Real2Sim supports interchangeable vision-language model backends. An open-weight 31B model (Gemma) achieves comparable 48/100 replay-success outcomes to proprietary models while reducing costs by up to 31.4× compared to frontier models, with total conversion costs as low as $2.62 per 100 episodes.
Generalization Across Domains: The same core conversion contract extends beyond rigid-body manipulation to deformable-object interactions (rope, cloth, soft materials) and humanoid locomotion, demonstrating that the framework handles diverse physical interaction types without domain-specific rewrites.
Deliberate Separation of Concerns: The architecture cleanly separates deterministic visual and simulation tools from agentic decision-making. VLMs make bounded, schema-constrained choices about object discovery, keyframe selection, and refinement strategies, while specialized perception and physics components handle geometry recovery, pose tracking, and grasp optimization.
Structured Evaluation Methodology: Episodes are evaluated using a VLM-based replay-success metric that compares real and simulated keyframes across four dimensions—target object identity, final location, action similarity, and gripper location—with three independent judges scoring each candidate.

Methodology

The framework processes DROID-style robot demonstrations through a multi-stage agentic pipeline. The visual processing agent extracts segmentation masks using SAM 3, recovers 3D geometry with SAM 3D, performs depth estimation via FoundationStereo, and tracks object poses using FoundationPose. The physical-prior inference agent infers material classes and mass properties from visual evidence. Scene preparation then calibrates camera extrinsics, optimizes robot base pose alignment, estimates ground planes, and loads the scene into MuJoCo. Finally, a simulator-in-the-loop grasp optimization stage evaluates candidate object placements to identify configurations enabling successful grasping. VLM queries are scoped to high-level decisions with bounded retry budgets rather than geometric computation, enabling backend interchangeability. The system outputs a standardized episode folder containing meshes, pose tracks, robot trajectories, camera metadata, and task semantics.

Results and Findings

On DROID-100 (100 diverse manipulation episodes), Gemma 31B achieved 48 successful replays, 8 partial successes, and 44 failures at a model cost of $2.62. Across four VLM backends tested—Gemma 4 (31B), Qwen (35B), GPT-5.4, and Claude Haiku—replay-success rates ranged from 37 to 48 episodes, but model costs varied dramatically: Gemma required $2.62, Claude $9.09, Qwen $13.00, and GPT-5.4 $82.30. The similar success rates across backends despite 31.4× cost differences suggest that remaining performance headroom lies primarily in visual and simulation components rather than VLM capability alone. Qualitative results demonstrate successful conversions across rigid manipulation, deformable materials (cloth, rope, soft packages), and humanoid motion, with representative visual comparisons showing real-to-simulated alignment. The framework‘s deterministic tools enable reliable geometry and physics computation while VLM orchestration handles ambiguous perceptual decisions.

Implications and Conclusions

Agentic Real2Sim addresses a critical bottleneck in robotics research by automating the labor-intensive conversion of real demonstrations into simulation assets suitable for policy learning and evaluation. By achieving comparable results with open-weight models at minimal cost while maintaining modularity across rigid, deformable, and humanoid domains, this work establishes a scalable foundation for leveraging large real-world robot datasets in simulation-based learning pipelines.

No Training, Better Flights: Test-Time Scaled VLMs for UAV Navigation

Authors: Feinan Cheng, Dongliang Xu, Wenli Nong, Zhiheng Zhang, Ang Liu, Tianyu Wang, Yue Yao

Source and references: https://arxiv.org/abs/2607.19288v1

Test-Time Scaled VLMs for UAV Navigation: Summary

Introduction

This paper introduces a test-time scaling approach to improve Vision-Language Model (VLM) performance for unmanned aerial vehicle (UAV) navigation without requiring additional model training. Rather than relying on a single inference pass, the authors propose an iterative refinement process that enhances navigation reasoning through parallel exploration, serial self-correction, and multi-criteria evaluation.

Key Points

Three-Stage Pipeline: The method implements an “Explore–Refine–Select” framework that generates multiple candidate trajectories in parallel, refines each through iterative self-correction, and selects the optimal path using a multi-criteria scoring function.
No Model Retraining Required: By operating entirely at inference time, the approach improves performance on frozen, pre-trained VLMs without modifying any parameters or requiring additional training data.
State-of-the-Art Performance: The method achieves superior results across all test sets, with 2.02% improvement in success rate (SR) on seen environments and 1.28-0.94% improvements on unseen objects and maps compared to baseline approaches.
Safety-First Evaluation: The scoring function prioritizes collision avoidance (50% weight) while balancing goal alignment (30%) and forward progress (20%), ensuring UAVs make deliberate, reliable decisions in complex environments.
Computational Budget Trade-off: Performance scales positively with inference-time token consumption, demonstrating that allocating more computational resources during inference translates directly to better navigation accuracy.

Methodology

The approach formulates UAV navigation as a three-stage process applied at test time. First, the model generates N distinct candidate coordinates through parallel inference calls, creating a diverse set of initial hypotheses. Second, each candidate undergoes M rounds of iterative self-correction via a self-reflective prompt strategy that explicitly instructs the model to reconsider its initial plan. Finally, a weighted multi-criteria scoring function evaluates all refined candidates across three dimensions—Safety (based on depth sensor data), Goal-Alignment (cosine similarity to target direction), and Forward-Progress (Euclidean distance with tanh normalization)—selecting the highest-scoring trajectory for execution.

Results and Findings

Experiments on the TRAVEL-UAV dataset demonstrate consistent improvements across multiple evaluation metrics. On the Test-Seen (TS) subset, the method improves success rate to 24.96% (+2.02%), overall success rate to 47.39% (+2.47%), and success-weighted path length to 20.93 (+1.43%), while reducing navigation error to 106.32 meters (-4.67%). On more challenging Unseen Object (UO) and Unseen Map (UM) subsets, SR improvements of 1.28% and 0.94% respectively demonstrate generalization capability despite distribution shifts. Ablation studies confirm that combining parallel exploration (Par=3) with serial refinement (Ser=2) yields superior performance compared to single-dimension scaling strategies. Token analysis reveals a clear positive correlation between inference-time computational budget (ranging from 6,705 to 38,315 tokens) and navigation success rates.

Implications and Conclusions

This work establishes test-time scaling as a practical paradigm for improving VLM-based UAV navigation without architectural modifications or retraining, offering a complementary approach to existing training-focused optimization methods. The framework‘s demonstration that increased inference-time computation directly translates to safer, more reliable autonomous navigation opens pathways for deploying robust aerial navigation systems in real-world applications including logistics, infrastructure inspection, agriculture, and emergency response without incurring significant training overhead.

Gradient Bottlenecks, Token Learning Dynamics, and Agentic Self-Evolution: Understanding LLM Training and Deployment

State of AI — Mon, 13 Jul 2026 18:03:33 GMT

Welcome to today‘s edition of State of AI 👋

This edition covers some fundamental breakthroughs in understanding how language models actually learn and optimize. We‘re diving deep into the mechanics of LLM training, from the surprising discovery that language model output layers destroy 95% of gradient information during backpropagation, to new insights showing that token learning follows concentrated sigmoid patterns hidden beneath smooth scaling laws. Beyond foundation models, we‘re also seeing exciting progress in agentic systems that can self-evolve, make cost-aware decisions, and learn from their own failures in ways that mirror human cognitive processes like sleep and memory consolidation.

Here‘s what caught our attention:

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories — Treats agent failures as temporal processes rather than static outcomes, discovering that 57.9% of failures are epistemic (misusing available information) and that decisive errors occur 9 steps before they become observable, creating a critical intervention window.
Lost in Backpropagation: The LM Head is a Gradient Bottleneck — Reveals that the output layer projects V-dimensional gradients through rank-D compression, destroying 95-99% of gradient information and causing ×16 convergence slowdowns independent of model architecture.
Smooth Scaling Laws Hide Stepwise Token Learning — Decomposes scaling laws into individual token learning events, showing that tokens follow concentrated sigmoid trajectories and that the observed power-law structure emerges from how learning times are distributed across training.
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories — Enables continuous post-deployment learning through biologically-inspired sleep phases that consolidate memories and generate synthetic training data, achieving 48.9% accuracy on new knowledge incorporation versus 46.7% for competing approaches.
SAGEAgent: A Self-Evolving Agent for Cost-Aware Modality Acquisition in Multimodal Survival Prediction — Clinical agent that autonomously decides which diagnostic modalities to acquire, achieving 55% reduction in acquisition burden while maintaining prediction accuracy through dual-memory self-evolution without gradient updates.
The LLMbda Calculus: AI Agents, Conversations, and Information Flow — First formally verified LLM agent harness with machine-checked security theorems in Lean, preventing prompt injection attacks through provenance-based information-flow control while maintaining utility parity with leading defense systems.
Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models — Formalizes the runtime decision between generating multiple samples from one model versus switching to another model under fixed cost budgets, achieving 22-31% cost savings with 2-24 point accuracy improvements.

Let‘s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Authors: Xiangxin Zhao, Han Li, Shuaiting Li, Tianyi Zhao, Earl T. Barr, Federica Sarro, He Ye

Source and references: https://arxiv.org/abs/2607.09510v1

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Introduction

This paper presents the first large-scale empirical study treating coding agent failures as temporal processes rather than static outcomes. The researchers analyze 1,794 execution trajectories from seven frontier language models across three coding agent scaffolds to understand how failures emerge, evolve, and become unrecoverable in terminal-based environments.

Key Points

Process-oriented framework: Introduces a three-timestamp decomposition (t_err, t_lock, t_obs) that captures when decisive errors occur, when they become unrecoverable, and when they become observable—moving beyond simple pass/fail metrics.
Epistemic dominance: Demonstrates that 57.9% of failures stem from epistemic errors (misusing available information) rather than competence gaps, with false premises being the single largest failure trigger (30.7%).
Early decisive errors: Reveals that median decisive errors occur at step 7, yet remain hidden until step 16—suggesting outcomes are determined much earlier than trajectory termination.
Recovery as a process: Shows that 82% of failed trajectories continue executing after recovery becomes impossible, with “repairing the wrong problem” accounting for 39% of all wasted computation.
Cross-system consistency: Finds that epistemic errors remain dominant across all 21 model-scaffold combinations (44-80%), though overall pass rates vary substantially (19-45%).

Methodology

Recursive Evidence Replay for Long-Context Reasoning, Multi-Turn Red Teaming for Code Security, and Neuron-Aware LLM Optimization

State of AI — Mon, 06 Jul 2026 19:12:35 GMT

Welcome to today‘s edition of State of AI 👋

This week brings exciting advances across multiple frontiers. We‘re seeing breakthroughs in how LLMs handle long contexts through recursive evidence selection, sophisticated automated security testing frameworks for code generation models, and a deeper understanding of internal neuron dynamics that enable more efficient model optimization. On the robotics front, researchers are cracking the data efficiency problem through world models and task-agnostic pretraining, while vision-language models are learning to ground their reasoning in visual evidence rather than relying on text-only chain-of-thought.

Here‘s what caught our attention:

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning — Uses model-internal attention signals to iteratively identify and emphasize relevant evidence across multiple rounds, achieving 24.6% relative accuracy improvement without retraining or external retrieval systems.
RedCoder: Automated Multi-Turn Red Teaming for Code LLMs — A four-agent framework that systematically discovers code security vulnerabilities through adversarial multi-turn conversations, achieving 39-65% vulnerability induction rates where single-turn approaches fail.
BLAgent: Agentic RAG for File-Level Bug Localization — Combines AST-aware code encoding with dual-perspective query transformation and two-phase reranking to achieve 78%+ top-1 accuracy on bug localization, improving downstream repair success by 25%.
DecompRL: Solving Harder Problems by Learning Modular Code Generation — Trains RL policies to decompose problems into independently solvable functions, generating K^N solutions while reducing GPU token consumption by 50× compared to sampling-based approaches.
Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning — Introduces Random Turn Masking and Buffered Roll-In to enable VLMs to learn error recovery from visual feedback, improving out-of-distribution accuracy by 3-25% without dataset-specific adaptation.
Online Safety Monitoring for LLMs — Demonstrates that simple threshold-based monitors calibrated with conformal risk control detect unsafe outputs 50% faster than complex baselines while providing statistical guarantees on false alarm rates.
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training — Reveals that RL improvements concentrate in middle-layer regions, with the best single layer recovering up to 114% of full-parameter gains and enabling 69.1% accuracy versus 66.4% for full training.
WorldSample: Closed-loop Real-robot RL with World Modelling — Augments physical robot rollouts with world-model-generated counterfactuals through policy-paced learning, achieving 82% success rates while reducing training steps by 59%.

Let‘s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

Authors: Yanjun Zhao, Ruizhong Qiu, Tianxin Wei, Yuanchen Bei, Zhining Liu, Lingjie Chen, Ismini Lourentzou, Hanghang Tong, Jingrui He

Source and references: https://arxiv.org/abs/2607.02509v1

ReContext: Recursive Evidence Replay for Long-Context LLM Reasoning

Introduction

This paper addresses a critical limitation in modern large language models: while they can now process extremely long contexts (128K+ tokens), they often fail to effectively utilize relevant evidence that‘s already present in the input. ReContext proposes a training-free inference method that bridges the gap between context access and effective context utilization by recursively identifying and replaying the most relevant evidence before final answer generation.

Key Points

Novel training-free method: ReContext uses model-internal attention signals to identify question-relevant evidence from long contexts without requiring model retraining or external retrieval systems
Recursive evidence selection: The approach iteratively builds an evidence pool across multiple rounds, with each round conditioning on previously selected evidence to surface deeper connections
Evidence preservation: Unlike compression methods, ReContext keeps the full original context available throughout generation while emphasizing selected evidence—it’s a selection-for-emphasis approach rather than pruning
Theoretical grounding: The authors provide an associative-memory framework showing that recursive evidence replay monotonically increases hidden representation similarity toward the answer embedding (Theorem 1)
Comprehensive evaluation: Achieves best average ranking across three model backbones (Qwen3-4B, Qwen3-8B, Llama3-8B) on eight long-context datasets, with 24.6% relative accuracy improvement over vanilla baselines

Methodology

ReContext operates through four main stages: (1) Evidence Sifting - the model reads the context and question, using question-conditioned attention weights from selected layer-head pairs to score token relevance; (2) Evidence Materialization - top-K scoring tokens are mapped back to their containing sentences to create grounded text spans rather than fragments; (3) Recursive Replay - these spans are inserted into a new prompt structure [Original Context; Evidence Pool; Question] and processed again, allowing the model state to condition the next round‘s attention scores; (4) Final Generation - after R rounds of recursive selection, the answer is generated from the accumulated evidence pool plus full original context.

The method crucially restricts evidence selection to the original context tokens while allowing the replayed scaffold to condition the model‘s hidden states. This maintains explainability and prevents evidence drift while enabling later rounds to discover evidence related to previously selected spans.

Results and Findings

ReContext demonstrates consistent improvements across all evaluation settings:

128K context benchmark (primary): Achieves best average rank of 1.00 on Qwen3-4B, 1.46 on Qwen3-8B, and 1.29 on Llama3-8B. Improves mean accuracy from 0.24 to 0.30 (24.6% relative gain)
Model-specific performance: On Qwen3-4B, achieves best scores on every reported metric; NQ accuracy improves from 0.02 to 0.08 over the strongest baseline. On larger models, shows best average rank while maintaining competitive performance even on metrics where other methods excel
Robustness across conditions: Maintains top-2 performance on 64K context budgets (35% relative gain over vanilla) and remains strongest on key tasks when model thinking is enabled (16.7% relative improvement)
Ablation insights: Two recursive rounds provide optimal improvements for most tasks; optimal K (evidence-token budget) is task-dependent; selecting evidence from original context only (not replayed scaffold) consistently outperforms full-prompt selection
Efficiency trade-off: Introduces modest latency overhead compared to vanilla baseline but remains substantially faster than attention-intervention methods that modify decoding logic; minimal memory overhead (<128 additional tokens)

Implications and Conclusions

Benchmark Saturation, Self-Evolving Agents, and Trillion-Parameter Performance at 35B Scale

State of AI — Tue, 30 Jun 2026 16:11:05 GMT

Welcome to today‘s edition of State of AI 👋

This weekbrings critical insights into AI evaluation, agent architecture, and efficient scaling. We‘re seeing how benchmark saturation fundamentally challenges our ability to measure progress, how runtime-level security enables self-evolving LLM agents without permission escalation, and how extending agent reasoning horizons can match trillion-parameter performance with just 35 billion parameters. Meanwhile, practical advances in video editing, multimodal control, and long-context document understanding show the field rapidly moving beyond single-task capabilities toward unified, multi-capability systems deployed at scale.

Here‘s what caught our attention:

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation — A rigorous framework showing that nearly half of widely-used LLM benchmarks are already saturated, with age and test set size as the dominant predictors rather than commonly-assumed safeguards like private test sets or adversarial design.
Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent — Demonstrates that extending agent reasoning trajectories to 45K tokens with domain-specific expertise allows a 35B MoE model to match trillion-parameter system performance, challenging the parameter-scaling paradigm.
Agent libOS: A Runtime Substrate for Capability-Controlled Self-Evolving LLM Agents — Introduces process-identity-based access control and primitive-level authorization boundaries that prevent self-evolution from becoming a permission-escalation vulnerability, enabling safe agent capability expansion during deployment.
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions — Reveals a 23-24 point performance gap between single-turn and multi-turn coding tasks, exposing that autonomous implementation ability doesn’t reliably transfer to interactive developer workflows with iterative refinement.
TraceLab: Characterizing Coding Agent Workloads for LLM Serving — First cross-provider trace of real coding-agent usage showing prefix-cache dominance (59.5% of costs), context windows exceeding 100K tokens with minimal output, and 12.8% cost savings available from better cache retention during human-paced gaps.
Self-Evolving World Models for LLM Agent Planning — Proposes training-free context evolution through episodic and semantic memory that enables frozen world models to improve predictions without parameter updates, with selective foresight filtering that prevents degraded decision-making from noisy predictions.
How to Train Your Long-Context Visual Document Models — Establishes that matching training and evaluation context lengths outperforms longer training contexts by 1.4-3.0 points, and demonstrates bidirectional long-context transfer between vision and text modalities at 344K context length.
Artificial Intelligence Index Report 2026 — Comprehensive Stanford analysis documenting capability acceleration without plateau, the effectively-closed US-China AI gap, critical TSMC supply chain concentration, and a stark 30% mismatch between expert optimism and public skepticism about AI’s labor impact.

Let‘s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Artificial Intelligence Index Report 2026

Source and references: https://arxiv.org/abs/2606.15708v3

AI Index Report 2026: Summary for Tech-Savvy Audiences

Introduction

The AI Index 2026 Report, released by Stanford University‘s Human-Centered AI Institute, documents AI‘s transformation from an emerging technology to a mainstream force reshaping governance, research, and commerce. The report reveals a critical gap: AI capabilities are advancing faster than the institutional systems designed to manage them.

Key Points

Capability acceleration without plateau: Industry produced over 90% of frontier models in 2025, with leading systems now meeting or exceeding human baselines on PhD-level science questions, multimodal reasoning, and competition mathematics. SWE-bench Verified coding performance jumped from 60% to nearly 100% in a single year.
U.S.-China AI gap effectively closed: The two superpowers have traded leadership multiple times since early 2025. As of March 2026, Anthropic’s top model leads by just 2.7% over DeepSeek-R1. Both nations dominate model production, with China leading in publications, citations, and patents while the U.S. maintains higher-impact patents and more top-tier models overall.
Supply chain vulnerability concentrated in Taiwan: The U.S. hosts 5,427 data centers (over 10x any other country), but a single company—TSMC—fabricates nearly every leading AI chip globally, creating a critical geopolitical dependency. A TSMC U.S. expansion began operations in 2025, though it’s too early to assess impact.
Responsible AI severely lagging technical capabilities: Almost all leading frontier labs report capability benchmarks, but responsible AI benchmarking remains spotty. Documented AI incidents surged to 362 in 2025 (from 233 in 2024). Worse, improving one safety dimension can degrade another (e.g., improving safety reduces accuracy), creating design trade-offs developers are only beginning to understand.
Historic environmental footprint with concentrated costs: Training Grok 4 produced approximately 72,816 tons of CO₂ equivalent—more than a lifetime of average car emissions. AI data center power capacity reached 29.6 GW (equivalent to peak New York state demand). Annual GPT-4o inference water use alone may exceed the drinking water needs of 1.2 million people.

Methodology

Diffusion Models Meet Reasoning: Learning to Schedule, Search, and Synthesize

State of AI — Tue, 23 Jun 2026 16:53:27 GMT

This week brings advances in how we train and optimize AI systems across reasoning, robotics, and multimodal domains. We‘re seeing a shift toward learning when and how to use compute primitives, from scheduling token generation in diffusion models to adaptively invoking code in vision-language systems. Simultaneously, researchers are cracking long-standing problems in model composition, safety alignment for robotic systems, and the surprising ineffectiveness of naive prompt optimization in multi-agent setups. The papers also reveal emerging tensions: training diversity improves safety but not semantic reasoning; longer context training helps but requires careful curriculum design; and compression techniques benefit substantially from second-order optimization principles.

Here‘s what caught our attention:

SPIRAL formulates multi-trace reasoning as a set RL problem, achieving 11× higher scaling efficiency by training models to generate diverse parallel reasoning paths whose usefulness is determined collectively rather than individually, a fundamental rethinking of inference compute.
Scheduling Thoughts derives an information-theoretic upper bound on diffusion language model performance and uses it to train an optimal unmasking policy via GRPO, improving Sudoku accuracy from 82% to 91.8% with a frozen base model.
dVLA-RL solves the intractable problem of computing action probabilities in discrete diffusion policies by reformulating to trajectory-level probabilities, enabling RL optimization of vision-language-action models with 30.6pp gains on bimanual manipulation.
VeriEvol decouples prompt evolution from answer verification as independent pipeline axes, treating verification as a dataset property rather than a training concern, scaling SFT data 25× while maintaining reliability through multi-source falsification.
AIR introduces group-constrained RL rewards that decouple tool invocation from accuracy rewards, preventing agentic training collapse while achieving 95%+ code execution reliability across mathematical reasoning benchmarks.
Randomized YaRN improves length generalization through randomized position sampling with curriculum-based training, achieving 90% accuracy on out-of-distribution 128K contexts, revealing the critical importance of curriculum rather than static extrapolation.
SVD-Surgeon brings Optimal Brain Surgeon to the singular-value basis with closed-form updates, substantially improving low-rank compression trade-offs without retraining (e.g., 70% compression perplexity: 944→46 on OPT-6.7B).
LIBERO-Safety establishes systematic evaluation of physical and semantic safety in VLAs through parametric scenario generation, revealing that the “generalization-safety tension” prevents existing models from scaling collision-free trajectory synthesis.

Let‘s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

SPIRAL: Learning to Search and Aggregate

Authors: Jubayer Ibn Hamid, Ifdita Hasan Orney, Michael Y. Li, Omar Shaikh, Yoonho Lee, Dorsa Sadigh, Chelsea Finn, Noah Goodman

Audio Reasoning, Hallucination Mitigation, and Efficient Inference: From Chain-of-Thought Speech Models to INT8 Diffusion Transformers

State of AI — Mon, 15 Jun 2026 07:27:07 GMT

Welcome to today’s edition of State of AI 👋

This week brings a fascinating convergence of advances across three critical frontiers. In audio and multimodal understanding, we’re seeing sophisticated approaches to reasoning and robustness—from deduplication-enhanced datasets for audio-language models to entropy-guided explainability in speech recognition and anti-spoofing systems. In parallel, the field is tackling a persistent problem: hallucinations in vision-language models, with solutions ranging from textual embedding refinement to stage-wise diagnostic frameworks for medical MLLMs. Meanwhile, a wave of efficiency innovations is making frontier models practical on consumer hardware, with fused INT8 kernels, KV cache compression, and quantization techniques that maintain quality while slashing memory requirements.

Here’s what caught our attention:

AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models — Uses acoustic similarity-based deduplication and chain-of-thought generation to construct a 191K-sample dataset that measurably improves complex audio reasoning beyond basic understanding tasks.
Gaze Heads: How VLMs Look at What They Describe — Identifies fewer than 100 specialized attention heads in VLMs that deterministically control which image regions the model describes, enabling causal steering with 83% accuracy on visual QA tasks.
Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings — Reveals that modality imbalance causes over-reliance on linguistic priors, then demonstrates that fusing visual context directly into textual embeddings before transformer processing significantly reduces hallucinations across multiple architectures.
Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs — Diagnoses why production INT8 implementations were actually “fake quantization,” then delivers a fused Triton kernel that properly engages Ampere’s integer tensor cores, achieving 2.8-4.2× speedups and enabling 1024px image generation on a single RTX 3090.
Sub-Token Routing for KV Cache Compression — Operates at finer granularity than token-level compression by selectively retaining portions of value vectors, showing particularly strong gains when combined with existing methods on vision-language models at aggressive budget constraints.
Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning — Formalizes abstention as a Markov decision process with dynamic value-thresholding, achieving 64% selective accuracy at 90% abstention on OlympiadBench—a 30-point improvement over baselines.
Planning with the Views via Scene Self-Exploration — Reveals a critical “planning gap” where frontier VLMs achieve ~70% accuracy on single-view transitions but collapse to <21% on multi-turn spatial planning, then fixes it with view-graph distillation, lifting Qwen2.5-VL from 2.5% to 47.8%.
TRACE: Trajectory-Routed Causal Memory for Delayed-Evidence Visuomotor Imitation — Addresses partial observability in robot manipulation by using path signatures as deterministic memory keys, enabling policies to leverage visual evidence that disappeared before decision points, with improvements from 25% to 69% success on long-horizon tasks.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models

Authors: Hui Geng, Yi Su, Han Yin, Tianjiao Wan, Qisheng Xu, Jiaxin Chen, Zijian Gao, Hengzhu Liu, Xie Chen, Kele Xu

Source and references: https://arxiv.org/abs/2606.14591v1

AudioDER: A Deduplication-Enhanced Reasoning Dataset for Large Audio-Language Models

Introduction

This paper introduces AudioDER, a post-training dataset specifically designed to improve complex audio reasoning capabilities in Large Audio-Language Models (LALMs). The research addresses a critical gap: while LALMs have achieved strong performance on basic audio understanding tasks, they struggle with reasoning-heavy problems that require compositional understanding and multi-step inference.

Key Points

Redundancy Problem: Existing audio-language datasets contain substantial acoustic similarity and overlap, resulting in redundant supervisory signals that increase annotation costs while limiting corpus diversity and post-training effectiveness.
Deduplication Pipeline: The authors implement an acoustic similarity-based deduplication approach across raw audio datasets to systematically improve corpus diversity before annotation.
Unified Format Integration: Multiple annotation types (captions, question-answer pairs) are consolidated into a unified multiple-choice format, creating a standardized structure for consistent training.
Chain-of-Thought Generation: The pipeline leverages Qwen3-30B to generate chain-of-thought (CoT) rationales, providing explicit reasoning explanations alongside answers for enhanced learning.
Comprehensive Dataset: AudioDER comprises 191,000 samples spanning sound events, speech, and music, with each sample containing an audio clip, multiple-choice question, four answer options, audio caption, and CoT rationale.

Methodology

The research employs a redundancy-aware data construction pipeline with three main stages. First, acoustic similarity-based deduplication is performed across raw audio datasets to identify and remove overlapping content, improving corpus diversity. Second, existing audio captions and question-answer pairs are standardized into a unified multiple-choice format, ensuring consistency across diverse annotation sources. Third, Qwen3-30B generates reasoning-oriented chain-of-thought rationales for each sample, providing explicit explanations that guide the model’s reasoning process during post-training.

Results and Findings

Post-training on AudioDER consistently improved Qwen2-Audio-7B-Instruct performance across multiple audio reasoning benchmarks. The model demonstrated measurable gains on MMAU-mini, MMSU, and MMAR benchmarks, validating that reasoning-oriented supervision and reduced dataset redundancy enhance complex audio understanding capabilities. The results indicate that the deduplication approach successfully increased effective training signal diversity, and the CoT rationales effectively transferred reasoning knowledge to the base model. The 191k-sample dataset size represents a substantial contribution to the audio-language modeling landscape, particularly for reasoning-focused applications.

Implications and Conclusions

This work establishes that dataset quality—specifically through redundancy elimination and reasoning-oriented annotations—plays a crucial role in post-training effectiveness for audio-language models. AudioDER provides both a practical resource for the research community and a methodological blueprint for constructing high-quality post-training datasets that prioritize diversity and reasoning capabilities over raw size.

From Self-Supervised Speech Models to Mixture-of-Experts for Robust Anti-Spoofing

Authors: Hugo Daumain, Driss Matrouf, Khaled Khelif, Mickael Rouvier

Source and references: https://arxiv.org/abs/2606.14639v1

Introduction

This paper addresses the challenge of detecting synthetic and manipulated speech by converting self-supervised speech models into Mixture-of-Experts (MoE) architectures. As modern speech synthesis techniques become increasingly sophisticated, traditional anti-spoofing systems struggle to generalize across unseen synthesis methods, motivating the need for more robust detection approaches.

Key Points

Full MoE conversion approach: The authors replace feed-forward networks in selected transformer encoder layers with multiple expert networks controlled by layer-wise gating mechanisms, preserving pretrained knowledge while improving generalization to unseen spoofing methods.
Comprehensive architectural analysis: The study systematically evaluates critical design choices including expert placement (early, late, full, or alternating insertion), pooling strategies for the gating network, number of experts, and top-k routing values to identify optimal configurations.
Superior performance over LoRA-based methods: The proposed dense expert approach outperforms low-rank adaptation (LoRA) alternatives by allowing joint fine-tuning of attention layers alongside experts, achieving better expressiveness and specialization.
Extensive evaluation across 14 datasets: Testing spans diverse spoofing conditions including text-to-speech, voice conversion, codec-based manipulation, diffusion-based generation, and real-world scenarios across multiple languages.
Expert activation analysis: Investigation of whether experts specialize for specific synthesizers reveals balanced activation patterns with only modest routing differences across synthesis methods, suggesting experts capture complex general acoustic patterns rather than method-specific artifacts.

Methodology

The approach builds upon WavLM-Large, a 24-layer self-supervised speech model with a convolutional feature extractor and transformer encoder. The authors convert six of the first 13 transformer layers (selected for their acoustic relevance) by replacing their feed-forward networks with four parallel expert networks. A gating network computes routing probabilities using statistical pooling of frame-level representations, selecting the single highest-scoring expert (top-k=1) via softmax. An auxiliary load-balancing loss prevents expert collapse during training. The model trains on 1.4M samples across six datasets using binary cross-entropy loss, with progressive unfreezing of SSL parameters and data augmentation including codec, noise, and reverberation perturbations.

Results and Findings

The best MoE configuration achieves a macro Equal Error Rate (EER) of 4.81% compared to the baseline’s 5.46%—an 11.9% relative improvement—with a micro EER of 12.34%. Statistical pooling outperformed attentive pooling, and the optimal configuration used four experts with top-k=1 routing. Notably, configurations with k≥2 degraded performance, suggesting that activating multiple experts simultaneously reduces beneficial specialization. The comparison with LoRA-based approaches showed consistent superiority across all tested ranks, with LoRA achieving only 6.66-6.84% macro EER compared to 4.81% for the full approach. Expert activation analysis revealed relatively balanced routing across synthesizers with mean Jensen-Shannon divergences of 0.086-0.299, indicating experts capture generalized acoustic patterns rather than method-specific features. Per-dataset results varied significantly, with particularly strong performance on ASVspoof2019 LA (0.04% EER) and ASVspoof2021 DF (0.30% EER).

Implications and Conclusions

This research demonstrates that converting pretrained self-supervised models into full Mixture-of-Experts architectures substantially improves robustness to diverse and evolving spoofing attacks, with particular relevance for practical deployment scenarios where detection systems encounter continuously changing synthesis techniques. The work advances the field by showing that dense expert networks with joint fine-tuning provide better generalization than parameter-efficient alternatives, though future work remains needed to develop interpretable mechanisms that explicitly guide experts toward distinct spoofing artifact categories.

Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models

Authors: Ravi Ranjan, Utkarsh Grover, Xiaomin Lin, Agoritsa Polyzou

Source and references: https://arxiv.org/abs/2606.14647v1

LEAF-X: Entropy-Guided Explainability for Transformer-Based Audio Models

Introduction

This paper introduces LEAF-X, a model-intrinsic explainability framework designed to interpret transformer-based automatic speech recognition (ASR) systems like OpenAI’s Whisper. The work addresses a critical gap in ASR transparency by providing faithful, temporally grounded explanations that reveal which acoustic regions support each transcribed word.

Key Points

Hyperbolic Embeddings, Sparse Attention Kernels, and Diffusion-Based Retrieval: Three Breakthroughs in Scaling AI Systems

State of AI — Mon, 08 Jun 2026 17:32:59 GMT

Welcome to today’s edition of State of AI 👋 This week brings a fascinating convergence: researchers are fundamentally rethinking how we represent knowledge in retrieval systems (moving from Euclidean to hyperbolic geometry), how we optimize inference efficiency (through programmable sparse attention and cross-layer routing), and how we augment language models with dynamic retrieval (leveraging the unique properties of diffusion-based generation). In parallel, the community is pushing forward on embodied AI with massive new datasets and frameworks for medical robotics, while breakthroughs in reasoning models, 3D scene understanding, and parameter-efficient adaptation suggest we’re entering a phase where specialized domain knowledge can be efficiently injected into foundation models. Here’s what caught our attention: HypRAG: Embedding documents in hyperbolic space rather than Euclidean space achieves 29% improvements in RAG performance by naturally capturing semantic hierarchies—a geometric insight that challenges decades of deep learning convention. Vortex: A programmable framework that abstracts sparse attention implementation complexity away through a Python DSL, enabling both human researchers and autonomous AI agents to design new sparse attention algorithms that achieve 3.46× throughput improvements. RAG Security and Privacy: The first formal threat model for retrieval-augmented generation systems, formalizing document-level privacy risks and attack surfaces that didn’t exist in traditional LLMs. Astra (Agentic Visual Spatial Reasoning): Vision-language models that actively generate imagined viewpoints through a world simulator during reasoning, improving spatial understanding tasks by 9-10 points through learned, selective tool use. SARDI (Self-Augmenting Retrieval for Diffusion): Exploiting the parallel denoising structure of diffusion models to dynamically refresh retrieved evidence at every generation step, enabling 8× faster multi-hop reasoning than autoregressive approaches. Open-H-Embodiment: 780 hours of medical robotics data across 50+ institutions and 20 platforms, enabling the first surgical foundation model that generalizes across different robotic systems with 25% end-to-end task success. PC Layer: A weight preconditioning technique using polynomial matrices that reshapes singular-value spectra during LLM training, achieving 2× token efficiency improvements with zero inference overhead. RREDCoT: A novel credit assignment algorithm that redistributes rewards across chain-of-thought segments for reasoning models, improving AIME performance from 85% to 90.8% through fine-grained intermediate learning signals. Let’s get into it 👇 Bi-Weekly AI Research Roundup Latest research summaries in ML, Robotics, CV, NLP and AI Contents HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents RAG Security and Privacy: Formalizing the Threat Model and Attack Surface Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding A Vision-language Framework for Comparative Reasoning in Radiology PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training Pretraining Recurrent Networks without Recurrence RREDCoT: Segment-Level Reward Redistribution for Reasoning Models You Only Index Once: Cross-Layer Sparse Attention with Shared Routing Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution Self-Augmenting Retrieval for Diffusion Language Models Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics TempoVLA: Learning Speed-Controllable Vision-Language-Action Policies PHUMA: Physically Reliable Humanoid Locomotion Dataset HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation Authors: Hiren Madhu, Ngoc Bui, Ali Maatouk, Leandros Tassiulas, Smita Krishnaswamy, Menglin Yang, Sukanta Ganguly, Kiran Srinivasan, Rex Ying Source and references: https://arxiv.org/abs/2602.07739v2 HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation Introduction This paper introduces hyperbolic dense retrieval as a geometric approach to improve retrieval-augmented generation (RAG) systems. The authors argue that natural language exhibits hierarchical structure that Euclidean embeddings fail to preserve, proposing instead to embed documents in hyperbolic space where negative curvature naturally captures semantic hierarchies from general topics to specific entities. Key Points Hierarchical Structure Alignment: Hyperbolic geometry’s exponential volume growth makes it inherently suited for representing branching topic hierarchies in language, whereas Euclidean embeddings suffer from crowding effects that make semantically distant documents appear spuriously similar. Two Architecture Variants: The paper presents HyTE-FH (fully hyperbolic transformer) operating entirely in the Lorentz model of hyperbolic space, and HyTE-H (hybrid architecture) that projects pre-trained Euclidean embeddings into hyperbolic space, offering flexibility between theoretical purity and practical efficiency. Outward Einstein Midpoint Pooling: A novel geometry-aware aggregation operator that explicitly amplifies token contributions based on radial distance from the origin, theoretically proven to preserve hierarchical structure during document-level representation aggregation—addressing a critical failure mode where naive pooling causes representational collapse. Significant RAG Performance Gains: HyTE-H achieves up to 29% improvements over Euclidean baselines in context relevance and answer relevance on RAG-Bench, while using substantially smaller models (149M parameters) than current state-of-the-art retrievers. Norm-Based Concept Specificity: Hyperbolic models encode document generality-to-specificity through radial distance from the origin, with the fully hyperbolic model showing a 20.2% radius increase from general to specific concepts—a property completely absent in Euclidean embeddings. Methodology The authors develop two complementary models using the Lorentz model of hyperbolic space. HyTE-FH employs fully hyperbolic transformer components including Lorentz linear layers, hyperbolic layer normalization, residual connections, and hyperbolic self-attention with geodesic distance-based similarity. Both variants use a three-stage training pipeline: hyperbolic masked language modeling, contrastive pre-training with large batches (16,384), and task-specific fine-tuning with hard negatives. The Outward Einstein Midpoint pooling operator is mathematically defined with a radius-dependent weighting function φ_p(x_i) = x^p_{i,0} that systematically prioritizes tokens farther from the origin, with theoretical proofs demonstrating superiority over standard Einstein midpoint and naive averaging approaches. Results and Findings On the MTEB benchmark, HyTE-FH achieves a mean score of 56.41 compared to 54.11 for Euclidean baseline (EucBERT), with hybrid variant HyTE-H_Qwen reaching 80.3 average performance across RAG-Bench datasets. More specifically, HyTE-FH demonstrates 76.5% answer relevance versus 64.7% for EucBERT across RAG benchmarks. The hierarchy-sensitive retrieval experiments using the MeSH taxonomy show HyTE-FH achieving 96.3% recall@1 and 97.0% specificity hit rate versus 81.0% and 88.3% for EucBERT—effectively eliminating grandparent-level confusions that plague Euclidean models. Approximate nearest neighbor search on MS MARCO reveals HyTE-FH outperforms EucBERT at all k values (6.4 point improvement at k=50) while achieving 3x faster latency due to improved tree-like embedding structure. Ablation studies confirm Outward Einstein Midpoint pooling delivers a 10% improvement over the next-best Einstein midpoint approach, validating the geometric design choice. Implications and Conclusions This work demonstrates that embedding geometry is a fundamental design choice for retrieval systems, with hyperbolic space providing substantial practical benefits for RAG faithfulness and grounding without requiring larger models. The finding that natural language embeddings exhibit intrinsically negative curvature properties validates the theoretical motivation, suggesting future dense retrievers should consider geometric alignment with semantic structure as a core architectural principle rather than defaulting to Euclidean conventions inherited from general neural network practice. Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents Authors: Zhuoming Chen, Xinrui Zhong, Qilong Feng, Ranajoy Sadhukhan, Yang Zhou, Michael Qizhe Shieh, Zhihao Jia, Beidi Chen Source and references: https://arxiv.org/abs/2606.06453v1 Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents Introduction As large language models (LLMs) generate longer sequences, sparse attention has become critical for reducing inference costs, yet implementing and validating new sparse attention algorithms remains engineering-intensive. Vortex addresses this challenge by providing a programmable system that enables rapid prototyping, deployment, and evaluation of sparse attention mechanisms while achieving real-world performance improvements. Key Points Programmable Frontend (vFlow): Vortex introduces vFlow, a Python-embedded domain-specific language that abstracts away low-level memory layout details, allowing researchers and AI agents to express diverse sparse attention algorithms as composable operations like GeMM, Top-K, and Gather without worrying about physical tensor layouts. Unified Tensor Abstraction (vTensor): The system employs a page-centric tensor abstraction that bridges the gap between logical tensor views and physical paged memory layouts used in modern LLM serving systems, enabling efficient execution on non-contiguous memory while maintaining programmer simplicity. AI Agent-Driven Algorithm Design: Vortex enables autonomous AI agents (Claude Code and Codex) to automatically generate and optimize sparse attention algorithms, with best-performing variants achieving up to 3.46× higher throughput than full attention while maintaining accuracy. System Integration and Compatibility: The framework seamlessly integrates with modern LLM serving stacks, supporting paged attention, prefix caching, and various attention architectures (GQA, MLA), ensuring theoretical speedups translate to practical end-to-end gains. Execution Optimizations: Vortex incorporates kernel fusion to reduce memory traffic, workload planning for balanced GPU scheduling, and stochastic top-k optimization that trades exactness for speed when needed. Methodology Vortex comprises three components working in concert. The vFlow programming model provides a logical single-request view of sparse attention algorithms, decomposing them into query-independent preprocessing stages and query-dependent execution stages. The vTensor interpreter translates vFlow programs into executable operations while handling the physical paged tensor layouts transparently. Finally, the execution backend integrates with existing serving systems, automatically selecting appropriate attention kernels and applying optimizations like kernel fusion and workload planning based on batch characteristics. The system is evaluated through two primary pathways: autonomous AI agent experimentation that generates diverse sparse attention variants, and traditional benchmarking across models of varying sizes (8B to 229B parameters) on NVIDIA H200 and B200 GPUs. Evaluation metrics include throughput (tokens/second), P95 latency, and accuracy preservation on standard benchmarks. Results and Findings Vortex demonstrates significant performance improvements across multiple dimensions. In autonomous AI agent experiments, the best generated sparse attention algorithm achieves 3.46× higher throughput than full attention on NVIDIA H200 GPUs while maintaining accuracy. For established algorithms, Vortex achieves up to 3.60× throughput improvements for block top-k attention and 2.98× for Quest compared to SGLang full attention baselines, reducing P95 latency by up to 11.7× and 12.8× respectively under long-input workloads. The system extends effectively to newer architectures: on GLM-4.7-Flash using multi-head latent attention (MLA), a rope-aware sparse attention design achieves 4.7× speedup over full attention on NVIDIA B200 GPUs. Even on massive models like MiniMax-M2.7 (229 billion parameters) distributed across four B200 GPUs, block top-k attention maintains a 1.37× speedup while exceeding full-attention accuracy. These results validate that Vortex translates theoretical efficiency gains into practical serving performance across diverse model architectures and scales. Implications and Conclusions Vortex significantly accelerates sparse attention research by reducing implementation complexity from thousands of lines of code to simple, composable operations, enabling both human researchers and AI agents to rapidly iterate on algorithm designs. By bridging the gap between algorithm research and production serving systems while maintaining compatibility with modern infrastructure optimizations, Vortex provides a scalable foundation for advancing sparse attention techniques and autonomous machine learning system design. RAG Security and Privacy: Formalizing the Threat Model and Attack Surface Authors: Atousa Arzanipour, Rouzbeh Behnia, Reza Ebrahimi, Kaushik Dutta Source and references: https://arxiv.org/abs/2509.20324v2 RAG Security and Privacy: Formalizing the Threat Model and Attack Surface Introduction This paper addresses a critical gap in AI security research by proposing the first formal threat model for Retrieval-Augmented Generation (RAG) systems. RAG combines large language models with external document retrieval to improve factual accuracy, but this architecture introduces novel privacy and security vulnerabilities that existing frameworks fail to adequately address. Key Points First Formal Threat Model for RAG: The authors propose a comprehensive framework that formalizes privacy and security threats specific to RAG systems, including a taxonomy of adversary types based on their access to model components, documents, and training data. Document-Level Privacy Risks: RAG systems expose new attack surfaces where adversaries can infer whether specific documents exist in the knowledge base or extract information about document content, structure, or authorship through carefully crafted queries—risks that don’t exist in traditional LLMs. Inherited LLM Vulnerabilities: RAG systems inherit existing LLM vulnerabilities including training data memorization, prompt injection attacks, gradient inversion, and model misalignment, all of which remain active threats during inference. Formal Definitions of Key Threats: The paper formally defines critical threat vectors including document-level membership inference, document reconstruction attacks, and data poisoning attacks that pose serious risks in real-world RAG deployments. Architecture-Specific Attack Surface: Unlike conventional LLMs that internalize knowledge in model parameters, RAG’s reliance on external knowledge bases creates distinct privacy concerns, particularly in sensitive domains like healthcare and finance where document existence itself may be confidential. Methodology The authors establish a formal system model for RAG pipelines, comprehensively documenting each component: the knowledge base (D) containing n documents, a retriever (R) that maps queries to top-k relevant documents, and a generator (G) that produces responses based on both user queries and retrieved content. The methodology formalizes the eight-step RAG pipeline from query submission through embedding transformation, document retrieval via similarity metrics, query augmentation, and final response generation. This systematic formalization enables precise threat modeling by explicitly defining the interaction points between users, retrieval mechanisms, and language models where security and privacy violations could occur. Results and Findings The paper does not present empirical attack demonstrations in the provided sections; rather, it establishes foundational theoretical frameworks. The authors systematically identify that RAG systems face two classes of threats: inherited vulnerabilities from underlying LLMs (memorization during training, prompt injection at inference) and novel vulnerabilities specific to the retrieval component (document existence inference, retrieval manipulation). Notably, the work demonstrates through scenarios that attackers could query a RAG-powered medical assistant to determine whether specific diagnoses appear in retrieval indexes, or probe financial systems to verify the presence of confidential audit reports—exploits impossible against traditional LLMs lacking external knowledge bases. Implications and Conclusions This work establishes essential foundational definitions necessary for rigorously analyzing and mitigating critical risks in RAG-based systems, particularly as major technology companies like Google and Microsoft integrate RAG into production search engines. By formalizing the threat landscape, this research enables the security community to develop principled defense mechanisms and privacy-preserving techniques, such as differential privacy implementations, that address both inherited LLM vulnerabilities and architecture-specific risks before widespread RAG deployment in sensitive domains like healthcare and finance. Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators Authors: Chenming Zhu, Jingli Lin, Yilin Long, Peizhou Cao, Tai Wang, Jiangmiao Pang, Xihui Liu Source and references: https://arxiv.org/abs/2606.06476v1 Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators Introduction This paper tackles a fundamental limitation in Vision-Language Models (VLMs): their struggle with spatial reasoning when asked to infer unobserved layouts, maintain cross-view consistency, or reason from alternative viewpoints with limited egocentric observations. The authors propose Astra, a framework that enables VLMs to actively acquire imagined visual evidence by interacting with a world simulator during the reasoning process, transforming spatial understanding from passive observation to active evidence acquisition. Key Points Two-component architecture: Astra combines Astra-VL (an RL-trained VLM policy based on Qwen3-VL) with Astra-WM (a Bagel-based world simulator) that generates spatially consistent novel-view observations from natural-language camera-motion instructions View Consistency Tuning: Rather than relying on generic image generation, Astra-WM is specifically trained to maintain pose and content consistency across views, improving pose consistency from 9.0/3.0 to 72.5/70.5 compared to off-the-shelf Bagel World-simulator-in-the-loop RL curriculum: A two-phase reinforcement learning approach where Phase 1 teaches valid simulator interaction mechanics, and Phase 2 encourages selective imagination by rewarding cases where tool use improves reasoning over direct answering Significant performance gains: Astra improves Qwen3-VL-8B from 29.8% to 38.8% on MMSI-Bench and from 36.8% to 42.7% on MindCube, while also improving Gemini-3-Flash from 45.1% to 49.5% on MMSI-Bench through better simulator quality Learned decision-making: The framework learns not only when to invoke the simulator, but also which camera motion to request and how to integrate imagined observations into reasoning, demonstrating that effective imagination requires both simulator quality and agentic policy control Methodology The framework models visual spatial reasoning as an interactive decision process where agents can choose to either answer directly or invoke a world simulator for imagined observations. Astra-WM is trained on 544k quality-verified samples from multi-view indoor scene datasets (IsaacSim, ScanNet++, Matterport3D, etc.) using view consistency tuning to ensure generated views maintain spatial fidelity. Astra-VL undergoes training using a two-phase RL curriculum with the world simulator in the loop, progressing from learning basic tool interaction to selectively using the simulator only when it provides genuine reasoning benefits over direct answering. Results and Findings Experimental validation across MMSI-Bench and MindCube benchmarks reveals several critical insights: Simulator Quality Matters: Off-the-shelf Bagel provided minimal improvements to Gemini-3-Flash (from 45.1% to 45.8%), whereas Astra-WM achieved 49.5% accuracy. This 4.4-point gap demonstrates that spatial consistency, not photorealism, determines simulator utility. Tool Access Alone Is Insufficient: Forcing Qwen3-VL to use the simulator zero-shot actually degraded performance (29.8% dropping to 28.6% on MMSI-Bench), showing that naive tool integration can backfire without proper policy training. Two-Phase RL Curriculum Essential: Single-stage training approaches failed—either collapsing to zero tool use (4.9% tool-call rate) or over-using the simulator (98.1% tool-call rate). The full two-phase curriculum achieved optimal performance with selective tool use (61.5% tool-call rate) and the best spatial reasoning scores. Task-Specific Benefits: Camera-centric spatial relations benefited most from imagined viewpoints (Cam.-Cam. improved from 39.1% to 47.9%), while object-centric relations sometimes degraded under forced tool-use, highlighting the importance of learned selectivity. Implications and Conclusions This work demonstrates that effective world-model-augmented spatial reasoning requires more than access to a visual generation tool—it demands both high-quality spatially-consistent simulation and a learned agentic policy that understands when, where, and how to imaginatively query alternative viewpoints. The framework’s success in improving open-source VLMs by 9-10 points across benchmarks suggests a promising direction for advancing spatial reasoning capabilities, while the detailed ablations reveal critical design choices (view consistency tuning, two-phase RL, agentic control) that practitioners should consider when building simulator-augmented reasoning systems. PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding Authors: Shaohui Dai, Yansong Qu, You Shen, Shengchuan Zhang, Liujuan Cao Source and references: https://arxiv.org/abs/2606.06485v1 Introduction PAR3D introduces a unified 3D multimodal large language model (3D-MLLM) designed to understand and reason about object parts in 3D scenes, addressing a critical gap in existing models that operate at only the object level. The paper also presents ScenePart, a synthetic dataset with part-level annotations that enables training and evaluation of part-aware 3D scene understanding systems. Key Points Part-Aware 3D-MLLM Framework: PAR3D extends existing 3D-MLLMs to handle fine-grained part understanding alongside object-level tasks, enabling models to ground, reason about, and segment both objects and their functional components in 3D environments. ScenePart Dataset: The authors introduce a synthetic 3D scene dataset containing 800 scenes, 21K object masks, 44K part masks, and 273K language annotations with object-part correspondences, providing essential supervision for part-aware learning at scene level. Part-Aware 3D Representation Learning: The framework incorporates part-aware contrastive learning and representation-preserving self-distillation to enhance the visual backbone’s ability to capture fine-grained geometric and semantic features of object parts while maintaining general 3D semantics. Hierarchical Segmentation Query Generation: Instead of using a single generic grounding token, PAR3D generates granularity-aware tokens ([OBJ] for objects, [OBJ]+[PART] for parts), preserving object-part relationships and reducing granularity conflicts in grounding. Strong Cross-Task Performance: PAR3D achieves state-of-the-art results on object-level benchmarks (ScanRefer, Multi3DRefer, ScanQA) while substantially outperforming existing 3D-MLLMs on part-aware tasks, demonstrating the effectiveness of the unified framework. Methodology PAR3D builds upon 3D-LLaVA, a foundational 3D-MLLM, and extends it through three key technical contributions. The framework adopts a two-stage training approach: Stage 1 performs part-aware 3D backbone pretraining using a frozen pretrained Point Transformer encoder with a trainable query decoder, optimizing instance segmentation on ScanNet alongside part-aware contrastive learning and representation-preserving losses on ScenePart. Stage 2 conducts instruction tuning on combined datasets (ScanRefer, Nr3D, Multi3DRefer, ScanQA, SQA3D, Scan2Cap, and ScenePart-200K) using LoRA-based parameter efficient fine-tuning while freezing the visual backbone. Results and Findings PAR3D demonstrates substantial improvements across both object-level and part-level evaluations. On object-level benchmarks, it achieves state-of-the-art results among 3D-MLLMs, improving ScanRefer mIoU from 43.3% to 49.9% (6.6% absolute gain) and Multi3DRefer from 42.7% to 53.4% (10.7% gain) compared to the previous best generalist model 3D-LLaVA. On the proposed ScenePart benchmarks, PAR3D significantly outperforms baselines: achieving 89.6% object mIoU, 60.9% coarse-part mIoU, and 46.0% fine-part mIoU on segmentation tasks, compared to 78.4%, 52.8%, and 37.6% for 3D-LLaVA trained with ScenePart data. For visual question answering, PAR3D achieves 191.1 CIDEr and 61.7 BLEU-4 scores, substantially outperforming the baseline’s 177.2 and 45.6 respectively. Ablation studies confirm that ScenePart supervision provides critical improvements, while part-aware representation learning and hierarchical query generation each contribute meaningfully to final performance. Implications and Conclusions PAR3D advances 3D scene understanding toward practical embodied AI applications by enabling fine-grained part-level reasoning essential for robotic manipulation, interactive scene editing, and spatial reasoning. The introduction of ScenePart fills a significant gap in 3D vision-language datasets, providing the first large-scale scene-level part supervision that will likely benefit future part-aware 3D perception research. A Vision-language Framework for Comparative Reasoning in Radiology Authors: Tengfei Zhang, Ziheng Zhao, Lisong Dai, Xiaoman Zhang, Pengcheng Qiu, Ya Zhang, Yanfeng Wang, Weidi Xie Source and references: https://arxiv.org/abs/2606.06407v1 Introduction This paper introduces MedReCo, a vision-language framework designed to enable comparative reasoning in medical imaging—a critical capability that current AI systems lack. Unlike existing approaches that interpret images in isolation, this work tackles two clinically central workflows: retrieving similar reference cases for differential diagnosis and generating natural-language descriptions of temporal changes in longitudinal follow-ups. Key Points Entity-aware comparative reasoning: The framework moves beyond global image-level matching by conditioning comparisons on specific clinical entities—anatomical structures, abnormal findings, and pathological conditions—allowing the model to distinguish between co-occurring features within the same image. Dual-component architecture: MedReCo handles reference-case retrieval through entity-conditioned visual embeddings, while MedReCo-VLM extends this foundation to a large language model for generative comparative interpretation, with both components sharing the same underlying visual representation. MedReCo-DB dataset: The authors constructed a large-scale, multi-institutional benchmark comprising 690,000+ images from 160,000+ patients across eight institutions, four countries, and seven imaging modalities, with systematic decomposition of clinical reports into structured entity-level annotations. Modality-specific architecture: A mixture-of-experts Vision Transformer with modality-aware routing handles heterogeneous imaging types (radiography, CT, MRI, ultrasound), while a lightweight reranker performs fine-grained token-level matching on top candidates for clinically confusable entities. Scalable supervision pipeline: Clinical reports are automatically decomposed into entity-specific descriptions and converted into ranking triplets for retrieval training and comparative VQA pairs for generation, minimizing reliance on manual annotation while providing dense structured supervision. Methodology The framework employs a two-stage training approach. In Stage 1, a mixture-of-experts Vision Transformer is trained with text-guided contrastive learning using triplet loss, where entity-specific report descriptions derived from clinical text provide ranking supervision. A cross-attention module learns to condition visual representations on clinical entities, and a coarse-to-fine architecture combines efficient dense retrieval with a lightweight reranker for interpretability. In Stage 2, the frozen vision encoder is connected to a large language model (Qwen2.5-7B-Instruct) through a projection layer and trained with instruction tuning on comparative visual question-answering pairs. Entity-level similarity is quantified during data construction using three complementary text-based metrics (BioLord, RaTEScore, and RadGraph-XL) applied to report-derived entity descriptions, with multiple filtering stages to ensure clinical validity and reduce language priors. Results and Findings Reference Comparison (Controllable Retrieval): MedReCo achieved the highest Recall@1 across all 12 internal retrieval settings and improved external validation by a mean of 6.0 percentage points compared to the strongest baselines. On clinically confusable differential groups, it consistently outperformed baselines with macro-average gains of 10.9 percentage points, with the largest improvements on hydronephrosis versus renal cysts (+17.2%) and complete collapse versus consolidation versus effusion (+12.5%). Cross-center retrieval—querying against entirely different institutional archives—showed MedReCo improved Recall@1 by 11.4% on chest radiographs and 7.8% on CT over existing methods. Temporal Comparison (Generative Interpretation): MedReCo-VLM ranked first across all 24 comparative interpretation evaluations on internal validation. On same-patient longitudinal follow-ups, it improved accuracy by 14.5–46.5 percentage points on chest radiographs and 13.0–27.9 percentage points on CT compared to modality-specific baselines. On independent public benchmarks (Medical-Diff-VQA and MMXU), it achieved 87.1% accuracy, exceeding the second-best model by 13.2% in RaTEScore. The model particularly excelled at interval-change interpretation, correctly identifying resolution, new abnormality development, and changes in anatomical distribution. Ablation studies confirmed that modality-specific routing improved Recall@1 by a mean of 5.2 percentage points across retrieval tasks, and entity-conditioned contrastive learning enhanced closed-ended accuracy by 2.3 percentage points on generation tasks. Implications and Conclusions This work demonstrates that entity-aware comparative reasoning can be learned at scale from routine clinical data, providing a more clinically aligned foundation for medical imaging AI than single-image paradigm approaches. The framework’s strong generalization across institutions, modalities, and distribution shifts—combined with consistent gains on clinically confusable differentials and longitudinal assessments—suggests that integrating reference comparison and temporal comparison into medical foundation models may better reflect radiological practice and support more trustworthy clinical decision-making. The authors acknowledge limitations in precise quantitative assessment tasks and suggest future integration with dedicated localization or segmentation methods. PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training Authors: Senmiao Wang, Tiantian Fang, Haoran Zhang, Yushun Zhang, Kunxiang Zhao, Alex Schwing, Ruoyu Sun Source and references: https://arxiv.org/abs/2606.06470v1 Introduction This paper proposes the Preconditioning (PC) Layer, a weight parameterization technique using polynomial preconditioners that maintains stable weight conditioning throughout large language model training. The method reshapes the singular-value spectrum of weight matrices without adding inference overhead, demonstrating consistent improvements in training efficiency across different optimizers and model scales. Key Points Polynomial Weight Preconditioning: The PC layer uses low-degree matrix polynomials to reshape weight matrix spectra by amplifying small singular values and saturating large ones, achieving “soft” spectrum conditioning rather than enforcing exact orthogonality. Zero Inference Overhead: Preconditioned weights can be merged back into the original architecture after training, eliminating any computational cost during inference while maintaining performance gains. Significant Token-Efficiency Improvements: On Llama-1B with AdamW, PC achieves a 2× speedup (50% fewer tokens to reach baseline loss); with Muon, a 1.13× speedup. Improvements scale consistently across the 271M to 1B parameter range. Theoretical Foundation: The authors prove that uniformly bounding layer-wise singular values ensures geometric convergence of gradient descent to global minima for deep linear networks, providing principled justification for spectrum control. Downstream Task Improvements: Zero-shot evaluation on standard benchmarks shows consistent gains across both optimizers—0.0206 point improvement with AdamW and 0.0125 with Muon—winning on 8 out of 9 tasks in each case. Methodology The PC layer implements Algorithm 1, which operates on selected weight blocks during training. For each weight matrix W, the method: (1) estimates the spectral norm s(W) via streaming power iteration to normalize weights into [0,1], (2) applies a polynomial preconditioner g(W) = p(WW^T)W to reshape singular values, and (3) performs norm recovery by multiplying the estimate s(W) back with an adaptive learnable scalar γ. The preconditioning polynomials are derived by solving weighted least-squares problems that fit piecewise-linear target mappings with different aggressiveness levels (pc_level = 1-4), corresponding to cutoff parameters b ∈ {0.8, 0.6, 0.4, 0.3}. The authors apply PC to FFN projections (W_gate, W_up, W_down) and attention output projections (W_O) in Llama-2 architecture. Results and Findings Pre-training Performance: On Llama-271M with AdamW, PC reduces final validation loss by 0.055 (1.63× speedup); on Llama-1B, the improvement reaches 0.070 loss reduction (2× speedup). Under Muon, the respective improvements are 0.006 (1.07× speedup) and 0.012 (1.13× speedup). These gains demonstrate consistent scaling without diminishing returns at larger model sizes. Spectral Analysis: The modified condition number (κ̃)—measuring effective spectral spread—improves substantially. PC reduces the global modified condition number from 42.4 to 25.0 (41% reduction) on Llama-1B. Notably, even non-preconditioned attention blocks (W_Q, W_K, W_V) show improved conditioning, indicating positive indirect effects. Singular-value histograms reveal that PC redistributes spectral mass away from the lower end, creating more regular middle-range distributions across depths. Computational Efficiency: FLOPs overhead remains minimal at 0.39% for AdamW (pc_level=4) and 0.24% for Muon (pc_level=2) on Llama-1B. Memory overhead increases by approximately 9.56% (AdamW) and 8.73% (Muon), a reasonable trade-off for the training efficiency gains. Implications and Conclusions The PC layer establishes weight-spectrum control as a practical and theoretically grounded principle for improving LLM pre-training efficiency. By balancing optimization stability (through spectrum conditioning) with representational capacity (through controlled spectral variation), this work demonstrates that systematic weight parameterization can yield consistent, non-trivial improvements in token efficiency across different optimizers and scales. The approach’s zero inference overhead and compatibility with existing architectures make it immediately adoptable, with potential implications for reducing the computational cost of training future large language models. Pretraining Recurrent Networks without Recurrence Authors: Akarsh Kumar, Phillip Isola Source and references: https://arxiv.org/abs/2606.06479v1 Pretraining Recurrent Networks without Recurrence: Summary Introduction This paper introduces Supervised Memory Training (SMT), a novel approach to training recurrent neural networks (RNNs) that bypasses the traditional backpropagation through time (BPTT) algorithm entirely. The authors propose a method that enables time-parallel RNN training while maintaining stable gradient propagation across long sequences, addressing fundamental limitations of RNNs that have historically hindered their ability to learn long-range dependencies. Key Points Novel Training Paradigm: SMT decouples learning what to remember from how to update memory, reducing RNN training to supervised learning on one-step memory transitions (m_t, x_{t+1}) → m_{t+1}, rather than unrolling the network through time. Gradient Path Reduction: SMT achieves O(1) credit path length between tokens compared to BPTT’s O(T), eliminating vanishing/exploding gradient problems by removing the need for recurrent credit propagation through the computation graph. Time-Parallel Training: By representing the past as a permutation-invariant set of timestamped events rather than a sequence, SMT enables fully parallelizable training using a Transformer encoder to generate optimal memory representations. Teacher-Student Architecture: A bidirectional Transformer encoder learns predictive state representations (retaining only information necessary for future prediction), which then supervises the RNN’s learning of memory dynamics through a secondary finetuning stage (DMT). Consistent Performance Improvements: Across synthetic tasks and real-world benchmarks (language modeling, pixel sequence modeling), SMT-trained RNNs substantially outperform BPTT-trained RNNs, particularly on tasks requiring long-range credit assignment and memory utilization. Methodology SMT operates in two stages. First, a Transformer encoder-decoder pair is trained on a predictive objective: the encoder compresses past context into memory representations m_t, while the decoder predicts future outputs using these representations. The complete training objective combines three losses: a decoding loss (L_dec) that supervises memory quality, a dynamics loss (L_dyn) that trains the RNN to perform one-step memory updates, and a uniformity loss (L_unif) that prevents representation collapse. Subsequently, a lightweight finetuning phase called DAgger Memory Training (DMT) addresses train-test mismatch by exposing the RNN to its own predicted memory states, allowing it to correct accumulated drift while keeping the encoder frozen. Unlike BPTT, SMT achieves O(M+T) memory complexity instead of O(MT), where M is memory size and T is sequence length. Results and Findings SMT demonstrates superior performance across diverse experimental settings. On synthetic tasks specifically designed to probe credit assignment (retrieval, string copy, state tracking, associative recall, and in-context learning), SMT-trained RNNs outperform BPTT consistently as sequence length increases, with BPTT showing clear performance degradation beyond length 64. On Attneave’s pixel sequence modeling task (raster-scan MNIST and Sketchy sketches), SMT captures long-range stroke dependencies that BPTT fails to recognize, producing coherent generated images versus fragmented BPTT outputs. In terms of computational efficiency, SMT and SMT→DMT require significantly fewer sequential FLOPs than BPTT across Transformer and MLP architectures on TinyStories, with comparable data efficiency on natural language but substantially better data efficiency on MNIST. Scaling experiments show smooth, predictable performance improvements with larger context lengths and memory state sizes. Notably, gradient analysis confirms SMT maintains stable gradient magnitudes independent of sequence position (Figure 11), while BPTT exhibits vanishing/exploding gradients dependent on sequence length. An unexpected finding shows that SMT-trained RNNs generalize better to longer sequences than their Transformer teachers, suggesting RNNs’ fixed-state inductive bias provides superior length generalization properties. Implications and Conclusions This work fundamentally challenges the necessity of BPTT for training nonlinear RNNs, opening pathways to scale recurrent models through time-parallel training while maintaining expressivity advantages over Transformers. The research has significant implications for long-horizon learning problems—such as lifelong learning agents processing unbounded experience—where fixed-memory RNNs offer superior inference efficiency (O(1) memory versus O(T)) and the ability to learn temporally compressed abstractions of past experience, properties crucial for artificial systems that must operate over extended horizons. The framework suggests that scaling along the compression axis—achieving equivalent performance with smaller memory—represents a promising future direction, potentially connecting neural network training dynamics to principles of intelligence itself. RREDCoT: Segment-Level Reward Redistribution for Reasoning Models Authors: Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger, Sepp Hochreiter Source and references: https://arxiv.org/abs/2606.06475v1 RREDCoT: Segment-Level Reward Redistribution for Reasoning Models Introduction This paper introduces RREDCoT, a novel credit assignment algorithm that addresses the delayed reward problem in reinforcement learning-based fine-tuning of reasoning language models. The method redistributes rewards across Chain-of-Thought (CoT) segments to improve training efficiency without requiring additional generation or separate models. Key Points Core Innovation: RREDCoT adapts RUDDER principles to CoT generation, enabling tractable approximation of optimal reward redistribution by leveraging the language model itself as a value function estimator. Hybrid Segmentation Strategy: The paper proposes a keyword-entropy based segmentation approach that starts with generic keyword splitting and iteratively merges segments with lowest combined entropy, balancing practical granularity with computational efficiency. Value Function Estimation: Introduces a probability-ratio (PR) style estimator that uses reference solution paths to estimate intermediate values without expensive Monte Carlo sampling, reducing computational overhead by 1.5-2x compared to MC methods while maintaining comparable accuracy. Attribution vs. Credit Assignment Distinction: Demonstrates through correlation analysis that traditional attribution methods (Leave-One-Out, gradient-based) are fundamentally different from credit assignment and correlate highly with gradient descent—suggesting explicit weighting based on attribution alone would be redundant. Integration with Existing RL Objectives: Shows RREDCoT can seamlessly integrate with Group Relative Policy Optimization (GRPO) and other common RL objectives while preserving return-equivalence and optimal policy properties. Methodology The authors formalize CoT generation as a Markov Decision Process where states consist of the original query and previously generated segments, with zero immediate rewards during reasoning and delayed rewards only upon answer verification. They derive the optimal reward redistribution using the CoT MDP’s unique structure—particularly its deterministic state transitions and partitioned action space separating CoT continuation from answer generation. The core technical contribution leverages the model’s predictive distribution to estimate value functions through importance sampling over reference solutions, avoiding expensive trajectory sampling while maintaining theoretical guarantees on bias. Results and Findings Experiments on mathematical reasoning datasets (Numina-CoT, MATH-500, AIME) demonstrate RREDCoT’s practical effectiveness. On the 4B parameter Qwen3 model with 25k-token generation length, RREDCoT achieved 90.8% on AIME24 compared to GRPO’s 85.0%, with consistent improvements across AIME25, AIME26, Minerva, and MATH500 benchmarks. Analysis of MC sampling revealed that truncating completions for computational efficiency introduces significant bias, losing 40-60% of valid trajectories including correct solutions. The correlation analysis confirmed RREDCoT’s credit assignments align better with MC estimates than traditional attribution methods, especially in later trajectory stages. Small-scale experiments using models’ own answers for redistribution yielded competitive results on open-rs datasets with reduced context windows. Implications and Conclusions RREDCoT addresses a critical bottleneck in reasoning model training: the lack of fine-grained learning signals for intermediate reasoning steps. By providing computationally tractable credit assignment without additional models or generation, the method enables more efficient fine-tuning of long-context reasoning—an increasingly important capability as models scale. The work also clarifies fundamental differences between attribution analysis and reward-based credit assignment, suggesting that explicit token-weighting based purely on attribution would be redundant with gradient-based training, while proper credit assignment conditioned on solutions provides complementary learning signals that improve convergence speed and final performance. You Only Index Once: Cross-Layer Sparse Attention with Shared Routing Authors: Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei Source and references: https://arxiv.org/abs/2606.06467v1 Introduction This paper introduces Cross-Layer Sparse Attention (CLSA), a novel architecture that addresses the critical challenge of efficient long-context inference in large language models. By extending the KV-sharing design of cross-attention architectures to also share routing indices across decoder layers, CLSA achieves substantial speedups while maintaining model quality. Key Points Unified efficiency solution: CLSA jointly improves three major inference bottlenecks—pre-filling, KV-cache storage, and long-context decoding—rather than optimizing only one aspect at the expense of others. Amortized routing overhead: The core innovation is computing token-level top-k sparse selection once and reusing the routing index across all cross-decoder layers, eliminating redundant expensive GPU operations. Quality preservation: CLSA maintains nearly lossless performance compared to dense attention across both short and long-context benchmarks, with negligible cross-entropy loss degradation. Practical efficiency gains: At 128K context length, CLSA achieves up to 7.6× decoding speedup and 17.1× overall throughput improvement over standard Transformer baselines. Foundation on proven architecture: The method builds on YOCO (You Only Cache Once), leveraging its existing advantages in pre-filling efficiency and KV-cache reduction while adding improvements to the decoding phase. Methodology The approach decomposes the model into a self-decoder and cross-decoder stack. The self-decoder performs efficient attention and constructs a shared KV cache once. A lightweight single-head query-aware indexer then computes a token-level top-k routing index that identifies the most salient tokens from the shared cache. Critically, this routing index is computed only once and reused by all subsequent cross-decoder layers, which maintain their own query states while attending only to the selected token positions. The shared indexer is trained using a multi-layer distillation objective that matches the consensus attention patterns across all decoder layers, ensuring selected tokens remain useful to multiple layers simultaneously. Results and Findings CLSA demonstrates strong performance across multiple evaluation dimensions. On standard benchmarks, it matches or exceeds dense baselines on reasoning tasks, particularly excelling on ARC-Challenge, GSM8K, and DROP. Long-context evaluation on RULER at 32K tokens shows CLSA achieves the best average score, with particular improvements on harder multi-needle retrieval tasks. Validation loss curves across Books, ArXiv, and StarCoder domains remain nearly overlapping with dense attention from 8K to 32K token contexts, confirming lossless quality preservation. The sparsity analysis reveals that selecting 2,048 tokens (approximately 1:16 of a 128K context) captures roughly 80% of dense attention mass while maintaining comparable or superior language modeling quality. Inference measurements on NVIDIA B200 GPUs show decode latency per layer is lowest among comparable sparse methods, with amortized top-k routing requiring only 0.08ms per layer at 128K context. Implications and Conclusions CLSA represents a significant step forward in making long-context LLM inference practical at scale, providing a more complete architectural solution that reconciles the traditional efficiency-quality trade-off. The work demonstrates that by carefully co-designing sparse selection mechanisms with KV-sharing architectures, it’s possible to achieve substantial real-world speedups without sacrificing model capability, suggesting this direction could become foundational for efficient long-context LLM deployment. Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution Authors: Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie Source and references: https://arxiv.org/abs/2606.06492v1 Code2LoRA: Hypernetwork-Generated Adapters for Evolving Code Introduction This paper introduces Code2LoRA, a hypernetwork framework that generates repository-specific LoRA (Low-Rank Adaptation) adapters for code language models, eliminating the need for expensive per-repository fine-tuning or context-heavy retrieval-augmented generation (RAG) approaches. The framework addresses a critical challenge in code understanding: efficiently incorporating repository-level knowledge—imports, APIs, and conventions—without incurring inference-time token overhead or becoming brittle to evolving codebases. Key Points Dual-Scenario Framework: Code2LoRA offers two complementary approaches—Code2LoRA-Static for stable codebases (maps repository snapshots to adapters in one forward pass) and Code2LoRA-Evo for active development (maintains adapters via GRU aggregation of sequential code diffs). Zero Inference-Time Overhead: Unlike RAG methods that prepend retrieved context to every query, Code2LoRA distills repository knowledge directly into model parameters, avoiding context-window inflation and per-query retrieval costs. Novel Hypernetwork Architecture: The framework uses a shared repository encoder that compresses entire codebases into dense embeddings, which a hypernetwork then converts into LoRA weights for all seven attention and MLP projection types (extending prior work that targeted only Q/V or down-projections). RepoPeftBench Benchmark: The authors curate a comprehensive benchmark of 604 Python repositories with static and evolution tracks, including 215K commit-derived training tasks and a temporal out-of-distribution holdout for robust evaluation. Strong Empirical Results: Code2LoRA-Static achieves 63.8% cross-repo exact match (+9.9 percentage points over the strongest baseline) and 66.2% in-repo exact match (matching the per-repository LoRA upper bound); Code2LoRA-Evo reaches 60.3% cross-repo exact match on evolving codebases (+5.2 pp over single shared LoRA). Methodology Code2LoRA comprises three core components: (1) a frozen repository encoder using Qwen3-Embedding-0.6B that compresses files into importance-weighted vectors via content distinctiveness, file size, and path heuristics; (2) a hypernetwork that projects repository embeddings to LoRA weights using a 2-layer MLP with dedicated output heads per module type; (3) a frozen base LLM (Qwen2.5-Coder-1.5B) that receives the generated adapters. For Code2LoRA-Evo, a GRU layer aggregates sequential code-diff embeddings, maintaining a hidden state that conditions adapter generation at each commit. The hypernetwork is trained end-to-end via cross-entropy loss on assertion-completion tasks, where the model must predict expected assertion values from test files using repository context. Evaluation uses three metrics—exact match, edit similarity, and CodeBLEU—across cross-repo and in-repo splits, with the in-repo setting allowing per-repository LoRA as an upper bound baseline. Results and Findings On the static track, Code2LoRA-Static substantially outperforms all baselines: achieving 63.8% cross-repo exact match versus 53.9% for FFT+RAG and 48.2% for dependency-resolved context. Notably, it reaches 66.2% in-repo exact match without any per-repository training, matching the per-repository LoRA upper bound (64.0%) and indicating that cross-repository transfer learned by the hypernetwork exceeds single-repository fitting. A strengthened Text2LoRA baseline—matched on input modality and target coverage—reaches only 45.8%, isolating the hypernetwork architecture as critical. The evolution track reveals substantial task difficulty under commit-derived evaluation: pretrained baseline performance drops from 45.7% to 31.5% cross-repo exact match, while RAG methods collapse below pretrained levels. Code2LoRA-Evo achieves 60.3% cross-repo exact match (+5.2 pp over single LoRA) and 64.5% in-repo exact match (exceeding the per-repository LoRA upper bound of 64.2%), demonstrating that recurrent aggregation of diffs better tracks repository evolution than static snapshots. Code2LoRA-Static scores 55.7% on evolution tasks, confirming that it degrades when repositories diverge from their snapshot state. On the temporal out-of-distribution holdout (92 repositories created post-cutoff), Code2LoRA-Evo achieves the highest exact match at 74.1%, with consistent advantages across edit similarity (0.866) and CodeBLEU (0.846) metrics, confirming generalization to unseen repository contexts. Implications and Conclusions Code2LoRA demonstrates that repository knowledge is best injected parametrically through learned hypernetwork-generated adapters rather than through context injection, fundamentally shifting how code language models can incorporate repository-level context. By enabling tracking of software evolution through commit-level updates, Code2LoRA addresses a practical gap in existing methods—the brittleness of static fine-tuned adapters to changing codebases—making it viable for real-world development scenarios where code evolves continuously. The framework opens new research directions in parameter-efficient adaptation for evolving domains and establishes RepoPeftBench as a valuable resource for future work in repository-level code understanding. Self-Augmenting Retrieval for Diffusion Language Models Authors: Paul Jünger, Justin Lovelace, Linxi Zhao, Dongyoung Go, Kilian Q. Weinberger Source and references: https://arxiv.org/abs/2606.06474v1 Introduction This paper introduces SARDI (Self-Augmenting Retrieval for Diffusion Language Models), a novel framework that dynamically refreshes retrieved evidence during the text generation process by leveraging intermediate predictions from discrete diffusion models. Unlike autoregressive language models that generate tokens sequentially, diffusion models denoise entire sequences in parallel, exposing tentative predictions across all positions that can guide more effective retrieval for multi-hop question answering tasks. Key Points Diffusion trajectories as lookahead signals: Intermediate denoising states surface salient entities (bridge entities) early in generation, enabling retrieval of relevant evidence before the final output is committed—a capability unavailable in left-to-right autoregressive generation. Separation of retrieval and commitment confidence: SARDI uses two distinct confidence thresholds (τq for retrieval, τc for committing tokens), allowing low-confidence speculative tokens to inform retrieval long before they’re reliable enough to finalize in the output. Dynamic evidence refresh at every step: Rather than retrieving once from the question alone, SARDI constructs increasingly informed queries from partially denoised responses, updating the retrieved document set throughout generation. Reduced inter-token dependence under RAG: Retrieved evidence significantly decreases conditional mutual information between adjacent tokens—particularly for entity spans—making parallel decoding more effective and coherent. Training-free and retriever-agnostic design: SARDI requires no additional training or learned components and works with any discrete diffusion language model and retriever (sparse or dense). Methodology SARDI operates through iterative cycles of denoising and retrieval refinement. At each denoising step, the diffusion model produces confidence scores for all masked positions. Tokens exceeding the query threshold τq are incorporated into a proxy sequence that, combined with the original question, forms a retrieval query. This query refreshes the document set before the next denoising step. Only tokens exceeding the higher commit threshold τc are finalized in the output; others are remasked for further refinement. The framework was evaluated on five multi-hop QA benchmarks using DREAM-7B as the diffusion backbone, with identical fine-tuning applied to autoregressive baselines to ensure fair comparison. Results and Findings SARDI substantially improves performance across all benchmarks: on 2WikiMultiHopQA, exact match (EM) increases from 43.7% (static retrieval) to 59.1%; on HotpotQA from 39.9% to 48.7%; on CofCA from 43.4% to 44.9%; and on MuSiQue from 11.1% to 20.6%. The method achieves these gains while running up to 8× faster than autoregressive iterative-retrieval baselines at comparable accuracy. Analysis reveals that aggressive lookahead (τq ≈ 0) yields the best results, with early-generation recall improving by +19 percentage points at 25% of generation completion compared to autoregressive baselines. Per-query document recall analysis shows SARDI surfaces gold passages earlier than AR methods, closing most of the gap toward oracle performance. Notably, gains concentrate on multi-hop reasoning questions (2.5× improvement on inferential and compositional questions) while leaving single-hop comparisons unchanged. Conditional mutual information measurements confirm that grounding reduces entity-pair dependence by 10× (from 0.588 to 0.060), validating that RAG strongly promotes parallel decoding. Implications and Conclusions This work demonstrates that diffusion language models’ parallel denoising structure enables fundamentally new approaches to retrieval-augmented generation unavailable to autoregressive models. The framework’s plug-and-play nature and training-free design make it immediately applicable to emerging discrete diffusion models, while establishing a new quality-latency frontier for multi-hop reasoning tasks. As diffusion models mature and acquire zero-shot chain-of-thought capabilities without fine-tuning, SARDI’s practical constraints will diminish and its advantages in both speed and accuracy position it as a compelling alternative for knowledge-grounded text generation systems. Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics Authors: Open-H-Embodiment Consortium, :, Nigel Nelson, Juo-Tung Chen, Jesse Haworth, Xinhao Chen, Lukas Zbinden, Dianye Huang, Alaa Eldin Abdelaal, Alberto Arezzo, Ayberk Acar, Farshid Alambeigi, Carlo Alberto Ammirati, Yunke Ao, Pablo David Aranda Rodriguez, Soofiyan Atar, Mattia Ballo, Noah Barnes, Federica Barontini, Filip Binkiewicz, Peter Black, Sebastian Bodenstedt, Leonardo Borgioli, Nikola Budjak, Benjamin Calmé, Fabio Carrillo, Nicola Cavalcanti, Changwei Chen, Haoxin Chen, Sihang Chen, Qihan Chen, Zhongyu Chen, Ziyang Chen, Shing Shin Cheng, Meiqing Cheng, Min Cheng, Zih-Yun Sarah Chiu, Xiangyu Chu, Camilo Correa-Gallego, Giulio Dagnino, Anton Deguet, Jacob Delgado, Jonathan C. DeLong, Kaizhong Deng, Alexander Dimitrakakis, Qingpeng Ding, Hao Ding, Giovanni Distefano, Daniel Donoho, Anqing Duan, Marco Esposito, Shane Farritor, Jad Fayad, Zahi Fayad, Mario Ferradosa, Filippo Filicori, Chelsea Finn, Philipp Fürnstahl, Jiawei Ge, Stamatia Giannarou, Xavier Giralt Ludevid, Frederic Giraud, Aditya Amit Godbole, Ken Goldberg, Antony Goldenberg, Diego Granero Marana, Xiaoqing Guo, Tamás Haidegger, Evan Hailey, Pascal Hansen, Ziyi Hao, Kush Hari, Kengo Hayashi, Jonathon Hawkins, Shelby Haworth, Ortrun Hellig, S. Duke Herrell, Zhouyang Hong, Andrew Howe, Junlei Hu, Zhaoyang Jacopo Hu, Ria Jain, Mohammad Rafiee Javazm, Howard Ji, Rui Ji, Jianmin Ji, Zhongliang Jiang, Dominic Jones, Jeffrey Jopling, Britton Jordan, Ran Ju, Michael Kam, Luoyao Kang, Fausto Kang, Siddhartha Kapuria, Peter Kazanzides, Sonika Kiehler, Ethan Kilmer, Ji Woong Kim, Przemysław Korzeniowski, Chandra Kuchi, Nithesh Kumar, Alan Kuntz, Federico Lavagno, Yu Chung Lee, Hao-Chih Lee, Hang Li, Zhen Li, Xiao Liang, Xinxin Lin, Jinsong Lin, Chang Liu, Fei Liu, Pei Liu, Yun-hui Liu, Wanli Liuchen, Eszter Lukács, Sareena Mann, Miles Mannas, Brett Marinelli, Sabina Martyniak, Francesco Marzola, Lorenzo Mazza, Xueyan Mei, Maria Clara Morais, Luigi Muratore, Chetan Reddy Narayanaswamy, Michał Naskręt, David Navarro-Alarcon, Cyrus Neary, Chi Kit Ng, Christopher Nguan, David Noonan, Ki Hwan Oh, Tom Christian Olesch, Allison M. Okamura, Justin Opfermann, Matteo Pescio, Doan Xuan Viet Pham, Tito Porras, Hongliang Ren, Ariel Rodriguez Jimenez, Ferdinando Rodriguez y Baena, Septimiu E. Salcudean, Asmitha Sathya, Preethi Satish, Lalithkumar Seenivasan, Jiaqi Shao, Yiqing Shen, Yu Sheng, Lucy XiaoYang Shi, Zoe Soulé, Stefanie Speidel, Mingwu Su, Jianhao Su, Idris Sunmola, Kristóf Takács, Yunxi Tang, Patrick Thornycroft, Yu Tian, Jordan Thompson, Mehmet K. Turkcan, Mathias Unberath, Pietro Valdastri, Carlos Vives, Quan Vuong, Martin Wagner, Farong Wang, Wei Wang, Lidian Wang, Chung-Pang Wang, Guankun Wang, Junyi Wang, Erqi Wang, Ziyi Wang, Tanner Watts, Wolfgang Wein, Yimeng Wu, Zijian Wu, Hongjun Wu, Luohong Wu, Jie Ying Wu, Junlin Wu, Victoria Wu, Kaixuan Wu, Mateusz Wójcikowski, Yunye Xiao, Nan Xiao, Wenxuan Xie, Hao Yang, Tianqi Yang, Yinuo Yang, Menglong Ye, Ryan S. Yeung, Nural Yilmaz, Chim Ho Yin, Michael Yip, Rayan Younis, Chenhao Yu, Sayem Nazmuz Zaman, Milos Zefran, Han Zhang, Yuelin Zhang, Yidong Zhang, Yanyong Zhang, Xuyang Zhang, Yameng Zhang, Joyce Zhang, Ning Zhong, Peng Zhou, Haoying Zhou, Xiuli Zuo, Nassir Navab, Mahdi Azizian, Sean D. Huver, Axel Krieger Source and references: https://arxiv.org/abs/2604.21017v3 Open-H-Embodiment: Foundation Models for Medical Robotics Introduction This paper introduces Open-H-Embodiment, the largest open-source dataset for medical robotics combining 780 hours of synchronized video and kinematic data from 50+ institutions across 20 distinct robotic platforms. The work addresses a critical bottleneck in surgical AI: the lack of large-scale, multi-embodiment datasets needed to train foundation models that can generalize across different surgical robots and procedures. Key Points Dataset Scale & Diversity: Open-H comprises 780 hours of paired video-kinematics data across 20 robotic platforms (including da Vinci, Versius, dVRK, MIRA, and Maestro), spanning surgical manipulation, robotic ultrasound, and endoscopy—representing a 39x increase over the previous largest surgical dataset (ImitateCholec at ~20 hours). GR00T-H Foundation Model: The first open-source vision-language-action (VLA) model trained specifically for surgical robotics, achieving 25% end-to-end task completion on structured suturing benchmarks versus 0% for all baseline models, and 64% average success on a 29-step ex vivo suturing sequence. Multi-Embodiment Generalization: GR00T-H demonstrates statistically significant performance gains across three distinct surgical platforms (dVRK-Si, CMR Versius, Virtual Incision MIRA), validating that domain-specific pretraining enables cross-platform transfer. Cosmos-H-Surgical-Simulator: The first multi-embodiment, action-conditioned world model for surgical simulation spanning nine robotic platforms, enabling in silico policy evaluation and synthetic data generation from a single checkpoint. Data Efficiency Improvements: GR00T-H achieves competitive performance with only 33% of fine-tuning data and shows improved robustness to hardware drift compared to specialized baselines, suggesting that surgical-domain pretraining reduces data requirements for adaptation. Methodology The dataset aggregates contributions from 50+ institutions following a unified LeRobot v2.1 format with standardized metadata documentation. Data spans five realism tiers: digital simulation (84 hours), benchtop/phantom (119 hours), ex vivo (75 hours), in vivo (3 hours), and clinical human procedures (499 hours). GR00T-H is developed through post-training NVIDIA’s GR00T-N1.6 model on a 601-hour surgical subset, using embodiment-specific action heads and relative 6D end-effector control normalized across platforms. Cosmos-H-Surgical-Simulator fine-tunes Cosmos-Predict 2.5 (a 2B-parameter latent video diffusion transformer) on 32 surgical datasets across 9 robotic platforms, with action conditioning via MLPs that accommodate all embodiments through zero-padding. Results and Findings GR00T-H Evaluation Results: End-to-end suturing: 5/20 trials (25%) completed vs. 0/20 for all baseline models (ACT, GR00T-N1.6, LingBot-VA) Out-of-distribution generalization: 54% average subtask success vs. 30% for GR00T-N1.6 and 5% for ACT Data efficiency at 33% fine-tuning data: GR00T-H matched ACT baseline while GR00T-N1.6 achieved only 20% Multi-embodiment performance: Statistically significant improvements (p<0.001) across Versius Peg Transfer, MIRA needle pickup, and dVRK-Si suturing Ex vivo suturing on real tissue: 64% average success across 29 subtasks, with near-perfect performance (>90%) on structured primitives (pickup, handover, knot-tying) but lower success (20-40%) on fine-contact and cutting tasks Cosmos-H-Surgical-Simulator Evaluation: Per-frame L1 error and SSIM maintained stable quality across 72-frame autoregressive rollouts on benchtop scenes (controlled environments) Tissue-based scenes showed predictable degradation reflecting visual complexity (variable endoscopic lighting, instrument occlusion, deformable anatomy) Model successfully generates multi-embodiment surgical trajectories without platform-specific retraining Implications and Conclusions This work demonstrates that large-scale, domain-focused dataset curation can successfully apply scaling laws from general robotics to the surgical domain, resolving the transfer gap that prevented prior general-purpose models from performing surgical tasks. The multi-embodiment design and open-source release position Open-H as critical shared infrastructure for the surgical robotics community, enabling advances in surgical autonomy, world modeling, and policy learning. However, absolute performance levels (25% end-to-end suturing, 64% ex vivo success) remain below clinical deployment thresholds, and the absence of failure examples, patient-specific variation, and closed-loop safety mechanisms indicates that substantial additional development and pre-clinical validation are necessary before any component could transition toward human surgical application. TempoVLA: Learning Speed-Controllable Vision-Language-Action Policies Authors: Dong Jing, Jingchen Nie, Tianqi Zhang, Jiaqi Liu, Huaxiu Yao, Zhiwu Lu, Mingyu Ding Source and references: https://arxiv.org/abs/2606.06491v1 Introduction TempoVLA introduces a framework that enables Vision-Language-Action models to execute robot manipulation tasks at variable, controllable speeds. Current VLAs inherit a single fixed execution speed from training data, limiting their ability to adapt between fast transit phases and slow, precise contact phases that real manipulation requires. Key Points Variable-Speed Trajectory Augmentation (VSTA): A data augmentation technique that re-times demonstrations to arbitrary target speeds by merging actions to accelerate or splitting them to decelerate while preserving motion semantics. Speed Conditioning Mechanism: Three lightweight injection schemes (textual prefix, modulated RMSNorm, soft prompts) that add explicit scalar speed control to existing VLAs without architectural changes. Bidirectional Speed Control: Unlike prior acceleration-focused approaches, TempoVLA enables both speedup (s > 1) and slowdown (s < 1) of the same policy without retraining from scratch. Performance Improvements: Speed-conditioned training acts as effective data augmentation, improving the default 1× success rate by 0.2-0.7% in simulation and 8% on real hardware compared to single-speed baselines. Dynamic Speed Scheduling: Integration with Vision-Language Models enables automated, phase-aware speed adjustment that accelerates through low-risk phases and decelerates for high-risk contact operations. Methodology TempoVLA combines two coupled components operating on both the data and model sides. On the data side, VSTA processes demonstrations through motion-consistent segmentation (identifying motion modes like still, translate, rotate), chunk-level speed transformation (accumulating motion and re-splitting into target frame counts), and online chunk-start sampling to maximize data utilization. On the model side, the speed scalar is injected via textual prefix, speed-modulated RMSNorm, or soft prompts, conditioning the policy’s action magnitude scaling while leaving low-level controllers unchanged. The approach was evaluated on LIBERO simulation benchmark using π0.5 (a flow-matching VLA) and real-world Franka robot experiments with five manipulation tasks. Results and Findings Simulation Results (LIBERO): VSTA reliably re-times demonstrations with success rates of 92.9-97.6% across speeds 0.75×-1.25×, with negligible motion error (< 5×10⁻⁸). All three speed-conditioning schemes achieved essentially identical performance (~96.8% average success rate). Training with seven speed anchors {0.5, 0.75, 1, 1.25, 1.5, 1.75, 2}× provided optimal coverage, with peak performance occurring at 1.25×-1.5× rather than the original 1× speed, suggesting teleoperation data contains rhythm padding that VSTA’s compression removes. Real-World Results (Franka): TempoVLA improved the 1× baseline success rate from 80% to 88%, matching the simulation augmentation effect. The realized speed ratio closely tracked commanded speeds (0.63× at 0.75× command, 1.29× at 1.25×, 1.48× at 1.5×). GPT-4o-based dynamic speed scheduling achieved 96% average success rate across five tasks while maintaining 1.21× average speedup—an 8-point improvement over the best fixed-speed configuration. Implications and Conclusions TempoVLA demonstrates that execution speed can be effectively controlled through lightweight data and model modifications without architectural changes or new data collection. By enabling phase-aware speed scheduling with external VLMs, the framework transforms execution speed into a control channel for higher-level reasoning, with practical implications for robot manipulation tasks requiring adaptation between rapid transit and precise contact phases. The work suggests future extensions could involve co-tuning low-level controllers to improve high-speed tracking and broader integration with more sophisticated planning systems. PHUMA: Physically Reliable Humanoid Locomotion Dataset Authors: Kyungmin Lee, Sibeen Kim, Youngdo Lee, Minho Park, Hyunseung Kim, Dongyoon Hwang, Donghu Kim, Hojoon Lee, Jaegul Choo Source and references: https://arxiv.org/abs/2510.26236v2 PHUMA: Physically Reliable Humanoid Locomotion Dataset Introduction This paper introduces PHUMA, a large-scale humanoid locomotion dataset designed to address fundamental limitations in motion imitation for humanoid robots. While existing datasets like AMASS offer high quality, they lack scale and diversity, whereas internet video-derived datasets like Humanoid-X provide scale but suffer from physical artifacts that prevent stable robot control. Key Points Two-stage curation and retargeting pipeline: PHUMA combines physics-aware motion filtering with a novel physics-constrained retargeting method to produce 73 hours of physically reliable motion—3.5× larger than AMASS while maintaining superior quality compared to the 231-hour Humanoid-X dataset. Physics-Aware Curation filters three artifact types: The pipeline eliminates high-frequency jitter through low-pass filtering, removes physically unstable motions (e.g., sitting on non-existent furniture) by analyzing center-of-mass stability, and fixes floating/penetration issues through robust ground plane estimation via majority voting. PhySINK retargeting method enforces physical constraints: Rather than prioritizing kinematic fidelity alone, PhySINK jointly optimizes motion fidelity while enforcing soft joint limits, ground contact alignment, and anti-skating constraints, producing the only method that performs consistently across all physical reliability metrics. Superior downstream performance: PHUMA-trained policies achieve 92.7% success rate on test motions versus 76.2% for AMASS and 50.6% for Humanoid-X, demonstrating that data quality substantially outweighs raw quantity—even though PHUMA filters out ~73% of raw video data. Real-world transfer validated: Zero-shot deployment on physical Unitree G1 robots shows 16.3% lower tracking error compared to AMASS-trained policies, confirming that simulation advantages translate to hardware. Methodology The research employs a four-stage pipeline beginning with large-scale human motion collection from videos and mocap. Physics-aware curation then filters problematic sequences through Butterworth filtering (jitter removal), center-of-mass stability analysis (physical instability detection), and ground contact scoring (floating/penetration detection). The filtered motions undergo retargeting via PhySINK, which optimizes a composite loss combining motion fidelity, joint feasibility, grounding, and anti-skating terms. Finally, reinforcement learning policies are trained using Masked-Mimic framework for simulation and BeyondMimic for sim-to-real transfer, with evaluation across motion categories (stationary, angular, vertical, horizontal) on Unitree G1 and H1-2 humanoid platforms. Results and Findings PHUMA achieves comprehensive improvements across all evaluation axes. On internal test sets, the PHUMA-trained policy reaches 92.7% success rate versus 76.2% (AMASS) and 50.6% (Humanoid-X). On 504 self-recorded videos, PHUMA maintains 82.9% success compared to 78.2% for AMASS. PhySINK retargeting demonstrates balanced performance across five physical reliability metrics (motion fidelity: 94.8%, joint feasibility: 100%, non-floating: 99.9%, non-penetration: 96.8%, non-skating: 89.7%), substantially outperforming prior methods. Real-world experiments on Unitree G1 show PHUMA-trained policies achieve 34.3mm mean per-joint position error versus 41.0mm for AMASS, with improvements sustained across all motion categories and metrics (degrees-of-freedom velocity and acceleration errors). Implications and Conclusions PHUMA demonstrates that humanoid motion imitation success depends critically on data quality rather than quantity alone, establishing a new paradigm where careful physics-informed curation and constrained retargeting outperform scaling raw internet video. The dataset’s successful real-world transfer indicates that addressing physical artifacts in training data directly improves robot control reliability, providing a foundation for deploying humanoid robots in diverse real-world scenarios and suggesting that future humanoid locomotion research should prioritize physical plausibility alongside data scale.

Inference-Time Memory in Video VLMs and Faithful Reasoning in Language Models

State of AI — Mon, 01 Jun 2026 16:35:36 GMT

Welcome to today’s edition of State of AI 👋

Featuring: Transformer vs Post-Transformer Debate

Before today’s edition, here’s a debate worth your time on where frontier model architectures go next.

State of AI readers have seen the Post Transformer question show up across papers on memory, context length, reasoning efficiency, and models that try to do more with less.

Pathway put the people behind the architectures on one public stage.

Łukasz Kaiser, co-author of the Transformer and co-creator of ChatGPT, argues why Transformers will stay dominant.
Adrian Kosowski, inventor of Dragon Hatchling (BDH), says AI has not yet had a PageRank moment for intelligence.
Llion Jones, also a Transformer co-author, argues against his own invention: the field may be stuck at a local minimum.
Mathias Lechner, co-inventor of Liquid Neural Networks, brings the hardware and deployment lens.

Together, the debate gets into the next architecture shift: memory, long horizon reasoning, latent reasoning, hardware limits, and the 10x bar for Post Transformer models. This is one of the more useful public conversations on where AI architectures go next, argued by the people building them.

Watch the full debate:

This week’s papers bring significant advances across three critical frontiers in AI systems. We’re seeing breakthroughs in efficient model scaling through adaptive expert routing that respects computational budgets, major improvements in long-context reasoning through structured search trees and explicit memory mechanisms, and important discoveries about the gap between what language models claim to reason versus what they actually compute. Beyond core LLM research, there’s meaningful progress in vision-language systems tackling real-world robustness—from handling dynamic environments in robotics to uncovering hidden biases in multimodal representations.

Here’s what caught our attention:

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training — Uses classical control theory (PI controllers) to dynamically adjust sparse expert routing while maintaining predictable computational budgets, solving the fundamental tension between adaptive allocation and strict FLOP constraints.
LinTree: Improving LLM Reasoning with Explicitly Structured Search Histories — Demonstrates that exposing tree structure through parent pointers in reasoning traces dramatically improves LLM search performance, revealing representation design matters as much as model capacity.
Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization — Provides mechanistic analysis proving that symbolic attention mechanisms generalize to longer sequences while positional mechanisms fail, with implications for understanding length extrapolation limits.
Linear Scaling Video VLMs for Long Video Understanding — Achieves O(N) complexity for video processing through importance-based token selection, enabling practical long-video understanding without architectural changes.
From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents — Reveals that LLM agents reconstruct real identities from anonymized data by synthesizing scattered cues with web retrieval, exposing a critical privacy failure mode distinct from direct leakage.
Chain-of-Thought Reasoning In The Wild Is Not Always Faithful — Shows frontier models including “thinking” variants rationalize predetermined answers rather than reasoning faithfully, with unfaithfulness rates persisting across architectures despite alignment training.
Vision-Language Models Suppress Female Representations Under Ambiguous Input — Demonstrates VLMs encode female associations internally yet systematically suppress them before generation, revealing output-level auditing misses representation-level biases that matter for downstream applications.
Representation Forcing for Bottleneck-Free Unified Multimodal Models — Eliminates external VAE bottlenecks by training decoders to predict discrete visual representation tokens, enabling end-to-end pixel-space generation without frozen components.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

Authors: Can Jin, Hongwu Peng, Mingcan Xiang, Qixin Zhang, Xiangchi Yuan, Amit Hasan, Ohiremen Dibua, Yifan Gong, Yan Kang, Dimitris N. Metaxas

Source and references: https://arxiv.org/abs/2512.13996v2

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

Introduction

This paper introduces DTop-p, a dynamic routing mechanism for Mixture-of-Experts (MoE) architectures that addresses the rigidity of fixed Top-k selection and instability of fixed-threshold Top-p routing. By combining proportional-integral control with dynamic routing normalization, DTop-p enables adaptive expert allocation while maintaining predictable computational budgets—a critical requirement for large-scale foundation model training.

Key Points

Fixed Top-p limitations identified: The paper demonstrates that existing fixed-threshold Top-p MoE implementations provide only marginal performance improvements over Top-k while suffering from hyperparameter sensitivity and unpredictable computational costs that range from 4-12+ activated experts.
PI controller mechanism: DTop-p employs a Proportional-Integral (PI) controller from classical control theory to dynamically adjust the probability threshold, treating target sparsity as a setpoint. This feedback loop ensures the model converges to a specified computational budget regardless of training dynamics.
Dynamic Routing Normalization (DRN): Layer-specific learnable scaling normalizes routing logits adaptively, allowing different layers to exhibit distinct sparsity patterns (fewer experts in shallow layers, more in deeper layers) while respecting a global budget constraint.
Comprehensive experimental validation: Testing across NLP (on DCLM-Baseline dataset) and computer vision (Diffusion Transformers) domains shows DTop-p consistently outperforms Top-k and fixed-threshold Top-p baselines while matching FLOPs.
Strong scaling properties: The method demonstrates robust performance improvements across varying expert granularities (32E4A to 128E16A), total expert capacities, model sizes (0.4B to 2.4B parameters), and dataset sizes (100B to 300B tokens).

Methodology

DTop-p combines two core technical components to achieve sparsity control. First, a PI controller continuously monitors the average number of activated experts per batch and adjusts the global probability threshold using proportional and integral terms that track both current errors and accumulated historical deviations. The controller leverages the monotonicity property of nucleus sampling—increasing the threshold strictly requires selecting more experts. Second, Dynamic Routing Normalization independently rescales routing logit distributions for each layer using learnable temperature parameters, decoupling global threshold constraints from local statistical properties. This two-level approach allows tokens to adaptively select varying expert counts based on difficulty while maintaining layer-wise flexibility within a global computational budget.

Results and Findings

NLP Performance: On the Dense-1.3B vs. MoE-1.3B-6.9B-64E8A comparison trained on 100B tokens, DTop-p achieves superior training efficiency with lower validation loss. On downstream evaluation across 13 benchmarks (SVAMP, MMLU, ARC, HellaSwag, etc.), DTop-p averages 50.9% compared to Top-k’s 49.0%—a 1.9% absolute improvement while maintaining identical FLOPs.

Sparsity Control Precision: Figure 6 demonstrates that DTop-p maintains the target activation level (8 experts/token) with mean stability and low standard deviation (≈1), converging within the first 1B tokens. In contrast, fixed-threshold Top-p exhibits high variance (≈4) and fails to stabilize until late training stages.

Layer-wise Activation Patterns: Analysis reveals DTop-p learns interpretable depth-dependent routing: shallow layers (L0-L2) activate ~1 expert while deeper layers (L12-L15) utilize more capacity—consistent with theoretical expectations about broad shallow processing versus specialized deep reasoning.

Scaling Results: Expert granularity experiments show DTop-p’s advantage widens at higher granularities (16/128 configuration), with performance gains of 1.2-1.5% over Top-k. Model size scaling (0.4B to 2.4B) and dataset scaling (100B to 300B tokens) both show consistent performance leads, indicating robust generalization.

Computer Vision Validation: On Diffusion Transformer experiments with 2 trillion pixel tokens, DTop-p similarly outperforms baselines while successfully constraining expert activation to target levels, confirming cross-domain effectiveness.

Implications and Conclusions

DTop-p establishes a practical framework for reconciling adaptive expert allocation with strict computational constraints—essential for production foundation model training where FLOPs budgets are non-negotiable. The incorporation of classical control theory into deep learning routing mechanisms opens a novel direction for managing sparsity dynamics, while the demonstrated scaling properties across multiple dimensions suggest the approach generalizes well to increasingly large models and datasets. As sparse MoE architectures continue serving as the standard for efficient capacity scaling, DTop-p provides both immediate practical utility and a principled foundation for future dynamic routing research.

LinTree: Improving LLM Reasoning with Explicitly Structured Search Histories

Authors: Liwei Kang, Yee Whye Teh, Wee Sun Lee

Source and references: https://arxiv.org/abs/2605.31492v1

LinTree: Improving LLM Reasoning with Explicitly Structured Search Histories

Introduction

This paper investigates whether Large Language Models can leverage full search history to outperform traditional heuristic-guided search, and demonstrates that making search tree structures explicit significantly improves reasoning performance. The research frames LLM reasoning as an implicit search process and proposes LinTree, a method that adds parent pointers to expose tree topology in reasoning traces.

Key Points

Trace access alone is insufficient: Despite having access to complete search history, LLMs with implicit trace representations fail to consistently outperform local-state heuristic baselines across Blocks World, Grid Navigation, and Sokoban domains.
Explicit parent pointers dramatically improve performance: Adding simple parent-pointer annotations that expose tree structure improves solve rates (e.g., 94.9% to 100% in Navigation) and reduces search expansions while using identical underlying training data.
Structured supervision enhances plan extraction: Models trained on explicit tree structures more reliably extract correct final plans from their own search traces, with extraction failure rates dropping significantly (e.g., from 80.78% to 54.17% in Sokoban at the SFT stage).
Better exploration patterns emerge: Explicit structure enables more effective state-space exploration, with trace-conditioned policies visiting more diverse regions compared to local-state heuristics (average pairwise distance of 4.19 vs. 3.99).
Minimal architectural changes yield substantial gains: The improvement requires only adding state identifiers (sid) to trace annotations, demonstrating that representation design matters as much as model capacity for LLM reasoning.

Methodology

The research employs a two-stage training pipeline combining supervised fine-tuning (SFT) on best-first search traces followed by reinforcement learning with GRPO. Two competing approaches are evaluated: trace-conditioned reasoning policies that observe full search history, and local-state heuristic-guided search using an external best-first search controller that only observes current state and goal.

The team generates 20k instances for SFT and 20k for RL across three fully observable domains with well-defined search trees. For fair comparison, both approaches use identical base models (Qwen3-0.6B), training procedures, and reward functions—with the key difference being trace representation (implicit vs. explicit parent pointers).

Results and Findings

Implicit vs. Explicit Traces (Table 3):

Blocks World: GRPO-explicit achieves 100% solve rate vs. 97.3% implicit, with 7.31 vs. 8.25 expansions
Navigation: GRPO-explicit reaches 100% solve rate vs. 94.9% implicit, with 14.28 vs. 14.80 expansions
Sokoban: GRPO-explicit achieves 89.6% solve rate vs. 85.9% implicit, with 52.82 vs. 63.54 expansions

Plan Extraction Analysis (Table 4): At the SFT stage, explicit annotations reduce extraction failures from 80.78% to 54.17% in Sokoban, indicating that explicit structure makes it easier for models to trace back found solutions.

Exploration Patterns (Table 5): The explicit policy explores more diverse state-space regions (average pairwise distance of 4.19) compared to implicit reasoning (4.11) and local-state heuristics (3.99), suggesting that parent pointers help models track visited areas and avoid redundant expansions.

The results hold even with generation constraints applied to Sokoban’s complex dynamics, where the explicit model matches the local-state heuristic’s 99.1% solve rate while requiring fewer expansions (54.70 vs. 64.08).

Implications and Conclusions

This research demonstrates that LLM reasoning improvements depend critically on trace representation, not merely on information access. By making implicit search trees explicit through minimal architectural additions, researchers can substantially enhance both reasoning accuracy and computational efficiency. The findings suggest that future work on LLM reasoning should prioritize better interfaces between reasoning processes and their underlying computational structures, moving beyond larger models or improved training objectives alone to incorporate search-structured supervision as a fundamental design principle.

From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents

Authors: Myeongseob Ko, Jihyun Jeong, Sumiran Singh Thakur, Gyuhak Kim, Ruoxi Jia

Source and references: https://arxiv.org/abs/2603.18382v2

From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents

Introduction

This paper demonstrates that LLM-based agents can reconstruct real-world identities from anonymized data by combining scattered, individually non-identifying cues with publicly available information—a capability that fundamentally weakens traditional assumptions about anonymization as a privacy safeguard. The research reveals that identity reconstruction can occur not only through deliberate re-identification attacks, but also as an unintended byproduct of ordinary AI-assisted analysis tasks.

Key Points

Inference-driven linkage formalized: The paper introduces a new privacy failure mode where agents synthesize fragmented cues from anonymized artifacts with auxiliary context to reconstruct specific identities, distinct from direct identifier leakage or attribute inference attacks.
Superior performance on classical attacks: Modern LLM agents reproduce and exceed historical deanonymization baselines—GPT-5 achieves 79.2% identity reconstruction on the Netflix Prize dataset (sparsest regime) versus 56.0% for hand-engineered classical methods, and successfully links individuals in AOL search logs through open-ended web retrieval.
Silent linkage during benign tasks: Agents reconstruct identities even when not explicitly prompted to do so, with Claude 4.5 exhibiting substantial linkage rates (0.70–0.80) during routine cross-source analysis tasks designed for legitimate purposes like customer analytics.
Systematic evaluation through InferLink benchmark: A controlled benchmark isolates three factors—fingerprint type (intrinsic attributes, spatiotemporal coordinates, or hybrid), task framing (benign vs. explicit re-identification), and attacker knowledge (zero-knowledge vs. membership knowledge)—enabling precise measurement of conditions that trigger linkage.
Privacy-utility trade-off with mitigation: Privacy-aware system prompts effectively suppress linkage (reducing success rates to near zero in some conditions), but often degrade legitimate task performance, revealing a fundamental tension in defending against inference-driven attacks without over-refusing benign requests.

Methodology

The research employs a three-tiered evaluation approach. First, classical case studies revisit the Netflix Prize dataset and AOL search logs, comparing modern LLM agents against historical baselines. Second, InferLink constructs synthetic paired-source instances with known ground-truth linkages, varying fingerprint type, task intent, and attacker knowledge across 180 controlled instances. Third, open-ended case studies examine redacted interviews from the Anthropic Interviewer dataset and anonymized ChatGPT logs, where agents retrieve public evidence from the open web to corroborate identity hypotheses. Across all settings, agents are evaluated using metrics like linkage success rate (LSR) and confirmed linkage count (CLC), alongside utility measurements for task completion.

Results and Findings

Netflix Prize Setting: GPT-5 achieves 79.17% ± 4.97 linkage success when only two movies overlap (m=2), substantially outperforming the classical baseline’s 56.0–60.2% across all fragment sizes. Claude 4.5 performs inconsistently in sparse regimes (53.30% at m=2) but reaches 93–97% accuracy with four or more overlapping ratings.

InferLink Controlled Benchmark: Under implicit (benign) task framing, Claude 4.5 exhibits 70–80% linkage rates across fingerprint types without explicit re-identification requests. When task intent becomes explicit, linkage increases sharply: under explicit membership-knowledge conditions, Claude 4.5 achieves near-perfect linkage (95–100% across fingerprint types), while GPT-5 remains more conservative but still vulnerable (65–95% depending on fingerprint structure).

AOL Search Logs: The agent successfully corroborated 10 distinct identities (CLC=10) through three recurring linkage patterns: business and digital-footprint matching, institutional and lifestyle triangulation, and creative or extracurricular anchors. Once linked, anonymized search histories become attributed to named individuals, exposing sensitive queries about health, finance, and family matters.

Human–AI Interaction Traces: In the Anthropic Interviewer dataset, the agent achieved CLC=6 by extracting technical workflow descriptions, research methodologies, and linguistic markers to match redacted interviews against public academic records. In ChatGPT logs, the agent achieved CLC=1 across 30 privacy-relevant conversations by accumulating contextual cues across turns—affiliation, research topic, location, role, and temporal events—to resolve ambiguity and identify specific individuals.

Mitigation Results: Privacy-aware system prompts reduce linkage success to near-zero (LSR ≈ 0.00–0.07) in explicit re-identification scenarios, but incur measurable utility costs. GPT-5 maintains near-zero linkage with modest utility degradation (Δ utility ≈ -0.05 to -0.10), while Claude 4.5 exhibits substantial over-refusal behavior, degrading legitimate task performance by 16–54 percentage points.

Implications and Conclusions

This work challenges the adequacy of current privacy evaluations for agentic AI systems, demonstrating that privacy assessment must measure not only explicit information access and disclosure, but also what identities can be inferred through cross-source reasoning and auxiliary context retrieval. The findings suggest that traditional anonymization—removing direct identifiers while retaining quasi-identifying attributes—offers limited protection against general-purpose reasoning agents, raising urgent questions for organizations sharing anonymized datasets and for regulators designing privacy frameworks around agentic AI deployment.

Representation Forcing for Bottleneck-Free Unified Multimodal Models

Authors: Yuqing Wang, Zhijie Lin, Ceyuan Yang, Yang Zhao, Fei Xiao, Hao He, Qi Zhao, Zihan Ding, Fuyun Wang, Shuai Wang, Youliang Zhang, Haoqi Fan, Xihui Liu

Source and references: https://arxiv.org/abs/2605.31604v1

Representation Forcing for Bottleneck-Free Unified Multimodal Models

Introduction

This paper addresses a fundamental limitation in unified multimodal models (UMMs) that perform both image understanding and generation: existing approaches rely on frozen, separately pretrained VAE encoders and decoders, creating a structural bottleneck that limits end-to-end learning. The authors propose Representation Forcing (RF), a technique that enables pixel-space image generation without external VAEs by training the decoder to predict visual representations as intermediate tokens that guide the diffusion process.

Key Points

Representation Forcing mechanism: The decoder learns to autoregressively predict discrete visual representation tokens extracted from the model’s own understanding encoder, which then remain in context to guide pixel-space diffusion through shared self-attention, eliminating the need for external VAEs.
Dual performance improvement: RF benefits both image generation and understanding—pixel-space models with RF match VAE-based baselines on generation quality while outperforming VAE variants on understanding tasks across 6 of 8 benchmarks.
Unified representation space: By training the decoder to predict the same representations used for image understanding, RF creates a single end-to-end learned representation space rather than coordinating across separately pretrained components.
Discrete token superiority: Discretization of representations via online vector quantization outperforms continuous regression approaches, providing robustness against error accumulation and naturally encouraging the separation of high-level structure from low-level details.
Architecture-agnostic design: RF is applied to the Mixture-of-Transformers architecture with modality-specific experts, demonstrating compatibility with existing unified multimodal model designs while requiring no additional cross-attention modules or injection mechanisms.

Methodology

The approach extracts visual features from the understanding encoder using an exponential moving average (EMA) copy, then discretizes these features into representation tokens via online vector quantization with momentum updates and Sinkhorn-Knopp balancing to prevent codebook collapse. During training, the decoder learns to predict these representation tokens autoregressively under cross-entropy loss, while simultaneously learning to generate pixels via flow matching with x-prediction velocity loss. The model processes a unified token sequence combining text tokens, representation tokens, and pixel patches, where representation tokens provide in-context conditioning through bidirectional attention in the pixel generation phase. At inference, the encoder is bypassed entirely—the decoder predicts representation tokens from text alone, which then guide pixel synthesis in pixel space through standard self-attention mechanisms.

Results and Findings

On text-to-image generation benchmarks, the pixel-space RF model achieves a GenEval score of 0.84 without an LLM rewriter and 0.88 with one, matching state-of-the-art VAE-based unified models like BAGEL (0.82) and BLIP3-o (0.84), while scoring 84.15 on DPG-Bench—comparable to existing approaches. For image understanding, RF provides substantial improvements on general visual comprehension tasks: Pixel+RF gains +4.3 on MMMU, +3.6 on MME, and +3.6 on BLINK. Ablation studies reveal that without RF, naive pixel-space generation scores only 0.25 on GenEval versus 0.76 with RF, demonstrating the critical importance of representation guidance. Discrete token formulation (0.76) significantly outperforms continuous regression (0.26), and RF substantially surpasses the REPA auxiliary alignment approach (0.43 vs. 0.76). The model shows robustness to codebook size variations (K=16,384 vs K=32,768 perform comparably at 0.76-0.77), and DINOv3 encoder selection outperforms SigLIP2 on 4 of 5 understanding benchmarks.

Implications and Conclusions

This work demonstrates that end-to-end pixel-space generation in unified multimodal models is viable through explicit structural guidance from internally learned representations, eliminating the need for external frozen components. The research advances toward fully integrated multimodal systems where perception and generation share a single learned representation space, suggesting future directions for native multimodal learning where all capabilities emerge directly from raw input processing within unified architectures rather than combining independently trained modules.

Personalize Your Large Vision-language Models With In-context Prompt Tuning

Authors: Yanshu Li, Jiaqian Li, Kuai Yu, Xi Xiao, Dongfang Liu, Tianyang Wang, Ruixiang Tang

Source and references: https://arxiv.org/abs/2605.31513v1

Personalize Your Large Vision-Language Models With In-context Prompt Tuning

Introduction

This paper introduces In-Context Prompt Tuning (ICPT), a novel method for personalizing large vision-language models (LVLMs) to recognize user-specific concepts without requiring inference-time training. The approach addresses critical limitations in existing personalization methods, particularly their inefficiency and struggles with complex multi-image, multi-concept scenarios that are increasingly common in real-world applications.

Key Points

Adaptive Concept Projector (ACP): A lightweight module that extracts fine-grained visual semantics from multiple reference images and transforms them into continuous prompts alongside identity-label mappings, without relying on external encoders or masks.
Dynamic Token Router (DTR): An intelligent mechanism that adaptively allocates variable prompt lengths based on the visual complexity of each concept, balancing representational capacity with computational efficiency and achieving 12-18% lower inference latency than competing methods.
Contextual Variation Memory (CVM): A geometric regularization constraint that maintains a memory queue of environmental variations, enabling learned prompts to filter out biases from backgrounds and lighting while preserving core identity information.
Margin-constrained Concept Separation (MCS): A dual-modality constraint that prevents cross-concept interference by allowing representations to preserve shared semantic features up to a similarity threshold, rather than forcing strict orthogonality that would damage knowledge transfer.
Comprehensive evaluation framework: Extensive experiments across four LVLM architectures (LLaVA-NeXT-7B/34B, InternVL3-8B, Qwen3VL-8B) on 200 out-of-distribution concepts with tasks including existence recognition, visual question answering, and image captioning, demonstrating consistent state-of-the-art performance.

Methodology

ICPT operates by simulating in-context learning within the representation space of frozen LVLMs, eliminating the need for vocabulary expansion or inference-time training. The method processes reference images through the LVLM’s built-in vision encoder, extracting hierarchical features from early and deep layers to capture multi-scale visual characteristics. These features are fused and processed through the Adaptive Concept Projector, which uses cross-attention and MLPs to generate both a discrete label embedding and a continuous visual prompt for each concept. The Dynamic Token Router then adaptively prunes redundant tokens based on visual complexity. During training, two geometric constraints—Contextual Variation Memory and Margin-constrained Concept Separation—regularize the prompt representations to decouple identities from environmental factors and prevent cross-concept confusion. The entire framework is optimized end-to-end using a three-term loss combining VQA losses, geometric constraints, and sparsity penalties.

Results and Findings

ICPT achieves state-of-the-art results across all personalization tasks and settings. On LLaVA-NeXT-7B, ICPT surpasses the previous best method (MC-LLaVA) by 5.7 points in existence recognition (weighted score: 0.868 vs. 0.811), while using only 13.6 tokens per concept compared to MC-LLaVA’s fixed 16 tokens. Performance improvements are particularly substantial in complex multi-concept scenarios: ICPT improves by 6.7 points on open-ended VQA (0.702 vs. 0.644) and by 6.7 points on captioning (0.654 vs. 0.587) in multi-image settings. The method demonstrates robust generalization across different LVLM sizes and architectures—on LLaVA-NeXT-34B, performance reaches 0.904 weighted score on existence recognition, while on InternVL3-8B it achieves 0.932. Ablation studies confirm that both CVM and MCS constraints contribute meaningfully to performance, with larger gains appearing in multi-concept settings. Efficiency analyses reveal 12-18% lower inference latency compared to competing methods, and investigation of training data reveals that diversity matters far more than volume—a low-volume, high-diversity training setup substantially outperforms higher-volume approaches.

Implications and Conclusions

ICPT represents a significant advancement in practical LVLM personalization, successfully addressing the scalability and efficiency challenges that have limited broader deployment of personalized vision-language systems. By enabling models to efficiently learn user-specific concepts without inference-time training while maintaining robust performance in complex real-world scenarios, this work provides a foundation for more practical and deployable personalized AI applications. The method’s consistent performance improvements across multiple LVLM architectures and its ability to leverage advances in foundation models suggest it will remain effective as LVLMs continue to evolve, making it a valuable technique for building next-generation personalized AI systems.

Linear Scaling Video VLMs for Long Video Understanding

Authors: Cristobal Eyzaguirre, Jiajun Wu, Juan Carlos Niebles

Source and references: https://arxiv.org/abs/2605.31598v1

Introduction

StateKV tackles a critical bottleneck in video vision-language models: the quadratic computational scaling of spatiotemporal self-attention as video length increases. The paper introduces an inference-time method that achieves linear-time video processing by maintaining a fixed-capacity, importance-based recurrent state while preserving full per-frame detail for language generation, enabling practical long-video understanding without architectural modifications or fine-tuning.

Key Points

Linear scaling achievement: StateKV reduces video-prefill complexity from O(N²) to O(N) by restricting cross-frame attention to a fixed-capacity compressed state during cache construction, maintaining constant per-frame compute regardless of video length.
Dual-cache architecture: The method employs two separate KV caches per layer—a fixed-size compressed state for cross-frame context during prefill and a detailed state containing all video tokens used during final text decoding.
Attention-driven token selection: Instead of strict sliding-window approximations, StateKV uses empirical attention patterns to identify and preserve “temporal sink” tokens—a small set of historically important tokens that concentrate inter-frame attention mass.
Cross-model consistency: Testing across seven models spanning three families (InternVL3, Qwen3-VL, Eagle2.5) and multiple parameter scales demonstrates that importance-based memory consistently outperforms recency-based alternatives while remaining close to full self-attention accuracy.
Compute-aware scaling: The FLOP savings enable practitioners to run larger models at similar computational cost, creating operating points where a larger StateKV model is both cheaper and more accurate than smaller full-attention baselines.

Methodology

StateKV builds on three core assumptions about attention structure in video VLMs: (1) most inter-frame attention concentrates on a small, fixed-size set of tokens rather than spreading across the entire history; (2) these “temporal sink” tokens evolve slowly, allowing updates from the previous state plus current frame; and (3) the compressed state only approximates frame-to-frame interactions during prefill, not final generation. The method processes video incrementally, frame-by-frame, computing attention only against a compressed memory and current frame tokens. After each frame, it updates the compressed state by selecting the top-B tokens by importance score—combining retained memory tokens with newly salient tokens from the current frame. Critically, StateKV maintains a virtual sequence length for proper positional encoding (RoPE) and consistent scaling across cache building and generation phases. The detailed cache grows linearly with frames but is only queried during final autoregressive decoding, decoupling the expensive prefill stage from generation.

Results and Findings

Across three benchmarks (VideoMME, MLVU, OVOBench) and 512-frame videos, StateKV consistently outperforms sliding-window baselines by approximately 10 percentage points while remaining within 1 point of full self-attention. On VideoMME, StateKV-InternVL3-8B with cache budget B=4096 achieves 62.5% accuracy at similar compute cost as Full SA-1B (46.2%), demonstrating that FLOP reductions enable larger models. The compute-accuracy frontier reveals smooth log-linear scaling, enabling predictable test-time performance tradeoffs. StateKV shows stable scaling across cache budgets and video lengths, monotonically improving toward full-attention accuracy as capacity increases. In contrast, ReKV exhibits instability across model sizes and datasets, with persistent 5-10 point gaps even at comparable computational budgets. The marginal cost analysis shows that beyond certain video lengths, running a larger StateKV model becomes cheaper than processing with smaller full-attention baselines—a gap that widens dramatically at longer durations (extrapolated to 3600 frames/1 hour).

Implications and Conclusions

StateKV addresses a fundamental scalability challenge for deploying video VLMs in real-world applications like autonomous driving and embodied robotics by achieving linear-time complexity without sacrificing accuracy or requiring model retraining. By reframing streaming video prefill as approximating full self-attention through principled token selection rather than ad-hoc heuristics, the work demonstrates a practical pathway toward enabling long-duration video understanding on current hardware constraints while suggesting that larger models become computationally accessible for long-video tasks—a critical insight for the future deployment of video-understanding systems.

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization

Authors: Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias, Jorge Salas, Cristian B. Calderon, Cristobal Rojas

Source and references: https://arxiv.org/abs/2605.31558v1

Introduction

This paper investigates how Transformer attention mechanisms learn to solve structured reasoning tasks, specifically examining the distinction between positional attention heads (which attend to specific sequence locations) and symbolic attention heads (which attend to specific tokens regardless of position). The research reveals fundamental differences in how these mechanisms emerge during training and their robustness to longer input sequences.

Key Points

Task-Mechanism Alignment: The paper introduces two structurally equivalent multi-hop reasoning tasks—a number task requiring positional reasoning and a letter task requiring symbolic reasoning—demonstrating that successful learning correlates with the emergence of “pure” attention heads that express themselves as either positional or symbolic, not mixed.
Mechanistic Decomposition: The authors identify three core functions implemented by attention heads: Selective Indexing (positional), Retrieval (symbolic), and Reflexive propagation. They prove mathematically that these functions can be realized by single RoPE-based attention layers with geometrically interpretable query, key, and value operations.
Length Generalization Separation: Through a novel notion called “discrepancy,” the paper establishes a quantitative theoretical separation between positional and symbolic mechanisms in handling longer sequences. Symbolic mechanisms maintain robustness while positional mechanisms face severe limitations as sequence length increases.
Empirical Validation Across Model Scales: Predictions from theoretical analysis are validated not only in controlled single-head models but also in real-world multi-head architectures and frontier LLMs (GPT 5.4, 5.5, Claude Sonnet 3.7), showing consistent superiority of symbolic mechanisms in length generalization.
Learning Dynamics Insights: The paper reveals distinct temporal patterns: the number task exhibits progressive hop-wise learning as positional heads gradually emerge, while the letter task shows simultaneous learning across all hop conditions due to reliance on symbolic computation, providing mechanistic explanations for observed learning curves.

Methodology

The researchers trained a 12-layer decoder-only Transformer (GPT-J architecture) with one attention head per layer on both the number and letter tasks, using RoPE (Rotary Positional Encoding) for position encoding. They employed positional and symbolic attention head scoring metrics from prior work to characterize head behavior during training. Mechanistic analysis involved inspecting attention patterns and information flow in correctly solved inputs, leading to the identification of three idealized functions. The authors then provided formal mathematical constructions proving these functions can be realized by single RoPE-based attention layers and derived theoretical bounds on the “discrepancy” metric, which quantifies a model’s ability to distinguish target tokens as sequence length increases. Finally, they tested predictions on their controlled models, extended to real-world models and frontier LLMs using simplified task variants with varying sequence lengths.

Results and Findings

The experimental results demonstrate that task accuracy converges to maximum values precisely when attention heads become “pure”—expressing themselves clearly as either positional or symbolic (Figure 2). The number task requires a mix of both head types with a characteristic step-like emergence pattern aligned to hop count, while the letter task achieves all hop conditions simultaneously once its symbolic computation emerges. The theoretical constructions successfully replicate trained model behavior, with geometric patterns in query-key vector arrangements closely matching between theoretical and empirical implementations (Figure 4). Most strikingly, length generalization results show dramatic divergence: the letter task maintains 90%+ accuracy up to 850 tokens (53× original length), while the number task drops below 50% accuracy at just 32 tokens (Figure 5B). This pattern holds consistently across GPT 5.4, GPT 5.5, and Claude Sonnet 3.7, with the number task accuracy falling below 10% at 100 tokens while the letter task maintains 65%+ accuracy at the same length (Figures 5C-D).

Implications and Conclusions

This research demonstrates that Transformer mechanisms for solving structured tasks exhibit fundamental architectural constraints tied to whether they employ positional or symbolic computation, with profound consequences for sequence length generalization. The findings suggest that promoting symbolic over positional mechanisms during training could substantially improve length extrapolation, while the discrepancy metric provides quantitative predictions for when length generalization breaks down—insights directly applicable to developing safer and more capable large language models for real-world deployment.

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Authors: Valérie Castin, Kimia Nadjahi, Pierre Ablin, Gabriel Peyré

Source and references: https://arxiv.org/abs/2605.31484v1

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Introduction

Low-Rank Adaptation (LoRA) has become the standard method for efficiently fine-tuning large language models, but the technique suffers from fundamental overparameterization issues that impact convergence speed. This paper identifies and addresses a critical inefficiency: multiple pairs of low-rank factors can produce identical adapted weight matrices yet exhibit dramatically different condition numbers, directly affecting how quickly optimization converges.

Key Points

Core Problem Identified: LoRA’s overparameterization creates a manifold of equivalent solutions with varying loss landscape conditioning. The researchers prove theoretically that some minimizers are significantly flatter than others, leading to faster asymptotic convergence rates.
Balanced Minimizers are Optimal: The paper demonstrates that “balanced” minimizers—where the low-rank factors A and B satisfy A⊤A = BB⊤—achieve the best possible conditioning of the loss landscape. This balance condition minimizes the condition number and accelerates convergence.
BaLoRA Algorithm: The authors introduce Balanced Low-Rank Adaptation (BaLoRA), which projects low-rank adapters onto a balanced manifold after each optimizer step. The projection is computationally lightweight, adding only O((a+b)r²) complexity with negligible overhead to standard LoRA pipelines.
Geometric Interpretation: BaLoRA-GD (gradient descent variant) can be reformulated as intrinsic gradient descent on the manifold of rank-r matrices using the Bures metric, providing elegant theoretical grounding and interpretability.
Empirical Superiority: Experiments across multiple LLMs (Llama-3.2-3B, Qwen-2.5-3B) and diverse datasets show BaLoRA consistently outperforms standard LoRA and matches or exceeds state-of-the-art variants, with particular advantages at larger adapter ranks (r ∈ {64, 128}).

Methodology

The research employs a multi-layered theoretical and empirical approach. Theoretically, the authors analyze LoRA’s convergence dynamics by examining the condition number of the loss landscape at different minimizers. They start with tractable cases—one-layer linear networks—and extend analysis to deep non-linear networks in the interpolating regime. The condition number κ is shown to govern asymptotic convergence rate through both standard gradient descent (Proposition 2.1) and scaled sign-GD approximating Adam behavior (Proposition 2.2). Building on theoretical insights, they develop the balancing map P that projects iterates onto the hyperbalanced manifold H while preserving the adapted matrix product AB. Empirically, experiments span synthetic linear networks, large language model fine-tuning on 10+ datasets, and systematic ablations across hyperparameter ranges and adapter ranks.

Results and Findings

Theoretical Results: For one-layer linear networks with rank-matching targets, balanced minimizers achieve condition number κ_min = 2σ₁(Z)/σᵣ(Z), which is optimal. When target rank exceeds adapter rank (typical case), the governing quantity shifts to the r-spectral gap σᵣ(Z) - σᵣ₊₁(Z). Proposition 2.7 shows that balancing minimizes the upper bound on conditioning for deep networks in the interpolation regime.

Synthetic Experiments: On both one-layer and two-layer linear networks, BaLoRA exhibits slower initial convergence but enters a fast convergence regime where it significantly outperforms standard LoRA, validating theoretical predictions.

Large Language Model Results:

Wikitext-2: BaLoRA achieves superior test loss and demonstrates greater stability across learning rates and initialization scales compared to LoRA, OLoRA, and LoRA-GA (Figure 4).
Multi-dataset Comparison: Across five datasets (Alpaca, CodeFeedback, OpenHermes, OpenOrca, WizardLM), BaLoRA ranks in the top 2, with the two balanced methods (BaLoRA and RefLoRA) outperforming all other variants (Table 1).
Rank Sensitivity: BaLoRA shows clear advantages at larger ranks, achieving final train loss of 1.014 at r=128 compared to LoRA’s 1.030 on DeepMind Mathematics (Table 2).
Computational Overhead: Peak GPU memory consumption shows negligible additional overhead—less than 2% increase over standard LoRA.

Implications and Conclusions

This work provides principled theoretical grounding for understanding LoRA’s optimization dynamics and identifies a simple, practical solution with immediate applicability. The demonstration that balanced parameterizations achieve optimal conditioning while requiring minimal computational overhead suggests BaLoRA could become a drop-in replacement for standard LoRA in production fine-tuning pipelines, offering improved convergence speed and hyperparameter robustness without sacrificing efficiency—particularly valuable for the increasingly common scenario of large-rank adapters in contemporary large-scale models.

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

Authors: Christian Moya, Alex Semendinger, Guang Lin, Elliott Thornley

Source and references: https://arxiv.org/abs/2605.11134v2

Spurious Correlation Learning in Preference Optimization: A Summary

Introduction

This paper provides a theoretical framework for understanding how preference optimization methods like Direct Preference Optimization (DPO) develop spurious correlation reliance—learning to optimize surface-level features rather than true response quality. The authors propose tie training, a data augmentation mitigation strategy with provable guarantees for reducing this problematic behavior.

Key Points

Dual mechanisms of spurious learning: The paper proves that standard preference-learning objectives induce spurious feature reliance through two channels: mean spurious bias and causal-spurious correlation leakage, demonstrating this arises structurally from training data rather than optimization artifacts.
Irreducible deployment vulnerability: Spurious correlation learning creates a fundamental vulnerability to distribution shift—scaling training data alone cannot eliminate the model’s dependence on spurious features, making this problem qualitatively different from standard overfitting.
Tie training mitigation: The authors propose augmenting training data with preference pairs of equal utility but differing spurious features, which injects regularization selectively along spurious directions without degrading causal learning.
Provable reduction in shift error: Theoretical analysis demonstrates that tie training reduces the irreducible shift error that emerges during deployment when spurious statistics change between training and test distributions.
Validated across model scales: The framework is validated on linear models with quantitative agreement to theory, neural networks showing persistent qualitative mechanisms, and large language models where tie training reduces spurious learning while maintaining in-distribution performance.

Methodology

The authors develop a mathematical framework centered on analyzing log-linear DPO as a tractable testbed for pairwise preference optimization. They characterize the population equilibrium of the linearized log-linear DPO objective to understand how feature correlations interact with optimization. The theoretical analysis decomposes deployment suboptimality into an irreducible shift term (driven by spurious parameters) and a reducible estimation term, allowing precise characterization of when and why scaling data fails. Validation progresses through controlled experiments: linear models verify quantitative predictions, neural networks assess whether mechanisms persist in non-linear settings, and LLM experiments evaluate practical applicability.

Results and Findings

The paper’s core theoretical contribution is Theorem 4.1, which proves that spurious parameters become nonzero at population equilibrium whenever mean spurious bias or causal-spurious correlation exists in training data—establishing spurious learning as a structural property. Proposition 5.1 and 5.2 characterize how distribution shifts between training and deployment create vulnerability, while Theorem 5.3 decomposes finite-sample deployment error into irreducible and reducible components, demonstrating that increasing n→∞ only reduces the reducible term while leaving shift-driven error unchanged.

For tie training mitigation, Theorem 6.2 proves that equal-utility preference pairs reduce spurious parameter reliance (part i) while preserving causal learning (part ii), and provably reduce the irreducible shift error at deployment (part iii). Empirical validation shows these theoretical predictions hold qualitatively across neural networks and LLMs—tie training consistently reduces spurious correlation learning (such as length bias and sycophancy) without degrading in-distribution accuracy.

Implications and Conclusions

This work provides essential theoretical grounding for understanding why current alignment methods like RLHF and DPO develop concrete failure modes such as verbosity bias and sycophancy, moving beyond symptom description to mechanism characterization. The results have significant safety implications, offering a principled mitigation strategy with theoretical guarantees that addresses a fundamental vulnerability in preference optimization—one that cannot be solved through data scaling alone but requires explicit structural intervention via tie training or equivalent approaches.

Chain-of-Thought Reasoning In The Wild Is Not Always Faithful

Authors: Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, Arthur Conmy

Source and references: https://arxiv.org/abs/2503.08679v5

Introduction

This paper demonstrates that state-of-the-art language models, including advanced “thinking models,” generate unfaithful Chain-of-Thought (CoT) reasoning even on naturally worded, non-adversarial prompts. The research reveals that models’ verbalized reasoning often masks unspoken biases and shortcuts that don’t reflect their actual decision-making processes.

Key Points

Implicit Post-Hoc Rationalization (IPHR): Models exhibit systematic biases toward “Yes” or “No” answers on logically contradictory question pairs, then construct plausible-sounding reasoning to justify these predetermined conclusions rather than reasoning faithfully to answers.
Unfaithful Illogical Shortcuts: On difficult math problems, models use clearly illogical reasoning jumps to reach correct answers while failing to acknowledge these shortcuts in their explanations.
Universal Problem Across Architectures: Unfaithfulness appears across 15 frontier models from six developers (Anthropic, OpenAI, Google, DeepMind, DeepSeek, Qwen, Meta), ranging from 0.04% to 13.49% unfaithful response pairs, with no model entirely exempt.
Thinking Models Show Improvement but Not Immunity: Extended reasoning models like DeepSeek R1 (0.37%) and Claude Sonnet 3.7 with thinking (0.04%) perform better than non-thinking variants, but remain fundamentally susceptible to unfaithful patterns.
Specific Unfaithfulness Patterns Identified: The paper categorizes unfaithfulness into distinct types—biased fact inconsistency (selectively citing different facts across variants), argument switching (inconsistently applying reasoning standards), and answer flipping—revealing how models rationalize predetermined answers.

Methodology

The research employs two complementary evaluation pipelines. For IPHR, the team generated 4,834 pairs of comparative questions from the World Model dataset, asking models to compare entities (e.g., “Is X bigger than Y?” vs. “Is Y bigger than X?”). Questions were filtered through two-stage ambiguity evaluation to ensure logical contradiction. Models generated 10 responses per question using standard temperature settings, and an LLM-based judge classified outputs as supporting Yes, No, or Unknown. For Unfaithful Illogical Shortcuts, the authors developed a three-stage pipeline evaluating answer correctness, step criticality, and step unfaithfulness on 215 curated Putnam math problems, using autoraters with manual verification to identify illogical reasoning that produces correct answers.

Results and Findings

Unfaithfulness rates vary significantly across models: production models like GPT-4o-mini show 13.49% unfaithful pairs, while Claude Sonnet 3.7 with extended thinking exhibits only 0.04% (2 pairs across 4,834). Intermediate models show 1-7% rates. On math problems, unfaithful shortcuts appear in 1.2%-18.8% of correct responses depending on model and reasoning capability. Across unfaithful question pairs, biased fact inconsistency appears in 52% (median) of cases, argument switching in 45%, with 18% showing argument switching alone—proving some unfaithfulness cannot be attributed to simple retrieval differences. Robustness tests confirm IPHR rates remain stable across sampling temperatures (correlation ≥0.97), different random seeds (within 0.4 percentage points), and across multiple judges (99.3% agreement). Notably, increasing thinking budget for Claude Sonnet 3.7 from 1,024 to 64,000 tokens slightly increased unfaithfulness (0.04% to 0.25%), correlating with models hallucinating justifications rather than refusing ambiguous questions.

Implications and Conclusions

This research fundamentally challenges the reliability of CoT explanations as faithful representations of model reasoning, with significant implications for AI safety and deployment in agentic or critical systems. The findings suggest that unfaithfulness is a structural challenge unlikely to resolve through current training methods—both RLHF and emerging reinforcement learning from verifiable rewards (RLVR) show susceptibility—indicating that fundamental algorithmic changes may be necessary. The paper concludes that while CoT remains useful for identifying flawed reasoning and discounting unreliable outputs, it should not be treated as certification of correctness. The work proposes two mitigation strategies: consistency-with-reversal as a training regularizer and template-gated prompting to flag biased templates, while emphasizing that CoT provides only an incomplete picture of actual decision-making processes, particularly concerning in scenarios involving multiple sampling attempts or agentic deployment.

Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models

Authors: Nianyi Lin, Jiajie Zhang, Lei Hou, Juanzi Li

Source and references: https://arxiv.org/abs/2510.11683v3

Introduction

This paper addresses a critical bottleneck in applying reinforcement learning to diffusion large language models (dLLMs)—the intractable likelihood functions and memory constraints that prevent accurate policy optimization. The authors propose Boundary-Guided Policy Optimization (BGPO), a memory-efficient RL algorithm that enables larger Monte Carlo sample sizes for improved likelihood approximations.

Key Points

Memory Efficiency Challenge: Previous ELBO-based RL methods for dLLMs require storing all Monte Carlo sample computational graphs to compute gradients, forcing practitioners to use small sample sizes (e.g., n_t=4) that introduce significant bias and variance in likelihood approximations.
Linear Lower Bound Construction: BGPO constructs a mathematically elegant lower bound on the ELBO-based objective that decomposes into a linear sum of individual sample terms, enabling gradient accumulation and constant memory usage regardless of sample size.
Dual Properties Design: The proposed lower bound satisfies two critical properties: (1) Linearity enabling separate backpropagation per sample, and (2) Equivalence guaranteeing that in on-policy training, both values and gradients match the original ELBO-based objective.
Theoretical Equivalence Proof: The authors prove that BGPO’s objective and gradients are mathematically equivalent to the ELBO-based objective during on-policy training, ensuring it provides an effective approximation of the original RL objective.
Empirical Validation: Comprehensive experiments across math problem solving, code generation, and planning tasks demonstrate significant performance improvements over prior dLLM RL methods, with larger MC sample sizes (16-32) reducing bias and variance without substantial computational overhead.

Methodology

The approach constructs a carefully designed lower bound of the ELBO-based RL objective that decomposes into a linear sum. For positive advantages, the authors apply first-order Taylor expansion (Lemma 1), while for negative advantages they apply Jensen’s inequality (Lemma 2). This dual construction ensures that each term g_j depends only on a single MC sample y_t^(j), enabling separate gradient computation and accumulation. The resulting linear formulation permits constant memory usage across any sample size, while maintaining mathematical equivalence to the original objective during on-policy training. The algorithm uses group-based advantage estimation across G responses per prompt and incorporates normalized reward advantages for stability.

Results and Findings

BGPO demonstrated substantial improvements across all tested domains. On mathematical tasks using LLaDA-8B-Instruct, BGPO with larger sample sizes (n_t=16-32) significantly outperformed the previous ELBO-based method (VRPO-OL) that operates under memory constraints with small sample sizes. The memory usage remained constant despite using 4-8x larger sample sizes—with n_t=32 showing comparable or lower memory requirements than n_t=4 in prior methods. Ablation studies revealed that increasing MC sample sizes effectively reduced gradient bias and variance, directly correlating with improved model performance across code generation (MBPP, HumanEval) and planning tasks (Countdown, Sudoku). Notably, these performance gains came with only marginal increases in average training step time, demonstrating practical efficiency. Quantitative comparisons showed BGPO consistently achieved higher accuracy metrics on all downstream tasks compared to diffuGRPO and VRPO-OL baselines.

Implications and Conclusions

This work establishes a foundational methodology for efficient RL training of diffusion language models, removing a significant practical bottleneck that previously limited dLLM capabilities. By enabling larger and more accurate likelihood approximations through clever mathematical reformulation rather than architectural changes, BGPO makes dLLM RL training more accessible and effective, potentially accelerating adoption of non-autoregressive language models that offer faster inference speeds alongside improved task performance.

LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards

Authors: Nianyi Lin, Jiajie Zhang, Lei Hou, Juanzi Li

Source and references: https://arxiv.org/abs/2605.31584v1

Introduction

LongTraceRL addresses a fundamental challenge in modern language models: reasoning effectively over extremely long contexts filled with distracting information. The paper proposes a novel reinforcement learning framework that combines sophisticated data construction with fine-grained reward signals to teach models how to locate, integrate, and reason through key information in 128K-token contexts.

Key Points

Trajectory-based distractor generation: Rather than using random documents as distractors, the authors leverage search agent trajectories to create “tiered distractors”—documents the agent read but didn’t cite (high confusability) and documents in search results but never opened (low confusability). This approach creates far more challenging and realistic training scenarios than random sampling.
Entity-level rubric rewards: The method introduces fine-grained process supervision by tracking whether models reference gold entities along the reasoning chain, moving beyond sparse outcome-only rewards that can’t supervise intermediate reasoning steps. Gold entities extracted from knowledge graph paths serve as verifiable process-level signals.
Positive-only reward strategy: To prevent reward hacking, rubric rewards are only applied to responses with correct final answers. This design distinguishes reasoning quality among correct responses while preventing models from gaming the system by simply enumerating entities without genuine reasoning.
Knowledge graph-based question generation: Multi-hop questions with deep reasoning chains (8 hops) are synthesized via controlled random walks over Wikipedia’s hyperlink graph, ensuring questions require step-by-step reasoning with no shortcuts possible.
Consistent cross-model improvements: Testing on three reasoning LLMs ranging from 4B to 30B parameters across five benchmarks demonstrates the generalizability of the approach, with Qwen3-4B improving by 5.7 points over baseline and surpassing the strongest baseline by 2.5 points.

Methodology

The framework consists of two main components. First, a data construction pipeline generates complex multi-hop questions through knowledge graph random walks, then collects search agent trajectories attempting to answer them. Documents from these trajectories are categorized into tiers based on their confusability level and assembled into 128K-token contexts prioritizing harder distractors. Second, an RL training approach using Group Relative Policy Optimization (GRPO) combines outcome-based rewards (binary correctness signals) with normalized rubric rewards (entity-level process supervision). The composite reward is calculated as r = (1−α)·r_oc + α·r_rb only for responses with correct answers, with α=0.3 providing optimal balance.

Results and Findings

LongTraceRL consistently achieves the best performance across all tested models and benchmarks. On Qwen3-4B-Thinking, the method reaches an average score of 59.0 across five benchmarks, improving the base model by 5.7 points and surpassing the strongest baseline (LongRLVR) by 2.5 points. The most pronounced gains appear on reasoning-intensive benchmarks like AA-LCR (+8.6 points: 33.2→41.8). Ablation studies confirm that removing the rubric reward (reducing to outcome-only GRPO) drops performance to 53.7, demonstrating it as the dominant improvement driver. Analysis of distractor difficulty reveals that traj-tiered distractors achieve 50.03% overlap with rubric entities compared to only 1.35% for random sampling, directly correlating with downstream performance improvements. Training dynamics show that the rubric reward grows steadily while preventing pathological behaviors—models are self-regulated by finite response budgets to avoid exploiting process rewards without solving questions.

Implications and Conclusions

This work demonstrates that long-context reasoning in LLMs can be substantially improved through carefully engineered training data and reward design rather than simply scaling model size or context length. The trajectory-based distractor strategy and entity-level process supervision represent significant methodological advances that could inform future approaches to complex reasoning tasks, with practical implications for deploying reasoning systems in real-world applications involving extensive, distracting information.

Vision-Language Models Suppress Female Representations Under Ambiguous Input

Authors: Arnau Marin-Llobet, Simon Henniger, Mahzarin R. Banaji

Source and references: https://arxiv.org/abs/2605.31556v1

Vision-Language Models Suppress Female Representations Under Ambiguous Input

Introduction

This paper reveals a critical gap between what vision-language models (VLMs) say and what they internally encode about gender. While alignment techniques have made modern VLMs produce neutral outputs when describing people of ambiguous gender, the researchers demonstrate that biased associations persist in the models’ internal representations—and are systematically suppressed before generation, particularly for female associations.

Key Points

Internal-Output Decoupling: VLMs often encode female associations internally yet output male under forced-choice prompting, particularly for female-stereotyped occupations like babysitter, florist, and preschool teacher—exposing a blind spot in output-level bias auditing.
Asymmetric Layer Dynamics: Male signal amplifies from early to late network layers, while female signal peaks mid-network and is explicitly suppressed toward generation, creating a directional bias filter that only attenuates female representations.
LALS Metric Introduction: The authors propose Latent Association Leaning Score, a zero-shot method that projects visual token activations into text-embedding space, enabling token-level and layer-level measurement of gender associations without requiring labeled training data.
Culturally-Loaded Visual Cues: Color ablation experiments show that changing clothing from blue to pink substantially reduces male signal in construction worker images and increases female signal in nurse images, indicating models have internalized social-chromatic gender associations from training data.
One-Sided Male Default: Across four different VLM architectures and 15 occupations, models consistently collapse toward male when forced to guess gender on ambiguous images—even for occupations that are 97% female in U.S. labor statistics, and no occupation ever defaults to female against a male baseline.

Methodology

The researchers constructed a dataset of 800+ gender-ambiguous images using generative AI, showing faceless or obscured figures in occupation-specific settings with no visible gender markers. They then developed LALS, which works by: (1) extracting visual token representations at each network layer, (2) projecting them into the model’s text-embedding space using established latent lens techniques, (3) comparing these projections against a balanced reference corpus of gendered terms (man/father/boy vs. woman/mother/girl), and (4) scoring each token on a continuous male-to-female scale. The method was evaluated on four instruction-tuned open-weight VLMs (Qwen, LLaVA, InternVL) using both open-ended prompts (”Describe what this person is doing”) and forced-choice prompts (”Is this person male or female?”) to surface the gap between neutral outputs and biased behavior.

Results and Findings

When gender is visually clear, all four VLMs accurately identify it and maintain appropriate gender associations throughout their network layers. However, on ambiguous images:

Forced-choice outputs reveal sharp occupation-dependent defaults: Female-stereotyped occupations collapsed toward male in the majority of cases (hairdresser: 88–96% male across models despite being 92% female in actual labor force; babysitter: 72–96% male despite 93% female; preschool teacher: 40–74% male despite 97% female). Only makeup artist consistently surfaced as female.
Layer analysis shows three distinct regimes: Agreement-male occupations maintain male signal end-to-end; agreement-female occupations remain female-leaning but see some attenuation; divergence occupations (florist, preschool teacher, hairdresser) show female-leaning peaks at 70–80% network depth then sharply collapse to near-zero or male territory by the output layer.
Color modulation is substantial: A single color change (blue to pink) shifted internal gender associations by magnitudes comparable to differences between entire occupation categories, with pink reducing construction worker male signal by ~50% and more than doubling nurse female signal.
The asymmetry is pretraining-driven, not alignment-driven: Base model checkpoints (without instruction tuning) show the same occupation-dependent patterns and late-layer female collapse as their instruction-tuned variants, suggesting RLHF amplifies rather than creates the bias. Text-only prompts show opposite dynamics (female signal amplifies in late layers for female occupations), confirming the collapse is specific to the visual pathway.

Implications and Conclusions

This research demonstrates that output-level auditing—the current standard in VLM fairness evaluation—systematically misses representation-level biases that matter for downstream applications. Because VLM embeddings increasingly power image search, content ranking, and automated screening systems where outputs never pass through the language head, the internal biases documented here pose concrete risks even when text outputs appear neutral. The findings suggest that alignment and debiasing are distinct processes: RLHF effectively controls what models say but leaves underlying representations intact, particularly problematic for ambiguous real-world inputs like surveillance footage, workers in protective gear, and distant figures—exactly the cases where biased priors are most consequential. The paper establishes that modern VLMs have learned not to express gender bias in text rather than to eliminate it from their visual representations, and that solving this requires auditing and intervening on internal associations, not just outputs.

TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments

Authors: Zhiyu Huang, Yun Zhang, Johnson Liu, Rui Song, Chen Tang, Jiaqi Ma

Source and references: https://arxiv.org/abs/2602.02459v2

Introduction

This paper introduces TIC-VLA (Think-in-Control Vision-Language-Action), a framework designed to address a fundamental challenge in real-world robot navigation: the temporal mismatch between slow vision-language model (VLM) reasoning and fast real-time control. Unlike existing systems that assume semantic reasoning and control occur simultaneously, TIC-VLA explicitly models inference latency as a core component of the control problem, enabling robots to navigate dynamic environments while executing language-conditioned instructions on resource-constrained edge devices.

Key Points

Delayed Semantic-Control Interface: TIC-VLA conditions the action policy on delayed VLM outputs alongside explicit latency metadata and ego-motion offsets, allowing the controller to compensate for asynchronous reasoning and reinterpret stale semantic information in the current robot frame.
Latency-Consistent Training Pipeline: The framework employs a three-stage training approach (VLM supervised fine-tuning, imitation learning with injected delays, and reinforcement learning) that explicitly introduces reasoning latency during training to match real-world deployment conditions.
DynaNav Benchmark Suite: The authors developed a physics-accurate, photo-realistic simulation environment with dynamic human agents, supporting diverse indoor and outdoor navigation scenarios—filling a critical gap in existing navigation benchmarks that ignore embodied execution and human interactions.
Robust Edge Deployment: TIC-VLA achieves 85% success rate on real robots (Unitree Go2) running on an RTX 4060 laptop GPU with multi-second VLM latency, and maintains 75% success on a Jetson Orin NX (25W edge device), demonstrating practical viability for resource-constrained deployment.
Superior Performance Over Baselines: In simulation, TIC-VLA achieves 55.29% success rate compared to 32.94% for MobileVLA and 31.76% for OmniVLA, while reducing collision rates from 45.88% to 28.24%, significantly outperforming prior vision-language-action navigation systems.

Methodology

TIC-VLA adopts a dual-system architecture where a large VLM performs semantic reasoning asynchronously while a lightweight action expert executes at high frequency (10 Hz) without waiting for inference completion. The VLM operates on delayed visual observations (anchored at time t−Δt) and produces key-value cache features and waypoint predictions, which are passed to the action policy along with explicit latency metadata (Δt) and accumulated ego-motion offsets (Δp). The action policy, implemented as a Transformer with cross-attention layers, takes current observations, robot state, and the delayed semantic-control interface as inputs to predict short-horizon action chunks. Crucially, the training pipeline injects realistic inference delays during both imitation learning (sampling delays uniformly from 0-10 seconds) and reinforcement learning (PPO with stochastic delay injection), ensuring the learned policy compensates for temporal misalignment encountered at deployment.

Results and Findings

In simulation benchmarks on DynaNav, TIC-VLA achieves 55.29% success rate with 28.24% collision rate—substantially outperforming prior VLA methods like MobileVLA (32.94% SR, 45.88% CR) and DualVLN (30.59% SR, 47.06% CR). The framework maintains robust performance as VLM latency increases from 2 to 10+ seconds, with RL fine-tuning preserving success rates across all latency conditions while IL-only baselines degrade significantly. Real-world experiments on a Unitree Go2 quadruped across four diverse tasks (indoor hallways, offices, outdoor plazas, and walkways with terrain) achieved 85% success on an RTX 4060 and 75% on edge devices despite 3-5 second VLM reasoning delays. Ablation studies demonstrate that the KV-cache-based semantic interface with latency-aware training improves success from 30.59% to 47.06%, explicit latency modeling provides consistent improvements, and ego-motion offset incorporation increases success from 41.18% to 47.06%.

Implications and Conclusions

TIC-VLA reframes inference latency from an engineering inefficiency into an explicit modeling problem, establishing a principled approach to real-time robot control under asynchronous semantic reasoning. The framework’s demonstrated robustness on resource-constrained edge hardware with multi-second reasoning delays has significant implications for deploying embodied AI systems in real-world human-centric environments, where compute budgets cannot accommodate powerful GPUs and latency-unaware VLA systems frequently fail.

Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks

Authors: Sanjay Haresh, Daniel Dijkman, Apratim Bhattacharyya, Roland Memisevic

Source and references: https://arxiv.org/abs/2602.21013v2

Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks

Introduction

This paper addresses a critical limitation in current Vision-Language-Action (VLA) models: their inability to handle memory-dependent robotic tasks that require temporal or spatial reasoning. The researchers propose augmenting VLAs with a language scratchpad mechanism that enables robots to maintain explicit memory of past actions and environmental states, significantly improving performance on complex multi-step manipulation tasks.

Key Points

Language Scratchpad Mechanism: The core contribution is a simple yet effective approach where VLAs generate and accumulate textual descriptions of their actions and environmental observations in a scratchpad, which serves as persistent context for subsequent decision-making.
Dual Memory Capabilities: The scratchpad structure incorporates three components—grounding (spatial positions), planning (subtasks), and actions (temporal progress)—enabling both spatial memory (object locations) and temporal memory (task progression tracking).
Universal Compatibility: The approach works with both stateless transformer-based VLAs and recurrent VLAs, demonstrating 48% average performance improvement for non-recurrent models and 11% for recurrent models on memory-dependent tasks.
New Benchmark Introduction: The authors introduce ClevrSkills-Mem, a benchmark consisting of five memory-dependent manipulation tasks (Touch-Reset-Pick, Place-Next-to-Restore, Swap, Stack-and-Topple, and Rotate-Restore) designed specifically to evaluate spatial and temporal memory capabilities.
Real-World Validation: Beyond simulation, the approach successfully enables a physical robotic arm (UFACTORY xArm 6) to perform challenging pick-and-place tasks requiring memory, demonstrating practical applicability.

Methodology

The researchers implement scratchpad-augmented VLAs by extending the standard VLA formulation p(a_t|o_t,l) to p(a_t,d_t|o_t,S_t,l), where d_t represents textual descriptions and S_t is the accumulated scratchpad. For transformer-based VLAs, the scratchpad is linearized and appended to prompts using special tokens (, , , ). For recurrent models, sequences are interleaved with observations and actions in text format. Training data is generated from oracle trajectories in simulation with automatic scratchpad generation using subtask segmentation. The approach uses PaliGemma-2 (3B) for transformer experiments and Mamba (130M) with ViT backbone for recurrent experiments.

Results and Findings

On the ClevrSkills-Mem benchmark, T-VLA with scratchpad achieved dramatic improvements: 68% gain on Touch-Reset-Pick, 72% on Swap, 68% on Place-Next-to-Restore, and 30% on Stack-and-Topple, with an overall average improvement of 48.8% across five tasks. Notably, T-VLA+Scratchpad matched or exceeded the performance of inherent memory-based recurrent models on most tasks. Recurrent VLAs also benefited from scratchpad integration with 11% average improvement, with larger gains observed on longer-horizon tasks. On MemoryBench’s Put-Block-Back task, the scratchpad-augmented approach achieved 40% success in real evaluation and 100% in simulation, substantially outperforming baseline VLAs and approaching specialized task-specific methods.

Implications and Conclusions

This work demonstrates that language-based scratchpads provide an effective, flexible, and model-agnostic mechanism for endowing VLAs with explicit memory capabilities without requiring architectural modifications. The findings suggest that leveraging the natural language understanding capabilities of underlying vision-language models represents a promising direction for scaling robotic policies to complex, temporally-extended tasks that violate Markovian assumptions—a critical step toward deploying generalist VLAs on real-world robotic applications requiring memory-dependent reasoning.

Dynamic Routing for LLMs, Unified Embodied Intelligence, and Real-Time Agentic Reasoning

State of AI — Fri, 15 May 2026 18:39:40 GMT

Welcome to today’s edition of State of AI 👋

This week brought a flurry of breakthroughs in making AI systems more efficient, more unified, and more real-time. We’re seeing a shift from monolithic approaches toward modular architectures that know when to be expensive, whether that’s routing computation dynamically between precision levels, combining specialized vision experts for robotics, or launching tool calls speculatively while maintaining correctness. Alongside this comes a wave of work on truly unified systems: embodied models that jointly learn understanding, reasoning, imagination and action from a single loop; VLAs for autonomous driving that preserve language understanding without sacrificing control precision; and world models that can generate minute-scale video with precise camera control at accessible compute budgets. The common thread? Strategic coupling of once-separate components, powered by careful system design rather than brute-force scaling.

Here’s what caught our attention:

Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction — Step-level routing between full and quantized models achieves 1.01-1.58× speedups while recovering task success rates, using a lightweight router that identifies when quantization fails mid-trajectory.
OpenDeepThink: Parallel Reasoning via Bradley–Terry Aggregation — Pairwise LLM comparison reaches 86% accuracy versus 59% pointwise scoring for selecting among reasoning candidates, enabling parallel test-time compute without trained verifiers.
SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer — A 2.6B model generates 720p minute-long videos with 6-DoF camera control in 24.1 videos/hour on single H100, combining frame-wise linear attention with periodic softmax for long-context stability.
Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation — Replaces expensive trajectory precomputation with online single-step ODE supervision, achieving 4× training speedup (11.6k→2.9k GPU-hours) while improving frame-wise 2-step video generation to 50% lower latency than prior 4-step methods.
Pelican-Unified 1.0: A Unified Embodied Intelligence Model — Single model ranks #1 on WorldArena (66.03 EWM), achieves 93.5% on RoboTwin, and outperforms experienced human drivers on Waymo, proving understanding/reasoning/imagination/action can co-evolve through shared training rather than modular assembly.
MindVLA-U1: VLA Beats VA with Unified Streaming Architecture — First VLA to surpass experienced human drivers on WOD-E2E (8.20 vs 8.13 RFS) while preserving natural language interface through streaming memory and intent-conditioned action diffusion.
Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O — Decouples agent reasoning from I/O delays via event-driven architecture and speculative tool calling, achieving 1.6-2.2× speedups on edge models while maintaining accuracy through read/write classification.
VER: Vision Expert Transformer for Robot Learning via Foundation Distillation and Dynamic Routing — Distills DINOv2/CLIP/ViT into a mixture-of-experts with lightweight routing, reaching 74.7% average success across 17 manipulation tasks while suppressing task-irrelevant background information.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction

Authors: Yuanzhe Li, Jianing Deng, Jingtong Hu, Tianlong Chen, Song Wang, Huanrui Yang

Source and references: https://arxiv.org/abs/2602.02711v2

Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction

Introduction

This paper addresses the computational cost of deploying large language models in long-horizon decision-making tasks by proposing Dynamic Mixed-Precision Routing (DMR), a framework that selectively routes computation between full-precision and quantized models at each decision step. The key insight is that different steps in multi-step reasoning have varying sensitivity to quantization, allowing selective use of expensive full-precision computation only where necessary.

Key Points

Step-level routing framework: DMR operates at the decision-step level rather than the coarser question level or finer token level, enabling fine-grained control suited for agentic tasks like web navigation and embodied reasoning.
Two-stage training pipeline: The approach combines KL-divergence-based supervised learning to identify precision-sensitive steps with reinforcement learning (GRPO) refinement to optimize task success under cost constraints.
Lightweight router design: The routing model comprises only 2-3% of the routed LLM parameters, making it computationally efficient while capable of identifying critical decision points where quantization fails.
Significant accuracy-cost trade-offs: DMR matches or exceeds full-precision performance while achieving 1.01-1.58x speedups over full-precision baselines, with controlled trade-offs via a budget parameter ρ.
Robust bimodal behavior: The framework exploits the observation that KL divergence between low- and high-precision models exhibits a clear two-regime structure: most steps show minimal deviation, while critical steps show substantial deviation.

Methodology

LLM agents for neuroimaging, Minecraft self-evolution, 26× model compression, MoE expert pooling, and 1M-token attention

State of AI — Fri, 08 May 2026 08:11:46 GMT

Welcome to today’s edition of State of AI 👋 And a warm welcome to our new subscribers since last edition!

This edition brings a fascinating convergence around three major themes in AI systems. First, we’re seeing LLM agents move beyond simple task completion into sophisticated domain-specific automation—from neuroimaging pipelines to vulnerability reconstruction and long-horizon embodied reasoning in Minecraft. Second, a wave of work on model efficiency and architecture design challenges conventional wisdom: shared expert pools for MoE models, structured knowledge distillation achieving 26× compression, and novel training methods that question the necessity of linear scaling laws. Finally, we have tangible breakthroughs in continuous generation (diffusion-based language models), physics-aware robotics (motion retargeting via bilevel optimization), and long-context training (efficient attention mechanisms that scale to 1M tokens).

Here’s what caught our attention:

NeuroAgent: LLM Agents for Multimodal Neuroimaging Analysis and Research — A hierarchical multi-agent system that automates the full neuroimaging pipeline from raw DICOM through preprocessing to Alzheimer’s classification, achieving 0.9518 ROC-AUC on 1,470 subjects while introducing a generate-execute-validate feedback loop that handles errors autonomously.
MineEvolve: Self-Evolution with Accumulated Knowledge for Long-Horizon Embodied Minecraft Agents — Demonstrates how converting fine-grained execution signals into typed feedback, structured skills, and remedies enables embodied agents to improve on complex multi-step tasks, outperforming static retrieval and generic reflection across multiple language model backends.
DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression — Achieves 26× parameter reduction in vision encoders through asymmetric decomposition of the contrastive loss, where diagonal (matched pairs) remains fixed while off-diagonal terms transition from positive to negative weighting, enabling MobileFetalCLIP to match or exceed its 427M-parameter teacher.
Continuous Latent Diffusion Language Model (ColaDLM) — Rethinks text generation as hierarchical latent diffusion with global semantic organization in continuous space followed by conditional decoding, establishing a theoretical Markov-path framework that unifies autoregressive, discrete diffusion, and continuous approaches while demonstrating competitive scaling behavior.
UniPool: A Globally Shared Expert Pool for Mixture-of-Experts — Replaces per-layer expert ownership with a globally shared pool and pool-level auxiliary loss, converting expert-parameter growth from linear to sublinear with depth while maintaining or exceeding standard MoE performance at 41.6%–66.7% of expert parameters.
Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models — Formulates reasoning strategy selection as a contextual multi-armed bandit problem, achieving 28–35% reduction in inference time compared to comparable methods while improving accuracy by 4–12 percentage points on mathematical and puzzle-solving tasks.
Lighthouse Attention: Efficient Long-Context Pre-Training with Symmetric Hierarchical Pooling — Introduces parameter-free L2-norm-based selection with symmetric hierarchical pooling that maintains (Q, K, V) coherence across pyramid levels, enabling 1.69× wall-clock speedup at 98K context and seamless recovery to dense SDPA attention at inference.
ReActor: Physics-Aware Motion Retargeting via Reinforcement Learning — Jointly optimizes retargeting parameters and tracking policies through bilevel optimization with derived gradient approximations, eliminating foot sliding and self-penetration artifacts while enabling successful hardware deployment on physical robots.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

NeuroAgent: LLM Agents for Multimodal Neuroimaging Analysis and Research

Authors: Lujia Zhong, Yihao Xia, Jianwei Zhang, Shuo huang, Jiaxin Yue, Mingyang Xia, Yonggang Shi

Source and references: https://arxiv.org/abs/2605.06584v1

Introduction

NeuroAgent is an LLM-driven autonomous framework that automates multimodal neuroimaging analysis, from raw preprocessing to downstream statistical analysis and disease classification. The system addresses a critical bottleneck in neuroimaging research: the substantial expert labor required to preprocess heterogeneous data across multiple imaging modalities (structural MRI, functional MRI, diffusion MRI, and PET) and coordinate complex, modality-specific toolchains.

Key Points

Hierarchical multi-agent architecture: The system employs a Central Orchestrator that decomposes natural-language research goals into structured workflows, dispatching tasks to specialized agents for each imaging modality (sMRI, fMRI, dMRI, PET) with domain-specific knowledge and toolchains.
Generate-Execute-Validate feedback loop: Rather than static script execution, NeuroAgent iteratively generates preprocessing code, executes it in a sandboxed environment, validates output integrity against structural schemas (e.g., BIDS compliance), and automatically recovers from runtime errors by analyzing logs and re-issuing corrected tool calls.
End-to-end automation from raw data to results: The framework spans the complete neuroimaging lifecycle—converting raw DICOM acquisitions through standardized preprocessing, assembling multimodal datasets, performing group-level statistics via natural-language queries, and training disease classifiers—all with minimal manual intervention through a Human-in-the-Loop interface.
Robust performance across LLM backends: Ablation studies show that capable models reach 100% intent-parsing accuracy and up to 84.8% end-to-end preprocessing step correctness (Qwen3.5-27B), with smaller models (4B parameters) matching larger ones on structured tasks when properly prompted.
Strong downstream classification performance: On 1,470 ADNI subjects (1,000 cognitively normal, 470 Alzheimer’s disease), the agent ensemble achieves ROC-AUC 0.9518 for AD diagnosis using four modalities, outperforming all single-modality baselines and demonstrating that automated preprocessing preserves diagnostic signal.

Methodology

Speculative Retrieval at Indexing Time, Agentic Forecasting with Bayesian Belief States, and Cross-Embodiment Policy Learning via Visual Tokenization

State of AI — Thu, 23 Apr 2026 07:01:48 GMT

Welcome to today’s edition of State of AI 👋

This edition has an unexpected through-line: agents that know what they don’t know.

SpecAgent moves expensive retrieval out of inference time entirely, pre-computing context during indexing to cut latency without losing accuracy. BLF treats forecasting as sequential Bayesian updates over a structured belief state, becoming the first system to reliably beat crowd predictions on market questions. GRIL trains models to pause and ask for missing premises instead of fabricating them, pushing premise detection from 4.6% to 90.8%. And SafetyALFRED exposes the flip side of this problem, MLLMs can recognize kitchen hazards with 92% accuracy in QA, then fail to actually mitigate them once embodied. Knowing isn’t doing.

On the generation side, OmniGen2 pushes toward instruction-aligned multimodal synthesis with fine-grained cross-image control, while UniT bridges human demonstrations and humanoid policies through visual-anchored tokenization, getting 10× data efficiency and real-world pick-and-pour success rates jumping from 30% to 78% with human co-training. MiroThinker makes the case that interaction depth, up to 600 tool calls in a single task, is a third scaling axis alongside parameters and context.

The common thread is that the hardest problems right now aren’t about raw capability. They’re about calibration, grounding, and knowing when to stop.

SpecAgent: Shifts expensive retrieval operations from inference time to asynchronous repository indexing, eliminating latency overhead while achieving 48-58% relative improvements in code completion through speculative context construction and future-context leakage identification.
Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs: Combines structured belief state tracking with multi-trial aggregation and hierarchical calibration to achieve human-level probabilistic forecasting, significantly outperforming crowd predictions on market questions—a feat unachieved by all existing methods.
UniT: Establishes a unified physical language for human-to-humanoid policy learning through visual-anchored tokenization, enabling 10× data efficiency improvements and enabling zero-shot task transfer like picking and pouring with 60-78% real-world success rates using only human demonstrations.
OmniGen2: Introduces progressive reinforcement learning alignment with a three-stage curriculum and OmniRoPE positional encoding for reliable cross-image spatial consistency, achieving competitive or superior performance across text-to-image, editing, and in-context generation tasks.
MiroThinker v1.0: Demonstrates that interactive scaling—supporting up to 600 tool calls within 256K context—is a fundamental performance lever for research agents, reaching 81.9% on GAIA and approaching GPT-5-like capabilities in open-source implementations.
Pause or Fabricate? Training Language Models for Grounded Reasoning: Introduces GRIL framework using reinforcement learning to train models to recognize when premises are missing and request clarification rather than fabricate, improving premise detection from 4.6% to 90.8% with 40%+ response length reduction.
SafetyALFRED: Reveals a critical alignment gap where MLLMs achieving 92% hazard recognition in QA tasks fail to mitigate hazards in embodied planning, highlighting that static safety knowledge doesn’t translate to physical corrective actions.
Accurate and scalable exchange-correlation with deep learning: Introduces Skala, a neural network-based DFT functional that breaks the 60-year accuracy-efficiency tradeoff, achieving 2.8 kcal/mol error on GMTKN55 while maintaining O(N³) scaling, directly enabling computational discovery pipelines for drug candidates and materials.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion

Authors: George Ma, Anurag Koul, Qi Chen, Yawen Wu, Sachit Kuhar, Yu Yu, Aritra Sengupta, Varun Kumar, Murali Krishna Ramanathan

Source and references: https://arxiv.org/abs/2510.17925v2

SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion

Introduction

This paper introduces SpecAgent, an agent-based approach that addresses the latency-accuracy trade-off in code completion systems by shifting expensive retrieval operations from inference time to repository indexing time. The method combines speculative context construction with predictive code generation to enhance LLM performance on real-world code completion tasks while eliminating inference-time latency overhead.

Key Points

Indexing-Time Context Construction: SpecAgent performs expensive repository exploration and context retrieval asynchronously during indexing rather than at inference time, eliminating the latency-accuracy trade-off that plagues existing retrieval-augmented approaches.
Hybrid Agent Architecture: The system combines three complementary agents—a Retriever Agent (fetching relevant code snippets), a Forecaster Agent (predicting likely future functions), and SpecAgent (synthesizing both)—to construct diverse context blocks that support code generation.
Future Context Leakage Identification: The paper identifies and formally addresses a critical evaluation flaw in existing benchmarks where target function invocations inadvertently reveal future code information, inflating reported performance metrics.
Synthetic Leakage-Free Benchmark: The authors introduce a rigorously constructed benchmark that eliminates future context leakage, enabling more realistic evaluation of code completion systems in repository settings.
Substantial Performance Gains: SpecAgent achieves 9–11% absolute improvements (48–58% relative gains) over strong baselines while significantly reducing inference latency, with no additional computational cost during code generation.

Methodology

You Don't Have a Model Problem. You Have a Data Layer Problem.

State of AI — Thu, 09 Apr 2026 17:58:44 GMT

There is a recurring pattern in AI postmortems. The team built a RAG system or an agent pipeline. It worked well in testing. In production, it started returning wrong answers. Everyone looked at the model, the retrieval logic, the chunking strategy. Nobody looked at the data itself.

The data was just old.

This is the quiet failure mode that doesn’t get talked about enough. The model is often fine. The architecture is often fine. The problem is that AI systems are being built on top of data infrastructure that was designed for a different era, and the gap between what that infrastructure provides and what these systems actually need is getting harder to ignore.

The Solution!

What Static Data Does to AI Systems

Most production AI systems still rely on some version of a static corpus. You crawl, you index, you embed. Then you answer questions against that snapshot.

This works until the world changes. Pricing shifts. A company pivots. A product gets discontinued. A story breaks. The corpus doesn’t know. The model doesn’t know. The user gets a confident answer that was accurate three months ago.

The problem is far more than just staleness. It’s the shape of what you’re indexing. A standard search API gives you web results. That’s one lens on reality. But a competitive landscape lives in Reddit threads, Twitter discussions, niche forums, and increasingly in what AI answer engines like Perplexity are surfacing to users. None of that shows up in a basic search index.

Agents have it worse. An agent that needs to browse the web mid-task is depending on scraping infrastructure to work reliably. In practice, it hits CAPTCHAs, gets blocked, returns partial results, and silently degrades. The plan looked reasonable. The execution failed at the data layer.

The Shift That’s Already Happening

The better AI products being built right now treat data access as infrastructure, not as an afterthought.

The framing is shifting from “what dataset should I train on” to “what data can my system reach at runtime.” Static corpora are giving way to live retrieval. Single-source integrations are giving way to multi-source pipelines. And the teams doing this well are not building that infrastructure themselves. They’re treating it the same way they treat compute: something to procure, not something to engineer from scratch.

This is the layer that most AI frameworks don’t address. LangChain and LlamaIndex give you the retrieval logic. Vector databases give you the storage. What they don’t give you is reliable, real-time access to the actual web, across all the sources that matter, without spending months building and maintaining the pipelines to get there.

The Layer That’s Been Missing

What these systems need is a real-time web data layer that sits underneath the AI stack. Not a scraper you maintain. Not a single API with a narrow lens. Something that handles the full surface of the public web, keeps it fresh, and exposes it through interfaces that AI systems can actually use.

This is what Bright Data is.

It’s infrastructure, not a tool. The distinction matters. A tool solves a specific task. Infrastructure changes what’s possible across your entire stack.

What Bright Data Actually Gives You

Web Discovery is Bright Data’s unified access layer for real-time public web data. It covers search engines (Google, Bing), social platforms (Reddit, Twitter/X, Instagram, TikTok), web archives going back years, and answer engines (ChatGPT, Perplexity, Gemini). One integration. All of it, live.

The value here is not just breadth. It’s that each of these sources reveals something different. Search results tell you what ranks. Reddit tells you what practitioners actually think. Answer engines tell you what AI is telling your users right now. Web archives let you track how narratives change over time. These are different signals. A system that only reads one of them is working with a partial picture.

The MCP Server is the agent-facing layer. MCP (Model Context Protocol) is the protocol that lets AI models call out to external tools during a task. Bright Data’s MCP Server gives any agent the ability to search, navigate, and extract real-time web data in a single tool call, with CAPTCHA handling and block bypassing built in.

The practical consequence is that your agent can actually do what it plans to do. The failure mode of “agent makes a reasonable plan, data retrieval silently fails, outputs degrade” is the thing this removes. Details and setup are at brightdata.com/ai/mcp-server.

What This Looks Like in Practice

Competitive intelligence agent. An agent tasked with tracking a competitor monitors their site for pricing changes, pulls recent Reddit threads where users discuss the product, checks what Google is surfacing for relevant queries, and reads what Perplexity’s answer engine is telling people who ask about alternatives. This used to require four separate integrations, each brittle in its own way. With Web Discovery, it’s one.

RAG with live context. Instead of answering against a corpus indexed weeks ago, a retrieval pipeline pulls current search results and recent forum discussions at query time. The answers reflect the state of the world today. For anything in a fast-moving domain, this matters more than embedding quality.

Building without building infra. A founder building an AI product that depends on web data doesn’t spend the first three months engineering scrapers. They connect to Bright Data, get reliable access to the sources they need, and build the actual product. Bright Data’s blog has a practical walkthrough of this pattern for fine-tuning Llama 4 on web data, and integration examples for LlamaIndex and CrewAI.

The Leverage Argument

The reason this is worth paying attention to is not that Bright Data is uniquely clever. It’s that the data layer is where a lot of AI products are quietly losing time and quality, and most teams don’t realize it until they’re already deep in maintenance work they didn’t plan for.

Getting the data layer right doesn’t make your model smarter. It makes everything downstream of the data layer actually work as designed. That’s a different kind of leverage, less visible than a model improvement, but often more consequential.

If you’re building anything that depends on knowing what’s happening in the world right now, it’s worth looking at what Bright Data provides before you build the infrastructure yourself.

Explore the platform at brightdata.com.

This edition is sponsored by Bright Data. State of AI partners with products we consider directly relevant to our readers’ work.

Vectorizing Figures, Optimizing Workflows, and Enhancing Multilingual Watermarking in AI

State of AI — Sat, 28 Mar 2026 20:25:21 GMT

Welcome to today’s edition of State of AI 🚀

👋 And a warm welcome to our new subscribers since last edition!

This edition features a diverse range of AI research topics, from leveraging vision-language models to vectorize complex figures, to using large language models to optimize software development workflows, and enhancing the robustness of multilingual watermarking techniques. We’ll also explore breakthroughs in diffusion model scaling, surgical procedure understanding, and more.

Here’s what caught our attention:

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models - A novel approach to converting raster figures into high-fidelity vector graphics using a coarse-to-fine vision-language model training strategy.
LLM-Powered Workflow Optimization for Multidisciplinary Software Development - A practical case study demonstrating how large language models can automate translation and coordination tasks to significantly accelerate software development workflows.
Is Multilingual LLM Watermarking Truly Multilingual? - An in-depth examination of the limitations of current multilingual watermarking techniques and the introduction of a robust, scalable defense called STEAM.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Relationship-Aware Safety Unlearning for Multimodal LLMs

Authors: Vishnu Narayanan Anilkumar, Abhijith Sreesylesh Babu, Trieu Hai Vo, Mohankrishna Kolla, Alexander Cuneo

Source and references: https://arxiv.org/abs/2603.14185v3

Introduction

This research paper proposes a framework for “relationship-aware safety unlearning” to mitigate unsafe relationships in multimodal large language models (LLMs) while preserving the model’s utility.

Key Points

Formalize a schema for unsafe relational tuples with context (actors, actions, objects, attributes, spatial/temporal cues).
Develop fine-tuning or parameter-editing procedures (e.g., contrastive unlearning, LoRA masking, causal trace edits) targeted at object-relation-object (O-R-O) tuples.
Introduce counterfactual preservation losses and safe exemplars to retain utility on allowed and safe context examples.
Evaluate resistance to prompt fuzzing, synonym swaps, and compositional adversaries.
Provide testable acceptance criteria and unit tests for safety regression.

Methodology

The method introduces a systematic framework with two components: 1) construction of a relational graph that explicitly represents the unsafe object-relation structures, and 2) a targeted parameter-editing procedure using low-rank adapters (LoRA) to selectively weaken the model’s representations of the unsafe relations while preserving the model’s performance on other concepts.

Results and Findings

Experiments on the CLIP model showed that the proposed “FULL OPTIMAL” method significantly reduced cosine similarity for unsafe relationships across paraphrase, contextual, and out-of-distribution image attacks, with ∆cos values of 0.6878, 0.4881, and 0.7012 respectively. Simultaneously, the absolute drift in cosine similarity for safe and neutral knowledge preservation cases remained minimal, ranging from 0.0115 to 0.0608. Ablation studies confirmed the importance of the balanced multi-objective loss function in achieving both accurate forgetting and essential knowledge retention.

Implications and Conclusions

The research establishes a novel framework for relation-aware unlearning in multimodal LLMs, opening up several future directions, such as extending the unlearning mechanism, improving robustness and adversarial mitigation, scaling to larger generative models, and formalizing auditable safety evaluation criteria.

LLM-Powered Workflow Optimization for Multidisciplinary Software Development: An Automotive Industry Case Study

Transformer Architectures, Discrete Diffusion, and Materials Discovery

State of AI — Sat, 21 Mar 2026 04:35:02 GMT

Something keeps nagging at us this week.

We’re drowning in demos. Every day brings another benchmark broken, another capability unlocked, another “this changes everything” tweet. It’s easy to mistake motion for direction.

But buried under the noise, a quieter story is unfolding, researchers are going back to first principles. Not asking “how do we make models bigger?” but “are we even building these things right?”

That tension runs through everything in this edition. Video diffusion models that are fast not because of more compute, but because someone finally looked at where the attention is actually going. Image generators that fix their own mistakes without needing external feedback. A materials discovery framework that combines LLM intuition with chemistry that actually works in a lab.

And then there’s the architectures piece, which we’d encourage you not to skip. Turns out some of the strangest behaviors in transformers aren’t bugs, or emergent intelligence. They’re just artifacts of how we built the thing.

Here’s what we’ve got:

Accelerating Text-to-Video Generation with Calibrated Sparse Attention — a training-free method that maps the sparsity and repetition patterns inside video diffusion attention, then exploits them. Significant speedups, no retraining required.
OSPO: Object-Centric Self-Improving Preference Optimization — a self-improving loop that sharpens object-level alignment in text-to-image generation, no external models or labeled data needed.
LLEMA: Evolutionary Search with LLMs for Multi-Objective Materials Discovery — LLM scientific knowledge meets chemistry-aware evolutionary search. The goal: materials that work and can actually be synthesized.
The Spike, the Sparse and the Sink — a close anatomical look at massive activations and attention sinks in transformers. The finding is humbling: these aren’t deep properties of intelligence. They’re architectural side effects.
Cubic Discrete Diffusion — discrete visual generation on high-dimensional representation tokens, hitting state-of-the-art on ImageNet. Worth understanding before the next wave of image model papers land.

Let’s get into it 👇

Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

HydroGEM: A Self Supervised Zero Shot Hybrid TCN Transformer Foundation Model for Continental Scale Streamflow Quality Control

Authors: Ijaz Ul Haq, Byung Suk Lee, Julia N. Perdrial, David Baude

Source and references: https://arxiv.org/abs/2512.14106v3

Introduction

This paper introduces HydroGEM, a self-supervised zero-shot hybrid TCN-Transformer foundation model for continental-scale streamflow quality control.

Key Points

HydroGEM is trained on 3,724 USGS sites with 6.03 million sequences, an order of magnitude larger than prior multi-site hydrological studies.
HydroGEM uses a two-stage training approach combining self-supervised pretraining on clean data with synthetic anomaly injection for detection and reconstruction.
HydroGEM employs hierarchical normalization to enable learning across six orders of magnitude while preserving scale-dependent physical behavior.
HydroGEM achieves F1 = 0.792 for detection and 68.7% reconstruction error reduction on a held-out synthetic test set.
HydroGEM demonstrates cross-national transfer from USGS (USA) to ECCC (Canada) data with Tolerant F1 = 0.70.

Methodology

HydroGEM uses a two-stage training approach. Stage 1 pretrains a hybrid TCN-Transformer backbone on clean USGS data using masked reconstruction. Stage 2 fine-tunes the backbone with a lightweight detection head using synthetically generated anomalies.

Results and Findings

On the synthetic test set, HydroGEM achieves F1 = 0.792 for anomaly detection and reduces reconstruction error by 68.7% relative to injected corruptions, outperforming the strongest baseline by 36.3%. For cross-national validation on 100 ECCC stations, HydroGEM achieves Tolerant F1 = 0.70, with stable precision across all buffer sizes and 90.1% segment-level detection of anomaly events.

Implications and Conclusions

3 Out of 4 AI Coding Agents Will Break Your Code

State of AI — Mon, 16 Mar 2026 06:36:50 GMT

There is a single question that has quietly organized almost all AI coding research for the past several years: can a model fix this bug?

Fix the issue. Pass the tests. Ship the patch. The leaderboards fill up. The papers get written. The progress looks real.

It is the wrong question.

A paper out of Sun Yat-sen University and Alibaba Group just made that argument with unusual clarity. SWE-CI introduces a benchmark built not around a single snapshot of code, but around the full sweep of time. Repositories that evolved over 233 days and 71 commits on average. The agents do not just fix a bug. They maintain a living codebase, round after round, as new requirements keep arriving.

What the researchers found should make anyone building AI coding tools uncomfortable.

The Flaw in Every Benchmark You Trust

A Researcher Just Killed the Training Paradigm

State of AI — Tue, 24 Feb 2026 18:00:41 GMT

Every model you’ve ever used was built in a very similar way.

Feed it data. Measure how wrong it is. Nudge it slightly in the right direction. Repeat, hundreds of thousands of times, until the loss stops falling. This process, gradient descent, backpropagation, iterative optimization, is so universal we’ve stopped thinking of it as a choice. It’s just how machine learning works.

This paper argues that it doesn’t have to be this way.

This framework, “Learning Without Training,” proposes skipping optimization entirely. Don’t train. Don’t iterate. Construct the model directly from data using mathematical theory, derive explicit error bounds, and stop. It sounds like it shouldn’t produce useful results. It does.

The training paradigm persisted for so long partly because nobody stopped to ask whether it was the only option. Web data for agents has the exact same problem and Nimble looks to solve it!

Nimble: Live Web Data Your AI Agents Can Actually Use

You’ve built the agent. The logic is clean, the prompts are solid. Then it confidently tells you an iPad costs $329, a price from six weeks ago.

Instead of going through this Nimble turns the live web into structured data tables that AI agents can actually use!

Every approach to web data for agents has a huge flaw. Browser agents are slow, token-hungry, and fall apart trying to parse HTML at scale. AI search tools (Tavily, Exa, and the rest) return text summaries of the web, not data. Legacy scrapers are rigid, require specialized setup, and need constant maintenance. Nothing hurts more than a 2am data pipeline debug session.

Nimble engineered the infrastructure layer that none of these are: a multi-agent system that turns the live web into a structured dataset. Instead of querying a stale index, it sends a fleet of headless browsers to the actual page right now, with a proprietary proxy layer handling rate limiting, JavaScript rendering, and sites that actively block standard crawlers. Data age: 0–5 seconds.

The output isn’t a wall of markdown. It’s a structured table: schema plus rows, consistent across sources, that your agent can query, compare, and act on directly. Ask it to search 1,800 local stores, compare 8,100 products, and return the lowest price. It will !!!

Here is what that unlocks in practice:

“Which stores within 25 miles have a PS5 in stock right now — with price, pickup time, and address from each retailer’s site?” Live inventory, not last night’s cache.
“Every 2BR rental in San Francisco posted in the last 48 hours — price, sqft, fees, listing URL — across all major platforms.” Time-filtered, consistent schema across sources.
“All new RFPs this week for ‘data platform’ and ‘AI analytics’ — issuer, deadline, requirements, budget, submission link.” Across government procurement portals that block standard crawlers.

Install the Nimble skills directly in Claude or Cursor and your coding agents get live web access without building the infrastructure. Free trial, no credit card required.

Try It Now!

What Universal Approximation Theorems Don’t Tell You

The theoretical foundation of deep learning rests on a beautiful result: neural networks can approximate any continuous function to arbitrary precision. Given enough parameters, the right model exists.

What the theorem doesn’t tell you is how to find it.

That’s gradient descent’s job. And gradient descent is genuinely good at that job, good enough that the entire industry runs on it. But it carries costs that are easy to overlook when the benchmarks keep improving. It gets stuck in local minima. It’s sensitive to initialization. It requires careful tuning of learning rates and schedules. And perhaps most importantly, it produces models whose behavior is almost impossible to analyze from first principles. We know they work. We struggle to explain precisely why, or to predict when they’ll fail.

Universal approximation guarantees existence. It doesn’t guarantee that your optimization procedure will find what exists.

This approach bypasses the search entirely. If you can construct the approximator directly from data using mathematical theory, you don’t need to wander a loss surface hoping to land somewhere useful.

Direct Construction

The classical pipeline looks like this:

Data → Loss function → Optimization → Model

Learning Without Training shortens it:

Data → Mathematical construction → Model

The key machinery is localized kernels drawn from approximation theory and harmonic analysis. These are functions that concentrate their influence near specific points in the input space, with precise decay properties that can be analyzed theoretically. Rather than learning how to represent your data through gradient updates, you construct a representation whose approximation error can be bounded in advance.

The result is a model you didn’t train so much as derive.

The dissertation applies this idea to three distinct problems, each chosen to stress-test a different aspect of the framework.

Manifold Learning Without the Manifold

When data lies on a low-dimensional manifold embedded in high-dimensional space — images of a rotating object, sensor readings with redundant channels, gene expression data constrained by biological pathways — the standard approach is a two-step process: learn the manifold geometry, then approximate the target function on it. Two steps means two sources of error compounding each other.

The approach collapses this into one. Project the manifold data onto a hypersphere, apply localized kernels designed for spherical geometry, and produce an approximation of the form:

The error bounds on this construction scale with the intrinsic dimension qq q of the manifold, not the ambient dimension of the space it’s embedded in. If your images live in a million-dimensional pixel space but lie on a three-dimensional manifold of rotations, the error behaves like a three-dimensional problem. The extra dimensions don’t hurt you.

This holds with high probability, as a theorem, derived in advance and not observed empirically across a set of test cases.

A Map for Transfer Learning

Transfer learning is one of the most practically useful ideas in machine learning and one of the least theoretically understood. Fine-tuning works. The explanation for when and why it works, and when it will silently fail, remains mostly empirical.

This framework introduces data spaces, abstract mathematical objects encoding the geometry of a domain, and connects source and target domains through a “joint data space” that quantifies how information flows between them. The transfer operator lifts functions from one domain to the other using a joint localized kernel:

The particularly useful result is for local transfer learning, where you only know the source function on a subset of the source domain. The framework tells you which corresponding subset of the target domain can be reliably predicted, and what smoothness properties will be preserved. You get a map of where your knowledge transfers — not a guess, but a derivation from the joint structure of the two domains.

Classification as Signal Separation

The third contribution reframes classification from the ground up.

Standard classification treats the problem as function approximation: learn a function mapping inputs to labels. The paper argues this is the wrong frame. Classification is signal separation, your data is a mixture drawn from multiple source distributions, one per class, and the task is to unmix them.

This reframing isn’t just aesthetic. It changes what tools you reach for. Instead of learning a decision boundary, you estimate the support of each class distribution using a positive localized kernel:

The squared kernel ensures positivity, which prevents cancellations that would corrupt the support estimate. Once you’ve identified a cluster as belonging to one class distribution, the mathematical structure tells you that nearby unlabeled points belong to the same cluster — you don’t need to label each one individually.

The resulting algorithm, MASC (Multiscale Active Super-resolution Classification), uses this structure to achieve competitive accuracy on hyperspectral imaging datasets while requiring fewer labeled examples than existing active learning methods. Not because of a clever heuristic. Because the mathematics says the labels should propagate that way.

What a Guarantee Actually Buys You

The manifold approximation error bound looks like this:

The error is controlled by three quantities: the smoothness of the target function (γ), the noise level in the labels (‖z‖), and a localization parameter (n) you choose. Given any target accuracy, you can compute how much data you need. Given any noise level, you can compute how the error degrades. These calculations happen before you collect a single data point.

Trained models don’t offer this. Benchmark performance tells you how a model behaves on a specific distribution of test cases. It doesn’t tell you how performance changes under distribution shift, or what error to expect on inputs that look slightly different from the training set. The gap between empirical performance and theoretical understanding is, for most deployed models, essentially total.

For applications where that gap matters — medical imaging, structural monitoring, anything with real failure costs — theoretical guarantees aren’t a nice-to-have. They’re the point.

Where It Doesn’t Apply

None of this is a case against gradient descent, and it is not framed it as one.

The framework requires mathematical structure in the problem. Manifold assumptions, smoothness conditions, geometric relationships between source and target domains, these need to hold, at least approximately, for the constructions to work. Data that genuinely lives in high-dimensional space without manifold structure doesn’t benefit from the intrinsic dimension argument. The method exploits structure, so it needs structure to exploit.

The dissertation addresses supervised learning and classification. Language modeling, image generation, and reinforcement learning are different problems with different geometry, and the kernel-based constructions don’t translate directly. A large language model isn’t going to be derived from harmonic analysis anytime soon.

Computational scale is also an open question. Gradient descent parallelizes naturally on GPU clusters. Kernel methods have known scaling challenges. The dissertation demonstrates competitive or superior performance at research scale; whether that holds at the scale of modern foundation models hasn’t been tested.

Why Now

The scaling era has produced a curious situation: systems that clearly work, built on processes we can’t fully analyze. The response has been to keep scaling, more data, more parameters, more compute, and let empirical performance carry the argument.

That works until it doesn’t. The cases where we most want guarantees are exactly the cases where “it worked on the benchmark” is least satisfying as an answer.

This framework is a concrete demonstration that for a real class of machine learning problems, you can have the guarantee. You can know, analytically, what your model will do and why. The construction replaces the search, and the theory does the work that experimentation usually has to.

Whether this paradigm expands, into more domains, larger scales, more complex architectures, is genuinely unknown. The dissertation mentions feature extraction analysis in deep networks and operator approximation as natural extensions. There’s real research to be done.

But as a proof of concept that optimization isn’t the only path from data to model, it’s convincing. And in a field that has spent fifteen years treating gradient descent as the only game in town, that’s worth sitting with.