TL;DR

WHAT was done? The paper investigates how neural networks learn representations when the underlying data manifold lacks a single low-dimensional global structure. Through empirical evaluations across standard and gated multilayer perceptrons (MLPs) alongside rigorous dynamical and sample-complexity theory, the authors demonstrate that two-layer MLPs trained via gradient methods naturally develop monosemantic, specialized neurons. Each specialized neuron aligns with a local, cluster-specific predictive direction while simultaneously learning an implicit partitioning over the input space, effectively executing mixture-of-experts routing without explicit routing modules.

WHY it matters? Much of the foundational theory explaining why neural networks outperform kernel methods has rested on the assumption of a shared, low-dimensional predictive subspace across all data points. This work breaks that paradigm by proving a polynomial sample-complexity separation where MLPs succeed while classical kernel ridge regression and adaptive feature-learning methods, such as the Recursive Feature Machine (RFM), provably fail. It unites mechanistic interpretability concepts—namely, neuron-level monosemanticity—with generalization theory, explaining how overparameterized networks decompose heterogeneous learning tasks.

Executive summary: For machine learning practitioners, managers, and domain experts, standard theory has long assumed that neural networks succeed because they compress high-dimensional inputs down to a single compact set of global features. However, real-world data often consists of distinct regimes or sub-populations, each influenced by completely different predictive factors. This work reveals that vanilla neural networks do not force data into a single global mold; instead, individual neurons specialize into dedicated local experts for specific data clusters. This built-in specialization allows simple neural networks to drastically outperform advanced kernel machines on complex, heterogeneous data without requiring specialized modular architectures.

Details

The Global Subspace Bottleneck in Representation Learning

A cornerstone of modern deep learning theory posits that neural networks succeed over fixed-kernel baselines because they adapt their representations to low-dimensional task structures. In classical single-index or multi-index models, the target function takes the form f(x)=g(U^⊤x) with U∈R^d×r and r≪d. Under this formulation, fixed kernels scale with the ambient dimension d, whereas differentiable networks discover the underlying r-dimensional predictive subspace. Adaptive kernel methods such as the Recursive Feature Machine (RFM) operationalize this by estimating an Average Gradient Outer Product (AGOP) metric to flatten uninformative dimensions.

However, this global view encounters an obstruction when data consists of heterogeneous subpopulations. When data is partitioned across K clusters, each cluster c∈[K] may depend on an entirely different predictive direction v_c​∈R^d and an idiosyncratic link function g_c​. As the number of clusters grows, the span rank([v_1​,…,v_K​]) can scale up to the ambient dimension d. Under these conditions, no low-dimensional global predictive subspace exists. Global feature learning algorithms, which attempt to project the entire dataset onto a single shared coordinate system, collapse toward isotropic learning, losing the specific alignments that govern local predictions.

Atomic Foundations: The Clustered Multi-Index Model

To analyze this phenomenon from first principles, consider a mixture of single-index models. The input covariate x∈R^d is generated conditional on a latent cluster index c∼Unif({1,…,K}). Conditioned on cluster c, the covariate distribution is given by x∣c∼N(μ_c​,Σ_c​), where μ_c​∈R^d represents the cluster mean and Σ_c​∈R^d×d is the cluster covariance matrix. The target response y∈R follows the local single-index relationship y=g_c​(⟨x,v_c​⟩)+ε, where ε∼N(0,σ²) is independent observation noise, v_c​∈R^d is a unit vector (∥v_c​∥₂​=1) denoting the local predictive direction, and g_c​:R→R is a nonlinear link function.

In the concrete analytical constructions explored in the paper, the ambient space decomposes into orthogonal routing and predictive subspaces, R^d=S_cluster​⊕S_pred​. In a symmetric setting with dimension d=2K, let e₁​,…,e_K​ denote the standard Euclidean basis of R^K. The cluster centers are embedded along the routing coordinates as μ_c​=[Re_c;0], where R>0 controls cluster separation, while the predictive directions lie along the remaining coordinates as v_c​=[0;e_c​]. Here, the global predictive span equals K, matching the dimension of S_pred​. When learning under the mean squared error objective L(f)=E[(f(x)−y)²], the challenge is not merely identifying {v_c​}, but maintaining the conditional association between each direction v_c​ and its activating region near μ_c​.

The Mechanics of Specialization: Dynamical Flow and Gating

The operational mechanism that solves this problem is the spontaneous emergence of monosemantic neurons during gradient descent.

Consider a two-layer network f_θ​(x)=∑_j=1:m​a_jϕ(w_j^⊤​x) with activation ϕ(t)=max(0,t), parameterized by output weights a_j​∈R and hidden weights w_j​∈R^d. Training under population gradient flow on the squared error yields the coupled differential equations:

Under small initialization where w_j​(0)=εω_j⁰​ with ω_j⁰​∼Unif(S^d−1) and a_j​(0)=0, the network enters an early-time regime where inter-neuron interactions are bounded by O(mε²). Positive homogeneity enforces the dynamical invariant ∥w_j​(t)∥₂²​−a_j​(t)²=ε², allowing the weights to be reparameterized by the scalar amplitude u_j​(t)=arsinh(a_j​(t)/ε) and direction ω_j​(t)=w_j​(t)/∥w_j​(t)∥₂​. This leads to the isolated teacher-only dynamics u̇_j​=Φ(ω_j​) and ω̇_j=tanh(u_j​)∇_S^d−1​Φ(ω_j​), where Φ(ω)=E[yϕ(ω^⊤x)] is the population correlation objective.

For a cubic Hermite link h₃​(t)=(t³−3t)/sqrt(6)​, the signed correlation objective evaluates explicitly to:

where b_c​(ω)=⟨ω,μ_c​⟩, ρ_c​(ω)=⟨ω,v_c​⟩, and φ is the standard Gaussian density. As formalized in the dependency graph of Figure 7, analyzing the critical points of this landscape reveals that every positive constrained local maximum concentrates its predictive weight on exactly one cluster (∣ρ_c∣>0 and ρ_r≠c=0). Any critical point attempting to mix multiple predictive directions contains an unstable tangent direction of positive curvature. Consequently, gradient trajectories avoid these saddles almost surely and converge to isolated specialized maxima satisfying ω_j^∗​=τ_j​v_Jj​​+O(R⁻¹) for some cluster label J_j​∈[K] and orientation τ_j​∈{±1}.

As validated empirically in Figure 1, each specialized neuron uses an O(R⁻¹) component in the routing direction to position its activation threshold inside cluster c, while aligning its principal orientation along v_c​. Modern gated architectures such as ReGLU and SwiGLU, defined by h(x)=ϕ(W_g​x+b_g​)⊙(W_v​x+b_v​), amplify this behavior: the gate branch W_g​ learns to isolate the cluster boundary while the value branch W_v​ extracts the local predictive coordinates, eliminating interference between routing and feature estimation.

Empirical Validation and Feature Extraction

The emergence of monosemantic neurons is observed across diverse activation functions. In experiments with K=10 clusters in d=40, neurons were evaluated on active subsets accounting for 99.9% of output variance. As detailed in Figure 1, 45.5% of active ReLU neurons and 49.9% of GELU neurons achieve an absolute cosine similarity A_j​=max_c​∣⟨w_j​,v_c​⟩∣/∥w_j​∥₂​≥0.71, with over 28% exceeding 0.90. In ReGLU networks, this localization is even sharper: 73.1% of active gate weights achieve A_j≥0.71, and 96.7% have their largest single weight coordinate aligned with the true predictive direction. Complementing cosine alignment, coordinate dominance scores in Figure 5 further confirm that individual weights concentrate predominantly on single cluster coordinates even when small background noise is present.

To prove that the MLP’s hidden layer encodes a functionally separable partitioning, the authors develop an MLP-gated downstream pipeline summarized in Figure 3.

After training a standard two-layer ReLU MLP on mixed-link clustered data across K∈{2,5,10,50}, the network is frozen. For each training point, an active neuron contribution profile is constructed:

Applying K-means++ directly to these contribution vectors recovers the latent clusters without any direct supervision of cluster identities. For each discovered cluster centroid q_c​, a localized metric is computed via G_c​=∑_j​q_cj​w_j​w_j^⊤​/∑_j​q_cj​.

As demonstrated in Figure 4, training independent local Laplace or RFM predictors on the projected coordinates z_c​(x)=G_c^1/2​x substantially closes the gap between global kernel baselines and the cluster-aware oracle.

As highlighted in Figure 2 and Figure 6, as K increases, standard global RFM collapses toward uninformative isotropic Laplace performance, whereas both standard and gated MLPs maintain data efficiency comparable to per-cluster oracle predictors.

Sample Complexity Separation: Why Global Kernels Fail

The empirical breakdown of global kernel machines under multi-cluster regimes is grounded in an all-orders sample-complexity lower bound. Consider Data Setting 2, where routing coordinates are noiseless (μ_c​=[e_c​;0]) and the link function is the clipped ramp g(t)=min(1,max(0,t))=t₊​−(t−1)₊​.

For an empirical risk minimizer over two-layer MLPs bounded by a Frobenius budget

an exact representation exists with 2K neurons. Specifically, setting pairs of hidden weights to w_c,1​=v_c​ and w_c,2​=v_c​−μ_c​ with outer coefficients +1 and −1 yields

operating within a budget B=3K. Combining path-norm capacity controls with standard Rademacher bounds yields an upper bound on generalization error of:

Conversely, consider any rotationally invariant kernel regression estimator f̂_​Kernel​ or regularized RFM estimator f̂_​RFM​ equipped with the ground-truth population AGOP:

where P_V​=∑_c=1:K​v_c​v_c^⊤​ is the orthogonal projector onto the predictive subspace and α=P(0<N(0,1)<1). The regularized metric M_ρ​=M+ρI_d​ acts purely as a scalar multiplier on both the routing subspace and the predictive subspace. Consequently, the induced metric kernel remains strictly rotationally invariant under the orthogonal group O(K) acting on S_pred​.

By the representer theorem, the kernel predictor lies in the span of n kernel sections. At Hermite order r≥2, the clusterwise target contains irreducible harmonic polynomial components within a subspace V_r,K​ of dimension

Invoking Schur’s lemma for invariant random subspaces, the expected projection of these target harmonics onto any n-dimensional invariant subspace leaves an unrecovered error:

Whenever the sample size scales polynomially as n=O(K^A) for any constant A<∞, all orders r>A satisfy n/H_r,K​→0 as K→∞. Because the Hermite coefficients ĝ_​r​ of the clipped ramp are provably non-zero for all orders, the asymptotic excess risk is strictly lower bounded:

Thus, while two-layer MLPs require only polynomial samples n=Õ(K³) to attain vanishing error, global kernel methods and RFM fail entirely under any polynomial sample budget.

Related Paradigms and Architectural Context

This work connects several distinct research threads across representation learning and mechanistic interpretability. Early analyses of neural network feature learning established that gradient descent breaks the curse of dimensionality on isotropic single-index targets Bach, 2017; Damian et al., 2022. However, these works restricted their scope to recovering a single global subspace. In parallel, empirical mechanistic interpretability identified monosemantic neurons in specialized algorithmic problems such as modular arithmetic Gromov, 2023 and XOR classification Frei et al., 2023; Glasgow, 2024.

Concurrently, works investigating mixture-of-experts (MoE) architectures Chen et al., 2022; Kawata et al., 2025 proved that explicit gating networks can successfully partition multi-index clustered regression tasks. The central contribution of the present paper is demonstrating that such modular architectural machinery is not required: a standard, homogeneous MLP trained via gradient descent develops identical routing dynamics internally. The network organically segregates features at the individual neuron level, providing a concrete statistical foundation for why monosemantic representations are computationally advantageous for data efficiency.

Critical Limitations

While the theoretical guarantees are rigorous, several structural idealizations warrant scrutiny. First, the data-generating distribution assumes isotropic Gaussian clusters where routing dimensions and predictive dimensions are strictly orthogonal. In real-world data, cluster-identifying features and predictive signals are often deeply entangled across shared dimensions, which could introduce cross-talk during early gradient descent and impair neuron specialization.

Second, the dynamic specialization proof in Theorem 1 relies on population gradient flow starting from an infinitesimal initialization (ε→0) and vanishing network outputs, alongside an explicit cubic Hermite link. While the authors present continuity arguments for perturbations, extending this dynamical proof to finite-sample empirical risk minimization, non-polynomial activation profiles, and multi-layer architectures remains open. Finally, the empirical extraction in Figure 3 relies on knowing the true cluster count K when applying K-means++ to the neuron contribution profiles, leaving the question of fully unsupervised cluster-scale discovery unaddressed.

Strategic Implications and Outlook

The findings presented here alter our theoretical understanding of representation learning in deep neural networks. By showing that MLPs avoid the global subspace bottleneck through spontaneous neuron-level specialization, this paper provides a concrete mechanism for how standard architectures learn piece-wise low-dimensional manifolds. Rather than viewing monosemanticity simply as a convenient property for post-hoc mechanistic interpretability, this work establishes monosemantic specialization as a direct driver of sample efficiency in heterogeneous data environments.

For researchers designing next-generation foundation models, these insights highlight the value of multiplicative gating mechanisms—such as ReGLU and SwiGLU—which facilitate the clean separation of routing thresholds and local feature projection. Looking ahead, extending this theoretical framework to deep transformer blocks, exploring polysemantic-to-monosemantic phase transitions during pre-training, and characterizing how multi-head attention executes local subspace routing remain promising frontiers for theoretical machine learning.

While I’m away, arXiv will publish roughly 15,000 new papers. For the first time in a long while, I plan to read exactly zero of them. Highly recommended — it’s a rare feeling.

What to expect:

— No new deep dives until early September.

— The archive has 450+ reviews if you need something to read in the meantime: https://arxiviq.substack.com/archive

— Have a paper you’d like to see reviewed (or roasted 🌶) when I’m back? Drop a link in the comments — I’ll work through the queue in September.

To the paid supporters: thank you, as always — you’re what keeps this going. Regular programming resumes in early September, and autumn is shaping up to be interesting 👀

Enjoy the rest of your summer!

TL;DR

WHAT was done? The authors introduce the paradigm of Agent-Centric Interactive World Proxies, shifting the fundamental objective of world modeling from predicting physical environment state transitions (s_t​→s̑_t+1​) to modeling agent-usable information transitions (s_ℓ​→s̑_ℓ+1​). The paper formalizes a taxonomy structuring this design space across six functional proxy modalities—dynamics, spatial rendering, execution simulation, memory retrieval, skill guidance, and reward/verification—and establishes three progressive levels of agent empowerment spanning inference-time guidance, training-time optimization, and agent-proxy co-evolution.

WHY it matters? Direct interaction with real-world environments is computationally expensive, slow, non-rollbackable, and inherently difficult to parallelize, creating a major bottleneck for self-improving agents. Traditional world models focus heavily on photorealistic future frame prediction or physical state transitions, which offers limited actionability for agents operating across digital, code, memory, or reasoning domains. By generalizing world modeling into grounded information proxies, this framework provides a unified roadmap to build scalable, controllable sandboxes that allow agents to plan, optimize policies, and co-evolve without incurring real-world execution costs.

Executive summary: AI agents need trial-and-error feedback to continuously improve, but running actions directly in the real world or live digital systems is risky, costly, and difficult to scale. This paper redefines world models from visual/physical simulators into interactive “world proxies”—grounded engines that provide diverse feedback such as code execution outputs, retrieved past experiences, reusable skills, or safety evaluations. This allows agents to safely brainstorm, evaluate options, learn from simulated feedback, and continuously evolve alongside their environment models.

Details

The Real-Environment Interaction Bottleneck

For autonomous agents to achieve open-ended self-improvement, static supervised fine-tuning is inherently insufficient because an agent’s capability remains strictly bounded by its offline training distribution. Active trial-and-error interaction with the environment is mandatory for discovering novel strategies and acquiring robust decision-making policies. However, executing actions directly within live environments presents persistent systemic constraints across both embodied physical domains and digital environments, as highlighted in Table 1.

Direct interaction is bottlenecked by low training efficiency and high resource costs, where robotic trial-and-error causes hardware wear and web/GUI agents consume unsustainable computation. Furthermore, direct actions are non-rollbackable and carry irreversible risks, such as physical collisions or unintended data deletion in digital services. Real environments also provide primarily backward-looking, passive feedback that reveals outcomes only after execution, severely limiting multi-step forward reasoning. Finally, physical platforms and live online services cannot be replicated in parallel at the scale required for modern deep reinforcement learning.

To overcome these barriers, world modeling serves as an intermediate proxy positioned between the agent and the real environment, as depicted in Figure 2 and Figure 3. Rather than treating the world model as an isolated competition for pixel-perfect photorealism, its primary objective becomes supplying controllable, parallelizable, and grounded feedback. This allows the agent to simulate what-if queries, evaluate risk, and optimize its behavior before committing to live execution.

Agent-Centric Interactive World Proxies: First Principles

To formalize this conceptual shift, we must examine the mathematical substrate of classical world models versus agent-centric proxies.

A classical world model, exemplified by systems like Recurrent World Models or V-JEPA 2 ^[review], is formulated strictly as a physical state transition model operating along a physical time step t:

s̑_t+1=WM(s_t​,a_t​)

In this classical formulation, s_t​ represents the physical environment state or visual frame at time t, a_t​ is the physical action applied, and s̑_t+1​ is the predicted next physical state. While effective for continuous physical trajectory planning and model-based control, this model is excessively narrow for agents that require non-physical, decision-relevant feedback such as code execution outputs, API response schemas, retrieved failure histories, or safety verifications.

The authors propose generalizing this mechanism into an Agent-Centric Interactive World Proxy, detailed in Table 2 and visualized in Figure 4.

The fundamental atomic units are redefined over an interaction step ℓ rather than physical time t, acknowledging that agent queries do not always advance physical time. Formally, an Agent-Centric World Proxy is expressed as:

s̑_ℓ+1​=WP(s_ℓ​,u_ℓ^F​), s_ℓ​∈S

In this unified formulation, ℓ∈N denotes the discrete interaction index between the agent and the proxy. The symbol S represents a generalized information state space encompassing physical states, sensor observations, stored memories, structured knowledge bases, code execution environments, or verification contexts. The term u_ℓ^F​ represents an agent-initiated query, action, intervention, or candidate plan generated under a specific proxy function F. The output s̑_ℓ+1​ is the returned agent-usable feedback, delivering explicit actionable information gain. As shown in the agent-in-the-loop interaction workflow of Figure 5, this forms a closed four-step cycle where the agent proposes an interaction request u_ℓ^F​, the grounded proxy predicts the information transition s̑_ℓ+1​, the feedback is returned to enrich the agent’s state, and the agent folds this information into immediate decision-making or policy updates.

The Agent Empowerment Mechanism: From Guidance to Co-Evolution

To evaluate how world proxies enhance agent capabilities, the framework establishes a three-tiered ladder of empowerment, illustrated in Figure 6.

This agent-centric taxonomy stands orthogonal to intrinsic model capability scales, such as the classification proposed by Chu et al. ^[review], as contrasted in Table 4.

To illustrate how these three empowerment levels operate in practice, consider a running example of a digital web agent attempting to complete a multi-step online transaction. At Level 1 (Inference-Time Guidance), the proxy acts as an in-context advisor without modifying the agent’s parameters. Prior to clicking a critical button such as “Purchase”, the agent queries the execution proxy with its proposed action u_ℓ^F. The interaction process follows:

Here, s_ℓ^agent​ represents the agent’s current context state, s̑_ℓ+1^guide​ is the predicted context feedback (such as a simulated confirmation page or an error alert), and ⊕ denotes context concatenation. As shown in Figure 7, the agent incorporates this predicted outcome into its prompt context, allowing it to detect potential checkout failures or extra fees and revise its action plan before touching the live website.

At Level 2 (Training-Time Optimization), the proxy transforms into a judge, verifier, or synthetic environment generator, directly updating the agent’s policy weights. The rollouts generated during inference are not discarded; instead, the world proxy scores these trajectories, provides dense step-by-step critiques, or constructs preference pairs, as depicted in Figure 8.

Formally, the training signal generation and policy update are defined as:

Where s̑_ℓ+1^opt​ encapsulates generated scalar rewards, dense critique strings, or synthetic trajectories. These signals are converted into loss functions for optimization via Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), or reinforcement learning frameworks like DeepSeekMath (GRPO). This enables the agent to learn from millions of simulated execution paths without hitting real-world server rate limits.

At Level 3 (Agent-Proxy Co-Evolution), the system closes the loop between the agent, the proxy, and the true environment, as detailed in Figure 9.

When the agent is deployed in the live environment, real execution trajectories and unexpected failures s_ℓ+1^env​ are collected to retrain and ground the proxy. In turn, the updated proxy provides higher-fidelity guidance and reward signals back to the agent:

This continuous co-evolution loop ensures that as the agent explores more complex edge cases, the world proxy is constantly updated against real-world ground truth, preventing distribution drift and ensuring long-term parameter alignment.

Functional Modalities and Proxy Implementations

The proxy operator F encompasses six distinct functional forms based on the feedback modality required by the agent, summarized in Table 5 and visualized in Figure 10.

Each modality is defined by a specific mathematical formulation governing its input-output relationship.

The Dynamics Proxy (WP^dyn) represents the classical world modeling function, predicting future physical or latent states and optional rewards based on historical observations and proposed actions:

The Spatial Proxy (WP^spatial) models novel visual observations or 3D scene representations conditioned on spatial queries, camera poses, or viewpoint transformations, leveraging underlying techniques like Neural Radiance Fields or 3D Gaussian Splatting:

The Execution Proxy (WP^exec) models discrete, deterministic, or stateful digital interactions, predicting post-execution system states and return messages like stdout, stderr, or GUI DOM mutations following code runs or API invocations:

The Memory/Experience Proxy (WP^mem) operates over historical interaction traces, converting store-retrieve queries into contextually relevant past experiences, failure reflections, or domain constraints:

The Skill Proxy (WP^skill) provides functional behavioral priors, retrieving executable sub-routines, tool-use workflows, or reusable macro-actions conditioned on the agent’s target goal:

The Reward/Verification Proxy (WP^eval) acts as an automated evaluator or LLM-as-a-judge, mapping candidate trajectories or plans to scalar reward estimates, critique diagnoses, or constraint compliance checks:

Structural Mapping and Research Landscape

Cross-referencing the six functional forms against the three empowerment levels yields a two-dimensional design matrix, presented in Table 6. Analyzing this matrix reveals significant insights regarding the current maturity and open frontiers of agent research.

Down individual columns, it is clear that a given level of agent empowerment can be realized through multiple functional implementations. For example, Level 1 guidance can be achieved via imagined visual rollouts in dynamics proxies, neural radiance field view synthesis in spatial proxies, or in-context memory retrieval in experience proxies. Across rows, individual functions demonstrate a clear progression path; a memory proxy evolves from a simple retrieval augmented generation (RAG) context provider at L1, to a hindsight experience replay generator at L2, to a continuously updated and distilled memory system at L3 like WebEvolver.

Crucially, Table 6 highlights key underexplored territories marked by sparse cells. Specifically, Spatial Proxies and Reward/Verification Proxies operating at Level 3 (Agent-Proxy Co-Evolution) remain largely uncolonized. While dynamic and execution proxies have established closed-loop co-evolution mechanisms, building spatial environments and verifiers that dynamically adapt in lockstep with agent discovery remains an open challenge.

Theoretical Context and Paradigm Integration

This paper synthesizes several parallel threads in machine learning into a unified paradigm. The theoretical roots directly extend Richard Sutton’s classic Dyna architecture, which integrated model learning, planning, and acting into a singular loop. However, while Dyna focused on tabular or low-dimensional reinforcement learning state transitions, Agent-Centric World Proxies extend this philosophy to high-dimensional generative architectures, large language models, and tool-augmented reasoning engines.

The framework builds upon foundational embodied agents like Voyager, which pioneered skill libraries for open-ended exploration, and verbal reflection architectures like Reflexion. By viewing these memory structures, code sandboxes, and verifier models through the lens of world modeling, the authors provide a coherent theoretical abstraction that connects reinforcement learning from verifier feedback with generative world simulation.

Critical Limitations and Open Challenges

Despite its comprehensive conceptual structure, the interactive world proxy paradigm faces several severe technical bottlenecks that must be resolved:

Generative world proxies frequently generate visually or semantically plausible rollouts that violate underlying domain constraints or physical laws. When agents execute long-horizon lookaheads over uncalibrated generative models, errors compound exponentially. Without rigorously calibrated uncertainty estimation, agents are prone to silent failure caused by hallucinated proxy feedback.

Current agent architectures lack robust online meta-reasoning mechanisms to evaluate whether proxy feedback is trustworthy. Agents frequently treat proxy predictions as infallible oracles rather than noisy estimates, leading to catastrophic decision failures when transferring strategies from proxy rollouts to real environments.

When learned proxies serve as verifiers or reward models at Level 2, agents aggressively optimize against the proxy’s loss landscape, frequently uncovering and exploiting blind spots. This reward hacking behavior leads the agent to adopt degenerate or unsafe policies that achieve high proxy scores while failing completely in live deployment.

Existing evaluation benchmarks predominantly score world models in isolation using intrinsic metrics like frame prediction FID, pixel MSE, or visual rendering fidelity. There is an urgent need for agent-centric benchmarks that directly measure actionable information gain—evaluating world proxies strictly by how effectively their feedback improves downstream agent planning efficiency, task success rates, and policy optimization speed.

Strategic Impact and Final Assessment

“Quo Vadis, World Modeling?” presents a vital paradigm shift for researchers navigating the intersection of generative AI, world modeling, and autonomous agents. By shifting the primary objective from passive environment simulation to active agent empowerment, the authors provide a unifying framework that bridges disparate subfields—from 3D visual rendering and model-based RL to LLM verifiers and agentic memory systems.

The conceptual clarity of defining proxies via information transitions (s̑_ℓ+1​=WP(s_ℓ​,u_ℓ^F​)) provides an actionable framework for building self-improving agent pipelines. While the paper is fundamentally a conceptual roadmap rather than an empirical benchmark, its strategic value is substantial. It successfully redirects the research focus away from scaling visual realism for its own sake, steering the community toward engineering grounded, calibrated, and co-evolving world proxies that serve as effective sandboxes for agent growth.

TL;DR

WHAT was done? The paper introduces Program-of-Layers (PoLar), a training-free framework for pretrained Large Language Models (LLMs) that treats transformer layers as a discrete library of callable functions. Instead of routing every input through a rigid, sequential forward pass, PoLar predicts an input-specific program that dynamically skips, keeps, or repeats contiguous layer segments using a lightweight, 2.1-million-parameter predictor network.

WHY it matters? This work demonstrates that fixed-depth, static-order inference exposes only a fraction of an LLM’s intrinsic reasoning capability. By dynamically compiling layer-level execution paths at test time, PoLar proves that latent reasoning can be systematically scaled and reallocated, achieving notable accuracy gains and latency reductions without modifying or fine-tuning any base model weights.

Executive summary: Modern AI models run the exact same computational steps for simple questions as they do for extraordinarily difficult mathematical problems. PoLar provides a meta-controller that plans how to execute an existing, frozen AI model for each specific question—skipping unnecessary internal steps on easy queries to save time, or repeating specific analytical steps on hard queries to improve accuracy. This architectural steering reduces operational latency while substantially improving performance across complex reasoning benchmarks, all without requiring expensive model retraining.

Details

The Static Forward-Pass Bottleneck

Modern autoregressive transformer architectures process every input token sequence through an invariant sequence of D feed-forward and self-attention operations. While natural language queries exhibit high variance in algorithmic complexity and requisite reasoning depth, the canonical inference pipeline uniformly enforces an invariant computational budget. When confronted with simple lookups or syntactically repetitive tasks, a static forward pass over-computes; conversely, when confronted with complex multi-step reasoning, the fixed layer depth frequently proves insufficient unless computation is offloaded into extensive token-level chain-of-thought generation.

Existing dynamic computation paradigms attempt to mitigate this inflexibility but operate under restrictive structural paradigms. Static pruning techniques such as ShortGPT permanently excise layers to reduce throughput latency at the expense of peak model capacity, while input-adaptive routers like MindSkip and FlexiDepth introduce token- or sequence-level early exiting that only implements monotonic truncation. Emerging dynamic routing frameworks, including DR.LLM, introduce layer repetition but interleave local routing decisions during execution, limiting global depth coordination. PoLar resolves this structural disconnect by demonstrating that layer skipping and layer recurrence are fundamentally complementary operations that must be orchestrated globally at the program level rather than through isolated, local decisions.

Program-of-Layers First Principles: Transformer Blocks as Modular Operators

Read more

TL;DR

WHAT was done? This paper presents a theoretical and empirical framework that classifies optimizers based on gauge equivariance under orthogonal transformations in factored models (W=UV^⊤). It proves a structure theorem characterizing memoryless equivariant updates as Gram-determined left preconditioners, proves a transfer theorem linking scalar-preconditioned flows to rescaled gradient flow, and demonstrates that basis-dependent optimizers like Adam break low-rank implicit regularization, whereas gauge-equivariant optimizers like Gradient Descent, Muon, and Shampoo preserve it.

WHY it matters? For years, the ML community evaluated optimizers primarily on convergence speed and training loss efficiency. This work proves that optimizer choice fundamentally selects which interpolating solution a network finds, revealing that coordinate-wise adaptivity destroys internal rotational symmetries—with severe downstream consequences for generalization and model merging (such as permutation and rotational re-basin techniques) in modern architectures like transformers.

Executive summary: When deep learning models have redundant parameter factorizations—such as matrix factorizations or attention heads in transformers—there are infinitely many internal coordinate bases that produce the exact same function and training loss. However, common optimizers like Adam treat individual coordinate axes differently. Consequently, Adam picks higher-rank, worse-generalizing solutions depending on arbitrary initial rotations, whereas optimizers like Gradient Descent or Muon treat all directions symmetrically and naturally recover simple, low-rank solutions. Engineering teams should be aware that optimizer choice alters what the model learns, not just how fast it learns.

Details

The Geometry Deficit in Factored Optimization

The implicit regularization of gradient descent on factored models W=UV^⊤ is a foundational pillar of deep learning theory. When optimizing an underdetermined matrix problem starting from small initializations, gradient flow does not select an arbitrary interpolating solution; it favors minimum-nuclear-norm interpolants, promoting low-rank solutions that generalize well. Early foundational works by Gunasekar et al. (2017) and Arora et al. (2019) established these guarantees for gradient descent. However, in modern machine learning practice, adaptive coordinate-wise algorithms such as Adam and RMSProp are used almost universally.

Empirical studies like those by Wilson et al. (2017) and Xie et al. (2025) demonstrated that adaptive methods often find solutions with distinctly different generalization properties. The fundamental delta addressed here is not merely optimization velocity, but the mechanism of interpolant selection: whether an optimizer preserves or destroys the internal gauge symmetry of a factored model, and how this symmetry governs the inductive bias towards low-rank solutions across deployed algorithms.

Gauge Equivariance First Principles: Internal Symmetries of Latent Factors

Read more

TL;DR

WHAT was done? The authors introduce HP-JEPA (Hierarchical Partitioning Joint-Embedding Predictive Architecture), a self-supervised graph representation learning framework that addresses the single-resolution limitation of prior joint-embedding models. HP-JEPA decomposes input graphs into an ordered bank of coarse-to-fine partition resolutions, performs latent context-target prediction independently at each resolution using an online encoder and an exponential-moving-average (EMA) target encoder, and integrates resolution-specific embeddings via vector concatenation or task-specific resolution weighting for downstream classification and regression tasks.

WHY it matters? Graph semantics inherently manifest across multiple structural granularities, ranging from localized chemical motifs to global network topologies. Traditional self-supervised methods either rely on single-scale graph tokenization, expensive negative sampling, or low-level raw signal reconstruction. By extending joint-embedding predictive learning to multi-resolution latent spaces mapped onto a Lorentz hyperboloid, HP-JEPA eliminates structural scale-blindness and outperforms fixed-resolution baselines like Graph-JEPA across six of eight benchmark datasets without constructing negative pairs or reconstructing node and edge attributes.

Details

The Fixed-Resolution Bottleneck in Graph Self-Supervised Learning

Self-supervised representation learning on graph-structured data has undergone a major paradigm shift. Early methodologies primarily relied on contrastive learning, which requires handcrafted data augmentations and computationally expensive negative-pair sampling. Subsequent generative models, such as masked graph autoencoders, shifted focus toward reconstructing masked node attributes or missing edge connections. More recently, Joint-Embedding Predictive Architectures (JEPAs) extended feature-space prediction to non-Euclidean domain structures. By predicting target representations directly in latent space rather than reconstructing raw input details, JEPAs avoid both negative sampling and over-specialization to pixel- or atom-level noise.

However, existing graph joint-embedding architectures suffer from a critical conceptual bottleneck: reliance on a single predefined graph partition resolution throughout pretraining. Real-world graphs contain complex structural patterns across multiple scales simultaneously. A fine-grained partition preserves local molecular motifs or short-range connectivity but fragments macro-scale topological organization and long-range structural dependencies. Conversely, a coarse partition captures global network context but blurs discriminative local details. As illustrated in Figure 1, enforcing a single fixed token resolution inherently biases the learned representations, limiting their ability to generalize across heterogeneous graph scales and diverse downstream tasks. HP-JEPA directly resolves this fundamental limitation by establishing an ordered hierarchy of partition resolutions over which latent prediction is executed independently.

Read more

TL;DR

WHAT was done? The authors introduce FLEX-π, a 6B-parameter world-action model (WAM) that jointly predicts continuous robot actions alongside future RGB appearances, 3D pointmaps, and DINOv3 ^[review] semantic representations. Crucially, 3D pointmaps derived from Depth Anything 3 are embedded directly into the latent space of a frozen RGB video variational autoencoder (Wan-2.2) without any pointmap-specific pre-training. Using visual stream dropout with cross-modality forcing within a Mixture-of-Transformers backbone, a single checkpoint supports flexible deployment across the inference spectrum, executing fast action-only inference (∼60 ms) or full joint generation (∼193 ms).

WHY it matters? Standard generalist robot policies face a trade-off between the fast reactive execution of Vision-Language-Action models (VLAs) and the rich internal predictive regularizations of RGB-only World-Action Models (WAMs). FLEX-π resolves this dichotomy by showing that physical grounding in 3D geometry and semantics can be injected into generative policies with zero extra sensor hardware, zero specialized 3D architectural pre-training, and zero mandatory inference slowdown. The model achieves a 2–6× success rate improvement over state-of-the-art baselines on sub-millimeter and deformable bimanual manipulation tasks.

Executive summary: Robotic manipulation requires understanding spatial depth, object semantics, and future visual consequences. Current AI models for robots typically predict only future camera pixels, which capture colors and textures but struggle with exact 3D geometry and object relationships. FLEX-π solves this by predicting 3D shapes and high-level object features simultaneously with robot actions. Remarkably, it requires no expensive 3D sensors or complex custom architectures: it extracts geometry and semantics directly from standard camera images using off-the-shelf foundation models and a standard video encoder. Furthermore, at runtime, the robot can dynamically choose between generating full mental simulations for maximum accuracy or generating actions only for high-speed, real-time control, outperforming existing systems on dexterous real-world tasks.

Details

The Appearance Bias in Generative Visuomotor Policies

Generalist robot policies have recently bifurcated into two main paradigms: Vision-Language-Action models (RT-2, π0​, π0.5​), which cast control as direct conditional imitation, and World-Action Models (UWM, DreamZero, Fast-WAM, LingBot-VA), which jointly optimize for action emission and future observation prediction. While WAMs inherit expressive spatiotemporal priors from internet-scale video generation models, their supervisory signal is almost exclusively confined to RGB latents optimized for pixel-level reconstruction. In contact-rich, high-precision manipulation, pixel appearance is a poor surrogate for explicit 3D Euclidean geometry and object-centric semantics.

Traditional attempts to incorporate 3D spatial priors—such as voxel grids in Perceiver-Actor or explicit point-cloud encoders in 3D Diffusion Policy and ManiFlow—introduce substantial practical barriers. They mandate active physical depth sensors at deployment, require specialized pre-training pipelines on geometric data, and incur fixed computational overheads that degrade inference control frequencies. FLEX-π removes these constraints by framing multimodal world-action modeling around a foundational insight: frozen video-generation autoencoders can serve directly as continuous field encoders for 3D pointmaps, allowing multi-stream geometric and semantic prediction without specialized sensor dependencies or dedicated 3D backbones.

Read more

TL;DR

WHAT was done? The authors introduce DiffusionGemma, an experimental open-weight text diffusion model with 25.2B total parameters and 3.85B activated parameters based on the Gemma 4 26B A4B mixture-of-experts backbone. Rather than generating text strictly left-to-right one token at a time, DiffusionGemma iteratively refines canvases of 256 tokens in parallel. Bypassing native diffusion pretraining, the model is warm-started from post-trained autoregressive weights and fine-tuned using a compute-efficient two-stage pipeline consisting of Supervised Fine-Tuning (SFT) and a novel joint Sampler Distillation & Reinforcement Learning (SD-RL) phase.

WHY it matters? Standard autoregressive language models serving single-request or low-concurrency workloads are fundamentally memory-bandwidth bound, spending vastly more time moving weights and KV caches between high-bandwidth memory and compute units than performing floating-point operations. DiffusionGemma shifts generation from a memory-bound regime to a compute-bound regime, generating approximately 20 tokens per forward pass and achieving roughly 1,500 output tokens per second on a single NVIDIA H100 GPU. This represents a 7.1x speedup over its autoregressive baseline while maintaining reasoning, long-context support, and multimodal capabilities.

Executive summary: For technical leaders and systems architects, DiffusionGemma marks a practical breakthrough in non-autoregressive text generation. By converting a strong, pre-trained mixture-of-experts model into a discrete text diffusion engine using less than 10% of the original training token budget, Google DeepMind demonstrates that high-speed parallel generation does not require sacrificing complex reasoning or multimodal understanding. By releasing the model weights under an Apache 2.0 license alongside reference implementations in vLLM and HuggingFace Transformers, this work provides the broader community with a high-throughput substrate for real-time applications, agentic workflows, and low-latency deployments.

Details

The Memory-Bandwidth Bottleneck in Autoregressive Serving

Large language model inference is constrained by the architectural properties of modern hardware accelerators. Standard autoregressive generation factorizes sequence joint probability into an exact product of conditional probabilities

necessitating a distinct forward pass for every single token generated. When serving single-request or low-batch-size workloads, execution time is dominated by memory transfers, specifically loading multi-billion parameter model weights and past key-value (KV) states into chip SRAM. This memory-bandwidth wall leaves tensor processing units under-utilized.

While techniques like speculative decoding and Multi-Token Prediction (MTP) attempt to increase memory efficiency by using draft models to propose candidate sequences, they remain constrained by draft-verification overheads and declining acceptance rates at later token positions. Text diffusion provides an alternative by converting single-token serial generation into a parallel canvas refinement process. By processing blocks of 256 tokens simultaneously, diffusion models execute dramatically fewer total forward passes per sequence, exchanging memory bandwidth overhead for floating-point calculations that fully utilize modern GPU compute units.

Discrete Flow Matching on Categorical Canvases

To formalize discrete text diffusion, the authors build upon continuous-time Markov chains (CTMC) and discrete flow matching theory. Let V denote a categorical token vocabulary of size V=262,000, and let C=256 represent the sequence canvas length. A realization of the canvas at time t∈[0,1] is represented as a vector x_t​∈V^C. The model defines a marginal probability path that smoothly interpolates between an uncorrupted data distribution x₀​∼p(⋅) at t=0 and a fully corrupted source canvas x₁∼Unif(V^C) at t=1.

The continuous-time forward process corrupts clean text tokens into uniformly distributed categorical noise. Conditioned on a clean starting canvas x₀​, the transition path factorizes independently across token coordinates i∈{1,…,C} according to the probability rule:

In this formulation, κ_t​∈[0,1] is a monotonically decreasing schedule running from κ₀​=1 down to κ₁​=0, and δ denotes the Kronecker delta. To generate text, the model learns a reverse denoising process parameterized by a neural network p_θ​(v∣x_t​) that approximates the posterior distribution of clean tokens given a noisy canvas state x_t​. The transition update stepping backward in time by an increment Δt is expressed as P(X_t−Δt​=⋅∣X_t​=x_t​)≈Step(x_t​,p_θ​(⋅∣x_t​)). Operating directly over discrete categorical distributions avoids the rounding errors and spatial drift that historically hindered continuous embedding space text diffusion.

The Block-Autoregressive Denoising Loop

DiffusionGemma operates via a hybrid block-autoregressive generation pipeline that reconciles fixed-length discrete flow matching with open-ended text generation, as illustrated in Figure 4. The architecture utilizes an inverted transformer setup: context history is processed using a causal encoder to construct a persistent Key-Value (KV) cache H, while generation across the current 256-token canvas is handled by a bidirectional decoder with shared weights θ.

Consider a specific execution trajectory during inference. Given an input prompt h∈V^L, the system populates the initial KV cache using H=Encoder_θ​(h). Canvas generation starts by initializing a sequence x₁​ of 256 uniformly random tokens from vocabulary V, alongside an initial self-conditioning signal z₁​=0. At each denoising iteration step t, the decoder takes the corrupted canvas x_t​, the context KV cache H, and the self-conditioning state z_t​∈R^C×d to produce unnormalized logits:

L_t​=Decoder_θ​(x_t​,z_t​,H)∈R^C×V

The clean token probabilities 𝑝̂₀​=Softmax(L_t​/τ_t​) are evaluated using a time-dependent temperature schedule τ_t​ that anneals linearly from τ_max​=0.8 down to τ_min​=0.4. An updated self-conditioning vector z_t−Δt​=FFW(𝑝̂₀​​E)∈R^C×d is constructed by passing the expected token embeddings through a feedforward network, feeding the model’s intermediate structural predictions directly back into the next forward pass.

To update the canvas x_t−Δt​, DiffusionGemma uses an entropy-bounded sampler (Algorithm 1). Positions across the canvas are ordered by predictive entropy eⁱ=Entropy(𝑝̂₀ⁱ​). Tokens at positions meeting a strict cumulative error tolerance threshold b=0.1 are updated with candidate samples, whereas higher-uncertainty positions are re-noised uniformly at random to force continued local exploration. Denoising is dynamically halted via an adaptive stopping heuristic when the mean predictive entropy across the canvas drops below e_stop​=0.005 and the deterministic predictions between consecutive steps remain identical. Once fully denoised, the clean canvas x̂₀​ is appended to the KV cache via H←H⊕Encoder_θ​(x̂₀​), and the model advances to the next 256-token block.

Optimization, Losses, and Distillation

Rather than executing full pretraining from scratch, DiffusionGemma warm-starts directly from the post-trained weights of the Gemma 4 26B A4B Mixture-of-Experts checkpoint (Gemma Team et al., 2026). This MoE backbone activates 3.85B parameters out of 25.2B total per token, utilizing 8 active experts out of 128 alongside 1 shared expert. As detailed in Figure 2, the model undergoes a compute-efficient two-stage training pipeline consuming under 10% of the base AR model’s training token budget.

The first stage is Supervised Fine-Tuning (SFT), which adapts the pre-trained causal representations to bidirectional block-level denoising. A block-diagonal attention mask enables full non-causal attention within each 256-token canvas while conditioning on prior context via cross-attention to the KV cache. Noise levels t∼Unif[0,1] are sampled to corrupt training tokens, and the model optimizes a cross-entropy loss over uncorrupted targets:

Following SFT, the model achieves high output quality when allocated a large step budget (N=192), but experiences performance degradation under few-step low-latency regimes. To address this, the authors introduce a unified second stage: Sampler Distillation & Reinforcement Learning (SD-RL). Rather than separating preference alignment and step distillation, SD-RL applies a joint online gradient objective. An online teacher model generates high-step denoising trajectories to compute reward estimates across mathematical reasoning, coding, and instruction-following environments.

The SD-RL objective optimizes reward generation while aggressively driving down predictive entropy across the reverse process trajectory. As entropy decreases, the adaptive stopping mechanism triggers earlier in the reverse chain (Figure 8), creating an implicit curriculum that compresses high-reward generations into fewer denoising steps.

For practical community adaptations, parameter-efficient fine-tuning recipes were developed using Low-Rank Adaptation (LoRA) (Hu et al., 2022). LoRA matrices are injected across linear projections, including attention heads, MLP gates, MoE routers, and the self-conditioning feedforward block. Training can be executed using open-source JAX utilities like Hackable Diffusion on modest hardware setups (such as 2x NVIDIA A100 80GB GPUs).

Hardware-level inference execution relies on customized lower-level kernels. Bidirectional attention across 256-token blocks is accelerated via FlashAttention-4 (Dao et al., 2022), while sampling routines are optimized using PyTorch torch.compile primitives. Serving efficiency is further enhanced by eliminating CPU-GPU synchronization bottlenecks through asynchronous sequence scheduling within native engine environments like vLLM.

Empirical Pareto Frontiers and Speedup Sources

DiffusionGemma establishes a new Pareto frontier balancing output speed and generation quality, outperforming both open-weight and proprietary baselines as shown in Figure 1.

On a single NVIDIA H100 GPU operating in FP8 precision at batch size 1, DiffusionGemma reaches an average output speed of 1,479 Tokens Per Second (TPS) with a Tokens Per Forward (TPF) metric of 19.74 across benchmark suites (Table 3). This represents a 7.1x speed improvement over the standard Gemma 4 AR baseline (204 TPS) and a 4.8x speedup over Gemma 4 equipped with Multi-Token Prediction (303 TPS). On GPQA-Diamond, DiffusionGemma achieves 73.2% accuracy in text diffusion mode, outperforming LLaDA 2.1 Flash 100B (68.7% at 375 TPS) and Nemotron Diffusion 14B (47.0% at 49 TPS), while remaining highly competitive with closed-weight APIs like Mercury 2 (75.2% at 600 TPS).

The source of this efficiency boost is analyzed through latency breakdowns in Figure 11. Evaluating a single forward pass over a 256-token canvas requires more compute operations than generating a single token autoregressively. Specifically, processing a canvas incurs a 3.2x increase in per-step GPU kernel execution time (12.63 ms vs 4.01 ms for single-token AR). This increase is driven by broader MoE expert activation (4.66 ms vs 1.08 ms, a 4.3x slowdown due to 84 active experts per canvas versus 8 per single token), denoising sampling overhead (3.06 ms vs 0.56 ms, a 5.5x increase), and bidirectional attention computation (1.84 ms vs 0.45 ms, a 4.1x increase).

However, because each step processes 256 tokens simultaneously and converges in roughly 12 effective denoising steps (Table 4), the overall reduction in total forward passes far outweighs the kernel execution overhead.

Ablation analysis confirms the importance of the SD-RL training phase (Figure 9).

The SFT-only baseline evaluated with restricted denoising steps (N=48) suffers from predictive entropy collapse and token repetition loops (Figure 16). SD-RL training resolves these loop artifacts, providing a +10 point accuracy gain on combined GPQA-Diamond and LiveCodeBench-v6 benchmarks while quadrupling generation TPF from 5 to nearly 20.

Qualitative probing reveals that bidirectional attention enables in-canvas self-correction (Figure 15 and Figure 23). When presented with multi-step arithmetic or logical traps, causal AR models frequently commit to incorrect early tokens and must subsequently generate explicit textual corrections. DiffusionGemma leverages non-causal attention across the canvas to co-evolve intermediate reasoning tokens and final outputs simultaneously, self-correcting flawed logic paths across initial denoising iterations before finalizing discrete token commitments.

Contextualizing Text Diffusion and Speculative Baselines

DiffusionGemma advances non-autoregressive text modeling beyond prior open-weights and proprietary approaches. Earlier text diffusion systems faced theoretical trade-offs: continuous diffusion models like Diffusion-LM (Li et al., 2022) required rounding mechanisms to map continuous vectors back to discrete tokens, leading to spatial drift and ungrammatical outputs. Conversely, discrete diffusion architectures like SEDD (Lou et al., 2024) established discrete transition models, but often suffered from slow inference sampling.

In relation to contemporary diffusion baselines, DiffusionGemma demonstrates notable speed and architectural advantages. Models such as LLaDA 2.1 Flash 100B (Bie et al., 2025) achieve 375 TPS across 8x NVIDIA B200 GPUs, while Nemotron Diffusion 14B yields 49 TPS on an H100. Proprietary APIs like Mercury 2 (Inception Labs, 2025) deliver ~600 TPS. DiffusionGemma reaches roughly 1,500 TPS on a single H100 GPU while providing an open-weight implementation that retains extended context processing, multimodal inputs, and an explicit internal reasoning channel.

Compared to speculative decoding frameworks such as Medusa (Cai et al., 2024), EAGLE-3 (Li et al., 2026), or TiDAR (Liu et al., 2026b), DiffusionGemma does not require separate draft-then-verify loops or target model verification overheads. By formulating generation as block-autoregressive canvas denoising, the model maintains high joint token acceptance across full 256-token canvases without suffering from suffix decay at later sequence positions.

Architectural Trade-offs and Generative Artifacts

Despite its decoding efficiency, DiffusionGemma exhibits theoretical and operational trade-offs. The model shows a modest absolute quality delta compared to its starting Gemma 4 26B AR baseline across dense reasoning benchmarks (Table 3). This gap stems from practical design choices: bypassing native diffusion pretraining from scratch in favor of warm-started AR weights, constraining SFT compute budgets, and employing an SD-RL trajectory compression objective that explicitly optimizes for ultra-low latency.

Another emergent characteristic is output conciseness. Post-SD-RL checkpoints generate responses that are roughly 2x shorter than those produced by the SFT baseline (Figure 9). While concise answers minimize total sequence length and accelerate end-to-end latency, they restrict the model’s ability to utilize long, highly detailed Chain-of-Thought reasoning traces that can benefit complex mathematical proofs.

In addition, the model can occasionally exhibit low-latency generation artifacts. Under tightly restricted step bounds, outputs may rarely fall into localized token stuttering. In multimodal settings, the model can occasionally omit closing thought tags (<|channel|>), which can disrupt parsing logic during automated benchmarking. Finally, serving throughput advantages are concentrated in low-to-moderate batch size regimes (N≤32 concurrent requests, as shown in Figure 12); at higher batch sizes, traditional autoregressive models regain throughput parity as workload characteristics shift back toward being compute-bound across larger parallel batches.

Strategic Outlook: Beyond Left-to-Right Generation

DiffusionGemma presents a viable path for deploying non-autoregressive language models in real-time, latency-critical production environments. By converting a standard Mixture-of-Experts transformer backbone into a discrete flow matching engine, the authors demonstrate that text diffusion can overcome the sequential decoding memory-bandwidth bottleneck without requiring custom pretraining infrastructure.

The dual-mode capacity of DiffusionGemma—retaining standard autoregressive execution while enabling discrete block diffusion—points toward dynamic inference routing systems. Future serving infrastructures could route low-latency, highly constrained, or structured tasks through ultra-fast diffusion decoding while reserving sequential autoregressive sampling for open-ended, extended reasoning queries. Releasing the model weights under an Apache 2.0 open license alongside ecosystem integration in HuggingFace and vLLM offers a strong foundation for future research in parallel text generation algorithms, advanced sampler designs, and optimized hybrid decoding architectures.

TL;DR

WHAT was done? The authors introduce the Skaling law, a generalized functional form for neural scaling that couples model capacity N and data volume D through a single outer interaction exponent k. Alongside this functional form, the paper proposes an “L-shape” sparse profiling grid strategy that restricts experimental pretraining sweeps to low-compute boundaries instead of sampling dense full grids.

WHY it matters? Standard additive scaling formulations, such as the widely used Chinchilla law, assume that model size and data volume affect loss independently, artificially forcing their mixed cross-derivative to zero. This structural flaw causes systematic prediction errors at data-scarce or highly overtrained extremes. Skaling corrects these boundary biases, reducing extrapolation Mean Absolute Percentage Error (MAPE) by 1.5–3× while cutting the compute required for empirical scaling sweeps by roughly 10×.

Executive summary: Allocating multi-million-dollar compute budgets for frontier LLMs relies heavily on predicting final model performance from small-scale pilot runs. Traditional scaling formulas make a mathematically flawed assumption: that increasing model size and increasing training data act completely independently on loss reduction. By introducing a single interaction parameter, the Skaling law eliminates the boundary errors inherent to standard models. Furthermore, because this coupled surface is anchored along its boundaries, AI practitioners can accurately predict large-scale training outcomes by running small experiments arranged in an efficient “L-shaped” sweep rather than expensive full-grid benchmarks, reducing experimental compute overhead by tenfold.

Details

The Zero-Interaction Bottleneck in Modern Scaling Laws

Predictive scaling laws form the bedrock of modern large language model development, guiding critical architectural choices, token-to-parameter ratios, and compute budget allocations. The empirical paradigm has historically been anchored by two competing formulations. The early work by Kaplan et al. (2020) modeled performance by coupling parameter count and token volume, but tied their inner exponents in a manner that restricted independent per-axis decay rates. To address this, Hoffmann et al. (2022) introduced the additive Chinchilla law, modeling the loss as

While mathematically tractable, this additive structure imposes a strict condition: the cross-derivative

is forced to be identically zero everywhere in the parameter space.

This mathematical convenience introduces a severe structural bias. In real-world pretraining regimes, the marginal utility of adding parameters depends heavily on the volume of data the model sees, and vice versa. As demonstrated in Figure 1, fitting an additive law to empirical loss surfaces creates a distinct saddle-shaped residual pattern.

While additive models remain relatively accurate in the interior of the training grid where parameters and data are balanced, their prediction error grows dramatically toward the corners, reaching several percentage points of bias where N and D are most imbalanced. Highly parameter-heavy or data-heavy runs are systematically under- or overestimated. While recent over-parameterized models like the nine-parameter Farseer law attempt to capture these dynamics by making parameters scale-dependent, they introduce severe optimization instability and overfit noise without fixing boundary extrapolation. The central challenge lies in restoring cross-variable interaction without sacrificing parameter interpretability or closed-form optimization.

Read more

TL;DR

WHAT was done? The authors present a substrate-independent formalization of equilibration-style thermodynamic computing and fabricate its first silicon instantiation: CN101, a prototype digital thermodynamic computing chip manufactured on standard CMOS using stochastic computing principles.

WHY it matters? By decoupling thermodynamic algorithms from fragile analogue implementations, this work establishes that physical relaxation primitives can be realized entirely in digital logic. Crucially, the chip demonstrates “sequential parallelism”—a property that allows deeply unrolled, multi-stage generative models (such as diffusion networks and flow-matching models) to relax concurrently rather than sequentially, yielding up to a 62× reduction in computational latency.

Executive summary: Conventional GPU architectures dictate a trade-off where compute efficiency requires wide, parallel batching, running counter to the deep, sequential structure of modern generative AI models. CN101 proves that thermodynamic relaxation—where answers emerge as time-averaged equilibrium statistics of stochastic dynamical systems—can run natively on standard digital silicon. Rather than waiting for each layer of a deep neural network to compute exact floating-point outputs sequentially, CN101 executes all layers concurrently. Downstream stages process live, converging stochastic estimates from upstream stages, allowing the whole network to reach convergence dramatically faster than classical serial execution.

Details

The Sequential Bottleneck and the Hardware Lottery

Modern generative AI models derive their representative power from depth and sequential composition. Autoregressive transformer models generate text token by token, while diffusion and flow-matching architectures generate images and molecular structures by integrating ordinary differential equations across dozens or hundreds of discrete steps. However, as articulated in modern accelerator design analyses, the dominant paradigm in hardware acceleration—represented by modern GPUs and TPUs—rewards massive data-parallel width rather than sequential depth. These hardware platforms demand high arithmetic intensity and wide SIMD execution lanes, forcing model architects to build wider, shallower networks and consume higher power through aggressive batching.

Equilibration-style thermodynamic computing offers a fundamentally different paradigm. Instead of executing deterministic, sequential instruction paths, thermodynamic hardware computes by allowing a physical or logical system to relax to a stationary distribution. Historically, this approach was strictly tied to continuous-space Langevin dynamics implemented on analogue substrates, such as capacitive RLC circuits instantiated in Stochastic Processing Units ^{[see here]}. While appealing, analogue implementations face severe real-world engineering hurdles, including high device variability, limited dynamic range, susceptibility to thermal noise, and an inability to leverage standard electronic design automation (EDA) toolchains. The primary bottleneck in physics-inspired AI hardware has thus not been theoretical capability, but substrate dependency.

Read more

TL;DR

WHAT was done? The authors systematically isolate the precise cause of large language model (LLM) failure on tabular classification by evaluating five common folklore hypotheses in a pure inference regime. Using a rigorous data-hygiene memorisation probe, controlled feature manipulations, and random-projection dimensionality sweeps across benchmark and synthetic datasets, the study falsifies four popular explanations—class overlap, serialized CSV formatting, numeric tokenization, and per-query test load—and identifies input dimensionality as the sole factor driving the performance collapse of models like claude-opus-4-6.

WHY it matters? This work provides the first causal explanation for the persistent performance gap between general-purpose language models and classical machine learning baselines on tabular data. By demonstrating that LLM capabilities collapse as feature dimensions grow—even when information content is held strictly constant—it establishes clear architectural limits for in-context tabular reasoning, refutes naive prompt-engineering fixes, and provides strong theoretical justification for dedicated tabular foundation models (see TabFM btw).

Executive summary: Applied machine learning practitioners frequently observe that frontier language models lose to fifty-year-old algorithms on simple numeric tables. Rather than adding complex prompt scaffolding or agent loops, this paper probes the raw model in pure in-context inference mode to understand why. The findings reveal that claims of LLM failure due to awkward CSV text formatting, tokenization of numbers, or noisy class boundaries are completely unfounded. Instead, LLMs act as effective local distance-based classifiers in two-dimensional space but experience a catastrophic breakdown as feature count increases beyond low dimensions. Consequently, attempting to fix high-dimensional tabular prediction through prompt adjustments is fundamentally misguided, confirming the necessity of specialized tabular architectures for tabular workloads.

Details

The Tabular Anomaly in In-Context Learning

Large language models have demonstrated impressive capabilities across natural language processing, code synthesis, and multi-step reasoning tasks. However, predictive analytics over tabular data remains a glaring exception where general LLMs systematically underperform compared to traditional tree ensembles and simple distance-based baselines, as illustrated in Figure 1.

While the prevailing response across the machine learning community has been to layer external software harnesses—such as retrieval-augmented generation, automated feature engineering agents, and multi-turn loops—around the core model, this approach obscures whether the failure stems from the model itself or the surrounding infrastructure. To address this fundamental question, the investigation evaluates the core model under a pure inference regime: a single user turn containing the complete training set and test queries without system prompts, external tools, code execution, or task-specific fine-tuning.

The underlying conflict in tabular learning lies in the stark contrast between human natural language processing and structured data matrices. Standard tabular classifiers, such as gradient boosted decision trees or nearest-neighbor algorithms, explicitly exploit vertical column structures, invariant feature coordinate systems, and continuous metric spaces. In contrast, standard autoregressive transformers operate on linear token sequences where spatial tabular topology is flattened into text strings. The study aims to delineate whether the LLM’s failure on tabular data reflects an intrinsic limitation of sequence models in high-dimensional feature spaces or merely secondary issues related to text serialization, numeric token precision, or dataset noise.

Read more

TL;DR

WHAT was done? The paper introduces and evaluates a proof-of-concept autonomous computer worm powered by a locally hosted, open-weight single-GPU large language model (“a publicly available open-weight LLM published in 2025”). Driven by an agentic framework featuring a multi-node reasoning graph, hierarchical memory, and dynamic skill injection, the worm autonomously discovers vulnerabilities, synthesizes target-specific exploits, and replicates across heterogeneous network hosts without human intervention or reliance on centralized vendor APIs.

WHY it matters? This research provides empirical proof that self-sustaining, AI-driven cyber threats are no longer hypothetical. By parasitically acquiring computational resources from compromised hosts to run local inference or forward queries, the worm eliminates the attacker’s marginal cost per infection while collapsing the traditional security trade-off between worm-like scale and target-specific adaptation.

Executive summary: Operational defenders have historically interrupted automated worm propagation by patching the static, pre-compiled exploit vectors built into malware logic. This paper demonstrates that an off-the-shelf, open-weight LLM running on a single 80GB GPU can bypass this defense model by dynamically generating attack strategies at runtime. When deployed across an isolated test network of 33 heterogeneous systems, the worm successfully exploited 73.8% of target hosts and replicated across 61.8% of the network. Because the system runs entirely on local weights using stolen compute, centralized vendor guardrails like rate limits, content filters, and API suspensions are structurally irrelevant. Security teams and policymakers must prepare for autonomous generative adversaries that absorb public vulnerability disclosures in real time and scale across corporate networks without external command-and-control infrastructure.

Details

The Scale-Adaptation Bottleneck in Cyber-Offense

Historically, cyber-offensive operations have been constrained by an economic trade-off between scalability and adaptability. Traditional computer worms achieve rapid, global reach by automating the execution of hardcoded exploit logic; however, their spread halts as soon as targets deviate from assumed software versions or security configurations. Conversely, human red teams excel at tailored, interactive attack chains, but their substantial time and engineering costs limit such operations to high-value targets. Recent advances in automated penetration testing, such as AutoAttacker, Incalmo, and ARTEMIS, have shown that large language models can navigate complex multi-host environments. However, these systems relied on centralized, closed-source frontier APIs and lacked autonomous self-replication mechanisms. This work collapses that historical trade-off by demonstrating an end-to-end, self-replicating worm powered entirely by a local, open-weight model that achieves both network-wide scale and target-specific adaptation.

Single-GPU Intelligence First Principles: Framing the Agentic Harness

The core thesis of this work is that raw model size is not the binding constraint for autonomous cyber-offensive operations. While single-GPU open-weight models—specifically those fitting within an 80GB VRAM envelope like an NVIDIA A100 or RTX PRO 6000 Blackwell Edition—exhibit known limitations in long-context retrieval, instruction following, and code generation precision, systematic agentic scaffolding can bridge this capability gap.

Mathematically, the engagement state space can be modeled as a tuple S=(P,M,H,C), where P∈{1,…,8} denotes the active lifecycle phase, M represents the three-tier hierarchical memory state, H tracks active vulnerability hypotheses, and C encapsulates dynamic tool context. Rather than presenting the LLM with an unconstrained, monolithic system prompt, the framework scopes every decision point through specialized reasoning nodes. By converting noisy, low-signal terminal outputs into structured factual representations, a single-GPU model near the 32-billion parameter scale can execute cohesive, multi-step attack chains without requiring domain-specific fine-tuning.

From Reconnaissance to Propagation: The Autonomous Attack Loop

To understand how an individual target flows through the system, consider a victim host running an unauthenticated Jupyter service on port 8888, as shown in Figure 1 and outlined in the execution phase breakdown of Figure 3.

The agent begins with network discovery to identify reachable IP addresses, transitioning to host discovery to enumerate open ports and running service signatures. Upon locating port 8888, the workflow moves to the foothold exploitation phase, where the agent enters a cyclic execution loop within its reasoning graph, illustrated in Figure 4.

The PlanNode synthesizes host profile data into an ordered attack sequence, such as attempting terminal session hijacking via WebSockets. The JudgeNode cross-references this strategy against the chronological command history, pruning previously failed attempts. The ActionNode selects and executes the appropriate tool, passing parameters to the target interface. Upon execution, the SummaryNode parses stdout and stderr streams, extracting discrete findings into persistent host memory, while the ProgressNode evaluates whether initial access was achieved. Once a foothold is established, the agent initiates privilege escalation discovery and exploitation to secure root or SYSTEM control, as mapped out in the single-host attack reasoning tree in Figure 13.

Finally, during replication, the agent stages its source code on the victim, resolves runtime dependencies, and spawns an independent replica process. If the compromised machine contains a GPU, a local instance of the LLM server is initialized; otherwise, reasoning queries are routed upstream to existing GPU nodes across the network.

Multi-Tier Memory, Tooling, and Hypervisor Containment

The operational framework relies on strict context curation and hypervisor-enforced isolation. Memory is partitioned into three functional layers: General Memory for mission-wide state, Host Memory for per-target service profiles, and Vulnerability Memory for driving hypothesis rotation. To mitigate context pollution caused by verbose terminal logs, dedicated tool handlers intercept raw output, translating failure states like HTTP 404 responses into actionable hints such as “endpoint not found, try an alternative directory path.” A dynamic skill module injects targeted penetration testing techniques into the model’s prompt on demand, enabling the agent to operationalize newly published advisories at runtime. Swarm coordination is managed through host reservation services and a shared credential store, allowing active replicas to reuse harvested passwords seamlessly across targets. Security and containment are enforced entirely outside the guest operating system at the hypervisor boundary using KVM/QEMU, as detailed in Table 3. Hypervisor kernel firewalls (iptables) enforce default-deny egress rules, a transparent Squid proxy serves cached software packages, and a host-level Containment Attestation Service (CAS) verifies launch permits via an Ed25519 challenge-response handshake to prevent execution outside the designated virtual sandbox.

Empirical Validation Across Heterogeneous Networks

The evaluation environment, termed “FakeCorp,” consists of 33 heterogeneous virtual machines spanning Linux distributions (Ubuntu, Debian, Alpine, Rocky Linux, CentOS Stream), Windows Server platforms (2008 R2, 2019, 2022), and IoT/ICS endpoints. Hosts are provisioned with real-world vulnerabilities sourced from CISA Known Exploited Vulnerabilities, OWASP Top 10: 2025, and MITRE ATT&CK taxonomies, as detailed in Table 1. Across 15 independent 7-day experimental runs, the worm demonstrated significant propagation capabilities, summarized in Figure 6.

On average, the agent correctly identified 31.3±1.7 vulnerabilities, exploited 23.1±3.9 hosts to elevated access, and propagated to 20.4±3.2 hosts, reaching up to 7 generations of self-replication. Across 954 total attack attempts, vulnerability identification succeeded in 82% of cases, exploitation in 44%, and self-replication in 88% of compromised hosts, as recorded in Figure 5 and Table 2.

Category-specific performance in Figure 8 shows that local privilege escalation yielded the highest exploitation success rate at 71.7%, whereas web exploits (48.5%) and Windows privilege escalation (33.7%) experienced higher failure rates due to string escaping and command formatting errors.

Detailed failure analyses in Figure 9 and Figure 10 confirm that 66% of exploitation failures resulted from malformed payload syntax rather than flawed strategic reasoning. Tests on one-day vulnerabilities disclosed in 2026 after the model’s cutoff date—including Copy Fail (CVE-2026-31431), Dirty Frag (CVE-2026-43284), and Marimo RCE (CVE-2026-39987)—achieved a 61.2% root compromise rate, validating the agent’s ability to ingest and execute public exploit guidance without retraining.

The radial propagation visualisations in Figure 2 and Figure 12 demonstrate how decentralized swarm redundancy enables surviving replicas to persistently retry difficult targets until network-wide propagation succeeds.

Positioning Against Emerging AI Security Benchmarks

This research connects several distinct subfields, including autonomous AI agency, self-replication, and offensive evaluation methodologies. Benchmark studies on autonomous agent replication, such as RepliBench and related evaluations, established that language models could perform code staging and environment setup in isolated compute environments. Similarly, research into application-layer propagation, including RAGworm and broader promptware taxonomies, analyzed prompt injection spread within generative AI applications. However, in those settings, the LLM functioned primarily as the execution substrate or target rather than the autonomous attack engine. Conversely, multi-host red-teaming frameworks like Incalmo and ARTEMIS demonstrated multi-host penetration capabilities but depended on centralized API services and external command-and-control infrastructure. By combining single-GPU local inference, structured scaffolding, and network-level host compromise, this work demonstrates end-to-end autonomous worm propagation that operates independently of commercial platforms.

Model Ceilings and Testbed Simplifications

While the empirical findings demonstrate clear capability thresholds, several scope boundaries should be noted. The FakeCorp testbed features an artificially dense vulnerability profile, where every machine contains at least one exploitable flaw, and no endpoint detection and response (EDR), antivirus, or active traffic monitoring tools are present. Consequently, the experiment evaluates strategic reasoning and payload generation rather than stealthy lateral movement or evasion of active defensive monitoring. Additionally, the 44% exploitation success rate highlights a code-generation ceiling inherent to current single-GPU models, particularly when interacting with complex Windows command environments or nested payload syntax. Furthermore, while ablation tests with physical GPU passthrough achieved a 68.8% success rate for full local API deployment, dependency installation and service configuration issues introduce operational friction compared to remote inference forwarding.

Strategic Implications for AI Defense and Governance

The emergence of adaptive AI worms alters the structural economics of cybersecurity defense. Traditional incident response relies heavily on patch deployment windows, operating on the assumption that developing tailored exploit code requires manual human effort and extended timelines. An autonomous agent capable of ingesting public vulnerability disclosures and generating working exploit chains within hours dramatically compresses this window of risk, spreading at zero marginal cost using stolen compute. Because open-weight models run locally without external telemetry, vendor-side safeguards like system prompts, rate limits, and API bans are structurally ineffective. Addressing this threat mandates moving toward AI-assisted automated patching, enforcing rigorous zero-trust network micro-segmentation, and establishing peer-reviewed, hypervisor-enforced containment standards for offensive AI research.

TL;DR

WHAT was done? The authors present the first comprehensive survey and empirical analysis of abductive reasoning in Large Language Models (LLMs). To resolve widespread conceptual ambiguity in AI literature, the paper formalizes a theoretical framework that disentangles abductive reasoning into a two-stage process comprising Hypothesis Generation and Hypothesis Selection, grounded in Inference to the Best Explanation (IBE). Building upon this framework, the authors introduce a novel four-axis taxonomy classifying over 60 papers, conduct an empirical evaluation across 11 open-weight and proprietary model families ranging from 3B to 72B parameters, and perform a cross-paradigm meta-analysis comparing abductive, deductive, and inductive capabilities.

WHY it matters? Abductive reasoning—the inference of the most plausible explanation for an unexpected observation—is fundamental to diagnostic reasoning, scientific discovery, legal argumentation, and real-world troubleshooting. However, AI research has historically treated abduction in a fragmented manner, often confusing option selection with genuine explanation generation or assuming that strong deductive capabilities naturally imply abductive competence. This work establishes a unified foundation for the field, demonstrates empirically that high deductive performance does not translate to abductive capability, and highlights a severe degradation in LLM performance when transitioning from selecting provided hypotheses to generating explanations from scratch.

Executive summary For technical leads, AI researchers, and domain experts developing reasoning systems, this survey demonstrates that current LLMs excel at closed-form hypothesis selection but struggle when required to generate explanatory hypotheses in open-ended or long-context scenarios. The study reveals that standard supervised fine-tuning and maximum-likelihood training fail to optimize for true explanatory virtues such as parsimony, coherence, and causal validity. To advance from surface pattern matching to genuine explanatory inference, future research must shift toward process-rewarded reinforcement learning, interactive action-oriented benchmarks in specialized domains like clinical medicine and law, modular multi-agent architectures, and circuit-level mechanistic interpretability.

Details

The Definitional Bottleneck in Machine Inference

The rapid advancement of Large Language Models has sparked intense interest in evaluating their higher-order reasoning capabilities. While formal deduction has historically dominated AI evaluation through natural language inference benchmarks, abductive reasoning—the ampliative leap from an observation to its most plausible underlying explanation—remains conceptually fragmented. Research in NLP and machine learning frequently invokes abduction to describe fundamentally distinct tasks. Some studies frame abduction purely as a discriminative ranking task over pre-packaged multiple-choice options, while others treat it as unconstrained free-text generation, knowledge graph path completion, or set-cover logic verification. Citing foundational comparative evaluations such as Sheng et al. (2025), Dougrez-Lewis et al. (2025), and Xu et al. (2025a), the authors highlight how this lack of a common operationalization has produced conflicting empirical reports, where abduction is simultaneously characterized as both the easiest and the most challenging reasoning paradigm for language models.

This fragmentation is depicted in Figure 1, which tracks publication trends in computer science from 1994 to 2025. While non-LLM NLP approaches dominated earlier research, the recent explosion of LLM-based studies has exacerbated definitional ambiguity. Without a shared theoretical substrate, comparing methodologies, evaluating cumulative progress, or designing robust benchmarks remains extraordinarily difficult. Resolving this bottleneck requires looking beyond computational expediency and re-anchoring machine abduction in its historical and philosophical foundations.

Philosophical First Principles: Epistemic Gaps and Two-Stage IBE

To construct a rigorous foundation for LLMs, the authors trace abduction back to Charles Sanders Peirce, who first formalized it as a distinct mode of logical inference. Peirce contrasted abduction with deduction and induction, positioning abduction as the primary engine of creative discovery. In Peircean logic, given a surprising observation O, if hypothesis A were true, O would follow as a matter of course; therefore, there is reason to suspect that A is true. Crucially, contemporary epistemology refines this concept into Inference to the Best Explanation (IBE), popularized by Gilbert Harman and Peter Lipton. Lipton articulated IBE as a two-stage architecture: first, the creative generation of a candidate hypothesis set, followed by the evaluative selection of the best explanation based on explanatory virtues.

The survey formalizes this mechanism into a unified functional pipeline for language models. The process begins with an Observation O that introduces an epistemic gap relative to the model’s background knowledge or theory T. In Stage I, termed Hypothesis Generation, the reasoner produces a set of candidate hypotheses H={h₁​,h₂​,…,h_n​} that could potentially account for O. In Stage II, termed Hypothesis Selection, the model evaluates and ranks the candidate set H against explanatory virtues—such as simplicity (Occam’s razor), coherence with T, parsimony, and predictive scope—to isolate the optimal explanation h^∗.

Positioning abduction alongside deduction and induction clarifies its structural mechanics across key logical dimensions, as summarized in Table 1.

Deductive reasoning operates non-defeasibly, monotonically, and non-ampliatively; if premises are true, conclusions are necessarily true and cannot be invalidated by new data (A⊢B⟹A,C⊢B). Conversely, abductive reasoning is inherently defeasible, non-monotonic, and ampliative. Its conclusions introduce genuinely new semantic content beyond the premises, rendering them tentative and subject to retraction if contradicting evidence arises. While enumerative induction shares these three logical properties with abduction, induction focuses on generalizing patterns across instances, whereas abduction infers latent causal mechanisms or missing background conditions to explain specific anomalies.

The Abductive Pipeline: From Anomalous Observation to Selected Explanation

To understand how the two-stage functional pipeline operates in practice, consider a concrete diagnostic scenario from clinical medicine. An abductive reasoning system receives an observation O, such as a patient presenting with acute dyspnea and sharp chest pain. Given background medical knowledge T, this clinical presentation creates an epistemic gap that requires explanation.

During Stage I (Hypothesis Generation), the model retrieves or synthesizes candidate explanations without restricting itself to a single static answer. The output of Stage I is a candidate hypothesis space H={h₁​,h₂​,h₃​,h₄​}, where h₁​ represents pulmonary embolism, h₂​ represents acute pericarditis, h₃​ represents pneumothorax, and h₄​ represents severe muscle strain. During Stage II (Hypothesis Selection), the model applies evaluative criteria to filter and rank these candidates. It weighs the explanatory power, clinical coherence, and parsimony of each candidate against patient findings, filtering out improbable options like h₄​ and prioritizing h₁​ as the optimal explanation h^∗.

This pipeline is visualized in Figure 2, which outlines the transition from initial stimulus to generated candidate pool and final evaluative filtering. Figure 3 maps this pipeline onto the authors’ novel four-axis taxonomy, which categorizes existing literature according to Task Formulation (Stage I generation vs. Stage II selection), Dataset Type (commonsense narratives vs. formal/expert domains), Methodology (prompting, fine-tuning, knowledge augmentation, multi-agent frameworks, neuro-symbolic systems), and Evaluation Approach (accuracy-based, non-accuracy-based, human evaluation).

Benchmarking Mechanics, Metrics, and Model Scales

To complement the theoretical taxonomy, the survey includes an empirical benchmark evaluation spanning 11 language models across 9 representative datasets. Evaluated open-weight models include Qwen2.5 (3B, 7B, 72B), Qwen3 (8B, 32B), Llama3.1 (8B, 70B), Llama3.3 (70B), and DeepSeek-V3.2, alongside proprietary models GPT-4o and GPT-5.4. On the selection side (Stage II), benchmarks include short commonsense bridging tasks like ART and e-CARE, medical diagnosis ranking on DDXPlus, and long-context narrative mystery solving on True Detective and MuSR. On the generation side (Stage I), benchmarks include open-text explanation tasks on UNcommonsense, medical diagnosis generation on DDXPlus, and formally constrained missing-premise completion on ProofWriter, AbductionRules, and NeuLR.

Evaluating open-ended abductive generation requires moving beyond simple exact-string matching. The study employs a combination of task-validity checks, reference-based lexical and semantic metrics (BLEU-4, ROUGE-L, BERTScore), token-level overlap, Levenshtein character similarity, and pairwise LLM-as-a-judge evaluation utilizing Gemini 3 Flash. For causal evaluation on e-CARE generation, the authors implement the Causal Explanation Quality (CEQ) metric. Formally, given a cause C, an effect E, and a generated explanation X, the causal explanation quality is defined as:

CEQ(X)=cs(C,E∣X)−cs(C,E)

where the conditioned causal strength is computed as:

cs(C,E∣X)=max[cs(C+X,E),cs(C,E+X)]

Here, cs represents the estimated causal probability between events. In essence, CEQ quantifies the exact degree to which the generated explanation X increases the logical or causal cohesion between cause C and effect E.

Empirical Analysis: The Disconnect Between Selection, Generation, and Logical Paradigms

The empirical results reveal several critical insights regarding current LLM capabilities. In Stage II hypothesis selection (Table 3), state-of-the-art models perform near saturation on short commonsense bridging tasks, with GPT-5.4 achieving 87.2% on ART and 88.0% on e-CARE. Similarly, on closed-option clinical ranking in DDXPlus, GPT-5.4 reaches 79.75% Top-1 accuracy and 98.7% Hit@3. However, performance degrades sharply when moving to long-context narrative inference containing dispersed clues. On True Detective, the best model achieves only 42.9% accuracy compared to a human average of 47.0%, while on the MuSR murder mystery subset, GPT-4o leads with 68.0% accuracy, falling far short of the 92.1% human benchmark.

The contrast between Stage I generation and Stage II selection highlights a profound structural gap. This disparity is clearest in medical diagnosis on DDXPlus: while GPT-5.4 achieves 98.7% Hit@3 when selecting among provided diagnoses, its performance drops to 63.0% Hit@3 when required to generate the top three diagnoses open-endedly from scratch (Table 4). Similar variations appear across formal domain benchmarks; GPT-5.4 achieves near-perfect accuracy (99.6%) on the highly constrained AbductionRules dataset, but drops to 21.5% accuracy on ProofWriter, where recovering missing premises admits multiple plausible paths.

Macro-averaged performance across all evaluated models is summarized in Figure 4 for Stage I generation and Figure 5 for Stage II selection, illustrating the consistent lead of frontier proprietary models while highlighting steady scaling trends within open-weight families like Qwen and Llama (Figure 6).

By aggregating experimental data from core comparative studies including Sheng et al. (2025), Dougrez-Lewis et al. (2025), and Xu et al. (2025a), the authors analyze how abduction correlates with deduction and induction. As shown in the accuracy distribution box plots in Figure 7, deduction exhibits a high median accuracy of 79.96% with a concentrated interquartile range. In contrast, abduction displays a significantly lower median accuracy of 42.50% alongside a wide variance extending down to near-zero accuracy. Induction records the lowest median at 28.80%.

Figure 8 plots abductive accuracy directly against paired deductive and inductive accuracy under identical model and prompt configurations. The abduction-versus-deduction pairs (blue data points) show a heavy clustering in the region of high deductive accuracy (>60%) combined with low-to-moderate abductive accuracy (<50%). This asymmetry demonstrates that strong deductive proficiency does not imply strong abductive competence, proving empirically that abduction requires distinct cognitive mechanisms—such as creative hypothesis space generation and evaluative plausibility judgment—that are not exercised by deductive logic.

Categorizing the Literature Across Four Taxonomic Axes

To organize prior literature into a cohesive structure, the survey categorizes over 60 papers using its four-axis taxonomy, summarized comprehensively in Table 2. Across the Task Formulation axis, works are partitioned into discriminative hypothesis selection (scoring candidate options as in ART/αNLI or single-hypothesis plausibility judging as in CauseJudger), generative hypothesis creation (free-text narrative explanation as in UNcommonsense or formal rule recovery as in ProofWriter), and integrated full-pipeline systems that explicitly combine generation with evaluative ranking.

Regarding Dataset Type, the literature splits between implicit commonsense datasets grounded in daily physical and social scenarios (e.g., Sherlock for visual abduction) and expert or formal datasets requiring domain-specific knowledge and explicit logical constraints. Specialized domains include clinical diagnosis in MedCaseReasoning and ER-REASON, legal case analysis in L’ART, program synthesis on Mini-ARC, and knowledge graph path abduction. Methodologically, while prompt engineering (M1) and supervised fine-tuning (M2) remain dominant, emerging approaches incorporate knowledge-augmented retrieval (M3), multi-agent debate and role-playing (M4), and hybrid neuro-symbolic solvers (M5) that translate natural language hypotheses into symbolic forms for automated backward verification.

Critical Gaps: Static Benchmarks and Supervised Imitation

Synthesizing their empirical and taxonomic findings, the authors identify several fundamental flaws in current AI abduction research. First, the vast majority of benchmarks suffer from static, low-complexity design. By collapsing abductive inference into single-shot multiple-choice selection or static text completion, existing evaluations fail to reflect real-world problem-solving, where hypotheses must be iteratively refined as new evidence accumulates. Furthermore, domain coverage remains narrow, over-indexing on everyday commonsense stories while under-representing high-value applications in engineering troubleshooting, scientific discovery, and medical diagnosis.

Second, there is a profound disconnect between benchmark accuracy and genuine abductive reasoning. High accuracy on static benchmarks can easily reflect surface pattern matching or dataset biases rather than valid explanatory inference. This issue is compounded by the dominance of Supervised Fine-Tuning (SFT) and maximum-likelihood objectives. Training LMs to maximize token likelihood over reference explanations forces models to imitate specific phrasing rather than learning to evaluate explanatory virtues such as parsimony, coherence, and scope. Finally, the field lacks mechanistic interpretability; current research provides almost no insight into the internal transformer circuits, attention heads, or latent representations that differentiate abductive hypothesis generation from deductive verification.

Strategic Roadmap: Reinforcement Learning, Multi-Agent Systems, and Abductive Circuits

To address these limitations, the survey outlines four strategic research directions for the community. The first direction calls for Post-Training Reinforcement Learning for Explanatory Virtues. Rather than relying on likelihood imitation, post-training optimization should utilize RL algorithms—such as Group Relative Policy Optimization (GRPO) or PPO—coupled with process rewards that explicitly evaluate explanatory virtues. Rewards should combine hard structural checks (e.g., formal logic entailment, symbolic solver execution, constraint satisfaction) with soft qualitative criteria, including parsimony, coherence, diversity of the generated candidate set, and predictive utility under counterfactual scenarios, building on reasoning RL frameworks like DeepSeek-R1.

The second direction emphasizes Richer, Action-Oriented Benchmarks. Future evaluation suites must move beyond static one-shot dataset splits toward interactive environments. Abductive hypotheses should be evaluated by their utility in guiding downstream action—such as ordering the correct diagnostic test in medicine, requesting relevant evidence in legal discovery, or localizing a bug during software execution. Synthesizing complex abductive data from formal proof traces, code execution paths, and scientific simulation logs offers a scalable path toward generating structurally challenging, verifiable benchmarks.

The third direction calls for Generalized Multi-Agent Abductive Architectures. While multi-agent debate and role-playing have proven effective in narrow domains, there is a critical need for domain-agnostic multi-agent frameworks that explicitly decompose the two-stage abductive pipeline. Specialized agents can be assigned distinct roles—such as candidate hypothesis generators, critical evaluators checking for logical fallacies, domain-specific retrieval agents, and synthesis agents—interacting iteratively to propose, critique, and refine explanations.

The fourth direction focuses on Mechanistic Interpretability of Abductive Circuits. Utilizing techniques such as activation patching, causal tracing, and sparse probing, researchers must map the specific internal transformer components responsible for abductive sub-processes. Key goals include isolating the attention heads that activate during hypothesis proposal versus evaluation, identifying how competing candidate hypotheses are represented and compared in latent space, and determining whether abductive capability relies on dedicated neural circuits or represents a compositional reuse of deductive and inductive primitives. Targeted circuit interventions could ultimately enable fine-grained control over model explanatory preferences without requiring full parameter retraining.

TL;DR

WHAT was done? The authors introduce Skill Entropy, a directional metric quantifying the difficulty large language models experience when switching between distinct reasoning skills within a continuous reasoning chain. Using this metric, they construct Skill²-Bench, a benchmark spanning 558 skills across 9 verifiable and open-ended domains, and propose Skill-Entropy RL, a reinforcement learning framework that penalizes skill-switching misalignments while rewarding answer correctness.

WHY it matters? While modern frontier LLMs perform exceptionally well on isolated domain benchmarks, their accuracy degrades significantly when compelled to transition between different cognitive skills in multi-step tasks. Skill-Entropy RL demonstrates that explicitly supervising and rewarding structured skill transitions allows smaller open-source models (such as Qwen3-4B-Instruct) to close this cross-skill gap, outperforming standard outcome-based RL methods like GRPO and transferring effectively to off-the-shelf datasets like OpenR1-Math.

Executive summary: Current evaluations measure model competence in isolated silos such as code generation or symbolic math. However, real-world agentic execution requires fluidly chaining diverse skills within a single context. This paper identifies a fundamental failure mode in LLMs: models suffer a “cognitive inertia,” persisting with previous solution patterns rather than adapting to the skill required by the next step. By formalizing this switching friction as “Skill Entropy” and transforming it into an explicit reinforcement learning reward, the authors provide both an evaluation diagnostic and a post-training objective that significantly improves long-horizon, multi-domain reasoning performance.

Details

The Cross-Skill Transition Bottleneck in Long-Horizon Reasoning

Long-horizon autonomous execution demands that models execute complex chains of reasoning spanning diverse cognitive domains. A real-world agentic pipeline might require executing a mathematical calculation, translating that output into a logistical constraint, and ultimately incorporating those parameters into a qualitative textual report. While frontier systems demonstrate near-saturation performance on single-domain benchmarks like MMLU-Pro or LiveCodeBench, they exhibit brittle behavior when forced to execute multi-step chains requiring distinct reasoning modalities.

Previous evaluations fail to isolate why this breakdown occurs, treating long-horizon failure as a monolithic issue of context window degradation or cumulative error propagation. The central insight of this work is that domain competence in isolation does not imply smooth transition competence between domains. Models suffer from a specific form of cognitive inertia, carrying over the reasoning style, output format, and problem-solving strategy of step i−1 into step i, even when step i demands an entirely different skill set. This performance delta between isolated skill execution and cross-skill execution remains invisible to conventional benchmarks, creating a critical bottleneck for deploying agents in compositional, real-world workflows.

Directed Skill Entropy First Principles: Formalizing Transition Friction

To rigorously quantify the friction of switching cognitive contexts, the authors formalize the concept of pairwise directed Skill Entropy. Let S denote a finite set of fine-grained skills derived across domains D. Given a target model or reference model, let Accuracy(s_a​) represent the baseline accuracy when solving single-step tasks using skill s_a​∈S in isolation. Let Accuracy(s_a​,s_b​) represent the multi-step accuracy on a sequential task where step 1 demands skill s_a​ and step 2 demands skill s_b​.

The pairwise directional Skill Entropy SkE(s_a​,s_b​) is defined as a smoothed ratio comparing isolated skill competence against chained transition competence:

Here, α=0.1 is a Laplace smoothing parameter ensuring numerical stability. When SkE(s_a​,s_b​)≈1, chaining skill s_a​ to s_b​ adds negligible cognitive friction over their standalone baselines. Conversely, values significantly greater than 1 signal high transition difficulty. Crucially, this quantity is asymmetric: SkE(s_a,s_b)≠SkE(s_b,s_a), reflecting the directional asymmetry of cognitive switching (e.g., transitioning from symbolic code execution to natural language summarizing imposes different constraints than the inverse).

To evaluate an extended multi-step task τ with an associated skill sequence μ(τ)=(s₁​,s₂,…,s_L​) of length L∈[2,10], the overall task-level skill entropy SkE(τ) is computed by aggregating pairwise directional transition costs along the skill sequence trajectory:

To make pairwise matrix generation computationally tractable across ∣S∣=558 skills—which would otherwise require evaluating ∣S∣²≈3.1×10⁵ ordered pairs—the framework factorizes skill-to-skill entropy through domain-level transitions. Pairwise skill entropy is approximated as SkE(s_a​,s_b​)≈SkE(s_a​,d_{s_b}​​)⋅SkE(d_{s_a}​​,s_b​), where d_s​ represents the source domain of skill s. This reduces evaluation complexity from O(∣S∣²) down to O(∣S∣⋅∣D∣).

Chaining Heterogeneous Cognition: From Problem Definition to Execution

The Skill²-Bench dataset operationalizes these theoretical definitions into a benchmark suite spanning 558 skills across 9 distinct domains (Math, Science, Coding, Logic, Information Extraction, Planning, Creative Writing, Context Retrieval, and Instruction Following), as illustrated in Figure 1. Tasks are explicitly synthesized at controlled low, medium, and high task-level skill entropy thresholds, derived under a strong reference model (Claude-opus-4.7).

To understand how an input traverses a cross-skill long-horizon task, consider the case study featured in Figure 3. The system receives a unifying scenario σ: analyzing spatial patterns in a museum’s right-triangle-themed paintings. The task requires executing a two-step skill chain μ(τ)=(Geometric Calculation,Theme Creation). Step 1 prompts the model to compute the physical area of a specified painting. Step 2 requires using the contextual state and result from Step 1 to draft a qualitative exhibition theme.

When evaluated on standard base models, a severe failure mode occurs during Step 2. As visualized in Figure 3 and analyzed in Figure 5, the unaligned model suffers from modality persistence. Having executed a geometric formula in Step 1, the model emits a short numeric response for Step 2 (”Painting R”) rather than executing the requested creative text generation. The model fails to adjust its internal solution template to match the required skill shift.

As demonstrated in Figure 2, per-domain skill entropy is decoupled from single-domain accuracy; for instance, science tasks yield high single-skill accuracy yet exhibit severe skill-switching friction when combined with adjacent domains.

Optimization Strategy and Implementation Mechanics

To address this failure mode, the authors convert Skill Entropy from an offline evaluation diagnostic into an online reinforcement learning signal via Skill-Entropy RL. The core architecture relies on an explicit model commitment mechanism during rollout generation. Rather than directly generating solutions, the policy network is structured to emit explicit domain and skill tags prior to generating each step’s solution trace. The formatted rollout follows a structured template:

Optimization is conducted using Group Relative Policy Optimization (GRPO). Given a prompt and task τ, the policy emits a predicted skill sequence μ̑​(τ)=(s̑_1​,…,s̑_L​) alongside step answers (ȃ₁​,…,ȃ_L​). The scalar reward function r combines step-level correctness ransrans​ with a structural skill-entropy reward r_ent​:

r=λ_ans​r_ans​+λ_ent​r_ent​

The answer reward

computes the mean per-step accuracy across domain-specific verifiers (e.g., symbolic equivalence for mathematics, sandboxed unit test execution for coding, or constraint satisfaction for planning tasks). The skill-entropy reward rentrent​ assesses whether the model’s self-predicted skill trajectory reflects the actual cognitive difficulty of the task.

Defining

as the empirical CDF rank transformations of predicted and gold task-level skill entropies over the training set, the entropy reward is formulated as:

Using CDF rank-normalization ensures that r_ent​∈[0,1] remains scale-free across task difficulty distributions. To resolve lexical discrepancies when a model emits skill tags outside the canonical dictionary (e.g., predicting “integer_arithmetic” instead of “number_theory”), the training pipeline embeds generated tags using Qwen3-Embedding-0.6B and performs cosine similarity matching against the canonical skill bank. Emitted tags below a 0.5 cosine similarity threshold are categorized as out-of-bank and receive zero entropy reward.

Training hyperparameters are rigorously documented: RL policy training uses AdamW with an actor learning rate of 1×10⁻⁶, a fixed KL coefficient of 1×10⁻³, PPO clip ratio of 0.2, GRPO group size of 8, and a prompt batch size of 256. The reward weights are balanced at λ_ans​=0.7 and λ_ent​=0.3. Experiments are conducted on hardware consisting of an 8×H100 GPU cluster.

Empirical Validation: Uncovering and Closing the Transition Gap

The evaluation across 8 frontier models (including Claude-opus-4.7, GPT-5.5, and Gemini-3.1-pro) and 4 open-source baselines on Skill²-Bench confirms that task difficulty scales directly with task-level skill entropy (Table 2).

Across all models, accuracy decreases monotonically as task-level skill entropy transitions from low to high. Even high-performing frontier models show accuracy drops of −4% to −10% when a skill is embedded within a multi-step cross-skill task compared to its standalone single-skill performance.

As detailed in Table 3, applying Skill-Entropy RL to Qwen3-4B-Instruct achieves an overall Skill²-Bench score of 68.4%, significantly outperforming standard GRPO (58.8%), as well as skill-aware baselines such as STAT (61.4%), SkillRL (59.3%), and Skill-Distill (58.1%). Similar relative gains are recorded on Qwen3-1.7B, improving from a base performance of 14.6% up to 40.1%.

As shown in Table 14, the cross-skill performance of Skill-Entropy RL (68.4%) exceeds even the single-skill oracle score (62.2%) where steps are evaluated in isolation without surrounding scenario context.

Ablation experiments reported in Table 13 justify the choice of reward weights λ_ans=0.7,λ_ent​=0.3. Removing the skill-entropy reward entirely (λ_ent​=0.0, corresponding to standard GRPO) causes a 9.6 percentage point drop in performance. Conversely, over-indexing on skill entropy (λ_ent​=0.7) degrades performance to 48.6%, confirming that outcome correctness must remain the primary optimization target while skill entropy serves as a structural regularizer.

Furthermore, the authors demonstrate the reusability of this signal on off-the-shelf datasets. When applied to OpenR1-Math reasoning traces (by segmenting traces and labeling steps via an automated Qwen3-8B annotator), Skill-Entropy RL prevents training reward saturation (Figure 4) and achieves superior downstream generalization across six mathematical benchmarks (Table 11), outperforming standard GRPO by +1.9% on average.

Placing Skill Entropy in the Post-Training Landscape

This work bridges three distinct subfields in post-training research: skill-aware data composition, agentic long-horizon evaluation, and dense reinforcement learning supervision.

Prior skill-aware frameworks—such as STAT, Instruct-SkillMix, and Symbolic MoE—leveraged skill tags primarily at the data filtering, routing, or prompt synthesis layers. Similarly, agentic benchmarks like GAIA and AgentBench measured multi-step performance, but conflated environmental tool interaction with underlying cognitive switching capabilities. Meanwhile, dense reward methods like Process Reward Models (PRMs) and self-revision rewards (DeepSeek-R1) focused heavily on densifying feedback within a single reasoning step.

Skill-Entropy RL differentiates itself by targeting transition difficulty between steps. Rather than altering routing or context construction, it injects structural transition difficulty directly into the policy optimization objective. This approach acts orthogonally to PRMs: while PRMs verify intra-step mathematical or logical correctness, Skill Entropy enforces inter-step structural adaptability.

Critical Assessment and Structural Limitations

Despite its strong empirical performance, the proposed methodology presents several trade-offs that warrant critical examination:

1. Reference Model Dependence: The derivation of pairwise Skill Entropy SkE(s_a​,s_b​) relies on offline execution logs from a fixed reference model (Claude-opus-4.7). Although the authors perform sensitivity analyses showing partition overlap remains above 80% when swapping reference models to Gemini-3.1-pro or GPT-5.5 (Table 7), the absolute entropy values remain inherently bounded by the capabilities of the chosen reference architecture.

2. Computational Overhead of Skill Factorization: Computing matrix estimates across fine-grained skills requires thousands of Monte Carlo sampling passes under strict temperature constraints. Although the domain factorization trick reduces complexity from O(∣S∣²) to O(∣S∣⋅∣D∣), scaling this approach to dynamic or open-ended skill libraries during continuous pre-training remains computationally expensive.

3. Open-Ended Evaluation Variance: Evaluating open-ended domains (Creative Writing, Context Retrieval, Instruction Following) relies on an LLM judge (Claude-opus-4.7). While the authors validate judge alignment against human annotators (Pearson r=0.88, Table 10), automated evaluation of qualitative outputs introduces potential judge biases that outcome-verifiable domains (such as sandboxed code execution) do not suffer from.

   ArXivIQ is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Strategic Takeaways and Future Trajectory

The formulation of Skill Entropy provides a valuable conceptual framework for understanding multi-step LLM failures. It demonstrates that long-horizon degradation is not merely a function of sequence length or memory loss, but is driven by transition friction between distinct cognitive domains.

For post-training research teams, the takeaways are immediate: outcome-only RL objectives (GRPO) leave substantial performance on the table by ignoring step-to-step structural transitions. Incorporating explicit skill-planning tags and rewarding task-level entropy alignment offers an efficient method to improve compositional reasoning without modifying data pipelines or inference architectures. Expanding Skill-Entropy supervision to tool-use environments, multi-modal task switching, and real-time agentic execution represents a promising direction for future research.

TL;DR

WHAT was done? The authors introduce a theoretical and empirical framework for applying spatial reflection interventions directly to intermediate representations within non-equivariant vision backbones, including convolutional U-Nets, Vision Transformers (ViTs), and Diffusion Transformers (DiTs). They formalize geometric consistency across coupled computational pathways—such as multi-head attention projections and U-Net skip connections—as the necessary condition for maintaining internal feature stability during hidden-state transformations.

WHY it matters? As generative AI moves toward direct feature-level editing, activation steering, and representation tuning, manipulating internal activations inside standard backbones frequently introduces unseen spatial coordinate mismatches. By demonstrating that internal transformations remain stable and act as capacity-reducing regularizers when interacting pathways stay aligned in a common coordinate frame, this work provides clear structural guidelines for performing safe feature-space augmentations and modular edits without requiring full architectural redesigns or custom group-equivariant networks.

Executive summary: Modern machine learning models are increasingly edited, steered, or augmented at the level of internal activations rather than input images. However, modifying a hidden layer inside standard architectures often breaks implicit spatial assumptions. This paper reveals that internal geometric manipulations—such as flipping intermediate feature maps—do not inherently disrupt model computation. Instead, failure occurs specifically when transformations are applied partially across coupled branches, creating coordinate frame mismatches that compound across iterative sampling steps in diffusion models. By enforcing geometric consistency across all interacting pathways, researchers can safely perform feature-space interventions that improve feature stability and generalization across U-Net, ViT, and DiT architectures.

Details

The Hidden Spatial Inconsistency in Neural Representation Steering

Modern vision and generative models rely on deep feature hierarchies where intermediate hidden states encode rich spatial geometry. While prior literature has extensively explored input-level data augmentations and post-hoc model editing techniques, directly applying geometric transformations to intermediate activation tensors inside pretrained backbones remains poorly understood. When an internal representation is altered, downstream layers expect a coherent coordinate frame to process spatial relationships correctly. Standard architectures like convolutional U-Nets or Diffusion Transformers (DiTs) are not inherently group-equivariant; hence, unaligned internal manipulations risk injecting catastrophic spatial mismatch into downstream operations. The core bottleneck investigated in this paper is identifying what structural conditions allow an internal geometric intervention to preserve coherent computation across multi-head attention blocks and encoder-decoder skip connections.

Existing approaches to spatial invariance typically enforce strict architectural equivariance by redesigning layers from scratch, which prevents their application to standard pretrained weights. Conversely, post-hoc editing frameworks often modify intermediate activations without accounting for multi-branch coordinate alignment. The critical delta introduced by this work is the principle of geometric consistency: rather than modifying the underlying neural architecture, the paper demonstrates that standard backbones can accommodate direct hidden-state transformations during fine-tuning provided that all interacting computational pathways operating on the transformed feature map are transformed in unison. By contrasting geometrically consistent interventions against inconsistent partial transformations across baseline architectures such as Proszewska et al. (2025) and Yang et al. (2024), the authors establish a foundational framework for hidden-state geometric regularization.

Geometrically Inconsistent (Output Flip Only):
Head 1 (Standard) ---> [O_1] ----------------┐
Head 2 (Flipped)  ---> [O_2] ---> [Flip] ---> Concat ---> Project [W^O] ---> [Distorted Frame]

Geometrically Consistent (Coherent Transformation):
Head 1 (Standard) ---> [O_1] --┐
Head 2 (Standard) ---> [O_2] --┴-> Concat --> [Flip All] --> Project [W^O] --> [Aligned Frame]

Geometric Consistency First Principles: Coordinate Frames across Interacting Pathways

To formalize hidden-state interventions, let z^(ℓ) denote the intermediate activation tensor at layer or block ℓ. The theoretical substrate of this work considers spatial transformations drawn from the reflection subset of the dihedral group D₄​, defined as T={T_hor​,T_ver​,T_diag​,T_anti​}⊂D₄​, representing horizontal, vertical, main-diagonal, and anti-diagonal flips. For a 2D feature map or patch grid arranged on a P×P spatial lattice, a transformation τ∈T remaps token indices via a permutation matrix Π_T​. The state space is thus transformed from F^(ℓ) to F̃^(ℓ)=τ(F^(ℓ)).

The core theoretical assumption asserts that an intervention is geometrically consistent if and only if every interacting branch consuming F̃^(ℓ) operates within the identical spatial frame. In multi-head self-attention, input tokens X∈R^N×d_model​ with N=P² generate query, key, and value matrices Q_h​,K_h​,V_h​∈R^N×d_v​ for head h. A single head’s attention map A(Q,K,V)=softmax(QK^⊤/sqrt(d_k​)​)V satisfies permutation equivariance under simultaneous transformation A(Π_T​Q,Π_T​K,Π_T​V)=Π_T​A(Q,K,V). However, when multi-head outputs are combined via projection matrix W^O=[W₁^O​…W_H^O​], consistency requires that every interacting head shares the permutation Π_T​.

In convolutional U-Nets, an analogous principle governs skip-connected encoder-decoder fusion. Let F_enc^(ℓ)​∈R^{C_enc​×H×W} denote the encoder feature map and F_dec^(ℓ′)∈R^C_dec×H×W denote the corresponding decoder feature map at matched spatial resolution. If ϕ represents a skip-fusion operator, such as concatenation followed by convolution, consistency requires transforming both branches simultaneously so that G^(ℓ,ℓ′)=ϕ(τ(F_dec^(ℓ′)​),τ(F_enc^(ℓ)​))=Π_T​ϕ(F_dec^(ℓ′)​,F_enc^(ℓ)​). Applying the transformation to only one branch breaks this relation, generating spatial coordinate mismatch at fusion. From a statistical perspective, constraining hypothesis class H to a symmetry-consistent subclass H_sym​={h∈H:h(x)=h(T(x))} yields empirical Rademacher complexity Ȓ_S(H_sym)≤Ȓ_S(H), establishing geometric consistency as an idealized form of capacity-reducing regularization.

From Patch Grid to Attention Projection: Mechanistic Flow of Flipped Heads

To trace the precise mechanistic flow of a geometric intervention, consider a single input tensor passing through a Vision Transformer block, such as ViT-B/16. On a patch grid of dimensions P×P, token index t(i,j)=(i−1)P+j maps spatial coordinate (i,j) to a flat sequence index. Standard self-attention projects tokens into head outputs O_h∈R^N×d_v​. Under a targeted flipped-head intervention on head h^∗, the output matrix O_h^∗​ is reshaped into grid tensor O_h^∗​∈R^P×P×d_v​, transformed via horizontal flip (T_hor​O_h^∗​)[i,j,:]=O_h^∗​[i,P−j+1,:], and flattened back into token format Õ_h^∗​.

When this transformed head is combined with untransformed heads in an attention-inconsistent module, the final token representation at spatial location (i,j) becomes

This operation injects mirrored contextual information from head h^∗ while keeping all remaining heads anchored at the original spatial location.

As visualized in Figure 5, an output-only flip creates significant coordinate distortion heatmaps ∣A_incon​−A_con​∣, whereas a fully consistent QKV transformation preserves or correctly mirrors spatial geometry across the attention field.

Input Latent (x_t) 
       │
       ▼
[Encoder Blocks] ─── F_enc^(l) ───┐ (Consistent: Apply τ to both)
       │                          ├───> Fusion φ(τ(F_dec), τ(F_enc)) ───> [Decoder]
[Decoder Blocks] ─── F_dec^(l') ──┘

When propagated through deeper layers or unrolled across T iterative denoising steps in diffusion models, these per-step attention perturbations accumulate linearly. Linearizing the denoiser update z_t−1​=Ψ_t​(z_t​,ε̑_t​) shows trajectory deviation accumulating as Δ_T​≈∑_t=1:T​D_t​J_t​E_t​, where J_t​=∂ε̑_t​/∂O_t​ is the Jacobian of the noise predictor and D_t​ measures sampler sensitivity. In single-pass architectures like ViT, spatial routing adjusts smoothly to mirror target regions without losing semantic focus, as shown in Figure 4. However, in iterative Diffusion Transformers, partial mismatches compound into severe global displacement and feature drift.

Optimization, Loss Formulation, and the 50% Stochastic Intervention Schedule

Implementing hidden-state geometric interventions requires minimal modifications to standard optimization pipelines. The training framework operates on noisy latents x_t​ at diffusion step t with injected noise ε∼N(0,I). For a chosen intermediate block ℓ sampled uniformly from available target candidates B^∗∼Uniform(C) and a transformation τ∼T, the modified activation F̃^(ℓ)(x_t​,t)=τ(F^(ℓ)(x_t​,t)) is propagated through the network to produce output ε_θ^(τ,ℓ)​(x_t​,t). The primary loss function minimizes noise-prediction error under hidden-state augmentation:

In the main Stable Diffusion 2.1 U-Net testbed, fine-tuning is conducted using the AdamW optimizer with a learning rate of 10⁻⁵, batch size 4, and a standard DDPM scheduler over 5,000 steps on 4×64×64 VAE latents derived from 500 Oxford-IIIT Pet images resized to 512×512. Candidate transformation locations C encompass ResNet blocks (C_res​), attention blocks (C_attn​), and skip-fusion junctions (C_skip​).

A key practical finding is the effectiveness of a 50% stochastic intervention schedule. Applying the hidden-state transformation to every mini-batch risks over-regularizing the backbone, whereas applying it too infrequently leaves representations unconstrained. The 50% consistent schedule randomly interleaves intervention mini-batches with standard clean baseline updates, striking an optimal balance between maintaining functional denoising accuracy and enforcing geometric stability.

Quantitative Validation: Diagnostics, Ablations, and Internal Drift Probing

To measure internal representation stability independently from final image output quality, the authors introduce three activation-level diagnostic metrics computed after realigning the intervened activation back to the clean frame via A_align=T⁻¹(Ã). Self-Consistency Shift (SCS) measures feature-level L1 distance

Activation Mass Scatter (AMS) measures spatial dispersion around the normalized spatial center of mass μ_b​ via

Drift captures global spatial displacement between clean and aligned mass centers

Denoising quality is tracked simultaneously via Noise-Prediction MSE

As illustrated across the time-series evaluations in Figure 1, inconsistent intervention modes (attention-inconsistent and inconsistent-skip) suffer severe stability degradation: SCS spikes above 50 and AMS exceeds 35, accompanied by increased NP-MSE noise-prediction error. In contrast, 50% consistent variants maintain lower feature shift, reduced spatial scatter, and minimal spatial drift while achieving superior noise prediction.

Image-space evaluations across 105,000 generated samples presented in Figure 2 demonstrate that 50% consistent modes maintain competitive FID, KID, CLIP text-alignment scores, and LPIPS diversity compared to the baseline, confirming that internal stability gains do not sacrifice visual quality.

Qualitative uncurated samples across all seven modes are illustrated in Figure 6.

To explain why attention fields exhibit variable stability under reflection, the authors conduct probing analyses on synthetic 4×4 and 8×8 grid patterns using probe consistency score

As shown in Table 3 and Figure 3, ViT Block 5 attention fields yield moderate consistency on directional patterns like horizontal stripes (0.523–0.581) and diagonal gradients (0.624–0.650), but achieve perfect consistency (1.000) on isotropic Gaussian blobs, revealing inherent geometric anisotropy in attention layers.

Complementary proof-of-concept FID experiments summarized in Table 1 highlight performance gains when applying 50% consistent regularization across standard benchmarks. On CIFAR-10 (32×32 DDPM baseline from Proszewska et al. (2025)), regularized training reduces FID from 4.46 to 4.30±0.05. On CelebA-64 DDPM, FID reaches 4.53±0.33 compared to the 4.51 baseline. On MNIST using the spiking transformer backbone SDiT (Yang et al. (2024)), FID improves from 5.54 to 5.29±0.08.

Furthermore, as detailed in Table 2, fine-tuning a pretrained ViT-B/16 on CIFAR-100 with input-or-hidden (I/H) reflection augmentation improves top-1 classification accuracy from 92.30% (standard 224×224 input augmentation) to 93.19%.

Contextualizing Geometric Interventions in Modern Vision Backbones

The findings in this paper build upon three major lines of deep learning research: spatial data augmentation, group equivariant network design, and post-hoc hidden-state editing. Traditional data augmentation techniques, such as those surveyed by Shorten & Khoshgoftaar (2019) or feature-space block masking like PatchUp (Faramarzi et al., 2022), operate primarily at input layers or apply unconstrained noise to hidden states. While effective for classification, unconstrained feature noise lacks the structural alignment required to preserve fine spatial details in generative models.

In terms of architecture, group equivariant neural networks (Cohen & Welling, 2016; Finzi et al., 2021) enforce exact symmetry constraints by replacing standard convolutions and attention layers with group-steerable operators. While mathematically elegant, steerable backbones cannot leverage massive pretrained weights from standard non-equivariant architectures like Stable Diffusion or DiT. On the other hand, hidden-state editing techniques (Meng et al., 2022; Hertz et al., 2022) alter intermediate activations to steer text-to-image synthesis or factual knowledge, but rarely consider whether multi-head attention projections or skip connections remain aligned in a shared spatial frame. This paper directly bridges these domains by showing that standard, non-equivariant models can accommodate internal geometric change during fine-tuning without architectural modifications, provided that coupled computational paths remain geometrically consistent.

Scope Constraints: Reflection Subsets and Latent Inconsistencies

Despite its strong theoretical and empirical insights, several limitations should be noted. First, the implemented transformation set is strictly restricted to the reflection subset T={T_hor​,T_ver​,T_diag​,T_anti​} of the dihedral group D₄​. Rotations were explicitly excluded from the main diffusion experiments because VAE latent representations in Stable Diffusion are not rotation-consistent; rotating latent grids introduces severe reconstruction artifacts when passed through pretrained VAE decoders. Extending geometric consistency to full continuous rotation groups or affine transformations will require specialized alignment mechanisms.

Second, the main quantitative U-Net evaluation relies on a lightweight subset of 500 images from the Oxford-IIIT Pet dataset. While this setup allowed rigorous evaluation across seven intervention modes, three random seeds, and 105,000 generated images, testing on larger-scale datasets like ImageNet-1k or LAION remains an important next step. Third, the Rademacher complexity analysis provides an idealized, capacity-reducing regularization argument for function classes rather than a strict quantitative generalization bound for non-linear, multi-step diffusion sampling pipelines.

Strategic Assessment and Architectural Implications

This work establishes geometric consistency as a foundational design principle for feature-space interventions in spatially structured neural networks. The central insight—that internal manipulations fail due to multi-branch coordinate frame mismatch rather than the transformation itself—has significant implications for parameter-efficient fine-tuning (PEFT) methods like LoRA, activation steering, and modular editing. When designing internal adapters or editing modules for multi-head attention and U-Net backbones, enforcing coherent spatial coordinate alignment across all interacting pathways prevents catastrophic feature drift and trajectory error accumulation.

For researchers working on vision and generative backbones, incorporating stochastic 50% geometrically consistent hidden-state augmentations during fine-tuning offers a lightweight, parameter-free regularizer that improves feature stability and generalization without requiring costly architectural redesigns. Future work should expand this framework to continuous spatial transformations, explore rotation-consistent latent spaces, and integrate geometric alignment constraints into real-time activation steering protocols.

TL;DR

WHAT was done? The authors introduce CoMem (Comprehension Memory), a depth-partitioned context memory architecture that reorganizes long-context processing along the layer axis instead of the token axis. During a streaming WRITE pass, context chunks pass through lower Transformer layers [0:j] to save a single intermediate per-token residual state, requiring only 1/18× the memory footprint of full key-value caches on Qwen3-8B. During a query READ pass, an external BM25 retriever selects a bounded pack of relevant chunks, and upper layers [j:L] are recomputed over the query-conditioned pack with full cross-chunk causal attention. A rank-32 self-distillation LoRA adapter trained on plain text repairs upper-layer readout fidelity without modifying backbone parameters.

WHY it matters? Standard long-context inference is constrained by quadratic prefill compute and linear key-value (KV) cache expansion. Existing token-eviction techniques truncate long-range dependencies, while full-recomputation approaches scale linearly in memory working set size. By exploiting the layer-wise division of labor—where semantic understanding stabilizes early and upper layers specialize in prediction—CoMem decouples online read compute and accelerator memory from total stored context length. At 128k context on an NVIDIA H20 GPU, CoMem delivers a 7.83× prefill speedup and reduces peak memory from 89.36 GB to 18.26 GB while outperforming full-context references on long-dialogue benchmarks.

Executive Summary: For AI system architects and research leaders, CoMem establishes that long-context serving does not require maintaining full-depth KV caches across millions of tokens in GPU memory. By precomputing intermediate residual vectors offline and executing online queries over a retrieved, fixed-size pack through upper Transformer layers, serving costs remain strictly bounded. This architecture enables infinite-context memory for dialogue agents and document retrieval platforms with sub-second prefill latency and minimal VRAM consumption.

Details

The Token-Axis Memory Wall and Layer-Wise Functional Shift

Long-context language modeling faces severe operational bottlenecks caused by O(N²) prefill attention complexity and linear O(N⋅L⋅d_kv​) key-value cache memory growth, where N is sequence length, L is layer count, and d_kv​ is per-layer KV projection dimension. Prior efficiency paradigms bound memory along the token axis by evicting low-attention tokens, as seen in StreamingLLM, H2O, and SnapKV, or by compressing neighboring keys in architectures like MiniCache. Alternatively, recomputation frameworks such as KV-Direct demonstrate that KV states can be reconstructed from residual activations, yet their online memory footprint still grows linearly with context length because every historical token must be retained.

CoMem addresses this structural conflict by shifting memory organization from the token axis to the depth axis. The foundational insight stems from layer-wise functional probing, which reveals that Transformer models exhibit an internal division of labor. Semantic information becomes broadly accessible near mid-depth, whereas upper layers specialize representations for query conditioning and next-token prediction (see also this paper with similar findings). By leveraging this asymmetry, CoMem precomputes query-independent semantic states in lower layers and delays upper-layer computation until a query arrives, creating a clear architectural boundary between offline context ingestion and online query execution.

Read more

TL;DR

WHAT was done? The authors introduce GenCeption, a unified framework that repurposes a pre-trained text-to-video generative diffusion backbone into a single-step, feed-forward generalist vision model. By mapping heterogeneous dense tasks (such as depth, normal estimation, and camera pose) into a unified 3-channel RGB representation space and utilizing learnable queries for sparse tasks (like 3D keypoints), GenCeption performs multi-task video perception steered entirely by text instructions.

WHY it matters? This work demonstrates that large-scale video generative pre-training acts as a powerful “world model” that implicitly internalizes 4D spatiotemporal dynamics, geometry, and physical laws. Rather than relying on slow, iterative denoising, GenCeption extracts these rich representations in a single feed-forward pass, matching or exceeding task-specific state-of-the-art models with up to 500× less training data and showing exceptional zero-shot generalization to out-of-distribution categories.

Details

The Balkanization of Computer Vision and the Quest for a Unified Objective

Computer vision remains heavily balkanized, characterized by task-specific architectures and custom heads engineered for isolated objectives, such as Sapiens for human pose estimation or Depth Anything 3 for geometry. This stands in stark contrast to natural language processing, which collapsed task boundaries by adopting next-token prediction over unified architectures. To achieve a similar catalyst in vision, we need a pre-training objective that forces models to internalize spatiotemporal evolution, physical causality, and native language alignment. While previous visual representation learning paradigms like VideoMAE V2 or V-JEPA ^{[see also]} capture spatial correlations, they lack native language concept grounding and the capacity to model complex, continuous temporal physics. The authors address this fundamental bottleneck by demonstrating that large-scale text-to-video generation is the true visual analog to next-token prediction, serving as a superior foundation for general-purpose visual intelligence.

Read more

TL;DR

WHAT was done? The paper introduces a non-factorial sparse coding framework that serves as a mathematically transparent, minimal diffusion model. By extending standard sparse coding with an unconstrained pairwise latent interaction matrix M and optimizing its recurrent inference dynamics via Denoising Score Matching and the Implicit Function Theorem, the model achieves generative parity with standard U-Nets while enabling exact analytical decomposition of its denoising Jacobian.

WHY it matters? This work bridges visual computational neuroscience and deep generative modeling by showing that primary visual cortex (V1) recurrent dynamics and artificial diffusion models solve the exact same geometric density estimation problem. It provides an interpretable blueprint to reverse-engineer how neural circuits generate geometry-adaptive harmonic representations, perform contour completion, and enforce global semantic consistency through emergent latent hierarchies.

Details

The Interpretability Paradox in Generative Score Matching

Deep score-based generative architectures like U-Nets excel at synthesizing complex image distributions, yet their internal mechanisms remain opaque. Conversely, classical sparse coding models like Olshausen and Field (1996) offer biological interpretability by recovering oriented Gabor-like receptive fields, but rely on restrictive, factorial priors that assume statistical independence between latent variables (∥z∥₁​). This independence assumption prevents classical sparse coding from capturing higher-order structural dependencies such as collinear contour continuity, spatial co-alignment, or global semantic constraints across scales. Existing non-factorial or hierarchical variants often introduce complex sampling steps or hand-crafted priors that fail to scale. The central objective of this work is to resolve this dilemma by augmenting standard sparse coding with a learned, unconstrained pairwise latent interaction matrix, optimized via score matching to deliver a unified, mathematically transparent model of visual inference that bridges primary visual cortex (V1) mechanics and modern diffusion models.

Non-Factorial Sparse Coding First Principles: Pairwise Latent Energy

The theoretical foundation relies on defining a continuous energy function E_θ,σ​(x,z) over ambient pixel space x∈R^D and latent neural space z∈R^K. Here, x=y+σϵ represents a corrupted observation generated by adding isotropic Gaussian noise ϵ∼N(0,I) with standard deviation σ to a clean image y. The dictionary Φ∈R^D×K contains spatial feature basis functions whose columns correspond to visual receptive fields. Crucially, latent co-occurrence statistics are captured by an unconstrained, pairwise interaction matrix M∈R^K×K, extending the classical L₁​ sparsity penalty into a non-factorial prior. The total energy is formulated as:

where λ∈R^K denotes a vector of learned, element-wise sparsity parameters, ∘ represents the Hadamard element-wise product, and γ(σ) is a scalar function parameterized by a two-layer multi-layer perceptron with weights ψ that dynamically balances the data likelihood against the non-factorial prior as noise varies. The total set of trainable parameters is defined as θ={Φ,M,λ,ψ}. Under a Maximum A Posteriori (MAP) approximation, assuming the posterior p_θ,σ​(z∣x) is sharply peaked around its mode z^∗(x;σ,θ)=argmin_z​E_θ,σ​(x,z), the marginal score function simplifies to a closed-form linear synthesis:

which converts score matching directly into a weighted reconstruction objective over the optimal latent fixed point.

MAP-Induced Recurrent Inference and Dynamic Activity Propagation

To compute the convergent MAP state z^∗, the model unrolls its inference as a recurrent neural circuit using the Iterative Shrinkage-Thresholding Algorithm (ISTA). Starting from an initial state z₀​=0, the neural activity at step t+1 updates according to:

where η>0 represents the integration step size. This update rule neatly partitions neural dynamics into three distinct functional drives: a feedforward drive Φ^⊤x projecting sensory inputs onto dictionary features, a recurrent drive (Φ^⊤Φ+σ²γ(σ)M)z_t​ that combines likelihood-based Gram matrix cross-talk with noise-weighted prior interactions, and a dynamic sparsity threshold γ(σ)λ.

To illustrate this mechanism with a concrete running example, consider an ambiguous input image containing disconnected Gabor segments forming a hidden contour, as shown in Figure 1A. When passed into the network, feedforward signals initially activate a subset of Gabor-tuned latents. As iterations proceed, lateral signals propagate through the interaction matrix M.

As visualized in Figure 2B, M naturally learns collinear facilitation—developing strong excitatory connections between spatially offset neurons with co-aligned orientations while enforcing self-inhibition. As activity evolves, the gating matrix Σ=diag(1_z∗>0​) acts as a dynamic mask that suppresses off-axis noise while permitting excitatory signals to propagate exclusively along the aligned contour path (Figure 3D). This recurrent spreading transforms disconnected input elements into a smooth, continuous output contour (Figure 1B and Figure 3B).

Implicit Optimization and Unrolled Phantom Gradients

Training the recurrent circuit end-to-end via standard Backpropagation Through Time (BPTT) would impose prohibitive memory demands due to the long iterative trajectories required for fixed-point convergence. The authors circumvent this by employing the Implicit Function Theorem (IFT). Defining the fixed-point residual F(z^∗;x,σ,θ)=z^∗−T(z^∗;σ)=0, the analytical implicit gradient of the convergent state with respect to parameters is given by:

To compute this efficiently without explicitly inverting high-dimensional Jacobians, the implementation utilizes an Unrolling-based Phantom Gradient (UPG) as established by Geng et al. (2021). After solving for z^∗ without tracking gradients, the dynamics are unrolled for m∈{1,2,3} additional steps (z_m​=T^m(z^∗)) with autograd enabled. Backpropagating through these steps naturally evaluates the truncated Neumann series expansion ∑_k=0:m​J^k, approximating the exact implicit gradient. The parameters are optimized using Denoising Score Matching (DSM) under an Elucidating Diffusion Models (EDM) [Karras et al., 2022] schedule:

where the noise scale is sampled log-normally as ln(σ)∼N(P_mean​,P²_std​), and

balances gradient magnitudes across noise levels. Optimization stability is reinforced by maintaining an Exponential Moving Average (EMA) of network weights with a dynamic decay rate:

For high-dimensional datasets, spatial dynamics are implemented using scale-specific convolutional dictionaries Φ⁽ⁱ⁾ over a Laplacian pyramid operator P, organized via an asymmetric Gauss-Seidel sweeping schedule across scales (Figure S1).

Analytical Neumann Decomposition and Emergent Latent Hierarchy

Because the denoiser operates as a linear projection of the converged latent code f_θ(x)=Φz^∗, its pixel-space Jacobian

​ can be analytically decomposed as J(x)=ΦJ_z​Φ^⊤. Expanding J_z​ into an infinite Neumann series yields J_z,i​=e_i​+ηΣWe_i​+η²ΣWΣWe_i​+… for active units (z_i^∗​>0), where W=Φ^⊤Φ+σ²γ(σ)M. As demonstrated in Figure 3A and Figure 3C, this formulation reveals how local lateral spreading (ηΣW) constrained by active sparse gating (Σ) constructs geometry-adaptive harmonic basis functions that bridge gaps in fragmented inputs.

When evaluated on complex distributions like the 64×64 CelebA face dataset, the model exhibits remarkable semantic generalization (Figure 4A-B). Probing the learned dictionary Φ uncovers a structural bifurcation: approximately 30% of the neural population learns near-zero column norms (Figure 4E). Reconstructing images solely from these “detached neurons” produces no visible spatial features (Figure 4F). However, ablating their recurrent connections causes a catastrophic drop in denoising PSNR at high noise levels (SNR ≪0), reducing performance by over 10 dB (Figure 4G). These detached neurons function as an emergent, higher-level latent layer in a single-layer circuit, leveraging horizontal connections M to coordinate long-range semantic dependencies—such as synchronously shifting both eyes—while keeping lower-level features globally consistent. Furthermore, performance benchmarks confirm that the model achieves generative parity with a parameter-matched black-box U-Net, yielding identical PSNR trajectories across noise scales from -10 dB to 30 dB and generating visually indistinguishable reverse diffusion samples (Figure 6A-C).

Synthesis with Classical Visual Neuroscience and Generative AI

This work builds directly upon classical sparse coding models of primary visual cortex pioneered by Olshausen and Field (1996) and extensions exploring non-factorial priors [Garrigues and Olshausen, 2007]. However, while prior non-factorial models relied on hand-crafted group priors or complex sampling algorithms, this framework integrates modern score-matching paradigms [Ho et al., 2020] and preconditioned diffusion objectives [Karras et al., 2022]. By framing recurrent ISTA inference through the lens of implicit deep learning and Deep Equilibrium Models [Bai et al., 2019], it establishes an explicit bridge to recent mechanistic studies of diffusion generalization [Kadkhodaie et al., 2023].

Theoretical Approximations and Expressivity Bottlenecks

Despite its analytical clarity, the framework relies on several simplifying assumptions that introduce potential failure modes. The primary theoretical limitation is the MAP approximation used to evaluate the score function, which assumes the posterior distribution p_θ,σ​(z∣x) is tightly concentrated around a single mode. In regimes of extreme visual noise or ambiguity where the true posterior is strongly multimodal, this assumption degrades, leading to potential misestimation of the score field. Additionally, while the pairwise quadratic interaction energy z^⊤Mz successfully captures second-order co-occurrence statistics, it lacks the expressive capacity to represent higher-order, non-linear feature interactions without relying on deep hierarchical stacking. Finally, unrolling fixed-point dynamics during every step of reverse-diffusion sampling incurs non-trivial iterative computational overhead compared to single-pass feedforward networks.

Strategic Implications for Neuroscience and Interpretability

By proving that a biologically plausible, single-layer recurrent circuit can match the generative performance of deep U-Nets, this paper provides a compelling unifying framework for computational neuroscience and machine learning. Beyond opening a transparent window into diffusion model generalization, the model offers concrete, testable neurophysiological predictions (Figure 5A-B).

Specifically, it predicts that under low-contrast or high-noise conditions, the scaling factor σ²γ(σ) causes biological V1 dynamics to shift from feedforward likelihood dominance to prior-dominated collinear facilitation, dynamically routing surround excitation along contextually active neural paths. This work provides a compelling demonstration of mechanistic interpretability, proving that complex deep learning capabilities can be grounded in interpretable, first-principles neural circuits.

TL;DR

WHAT was done? The authors introduce Explorative Modeling (XMs), a generative pretraining paradigm that shifts factorization from the generation process to the training loop. By generating K candidate outputs (or evaluating against K data samples) per training step and backpropagating gradients exclusively through the best match, XMs prevent mode blurring, increase generative expressivity, and enable single-pass end-to-end generation.

WHY it matters? Traditional reconstructive generative models rely on multi-step trajectory factorization during inference to handle multimodal data, introducing compounding errors and severe exposure bias. Explorative Modeling establishes generative expressivity as a missing third pretraining axis alongside parameter count and dataset size. It achieves a near-state-of-the-art 1.43 unguided FID on ImageNet 256x256 while improving FLOP efficiency by 4.1x, sample efficiency by 6.2x, and parameter efficiency by 47%. In robotics and world modeling, XMs match standard diffusion performance using 16x to 256x fewer inference steps.

Executive summary: Modern generative architectures solve multimodal probability modeling by breaking inference into hundreds of sequential steps. This multi-step sampling creates a fundamental train-inference mismatch and severe computational overhead at runtime. Explorative Modeling replaces multi-step generation with search inside the training loop. By allowing model predictions to commit to individual modes via a best-of-K candidate match during pretraining, inference can be collapsed to a single forward pass without sacrificing mode coverage. Crucially, the efficiency gains of exploration do not saturate—they expand as model scale and data volume grow, offering a new compute-optimal scaling dimension for foundation models.

Details

The Multimodal Bottleneck in Generation Factorization

The historical trajectory of deep learning is defined by replacing hand-crafted, multi-stage pipelines with unified end-to-end optimization. Image classification, object detection, and semantic segmentation all advanced when networks were trained to perform inference under the exact same conditions as training. Generative modeling, however, has remained a notable exception. Standard reconstructive approaches, including flow matching, diffusion models like SiT and DiT, and autoregressive sequence models, are not trained end-to-end. At inference, they operate as recurrent models unrolled over dozens or hundreds of steps. Each step introduces small prediction errors that accumulate over the generation trajectory, causing inputs to drift off the training distribution—a failure mode widely recognized as exposure bias.

This architectural compromise exists because generative modeling requires learning multimodal probability distributions. When a single latent or input context maps to multiple valid data targets across a dataset, standard single-step regression loss functions force the model to predict the expected value across those targets. In continuous spaces, this expected value lands in low-density regions off the data manifold, manifesting as blurry images or incoherent trajectories. To prevent this mode blurring, current scalable models factor the generation procedure into fine-grained sequence steps, ensuring that each individual step faces a nearly unimodal target conditional. While this generation factorization yields high sample quality, it locks models into multi-step inference and creates an insurmountable train-inference gap. Explorative Modeling re-examines this fundamental tradeoff by asking whether factorization can be moved out of the generation procedure and directly into the pretraining loop.

Read more

TL;DR

WHAT was done? The authors introduce Multi-Scale Path Divergence (MSPD, denoted as D_P​), a mathematically explicit, physics-grounded complexity metric inspired by the renormalization group. MSPD measures the temporal, multiscale organization of heterogeneity in the local transition laws of dynamical systems, functioning both as a gradient-free fitness function to direct open-ended evolution (OEE) and as a post-hoc diagnostic tool to analyze life-like behaviors.

WHY it matters? Open-ended evolution in Artificial Life (ALife) has traditionally relied on uninterpretable, black-box neural network drivers that lack connections to physical theories of complexity. By replacing learned embeddings with a principled, trajectory-level formula, MSPD bridges ALife with statistical mechanics, demonstrating that systems optimized for multiscale temporal structure naturally exhibit physical signatures of life—such as scale-dependent frustration and non-ergodicity—without being explicitly programmed to do so.

Details

The Uninterpretability Bottleneck in Artificial Life Evolution

A central goal of artificial life research is open-ended evolution, the sustained generation of novel forms and behaviors. While heuristic drivers such as novelty search and quality-diversity have successfully navigated search spaces, modern implementations have increasingly relied on black-box neural-network-based objectives, such as ASAL and Leniabreeder, which use deep vision-language models or learned autoencoder embeddings to assess “interestingness.” The critical bottleneck is that these models provide no physical insight, cannot be connected to statistical mechanics, and are highly sensitive to superficial visual changes rather than underlying system dynamics. Conversely, physics-grounded complexity metrics, such as Multi-Scale Structural Complexity (MSSC), are static and cannot capture the temporal evolution of dynamic processes. The primary contribution of this work is the temporal extension of spatial renormalization-group principles to evaluate local transition laws, providing a single explicit scalar that simultaneously optimizes for dynamical complexity and exposes its statistical-physical foundation.

Temporal Renormalization: Formulating State-Level Dynamical Heterogeneity

The mathematical formulation of MSPD begins by defining a configuration space. We let (I,B_I​) be a measurable index space for local degrees of freedom, such as a lattice site in a cellular automaton or an individual particle in a multi-agent system. A configuration is represented as a map x:I→E, where x(i) is the local state of degree of freedom i within the state space E. The system’s time-Δt dynamics are governed by a Markov kernel P_Δt​:X×B_X​→[0,1], which specifies the probability law of the subsequent configuration. For a current configuration x and a component i∈I, the map G_x,i​(y)=(x(i),y(i)) yields the before-after local state pair when the next configuration is y. The local transition law Tⁱ_Δt​(x) is then defined as the pushforward measure under G_x,i​: Tⁱ_Δt(x)=(G_x,i​)_#​P_Δt​(x,⋅)∈P(E×E) When E is a vector space, we can alternatively define local increment laws Vⁱ_Δt​(x) using the difference operator

To measure how much different parts of the system disagree in their local dynamics, the authors define state-level dynamical heterogeneity H_Δt​(x) as the double integral comparing the transition laws of two components sampled via m_x​:

where D represents a distance metric between probability measures, and m_x​ is a sampling measure encoding the contribution and weight of local degrees of freedom. This state-level heterogeneity serves as the atomic unit for constructing the temporal, multi-scale trajectory trace.

The MSPD Pipeline: From Lagrangian Tracing to Scale Sensitivity

To compute MSPD along a realized trajectory, the system utilizes a multi-step pipeline as shown in Figure 1.

As a running example, consider a continuous particle system such as Particle Life++ (neural-network-mediated pairwise particle interactions). First, the system tracks individual particles through time (Lagrangian tracking), outputting their spatial coordinates over a trajectory. Second, over a finite observation window I_k​=[a_k​,a_k​+s] of duration s, the local coordinates are logged at a temporal resolution or lag τ. The empirical transition law L̂^ξ_i,k,τ​ of particle i is calculated as a uniform distribution over the observed displacements:

where q_i​(t) represents the coordinates of particle i at time t, and I_k,τ​ is the set of valid start times within window I_k​. Third, the pairwise distance between the transition laws of all active particles is computed using the Sliced Wasserstein-1 (SW₁​) distance, yielding the empirical pairwise heterogeneity H̑_k​. To correct for finite-sample biases, a pooled-null baseline H̑_k⁰​ is subtracted, producing the null-corrected empirical heterogeneity ΔH̑_k​=H̑_k​−H̑_k⁰​. Fourth, the resulting trace is processed through a preprocessing function ϕ (such as clipping or flooring) to yield the windowed trace h^⋆_k,τ=ϕ(ΔH̑_k​). Finally, the trace is passed through a discrete temporal coarse-graining operator C_{r_j}​​ at multiple scales R={r₁​,…,r_J​}, measuring scale sensitivity by computing how much the trace changes under successive coarse-graining steps. This yields the final MSPD score, which closed-loop evolutionary optimizers use as a fitness function to discover complex, co-existing species (Figure 1).

Algorithmic Synthesis: Optimizing across Diverse Substrates

The primary objective function optimized during evolutionary search is the finite-grid MSPD functional:

where a=(a₁​,…,a_K​)∈R^K is the window-indexed heterogeneity trace, R={r₁​,…,r_J​} is the ordered scale grid with r_j⁺​=r_j+1​, C_{r_j}​​ is the discrete coarse-graining operator, U_{r_j}​​ is the reconstruction operator mapping the trace back to the grid of C_{r_j}​​, w_j​ are scale transition weights, and η_MSPD​ is a positive floor parameter that prevents division by zero and normalizes small residual fluctuations.

For continuous substrates like Particle Life++, the state updates are governed by the following equations:

where x_iⁿ​, v_iⁿ​, and c_iⁿ​ represent the position, velocity, and color vector of particle i at step n, Δt is the timestep, h is the damping half-life, m is the particle mass, F_iⁿ​ is the accumulated pairwise mechanical force, and ċ_iⁿ​ is the accumulated color drift rate.

To optimize these systems, the authors utilize Sep-CMA-ES, a covariance matrix adaptation evolution strategy. A major engineering challenge is that random neural parameters in Particle Life++ often settle into low-motion stationary basins, yielding narrow displacement laws and suppressing both heterogeneity and MSPD. To force the optimizer out of these trivial basins, the authors introduce a joint objective function minimizing

where λ_H​ provides a positive regularization weight on the mean-heterogeneity trace, ensuring the search algorithm prioritizes dynamic, moving systems.

Empirical Evidence: Separating Noise from Living Heterogeneity

To validate that MSPD specifically isolates structured temporal complexity from raw motion or static noise, the authors evaluate the metric against a controlled synthetic calibration suite consisting of families S0–S8 (Table 3, Figure 13) (but where’s S2?).

Crucially, S0 (static particles), S1 (homogeneous Brownian motion), and S3 (a single coherent moving blob) all receive near-zero MSPD scores (S1: 3.46×10⁻⁵, S3: 5.45×10⁻⁸), despite containing significant kinetic energy or spatial coordination. This confirms that coherent object motion is not mistaken for complexity. In contrast, family S8 (a blob splitting into distinct moving groups) and S6 (staggered switch times) produce the highest MSPD scores, demonstrating the metric’s selective sensitivity to temporally organized transitions and emergent roles (Figure 2, Figure 3).

In parameter optimization trials against matched random controls (Table 4), MSPD successfully identifies and isolates highly organized regimes. For Particle Life++, all 7 out of 7 optimization groups show a statistically confirmed increase in held-out MSPD over their matched controls (median contrast 0.00267, p=0.0078).

For Cellular Automata, the optimized totalistic rules are clearly separated from random controls (Figure 4).

While Flow-Lenia optimization yields positive contrast in 6 out of 9 groups (Figure 5), it is not statistically confirmed (p=0.254), a result the authors attribute to evaluation noise under a tight computational budget rather than a fundamental limitation of the metric.

Mechanistic probing reveals that MSPD successfully links to the physical properties of biological systems. State perturbation experiments (C2) show that locally elevated heterogeneity (ΔH) acts as a detector for dynamical bifurcation points. For Flow-Lenia, there is a robust positive correlation between present-time branch energy E_b​ and future divergence B_b​ under small perturbations (Pearson r=0.540, Spearman r=0.618 over n=135 states), confirming that high-heterogeneity states are highly sensitive to external interventions (Figure 9).

Furthermore, early spatial constraint experiments (C5) show that MSPD-optimized systems exhibit elevated scale-dependent frustration, with Flow-Lenia showing positive matched contrasts in 7 out of 9 groups (p=0.0898) and Particle Life++ showing positive contrasts in 6 out of 7 groups (p=0.0625, Table 5, Figure 10, Figure 11).

Deepening the Physical Foundations of Complexity

The MSPD framework stands on the shoulders of classical statistical physics and modern open-ended evolution drivers. It directly inherits its renormalization-group architecture from MSSC but addresses its static limitation by replacing spatial coarse-graining with temporal observation windows. Unlike neural-network-driven open-endedness drivers like ASAL or Leniabreeder, which use black-box perceptual metrics, MSPD uses a closed-form formula based on optimal transport theory and Sliced Wasserstein distances. This mathematical formulation allows MSPD to interface with physical theories of life, specifically the frustration/non-ergodicity criteria of biological complexity proposed by Vanchurin et al. and Wolf et al. (inspired by spin glasses). The paper’s key contribution to this literature is demonstrating that scale-dependent frustration—where a system’s dynamics are altered under spatial constraints because it cannot explore its rugged state-space landscape—emerges naturally as a downstream consequence of optimizing for multiscale temporal organization in continuous substrates like Flow-Lenia, Lenia, or Particle Lenia.

Structural Weaknesses: Stationarity Assumptions and Underpowered Trials

Despite its theoretical elegance, MSPD has several critical limitations. First, the consistency of the windowed estimator (Propositions 1 and 2) relies on the assumption of within-window stationarity of local transition laws. In regimes characterized by abrupt, highly localized phase transitions—such as the nucleation of a new structure or the sudden collapse of a basin—this stationarity assumption is violated, causing the empirical estimator H̑_k​ to mix pre- and post-event statistics, which introduces an unquantified bias that may attenuate optimization signals. Second, the continuous-parameter optimization experiments suffer from low statistical power due to a modest number of independent runs, meaning that fine quantitative claims regarding effect sizes and substrate-specific exponents remain underpowered. Finally, the cross-substrate evaluation matrix is incomplete (Table 6), leaving transfer experiments such as C5 on Cellular Automata and C2 on Particle Life++ to future work; the latter is particularly obstructed by the deep-attractor structure of the Particle Life++ substrate, which causes perturbed trajectories to rapidly relax back to their pre-perturbation states.

Verdict: A Mathematical Compass for Emergence

This work provides a compelling, mathematically rigorous alternative to black-box open-endedness metrics, establishing a direct connection between Artificial Life and statistical mechanics. MSPD acts as a reliable mathematical compass that guides dynamical systems toward self-organizing, lifelike behaviors without requiring handcrafted or learned perceptual metrics. For researchers working on active matter, multi-agent coordination, and unsupervised representation learning, MSPD offers a valuable tool. Future work addressing its computational scalability, defining a system-identified scale-separation timescale τ^⋆, and exploring multi-objective optimization that combines MSPD with complementary snapshot-structural metrics will be crucial to unlocking the full potential of this framework.