We have seen tremendous amount of progress in LLM world, and also open models in the last two weeks. Just to recap last week, OpenRouter has the following list of models per usage as of this writing(2026/08/15):

Qwen 3.8 is released last week as well which has the following capabilities:

• Core Capabilities: Comprehensive improvements across coding, professional work, research, and long-horizon agentic tasks.

• Agent Execution: Stronger autonomous planning and better handling of environment feedback, leading to more reliable end-to-end task completion.

• Downstream Compatibility: Broader support for popular harnesses and development tools, making it easier to integrate into your existing stack.

• Flexible Thinking Control: Reasoning depth can be tuned with reasoning_effort, and reasoning context from historical messages is retained via preserve_thinking.

  

Model weights(fp8, bf16) can be retrieved from here.

DeepSeek published their own harness. In this harness, they consider all of the capabilities as plugins to be able to create multiple capabilities, you need to “install” these plugins.

The plugins live in the Cordis kernel which manages plugin mounting, unmounting, and dependencies. This is kind of a control plane where all of the plugins(capabilities) register against Cordis kernel and Cordis kernel is able to orchestrate all of the capabilities in consistent manner. Cordis paper is available here.

With all of the other harnesses, everything the model sees is recorded in an append-only session log: system prompts, reasoning, tool calls and results, subagent scheduling, and every context injection. In the Trajectory view, you can inspect these records by source. Resume, fork, search, and replay all operate on the same event stream.

The code is available in GitHub. Developer docs(a bit light) is also available in GitHub pages.

Grok 4.6 is also released last week. Grok is closely following all of the other closed-source models well for a while, but could not move to the driver seat where Fable 5 Max currently sits.

Grok 4.6 builds on Grok 4.5 with a particular focus on long-running agents and more interactive and visual work. It stays with complex tasks across many steps, whether researching a topic, analyzing information, working across a codebase, or turning an idea into a polished application or work artifact. It can be accessed through Cursor or Grok Build.

Google released Gemini 3.7 Flash, it has improved better reasoning and accuracy for knowledge-dense fields like finance, law, and biosciences comparing to 3.6. It significantly outperforms 3.6 Flash on the GDP.pdf benchmark (34.0% vs 22.0%), an eval for testing a model’s ability to process complex documents. It also surpasses 3.6 Flash in AutomationBench, demonstrating it can more effectively complete real-world business workflows (30.4% vs 17.0%).

It is also quite cost-efficient comparing to other proprietary models, the open source models are much more cheaper, though.

The following has much more detailed benchmarking comparing to other models as well:

Declaude is an interesting website that “declaude” the text that is generated by Claude. It has also a post on how the watermarking that Claude does works which is very interesting.

The detector operates by using a secret key to assign the same token coloring(red, green) to a given passage. It then checks whether “green” tokens occur more often than would be expected by random chance. When the text is unwatermarked or the key is wrong, the “green” to “red” token ratio should be close to 50/50. In watermarked text, the increased count of green tokens yields a statistical score. This procedure leads watermark detection through the following mechanisms:

• Key-gated: External parties may run the detector only if they possess the secret key or have access to the official detection service.

• Probabilistic: Longer passages supply more evidence. Short texts frequently lack sufficient data for a confident result.

• Content-agnostic: The system does not assess writing style or meaning. It only searches for the specific signal generated during creation.

In context-dependent systems, each token’s status relies on the text that directly precedes it. Editing a passage disrupts the context window around the alteration, rendering local evidence unusable. Nevertheless, portions of the original wording that stay unchanged can still be detected.

• Light editing or standard paraphrasing: These actions frequently weaken the watermark. A detector can still locate the signal if sufficient token sequences stay intact, particularly in long documents.

• Full recomposition: Rewriting a passage based on its meaning rather than preserving the original wording can eliminate the contextual overlap required by these systems.

A probabilistic scheme that can measure various degrees of confidence even in the presence of editing is a robust method to understand if the text is generated. However, because it also has a lot of control on the text, I’d imagine this would be hard to understand and track over multiple/many versions of the models as one has to understand the “green” and “red” tokens and their rough distribution per versions of different models.

ThinkingMachines wrote about how determinism can be accomplished in inference for LLMs. Deterministic LLM inference broadly refers to “bitwise identical completions for the same request, every time, regardless of server load,” and the core claim is that the main obstacle to the determinism is not random sampling or GPU race conditions, but non–batch-invariant kernels whose numerics change as batch size and scheduling change under real-world load. They show this through carefully redesigning RMSNorm, matmul, and attention kernels to be batch-invariant—and integrating them into vLLM via FlexAttention—they can make a real LLM endpoint deterministic with acceptable performance overhead and unlock cleaner and repeatable RL training regimes where randomness can play a large role to prevent reproducibility in the experiments.

Why determinism is a hard problem to solve

At a high level, there are several overlapping notions of “determinism” in play, which makes the problem relatively ambiguous and hard to solve:

• Individual kernels:

  • Some GPU kernels really are run-to-run nondeterministic because they use atomics whose accumulation order depends on timing.

  • Kernels in a standard LLM forward pass (RMSNorm, matmuls, attention, pointwise ops) are typically run-to-run deterministic when given identical inputs on the same hardware and software stack.

• End-to-end forward pass:

  • Given a fixed batch of requests and a fixed kernel configuration, the forward pass is deterministic: every token’s logits are identical across runs.

  • Even so, small numerical differences from floating point arithmetic exist; they’re just fixed for a given configuration.

• Inference service from the user’s perspective:

  • The system appears nondeterministic because the same user request, sent multiple times, experiences different batch compositions depending on current load.

  • Since key kernels are not batch-invariant, the numerics for a given request change with batch size and scheduling, so the output tokens can diverge even under greedy decoding at temperature 0.

The post emphasizes that this is not primarily about stochastic sampling or about GPU threads “finishing in random order” in the LLM forward path; the real culprit is that kernel numerics depend on batch-related parameters that fluctuate nondeterministically in production serving.

Floating point and where nondeterminism really comes from

The classic “original sin” is actually the floating‑point non‑associativity.

• Floating point uses a mantissa–exponent representation (e.g., 3.45×1033.45×103, 4.86×10−14.86×10−1), which allows dynamic range but only a fixed number of significant digits.

• When adding numbers with very different exponents (e.g., 1230 and 23.4), low-magnitude terms get rounded away, and the rounding pattern depends on the order of additions.

• In a toy example, summing a small set of positive and negative values in different orders yields 102 distinct sums due purely to rounding order.

However, this only explains why different addition orders produce different results, not why kernels would change the order across runs.

The standard “concurrency + floating point” hypothesis is:

• Threads finish in an unpredictable order, atomics accumulate in whichever order completions happen, and non-associativity then turns that into small numerical differences, giving run-to-run nondeterminism.

However:

• Modern ML libraries avoid atomics for most forward-pass reductions.

• A simple GPU matmul repeated 1000 times with identical inputs produces bitwise identical results, even though it uses floating point and massive concurrency.

• For typical LLM forward passes, there are usually no atomic adds at all, so the forward pass is run-to-run deterministic for a given batch and kernel configuration.

The real issue appears when you step back to the system:

• The forward pass numerics depend on batch size (and related shape details) because kernels are not batch-invariant.

• Batch size is effectively a random variable from the user’s point of view, driven by other traffic and scheduling decisions.

• Compose “non-invariance to batch size” with “nondeterministic batch size under load,” and you get inference-level nondeterminism.

So floating point is the underlying source of numerical variability, but the mechanism that makes it visible as nondeterminism for inference users is batch-dependent kernel implementations, not atomics in the LLM forward path.

autBatch Invariance

The central technical concept is batch invariance:

• A kernel is batch-invariant if each element’s result depends only on its own data and is numerically identical regardless of overall batch size or batch composition.

• Concretely, the reduction order for the data relevant to a given element must be independent of batch size and of how work is partitioned across batch dimensions.

A simple PyTorch example where:

• Compute out1 = torch.mm(a[:1], b) (matvec).

• Compute out2 = torch.mm(a, b)[:1] (first row from a bigger matmul).

• These give different outputs (max absolute difference on the order of 1669) even though they use the same data and are run-to-run deterministic.

This illustrates that:

• Matmuls are typically not batch-invariant; mats of different sizes lead to different tiling, different use of tensor core instructions, and different reduction orders, hence different numerics.

• If an inference server dynamically batches requests, the same prompt can land in different batch sizes, pushing it through different numeric paths.

And post also splits operations into:

• Pointwise ops: generally batch-invariant (e.g., elementwise activations, basic arithmetic), so they’re not the main concern.

• Reduction ops: where batch invariance is tricky and crucial:

  • RMSNorm (per-token reductions over hidden dimension).

  • Matmul (reductions over feature dimension).

  • Attention (reductions over feature and sequence dimensions).

These three drive most of the nondeterministic behavior under changing batch conditions and are the focus of the batch-invariant redesign.

How RMSNorm, matmul, and attention become batch-invariant

RMSNorm

RMSNorm typically looks like:

# x: [batch_size, hidden_dim]
# weight: [hidden_dim]
def rms_norm(x, weight):
    return x * torch.rsqrt(torch.mean(x ** 2, dim=-1, keepdim=True)) * weight

• For batch invariance, the reduction order over hidden_dim must not depend on batch size.

• Standard GPU strategy: assign each batch element to a core (SM), so each element’s reduction happens “locally” without cross-core communication.

• This works well when batch size is large enough; increasing batch size does not change the per-element reduction strategy.

The problem appears when batch size is small:

• You end up with more cores than batch elements; naïve optimization will try to parallelize further by using split reductions or atomics across cores, changing the reduction order and breaking batch invariance.

Two strategies:

• Simple but acceptable: tolerate suboptimal performance for very small batch sizes and keep a single-core reduction strategy that is independent of batch.

• If you must optimize small batches, design a reduction strategy that yields sufficient parallelism even for batch size 1 and keep that same strategy for all larger batches, sacrificing some peak efficiency for invariance.

Matmul

Matmul can be viewed as:

• Compute pairwise products (pointwise), then reduce along the reduction dimension KK to form each output tile.

• A data-parallel tiling strategy chunks the output in (M,N)(M,N) so that each core handles its tile’s reductions locally, preserving a fixed reduction order within tiles.

However, real kernels vary strategy based on shape:

• When M and N are small (e.g., matmuls with small batch or small feature dims), to keep the GPU saturated you often switch to Split-K matmul: multiple cores sum partial results over K and then combine them, changing the reduction order and breaking batch invariance.

• Stream-K takes this further, splitting K differently for different tiles for better load-balancing, making even batch position non-invariant: the numerics can depend on where in the batch a row lives.

• Different tensor core instructions (e.g., different tile sizes) get selected for different shape regimes; small-batch cases may even drop tensor cores entirely, further changing the reduction order.

The batch-invariant design choice:

• Compile and use a single kernel configuration and reduction path for all relevant shapes, especially avoiding Split-K for the main LLM matmuls.

• Since the model dimension NN is usually large in LLMs, you can often maintain enough parallelism without Split-K; the performance hit is acceptable in inference workloads.

In effect, you trade a bit of hardware utilization for numerical predictability across batch sizes.

Attention

Attention is the hardest piece, because:

• It involves two matmuls and reductions over both feature and sequence dimensions.

• In inference, sequence handling is highly optimized with chunked prefill, suffix/prefix caching, and paged KV cache layouts, all of which can change how much of the sequence is processed at once.

Two key requirements for deterministic attention:

1. Numerics must be invariant to how many requests are processed in parallel.

2. Numerics must be invariant to how a single request’s sequence is chunked, i.e., the reduction order for a token must not depend on whether its sequence is processed in one shot or in multiple chunks.

Standard FlashAttention-style strategies:

• Do data-parallel work over query tokens and heads while reducing over K/V.

• For efficiency with long KV caches or small query lengths (typical decode), they use Split-KV / FlashDecoding strategies that choose split factors dynamically to saturate cores, e.g., “balanced scheduling” that picks the largest split size that fully utilizes the GPU.

• Because split size and scheduling depend on batch composition and KV length, the reduction order for a given token changes with load, breaking batch invariance.

The batch-invariant attention design has two main components:

• Consistent KV layout:

  • Update the KV cache and page tables before the attention kernel so that keys and values are laid out identically regardless of how many tokens are processed in the current step (chunked prefill vs decode).

  • This ensures that, for a given token index, the sequence of K/V elements it sees is the same in memory ordering.

• Fixed split-size strategy:

  • Instead of choosing the number of splits dynamically from batch conditions, fix the split size and allow the number of splits to vary.

  • For a given token, this yields a fixed pattern of partial reductions and combination steps that does not depend on global batch size.

  • Internally, FlexAttention is modified to enforce this fixed split-size pattern, though these modifications are not yet fully upstreamed at the time of writing.

With batch-invariant matmuls and attention, plus RMSNorm and pointwise ops, the entire transformer forward pass becomes invariant to batch size and chunking strategy, which is what you need to make an inference endpoint deterministic under arbitrary load.

Training parallels and why determinism is hard

The post claims that it has strong parallels on the training side, and it helps explain why full determinism is hard in practice for training as well.

Run-to-run determinism vs. environment determinism

In training, several phenomena mirror the inference piece:

• Kernel-level determinism:

  • You can often make the forward and backward passes run-to-run deterministic by disabling specific nondeterministic ops (e.g., atomics in some backward kernels), fixing seeds, and pinning hardware/software versions.

  • However, different minibatch compositions, data orderings, and parallelism strategies (e.g., data vs. model parallel, sharded optimizers) change numeric paths and produce different trajectories even with the same seed.

• Batch- and schedule-invariance issues:

  • Backprop involves reductions across data-parallel replicas (e.g., all-reduce of gradients), where the reduction order and grouping can change across distributed configurations or NCCL scheduling, leading to different gradient numerics.

  • Optimizers like Adam or RMSProp maintain state across steps, so small numeric perturbations in gradients accumulate into different trajectories.

  • Mixed precision, dynamic loss scaling, and varying microbatching strategies all introduce shape- and schedule-dependent numerical behavior.

Similar to inference:

• The low-level kernels might be individually deterministic for fixed shapes and launching strategies, but the training system as a whole is not, because the effective “batch size and structure over time” is shaped by distributed scheduling, sharding, and data ordering, which are complex and often nondeterministic.

RL training and “true on-policy” behavior

The post explicitly discusses RL as a motivating example of why determinism matters:

• In on-policy RL, the policy that generates trajectories should match the policy used for training updates.

• In practice, if the inference stack and training stack are numerically different (e.g., different attention implementations, different precision, or non-deterministic inference), the training is effectively off-policy, because the samples were generated by a numerically different policy.

They run experiments in an RLVR setup on Bigmath with Qwen 2.5-VL 8B as the base policy.

• With no off-policy correction (no importance weighting), training can become unstable and reward collapses when numerics diverge.

• Adding importance weighting stabilizes training but adds complexity.

• If they enforce bitwise identity between sampler and trainer (i.e., deterministic inference plus matched training stack), then:

  • KL divergence between sampler and trainer policies stays at 0.

  • Training remains stable without importance weighting.

This “true on-policy RL” point generalizes:

• Any training regime that depends on a “current policy” (RL, DPO-like methods, online fine-tuning) benefits from determinism, because it lets you reason cleanly about which policy generated which data.

• However, matching numerics across training and inference stacks is difficult: any divergence in kernel choice, precision, or batch handling breaks equality.

So determinism is hard in training not only because of floating point, but because of:

• Varying data order and microbatch schedules.

• Distributed gradient reductions and communication patterns.

• Different kernel choices in forward/backward vs inference paths.

• Mixed-precision and quantization differences between training and serving.

The post’s inference work shows one ingredient: if you can make the inference stack deterministic and phaselocked with the training stack, you get a cleaner on-policy setting.

Practical contributors to nondeterminism in LLM inference

Putting things together, you can categorize the main contributors to nondeterminism in LLM inference:

1. Floating-point arithmetic properties

• Non-associativity of addition and multiplication means different reduction orders produce different results.

• Mixed precision (e.g., bfloat16 activations, fp32 accumulators) shapes where rounding happens and how sensitive results are to ordering.

• Different hardware or library versions can change numerics even if the algorithm description is the same.

2. Kernel implementation choices

• Reduction kernels:

  • Use of Split-K (matmul) or Split-KV / FlashDecoding (attention) introduces multiple partial reductions whose combination order can vary with shape and scheduling, breaking batch invariance.

  • Optimization for small shapes often toggles between tensor core and non-tensor-core paths, changing the internal numeric path.

• Use (or avoidance) of atomics:

  • While the LLM forward pass seldom uses atomics, any kernel that does (e.g., some scatter_add variants, FlashAttention backward) can be inherently run-to-run nondeterministic unless special care is taken.

  • Backward passes and some auxiliary operations may still rely on atomics.

• Tile sizes and instruction selection:

  • Different shapes trigger different tiling strategies and tensor core instructions, all of which correspond to different internal reduction orders.

3. Batch composition and server load

• Dynamic batching merges requests arriving around the same time, making each request’s effective batch size and shape depend on current load, queueing, and scheduling.

• If kernels are not batch-invariant, changing batch size changes reduction strategies and numeric results for individual requests.

• Chunked prefill, prefix caching, and KV paging alter how many tokens from a sequence are processed together, which affects reduction structure in non-batch-invariant attention kernels.

From the user’s perspective, this is the dominant source of nondeterminism: the same prompt is routed through a different batch and shape profile on each call.

4. Library and hardware version differences

• Different GPU architectures (e.g., Hopper vs Blackwell) and different CUDA/cuBLAS/cuDNN versions can change kernel implementations and numeric behavior, even if the API and high-level configuration are the same.

• Cross-version runs are deterministic within a version but differ across versions, complicating reproducibility across environments.

5. Training–inference stack mismatches

• In RL and other online methods, if the inference path used for sampling differs numerically from the training path (e.g., different attention kernels, precision settings, or KV cache handling), then even a “fixed” policy ID corresponds to slightly different behaviors in practice.

• This shifts the problem from simple nondeterminism to policy mismatch, which can destabilize algorithms that assume on-policy data.

The post’s implementation—batch-invariant RMSNorm, matmul, attention, plus a vLLM integration—directly addresses categories 2 and 3, and indirectly helps category 5 by making inference behavior reproducible and alignable with training.

Libraries

Tokasaurus is an LLM inference engine designed for high-throughput workloads. Features include:

• OpenAI chat, completions, and batch APIs.

• Data, pipeline, and tensor parallelism (with support for AsyncTP).

• Support for Llama3 and Qwen2 architectures.

• Paged KV caching with prefix caching.

• Efficient attention over shared prefixes with Hydragen, with automatic detection of shared prefixes across groups of sequences.

• End-to-end torch compile with dynamic shapes.

• CUDA graphs.

• Very low CPU overhead (important for small models/fast GPUs).

• A scheduler that can simulate the number of available KV cache blocks thousands of steps in the future, allowing us to aggressively onboard new sequences and keep our batch size as large as possible.

• No OOMs or recompiles in production: on engine startup, we launch a series of warmup inputs that trigger all torch recompiles ahead-of-time (torch will recompile whenever a tensor has an input dimension is 0 or 1) and make check for OOMs using the largest configured batch size.

OpenP5 is an open-source platform for LLM-based Recommendation development, finetuning, and evaluation.

CUDA-L2 is a system that combines large language models (LLMs) and reinforcement learning (RL) to automatically optimize Half-precision General Matrix Multiply (HGEMM) CUDA kernels. CUDA-L2 systematically outperforms major matmul baselines to date, from the widely-used torch.matmul to state-of-the-art NVIDIA closed-source libraries (cuBLAS, cuBLASLt-heuristic, cuBLASLt-AutoTuning).

mbake is a Makefile formatter and linter:

Features

• Smart formatting: Tabs for recipes, consistent spacing, line continuation cleanup

• Intelligent .PHONY detection: Automatically identifies and manages phony targets

• Syntax validation: Ensures Makefiles are valid before and after formatting

• Configurable rules: Customize behavior via ~/.bake.toml

• CI/CD ready: Check mode for automated formatting validation

• VSCode extension: Full editor integration available

Below The Fold

Cadova is a Swift library for creating 3D models through code, with a focus on 3D printing. It offers a programmable alternative to traditional CAD tools, combining precise geometry with the expressiveness and elegance of Swift.

Cadova models are written entirely in Swift, making them easy to version, reuse, and extend. The result is a flexible and maintainable approach to modeling, especially for those already comfortable with code.

witr exists to answer a single question:

Why is this running?

When something is running on a system—whether it is a process, a service, or something bound to a port—there is always a cause. That cause is often indirect, non-obvious, or spread across multiple layers such as supervisors, containers, services, or shells.

Existing tools (ps, top, lsof, ss, systemctl, docker ps) expose state and metadata. They show what is running, but leave the user to infer why by manually correlating outputs across tools.

witr makes that causality explicit.

It explains where a running thing came from, how it was started, and what chain of systems is responsible for it existing right now, in a single, human-readable output.

Czkawka is a simple, fast and free app to remove unnecessary files from your computer.

Wireit upgrades your npm scripts to make them smarter and more efficient.

Kyanos is an eBPF-based network issue analysis tool that enables you to capture network requests, such as HTTP, Redis, and MySQL requests.
It also helps you analyze abnormal network issues and quickly troubleshooting without the complex steps of packet capturing, downloading, and analysis.

2025 LLM Year in Review

Andrej Karpathy’s 2025 year-in-review captures six paradigm shifts that reshaped the landscape of large language model development and deployment. The post shows a recalibration of how the industry approaches LLM training, application architecture, and how humans can interact AI systems with new innovations that came in 2025.

1. Reinforcement Learning from Verifiable Rewards (RLVR)

The most consequential technical development of 2025 is the emergence of Reinforcement Learning from Verifiable Rewards (RLVR) as the dominant training methodology, fundamentally altering the established three-stage LLM production stack. Prior to 2025, the industry consensus consisted of 1. pretraining (rooted in GPT-2/3 methodology from ~2020), 2. supervised finetuning (InstructGPT approach from ~2022), and 3. RLHF (RLHF from ~2022). These three stages represented a stable, proven recipe for production-grade LLMs.

RLVR introduces a critical methodological change: training models against objective, automatically verifiable reward functions rather than human preference signals. Rather than relying on ambiguous human feedback, RLVR leverages well-defined environments such as mathematical and code puzzle-solving where correctness is deterministically verifiable. This shift is architecturally and computationally very important as it allows model to lean how to spontaneously develop complex problem-solving strategies—breaking down problems into intermediate computational steps and employing iterative recovery mechanisms—without explicit supervision. These emergent reasoning behaviors were exceedingly difficult to achieve under previous paradigms because the optimal reasoning traces were neither obvious nor easily specifiable a priori.

The compute economics of RLVR represent also a factor in 2025’s capability acceleration. Unlike supervised finetuning and RLHF, short optimization stages with modest computational overhead, RLVR permits substantially longer training runs against objective reward functions. This computational arbitrage proved to be cost-efficient, causing a significant reallocation of compute budgets away from pretraining and toward extended RL optimization. Consequently, 2025 witnessed model capability advances driven primarily through longer inference-time reasoning traces rather than larger pretraining datasets or model sizes. A new scaling dimension emerged: test-time compute through extended thinking traces, providing a distinct knob for controlling capability independent of model parameters. This itself was significant both in terms of effort and time allocation, but also it was important in terms of compute/GPU resources. Prior to this change, most of the compute resources were in pre-training and due to that, it was spent in the training part of the model development. After this change, inference cost also increased and one needed to allocate more resources to the inference part of the model deployment.

OpenAI’s o1 model (late 2024) demonstrated the initial feasibility of RLVR, but the o3 release in early 2025 represented the inflection point where the capability differences became clear.

2. Reconceptualizing LLM Intelligence

The second conceptual insight concerns the fundamental nature of LLM intelligence itself. Karpathy articulates an interesting reframing and analogy: LLMs are not evolved animals but summoned ghosts—entities optimized under entirely different constraints than biological intelligence.

Human neural networks evolved under evolutionary pressure for tribal survival in ancestral environments, producing a form of general-purpose biological intelligence shaped by natural selection. LLM neural networks, conversely, are optimized via entirely different mechanisms: imitating human text, collecting rewards on mathematical puzzles, and gaming human preference signals on benchmarking arenas. The optimization targets are fundamentally misaligned, producing radically different capability profiles.

This insight manifests empirically in what Karpathy terms “jagged intelligence.” LLMs simultaneously exhibit polymath-level sophistication and cognitively challenged grade-school reasoning, sometimes within moments of each other. They can solve complex mathematical reasoning problems while remaining vulnerable to elementary jailbreaks designed to exfiltrate sensitive data. This jaggedness is not a training deficiency but a structural consequence of optimization: LLMs spike in capability near domains where RLVR training occurs, producing uneven performance landscapes.

This conceptualization has critical implications for benchmarking practices. Karpathy argues that benchmarks are inherently verifiable environments and thus immediately susceptible to RLVR and synthetic data augmentation. The typical “benchmaxxing” process involves research teams constructing synthetic data pockets adjacent to benchmark distributions, then growing capability “jaggies” to cover these regions—a process Karpathy characterizes as “training on the test set” in a new art form. This renders traditional benchmark-based capability assessment increasingly unreliable as a proxy for general capability. The industry faces a paradox: models that achieve state-of-the-art benchmark performance may still lack fundamental general capabilities. Conversely, achieving AGI does not require crushing all benchmarks—these goals are somehow detached and should be ideally decoupled.

3. The Rise of Vertical LLM Apps

The third paradigm shift concerns the emergence of a distinct application layer mediating between foundational LLM capabilities and end-user value. Cursor’s rise in 2025 revealed what Karpathy characterizes as a new “layer” of LLM applications—specialized orchestrators that bundle and coordinate multiple LLM calls for specific domains, giving rise to the pattern “Cursor for X.”

LLM applications like Cursor perform several critical functions that foundational models cannot efficiently provide:

• Context engineering: preprocessing and packaging domain-specific context

• Multi-call orchestration: stringing together complex directed acyclic graphs (DAGs) of sequential LLM invocations, carefully balancing performance against cost

• Application-specific UI/UX: providing domain-customized interfaces for human oversight

• Autonomy control: exposing an “autonomy slider” allowing users to modulate the degree of autonomous agent behavior

The critical question for 2025 is whether foundational LLM labs will capture all value at the application layer or whether durable opportunities exist for specialized vertical applications. Karpathy argues that foundational models will mature into “generally capable college students,” while specialized applications will organize and customize these capabilities into “deployed professionals” within specific verticals through private data integration, sensor/actuator access, and domain-specific feedback loops. This represents a competitive positioning distinct from foundational model capabilities—the “thickness” of this application layer suggests meaningful economic value capture. This is also a large change from the approach of “one single to rule them all” to “backbone models are useful, but so does vertical capabilities on top of the backbone models”.

4. Agents on Localhost, Not the Cloud

This paradigm shift addresses the architectural deployment of LLM agents. Anthropic’s Claude Code (CC) emerged as the first demonstration of a functional LLM agent—systems that iteratively chain together tool use and reasoning for extended problem-solving workflows.​

The architectural distinction is whether agent systems run on the developer’s local machine versus cloud infrastructure. Karpathy criticizes OpenAI’s early agent strategy of containerized cloud deployments orchestrated through ChatGPT interfaces, arguing this misses the critical insight that given the current era of “jagged capabilities” and slow capability takeoff, local deployment is superior. Claude Code’s key innovation is positioning itself as a “ghost on your computer”—a persistent agent running on localhost with direct access to the developer’s environment, data, configuration, secrets, and low-latency interaction patterns.​

This design choice is not merely infrastructural but represents a fundamental paradigm shift in human-AI interaction. Rather than visiting a website, the developer experiences an agent as a constituent tool within their existing development environment, achieving a qualitatively different interaction model. The agent gains contextual awareness of the developer’s specific setup—repositories, dependencies, environment variables, private data—and can reason within these constraints. This is in contrast to stateless, generic cloud interfaces that require constant context re-transmission.​

5. Vibe Coding

The fifth shift—vibe coding—represents another significant LLM capability with socioeconomic implications. By 2025, LLM capabilities had matured sufficiently to enable individuals to write functional software purely through natural language specification, with code generation handled entirely by the model. The programmer describes intent in English, and the system materializes fully functional implementations.

Historically, new technologies concentrated value among professionals and institutions with specialized training; LLMs, conversely, disproportionately benefit regular people who lack formal programming training. Vibe coding democratizes software development, making programming accessible to non-specialists.

Beyond democratization, vibe coding transforms how professional developers work. When code is “free” (zero-friction to generate), developers create software at a different scale: single-use ephemeral applications built to debug specific issues, quick prototypes for proof-of-concept ideas, and custom tooling that wouldn’t previously be economically justifiable. Karpathy himself demonstrates this with examples like custom BPE tokenizer implementations in Rust and ephemeral applications (menugen, llm-council, reader3, HN time capsule) built to explore specific ideas.

This capability shift has profound labor market implications. “Vibe coding will terraform software and alter job descriptions.” The number of programs written may increase dramatically as the friction of creation drops, but the occupational role of “professional programmer” will necessarily transform when code generation is commoditized.

6. The LLM GUI and Multimodal Interfaces

The sixth paradigm shift talks about the user interface layer for LLMs. Karpathy puts LLMs as the next major computing paradigm—analogous to the shift from mainframes to personal computing in the 1970s-80s. In this historical analogy, contemporary text-based LLM interfaces (ChatGPT, Claude chat) are equivalent to 1980s command-line interfaces.

Text is the native representation for computers and LLMs, but not for humans. Human cognition preferentially processes visual and spatial information; reading text is cognitively effortful and slow. This motivates the need for an LLM-native GUI analogous to how graphical user interfaces revolutionized computer accessibility.​

Google’s Gemini Nano banana model represents a first demonstration of what LLM-native interfaces might entail—systems jointly optimizing text generation, image generation, and world knowledge within unified model weights. Rather than treating image generation as a separate model invoked as a tool, nano banana integrates multimodal generation capabilities within the core architecture. The interface layer can express outputs as images, infographics, slides, whiteboards, animations, or interactive web applications rather than text, matching human cognitive preferences.​

Current outputs—emoji, Markdown, tables, bold/italic formatting—are primitive precursors to more sophisticated visual and spatial output formats. The strategic question is: who will build the LLM GUI? This represents a substantial design and engineering problem distinct from model training.​

The six paradigm shifts reveal that the primary advances are not simply “bigger models” but rather incorporating of training methodology (RLVR), reframing of AI intelligence nature (ghosts vs. animals), emergence of application architecture (the app layer), deployment paradigm (localhost agents), utilization patterns (vibe coding), and interface design (multimodal GUIs).

Karpathy’s concluding observation captures an apparent paradox: he simultaneously expects rapid continued progress and believes substantial work remains before realizing even 10% of LLMs’ potential. This reflects the current state where LLMs are simultaneously more capable and more limited than anticipated—exhibiting unpredictable jagged intelligence profiles that resist simple capability modeling.​

From Friends

• Curious how far you can go with state-of-the-art #LLMs without a huge compute budget?

In the latest blog 👉 recsysml.substack.com/p/training-gpt-2-on-a-budget, Vishwanath Sangale (Meta Ads ML + MRS) shares a hands-on journey training GPT-2 on a budget — not just what works, but why it works. Too often we treat “SOTA” (state-of-the-art) as a destination, rather than a process. We build on models without ever understanding the assumptions, trade-offs, or design decisions underneath. That’s a big missed opportunity. Learning by doing — actually rolling up your sleeves with real models — gives you intuition you can’t get from papers or APIs alone.

Libraries

Nunchaku is a high-performance inference engine for low-bit neural networks. It implements SVDQuant, a post-training quantization technique for 4-bit weights and activations that well maintains visual fidelity. On 12B FLUX.1-dev, it achieves 3.6× memory reduction compared to the BF16 model. By eliminating CPU offloading, it offers 8.7× speedup over the 16-bit model when on a 16GB laptop 4090 GPU, 3× faster than the NF4 W4A16 baseline. On PixArt-∑, it demonstrates significantly superior visual quality over other W4A4 or even W4A8 baselines. "E2E" means the end-to-end latency including the text encoder and VAE decoder.

nanoGPT is simplest, fastest repository for training/finetuning medium-sized GPTs. It is a rewrite of minGPT that prioritizes teeth over education. Still under active development, but currently the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of training. The code itself is plain and readable: train.py is a ~300-line boilerplate training loop and model.py a ~300-line GPT model definition, which can optionally load the GPT-2 weights from OpenAI. That's it.

faster-whisper is a reimplementation of OpenAI’s Whisper model using CTranslate2, which is a fast inference engine for Transformer models.

This implementation is up to 4 times faster than openai/whisper for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU.

Below The Fold

PocketBase is an open source Go backend that includes:

• embedded database (SQLite) with realtime subscriptions

• built-in files and users management

• convenient Admin dashboard UI

• and simple REST-ish API

Generative Manim (GM) is a suite of tools that allows you to create videos with Manim using LLMs (Large Language Models) like GPT-4 or Claude. The idea is to enable anyone to create wonderful animations from text ✨.

It began as a prototype of a web app that uses GPT-4 to generate videos with Manim. The idea behind this project is taking advantage of the power of LLMs in programming, the understanding of human language and the animation capabilities of Manim to generate a tool that could be used by anyone to create videos. Regardless of their programming or video editing skills.

• 🖐️ Generative Manim Demo: Check out the demo of Generative Manim!

• 🔬 Generative Manim API: Build over the Animation Processing Interface, or API.

• 🧑‍💻 Generative Manim Developers: Join our Discord server, learn new things, share your creations and more!

• 🍎 Generative Manim Streamlit (Legacy): First LLM exploration of LLMs and Animation.

Foliate is an open source library to read ebooks in style.

Actual is a local-first personal finance tool. It is 100% free and open-source, written in NodeJS, it has a synchronization element so that all your changes can move between devices without any heavy lifting.

BunkerWeb is a next-generation, open-source Web Application Firewall (WAF).

Being a full-featured web server (based on NGINX under the hood), it will protect your web services to make them “secure by default.” BunkerWeb integrates seamlessly into your existing environments (Linux, Docker, Swarm, Kubernetes, …) as a reverse proxy and is fully configurable (don’t panic, there is an awesome web UI if you don’t like the CLI) to meet your own use cases. In other words, cybersecurity is no longer a hassle.

Traditional search relevance measurement at Pinterest relied on manual human annotations—a bottleneck that constrained everything downstream. Evaluating changes to ranking algorithms required hiring annotators to judge whether search results matched user intent on a five-point scale (Highly Relevant down to Highly Irrelevant). This process was expensive, slow, and limited the sample sizes that teams could evaluate, making it nearly impossible to detect small improvements or understand how changes affected different user segments.​

The crux of the challenge lay in relevance measurement for online A/B experiments. Each new ranking model required relevance assessment across control and treatment groups, but human labeling imposed such high costs that Pinterest could only afford small samples. This resulted in minimum detectable effects (MDEs) of 1.3–1.5%, meaning the system could only statistically distinguish experiments with very large effects. Smaller improvements even if they to meaningful gains, are not very to be observed in production settings.

Cross-Encoder Architecture

Rather than replace their entire ranking system with LLMs (computationally infeasible at Pinterest’s scale), they deployed a two-stage strategy. First, they built a cross-encoder teacher model fine-tuned on human-annotated data. Unlike bi-encoders that separately embed queries and documents, cross-encoders process both jointly, allowing the model to attend to fine-grained interactions between search intent and Pin content.

The architecture is straightforward: a query and Pin text are concatenated ([CLS] Query [SEP] Pin text) and passed through an encoder LLM, which outputs a five-dimensional score distribution corresponding to relevance labels. This is relatively important as cross-encoders outperform bi-encoders on ranking accuracy—studies show advantages of 4+ nDCG points—but at the cost of computational expense. Every query-Pin pair must be scored, while bi-encoders precompute embeddings once and reuse them.​

The team experimented with multiple backbone architectures:

• BERT_base: Baseline multilingual performance

• T5_base: Sequence-to-sequence variant

• mDeBERTa_v3_base: Improved pretraining

• XLM-RoBERTa_large: Final production choice

• Llama-3-8B: Highest accuracy but 6x inference cost​

XLM-RoBERTa_large won the efficiency-accuracy tradeoff: it achieved near-Llama performance while running 150,000 labels in 30 minutes on a single A10G GPU. This presented the speed and the ability to continuously re-evaluate ranking changes without waiting weeks for human annotation while not sacrificing the accuracy as much as possible.

Pinterest feeds it a rich feature-set into LLMs which are somehow curated and requires some understanding of their product and what matters for pin ranking:​​

• Pin titles and descriptions: Seller or creator-provided text

• BLIP image captions: Synthetic captions generated by a vision-language model, capturing visual semantics

• Linked webpage titles and descriptions: Content from external sources that Pins link to

• User-curated board titles: Titles of boards where the Pin has been saved (implicit relevance signals)

• Highly-engaged queries: Queries that users have previously searched with when interacting with this Pin

This multi-modal text representation is interesting because it converts Pinterest’s heterogeneous data into a language-native format that LLMs are really good at. The BLIP captions can also be considered to provide similar modality—they represent an intermediate step for feature engineering, where a separate VLM distills visual content into structured text.

The distillation addresses the tradeoff between model accuracy and serving latency. The fine-tuned teacher LLM works well for evaluation and labeling, but you cannot run every search query against XLM-RoBERTa_large in production. Both latency and cost of inference would present a large challenge to do this in Pinterest scale.

In order to address this challenge, Pinterest adopted knowledge distillation; train a smaller, faster student model to mimic the teacher’s outputs. The teacher LLM labels billions of query-Pin pairs during an offline phase and then the student model is trained on these synthetic labels using standard supervised fine-tuning, learning to replicate the teacher’s relevance judgments without access to the original human annotations. The result is a model that combines teacher insight with student efficiency—a common pattern across many applications with different companies (Google uses this for BERT variants, Amazon for product matching, OpenAI for scaling inference with some routing logic based on the query).

This approach also enables multilingual generalization. The teacher LLM learns from primarily English human labels but leverages its pretrained multilingual knowledge. When applied to queries and Pins in French or German, the cross-lingual transfer properties of XLM-RoBERTa activate. The student model inherits this capability without language-specific fine-tuning.​

Before deploying LLM-generated labels into production metrics, Pinterest proved they reliably estimate true relevance.

Exact Agreement: 73.7% of LLM labels matched human labels exactly; 91.7% differed by no more than one point on the five-point scale.

Rank Correlation (at query level):

• Kendall’s τ: 0.652 overall (0.539–0.620 across query popularity segments)

• Spearman’s ρ: 0.817 overall (0.668–0.801 across segments)

These correlations matter because they validate ranking order rather than absolute scores. If an experiment improves the ranking quality, would LLM-based measurement detect it? Yes—the τ and ρ values confirm the LLM preserves the relative ordering of results.

For non-English markets, performance degraded slightly but remained strong: Kendall’s τ of ~0.48 and Spearman’s ρ of ~0.62 for France and Germany. The authors acknowledge this gap as a frontier for future work—larger fine-tuning datasets in these languages could close it.

Error Distribution: The query-level sDCG@K metric (a ranking-aware relevance score) showed mean error of 0.005 with 90% of errors between -0.047 and 0.059. Critically, paired differences (the metric differences between control and treatment) had even tighter distributions, centering around zero. This is exactly what you want for A/B testing—any bias cancels out, and noise is minimal.

Perhaps the deepest innovation isn’t the LLM itself but how Pinterest used cheap LLM labels to redesign sampling strategy. Before, cost constraints forced simple random sampling of ~2,000 queries. Now, with labels costing cents rather than dollars, they could stratify by query interest (via an in-house DistilBERT model) and popularity (head/torso/tail/single queries), allocating samples proportionally to variance within each stratum.

The impact was staggering: variance reduction of 94–99.6%, translating to MDEs dropping from 1.3–1.5% down to ≤0.25%. This sixfold improvement means teams can now detect small wins that previously disappeared into experimental noise. Methodologically, this leverages a classic insight from survey sampling (Hansen-Hurwitz, Horvitz-Thompson estimators) that stratification removes between-strata variance, leaving only within-strata noise.​

The formula (from their paper) illustrates the principle:

By choosing strata such that σ_k is small within each group (e.g., “beauty queries” form a homogeneous stratum), overall variance plummets.

Library

Arch handles the pesky plumbing work in building AI agents — like applying guardrails, routing prompts to the right agent, generating hyper-rich information traces for RL, and unifying access to any LLM. It’s a language and framework friendly infrastructure layer designed to help you build and ship agentic apps faster.

Flux contains minimal inference code to run image generation & editing with our Flux open-weight models.

kotaemon is an open-source clean & customizable RAG UI for chatting with your documents. Built with both end users and developers in mind.

Voice-Pro is a state-of-the-art web app that transforms multimedia content creation. It integrates YouTube video downloading, voice separation, speech recognition, translation, and text-to-speech into a single, powerful tool for creators, researchers, and multilingual professionals.

• 🔊 Top-tier speech recognition: Whisper, Faster-Whisper, Whisper-Timestamped, WhisperX

• 🎤 Zero-shot voice cloning: F5-TTS, E2-TTS, CosyVoice

• 📢 Multilingual text-to-speech: Edge-TTS, kokoro (Paid version includes Azure TTS)

• 🎥 YouTube processing & audio extraction: yt-dlp

• 🌍 Instant translation for 100+ languages: Deep-Translator (Paid version includes Azure Translator)

VizSeq is a Python toolkit for visual analysis on text generation tasks like machine translation, summarization, image captioning, speech translation and video description. It takes multi-modal sources, text references as well as text predictions as inputs, and analyzes them visually in Jupyter Notebook or a built-in Web App (the former has Fairseq integration). VizSeq also provides a collection of multi-process scorers as a normal Python package.

Archimedes is an open-source Python framework designed for deployment of control systems to hardware. To make this possible, it provides a comprehensive toolkit for modeling, simulation, optimization, and C code generation.

More documentation on the library and design principles behind of the library is here.

Eval Protocol (EP) is an open solution for doing reinforcement learning fine-tuning on existing agents — across any language, container, or framework.

More information is in their documentation page.

Below The Fold

Penpot is the first open-source design tool for design and code collaboration. Designers can create stunning designs, interactive prototypes, design systems at scale, while developers enjoy ready-to-use code and make their workflow easy and fast. And all of this with no handoff drama.

TidesDB is a fast and efficient key value storage engine library written in C. The underlying data structure is based on a log-structured merge-tree (LSM-tree).

It is not a full-featured database, but rather a library that can be used to build a database atop of or used as a standalone key-value/column store.

Walrus is a distributed message streaming platform built on a high-performance log storage engine. It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination.

Key Features:

• Automatic load balancing via segment-based leadership rotation

• Fault tolerance through Raft consensus (3+ nodes)

• Simple client protocol (connect to any node, auto-forwarding)

• Sealed segments for historical reads from any replica

• High-performance storage with io_uring on Linux

PyTogether is a Google docs for Python. A fully browser-based collaborative Python IDE with real-time editing, chat, and visualization.

Stardust is a unikernel operating system designed to run Cloud applications in a protected, single-address space environment. It delegates the management of physical resources to an underlying hypervisor which is treated as a trusted platform. Stardust has a small code base that can be maintained easily, and relies on static linking to combine a minimal kernel with a single application, along with the libraries and associated programming language run-time required for the execution of the application. Due to static linking, an executable binary of Stardust is packaged within an immutable single-purpose virtual machine image. Stardust supports multiple cores, preemptive threads, and basic block and networking drivers, and provides a collection of standard POSIX-compatible libraries.

Stanford researchers wrote about AMD GPUs and how they have developed a new library(HipKittens) to make AMD GPUs to be more efficient through the library that they have developed.

I covered a similar library last year for NVIDIA GPUs and name of that library was in a similar name ThunderKittens, they like kittens in the library name.

Main problem that they are trying to solve is that how do we improve the maturity level of AMD software that builds on top of AMD GPUs on the order of NVIDIA GPUs. The main root cause such a gap in performance(at least in the software layer) exists for AMD is that AMD is relatively new for SW-HW code sign that NVIDIA has spent more time and drove more community adoption through the lead time that they have.

The post is written in such a way that Stanford researchers try to prove that HipKittens (HK) unlocks the performance of modern AMD CDNA GPUs (like MI355X) through memory access, intra-GPU scheduling, and chiplet-aware grid scheduling, rather than simply copying NVIDIA-style kernel designs that they have in the ThunderKittens library(which they also are the author for that library).

The latest and greatest chip from AMD is: MI355X GPU which has 256 compute units (CUs), each with four SIMD units, with a 64‑thread “wave” occupying one SIMD, analogous to a 32‑thread warp on NVIDIA. An MI355X offers about 2.5 PFLOPs BF16 tensor performance and up to 10.1 PFLOPs in MXFP6/4, with 288 GB HBM and 8 TB/s bandwidth—comparable or better than NVIDIA B200’s raw specs in several formats, though with different on‑chip resources. However, AMD has only about 70% of the SRAM per processor relative to B200 and lacks NVIDIA-specific features like wgmma/tcgen05 async matmuls from shared/tensor memory, tensor memory accelerator, mbarriers, and register reallocation, which heavily shape NVIDIA kernel design.

The key advantage that AMD has is much larger register files and more processors per GPU: MI355X has roughly 2× larger registers per CU and 60% more processors (256 CUs vs 160 SMs on B200), plus fine‑grained MFMA matrix core instructions rather than large, monolithic tensor ops. AMD also moves to chiplets: MI355X’s 256 CUs are split into 8 XCDs, each with a private L2 and a shared last‑level cache (LLC) in front of HBM, creating a disaggregated memory hierarchy unlike monolithic dies.

Here the library comes to make this powerful chip to be easily programmable; HipKittens adopts the same high‑level tile programming model as ThunderKittens: users can program in terms of tiles in shared memory or registers, parameterized by data type, dimensions, and layout, then use PyTorch‑like operations for compute (matmul, reductions) and memory (load/store). Each tile is collectively owned by threads in a wave, and tile operators are thin C++ templates that wrap AMD CDNA ISA and C++ code directly, giving full control instead of hiding details behind a compiler like Triton. This also makes the programming harder, but also allows more headroom for efficiency as well.

While the tile interface is conceptually shared across vendors, the backend implementations differ substantially between AMD and NVIDIA: AMD requires different register layouts, swizzle patterns, and scheduling patterns, and as post points out that these must be codified as reusable primitives rather than hand‑rolled assembly for each kernel. HK’s goal is to offer AMD‑centric abstractions—optimized tile layouts, memory access patterns, and scheduling recipes—that let users of the library write high‑performance kernels in a few hundred lines of C++ instead of large assembly codebases.

One of the interesting part is how HK handles memory access, given AMD’s relatively difference chip design and chip composition comparing to NVIDIA chips. Shared memory is banked (4‑byte banks), and as usual, concurrent accesses from a wave to the same bank serialize; well‑designed kernels therefore use swizzle patterns to map logical matrix tiles to bank‑friendly layouts. On NVIDIA, tensor‑core register layouts are relatively regular across data types and shapes, so a single unified swizzle rule can work across many tensor layouts; frameworks like TK exploit this. On AMD, register layouts vary substantially with data type and MFMA shape, so HK shows that no single swizzle pattern can avoid conflicts across all layouts, especially when a kernel mixes shapes.

One example that outlines how special handling of memory is important is that attention backward’s shared‑memory patterns: a 16×16 bf16 row‑layout tile is written with a 64‑bit‑per‑thread store instruction that needs a 64‑bit‑based XOR swizzle to avoid bank conflicts, while a 16×32 tile read uses a 128‑bit‑per‑thread load instruction that requires 128 bits to remain contiguous. The 64‑bit oriented swizzle that works for the store fundamentally conflicts with the 128‑bit contiguity needed for the read, so different swizzles must be used, showing why AMD needs multiple layout‑specific swizzling schemes.

The HK team also reverse‑engineers AMD’s memory instruction phases. Waves execute shared‑memory instructions in phases where only subsets of threads access banks at once, but unlike NVIDIA’s sequential phase grouping (threads 0–7, 8–15, etc.), AMD’s phase groupings vary by instruction and are non‑sequential and poorly documented. For example, one 128‑bit load per thread uses four phases over 64 banks, while another 64‑bit store uses four phases over 32 banks, changing conflict patterns; HK releases “solvers” that infer these behaviors to design conflict‑free layouts.

AMD’s large register file is split into accumulator general purpose registers (AGPRs) and vector general purpose registers (VGPRs) when a single wave maps to a SIMD—roughly 256 of each per SIMD, for a total of 512. AGPRs are restricted: vector ALU instructions cannot operate directly on them, but MFMA instructions can use them as inputs or outputs, making them ideal accumulators for matmuls. However, the HIPCC compiler cannot generate code that feeds AGPR data directly into MFMA instructions in high‑pressure settings, instead inserting extra accvgpr_read/write moves between VGPRs and AGPRs, which wastes bandwidth and registers.

To address this, HK introduces explicit register scheduling: developers can pin specific registers for register tiles, manually controlling the allocation of AGPRs and VGPRs instead of leaving decisions to HIPCC. This approach is crucial for register‑heavy kernels—like large GEMMs or attention—because static allocation and compiler limitations otherwise lead to spilling into scratch memory and underutilized hardware. Also note that, HK team makes a strong recommendation on improving HIPCC’s register scheduling model would be one of the most impactful upgrades AMD could make for its kernel software stack.

AMD supports direct asynchronous loads from HBM to shared memory, analogous in spirit to NVIDIA’s tensor memory accelerator (TMA), but with a key difference: the instruction takes per‑thread global addresses and writes directly into shared memory, bypassing the register file. While TK on NVIDIA can often swizzle shared‑memory addresses directly, on AMD the effective swizzling is done by transforming the HBM addresses that these async loads use. HipKittens encapsulates these address‑generation schemes, including swizzle computations, inside its tile‑load primitives so that kernel authors can request tiles in logical coordinates and rely on HK to map them to hardware‑friendly address streams.

On NVIDIA, wave/warp specialization (producer‑consumer) is the dominant strategy: some warps focus on memory movement while others compute, enabling deep pipelines with features like register reallocation, large shared memory, and async matmuls (wgmma/tcgen05) to keep tensor cores fed on a continuous manner. This pattern underlies high‑end kernels like FlashAttention‑3, COMET for MoE, and many GEMM pipelines, as well as DSLs like ThunderKittens and TileLang.

AMD’s MI355X, however, lacks register reallocation: registers are statically partitioned among waves on a SIMD, so producer waves that only need a few registers still get a full share, and consumer waves cannot borrow unused registers to increase arithmetic intensity. This leads either to spilling registers to memory or to smaller output tiles per thread block, both of which reduce arithmetic intensity and increase memory‑bound behavior, degrading performance for GEMMs where compute grows as O(MNK)O(MNK) but memory as O(MK+NK)O(MK+NK).

GEMM benchmark table on MI355X showing that various producer/consumer configurations underperform compared to non‑specialized or lightly specialized patterns per the post. For example, a 4‑producer/8‑consumer configuration with a 128×256 output tile delivers much lower TFLOPs than a configuration with no producers and a larger 256×256 tile, where HK achieves around 1.6 TFLOPs and matches or beats NVIDIA kernels from TK and CUTLASS tuned on B200 for similar tile shapes. This result is notable given AMD’s lack of NVIDIA’s advanced async matmul and memory features, illustrating that AMD’s larger register file and smaller MFMA shapes provide an alternate path to high throughput via fine‑grained pipelining.

Instead of wave specialization, HipKittens proposes two AMD‑centric intra‑processor scheduling patterns that use the tile abstraction:

• 8‑wave ping‑pong:

  • Two waves per SIMD operate in a ping‑pong pattern; at any moment, one wave issues a cluster of memory instructions while the other issues a cluster of compute instructions.

  • This allows each thread to issue long runs of identical instructions over large tiles, leading to compact, easy‑to‑maintain code with large HK tiles that keep the hardware busy.

• 4‑wave interleave:

  • One wave per SIMD alternates finely between memory and compute ops, using small tiles that approximately match the MFMA instruction shape.

  • This results in finer‑grained pipelining and potentially higher performance but at the cost of larger code size and more intricate instruction interleaving.

8‑wave ping‑pong already achieves state‑of‑the‑art performance for GEMMs and attention forwards on MI355X, outperforming AMD’s own baselines and competing with well‑tuned NVIDIA Blackwell kernels. For more complex workloads like GQA non‑causal attention backwards, 8‑wave beats all AMD baselines, and 4‑wave interleave delivers further gains, demonstrating the value of both patterns as options in the HK library.

A specific example of this can be demonstrated through is that HK attention forward kernel hot loop using the 8‑wave ping‑pong schedule, divided into clusters that alternate between compute and memory, with explicit AMD barriers and priority settings. In “compute” clusters, the code performs operations like:

• QK matmul using MFMA (mma_AtB) on register tiles for Q and K, with layout swaps and transposes to match MFMA expectations.

• Softmax computations over attention tiles, including exponentiation, scaling with normalization vectors, column maxima and sums, and BF16 conversions, all implemented as tile operations.

In “memory” clusters, the kernel:

• Uses HK’s G::load to asynchronously move K and V tiles from global memory into shared memory or registers using swizzled offsets.

• Employs s_waitcnt, vmcnt, and amdgcn barriers to ensure loads complete before subsequent MFMA instructions consume the data, while keeping the pipeline full.

The control flow shows multiple compute–memory–compute–memory clusters within a single loop iteration, where attention tiles for QK and AV are overlapped, and software barriers carefully schedule instruction phases across waves. The overall effect is a deeply pipelined loop in which each wave alternates between matmul, softmax segments, and K/V tile loads, making heavy use of the register file and small MFMA shapes rather than large shared‑memory staging and producer/consumer specialization.

The HK BF16 GEMM kernel hot loop follows a similar clusterized pattern under 8‑wave ping‑pong, but with a simpler computation structure tuned for dense matmul. In the “memory” clusters, the kernel:

• Loads sub‑tiles of B and A from shared memory into register tiles using subtile_inplace, which selects the slice for each warp based on its row/column coordinates.

• Simultaneously fetches future tiles of A or B from global memory into shared memory using G::load with chiplet‑aware swizzled offsets, overlapping global–shared transfers with register‑level work.

In “compute” clusters, the kernel:

• Performs MFMA matmuls (mma_ABt) on current A and B sub‑tiles, accumulating into C_accum tiles in AGPRs.

• Manages priority hints (s_setprio) and barriers to ensure MFMA execution flows smoothly without hazards, while loads for subsequent MFMA steps complete in the background.

The loop toggles between two shared‑memory buffers with tic and toc, implementing a double‑buffering scheme that overlaps loading future tiles with computing on current ones, and uses multiple sub‑tiles per full tile to match MFMA shapes and avoid bank conflicts. Despite relying only on C++ with intrinsics and HK primitives, this kernel’s hot loop is relatively short yet reaches peak GEMM performance, competing with hand‑optimized assembly from AITER and hipBLASLT.

At the chip level, MI355X’s eight XCD chiplets each have private L2 cache and a shared LLC to HBM, so thread block placement affects how well caches are reused. The default row‑major grid schedule for GEMM, which assigns blocks to tiles of the output matrix in simple order, often yields poor L2 locality because blocks mapped to the same XCD load non‑overlapping A and B tiles, causing redundant HBM traffic and underutilized bandwidth.

HipKittens defines chiplet‑aware grid scheduling strategies that regroup blocks so that those operating on spatially nearby output tiles—and therefore sharing input A and B tiles—are more likely to run on the same XCD and reuse data in L2 and LLC. For large GEMMs (e.g., M=N=K=9216 and 14592) comparing:

• Row‑major ordering, which has moderate L2 hit rates and high LLC hits but only modest memory bandwidth and TFLOPs.

• Several “XCD‑aware” layouts that tune the per‑XCD worker counts (W) and blocks per chiplet (C), achieving better balance between L2 and LLC hit rates, higher effective memory bandwidth (e.g., over 16 TB/s), and higher TFLOPs compared to row‑major.

The key idea is that optimizing solely for L2 locality can backfire by having each chiplet fetch disjoint A/B regions, increasing LLC and HBM activity, so HK’s schedulers aim for a compromise that reuses tiles both within each XCD and across chiplets via LLC. These chiplet‑aware patterns are embodied as grid‑launch policies in HK, so kernel authors can select them without manually micro‑managing block indices.

HK is itself open-source and code is available in GitHub.

Library

Tile Language (tile-lang) is a concise domain-specific language designed to streamline the development of high-performance GPU/CPU kernels (e.g., GEMM, Dequant GEMM, FlashAttention, LinearAttention). By employing a Pythonic syntax with an underlying compiler infrastructure on top of TVM, tile-lang allows developers to focus on productivity without sacrificing the low-level optimizations necessary for state-of-the-art performance.

AITER is AMD’s centralized repository that support various of high performance AI operators for AI workloads acceleration, where a good unified place for all the customer operator-level requests, which can match different customers’ needs. Developers can focus on operators, and let the customers integrate this op collection into their own private/public/whatever framework.

VibeThinker-1.5B is a 1.5B-parameter dense model that challenges the prevailing notion that small models inherently lack robust reasoning capabilities. Developed with an innovative post-training methodology centered on the “Spectrum-to-Signal Principle (SSP)”, VibeThinker-1.5B demonstrates superior reasoning capabilities compared to closed-source models Magistral Medium and Claude Opus 4, while achieving performance on par with open-source models like GPT OSS-20B Medium.

Repomix is a powerful tool that packs your entire repository into a single, AI-friendly file. It’s perfect for when you need to feed your codebase to Large Language Models (LLMs) or other AI tools like Claude, ChatGPT, and Gemini.

Below The Fold

🌪️ Vortex

Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:

• ⚡️ Blazing Fast Performance

  • 100x faster random access reads (vs. modern Apache Parquet)

  • 10-20x faster scans

  • 5x faster writes

  • Similar compression ratios

  • Efficient support for wide tables with zero-copy/zero-parse metadata

• 🔧 Extensible Architecture

  • Modeled after Apache DataFusion’s extensible approach

  • Pluggable encoding system, type system, compression strategy, & layout strategy

  • Zero-copy compatibility with Apache Arrow

• 🗳️ Open Source, Neutral Governance

  • A Linux Foundation (LF AI & Data) Project

  • Apache-2.0 Licensed

• ↔️ Integrations

  • Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more

  • Apache Iceberg (coming soon)

Myna (Gracula religiosa 🐦‍⬛) is a monospace font which aims to bring harmony to your editor by treating symbols as first-class glyphs alongside alphanumeric characters.

Myna was born of a need to scratch a persistent typographical itch. While I’ve tried many otherwise well-crafted monospace fonts, I always found myself wanting to tweak a glyph here or adjust a shape there. After developing Myna and using it almost exclusively in my professional and personal work, I’m sharing it as a small contribution to the wonderful community of monospace typography enthusiasts.

Marko re-imagines HTML as a language for building dynamic and reactive user interfaces. Nearly any valid HTML is also valid Marko, but Marko extends the HTML language to enable building modern applications in a declarative way.

rackstack is a modular 3d-printable mini rack system

• Mount Anything: Perfect for organizing SBCs, mini PCs, small switches, power hubs, etc.

• Fully customizable: Fully written in OpenSCAD. Everything, from the dimensions of the rack, to the roundness of the corners, can be modified with a simple code change.

• Printable from home: Designed to be printed with conventioenal FDM printers. Requires minimal supports when printing, and final assembly needs only a few easy-to-source parts.

• No cage nuts: Sliding hex nut design for the front rails allows one to easily mount items, without dealing with cage nuts.

• Stackable: Individual racks can be easily stacked and fastened together. Mix and match different color and design combinations!

Netflix wrote an excellent in depth article on the engineering experience of an ML Engineer in Netflix and how they develop and iterate on the ML models in Netflix with the technology stack that they use.

That stack centers around Metaflow, an open-source framework that allows engineers to build, iterate, and deploy models. Netflix cares about understanding of engineering productivity, design principles from development with infrastructure tooling eventually that transforms the complex world of machine learning operations into something much more accessible, especially for junior engineers.

Netflix’s approach recognizes a fundamental problem in modern ML development: ML engineers need freedom to experiment and iterate rapidly, yet their workflows must eventually integrate with production systems(ideally without issues!). Unlike traditional software engineering where explicit control flow and deterministic execution are the norm, machine learning development operates in fundamentally different territory where control flows and execution are anything but deterministic. Models require experimental iteration with visible state, versioned and persisted artifacts to support reproducibility and observability, and the ability to test individual components without re-executing entire pipelines. Metaflow comes into play in this problem and provides what various good abstractions that manage the complete lifecycle of ML projects—from rapid local prototyping in notebooks to reliable, maintainable production deployments scaling across cloud infrastructure.

Metaflow code looks like a workflow — similar to Airflow: each Metaflow @step serves as a checkpoint boundary. At the end of every step, Metaflow automatically persists all instance variables as artifacts, allowing the execution to resume seamlessly from that point onward.

However, this has an issue in the reproducibility in the production setting as MetaFlow cannot put the same workflow to production the way experimentation takes it to.

To solve that problem, Netflix recently shipped a feature called Spin that solves that pain point of making the experimentation code to be ready for production ready. Spin allows engineers develop Metaflow workflows incrementally, step by step, with local iterations—by providing the notebook experience while building production-ready systems that scale across distributed infrastructure. The feature follows an incremental pattern: develop a stub flow with start and end steps, execute it to generate initial inputs, then use python myflow.py spin somestep to test changes quickly using artifacts from previous runs.

Each step can be tested independently with appropriate input data without orchestrating metadata or persisting artifacts globally, allowing engineers to eyeball logs and optionally access output artifacts locally. Once a step functions correctly, engineers run the complete flow normally to capture comprehensive metadata and artifacts. This mirrors the workflow patterns that made Jupyter notebooks successful —visual output and analysis attached to the Python code, and incremental development with cached state.

Metaflow can also integrate Claude, which can construct complete ML workflows using Spin, detecting and fixing issues with reduced human intervention and faster completion times than traditional manual development. It creates “human in the loop” experience by augmenting the systems by handling routine pattern application and execution cycles.

Claude integrated Spin can write tests based on input/output pairs, run them to confirm failure, implement code to pass tests, and iterate until all tests pass—all without human prompting between cycles. This test-driven development approach accelerates not just individual development but creates self-validating systems that resist common ML pitfalls like overfitting or architectural inconsistencies within the model.

This framework currently powers over 3,000 ML projects at Netflix, executing hundreds of millions of data-intensive compute jobs annually while processing petabytes of data.

Libraries

Helion is a Python-embedded domain-specific language (DSL) for authoring machine learning kernels, designed to compile down to Triton, a performant backend for programming GPUs and other devices. Helion aims to raise the level of abstraction compared to Triton, making it easier to write correct and efficient kernels while enabling more automation in the autotuning process.

OpenPCC is an open-source framework for provably private AI inference, inspired by Apple’s Private Cloud Compute but fully open, auditable, and deployable on your own infrastructure. It allows anyone to run open or custom AI models without exposing prompts, outputs, or logs - enforcing privacy with encrypted streaming, hardware attestation, and unlinkable requests.

DataChain is a Python-based AI-data warehouse for transforming and analyzing unstructured data like images, audio, videos, text and PDFs. It integrates with external storage (e.g. S3) to process data efficiently without data duplication and manages metadata in an internal database for easy and efficient querying.

1. ETL. Pythonic framework for describing and running unstructured data transformations and enrichments, applying models to data, including LLMs.

2. Analytics. DataChain dataset is a table that combines all the information about data objects in one place + it provides dataframe-like API and vectorized engine to do analytics on these tables at scale.

3. Versioning. DataChain doesn’t store, require moving or copying data (unlike DVC). Perfect use case is a bucket with thousands or millions of images, videos, audio, PDFs.

4. Incremental Processing. DataChain’s delta and retry features allow for efficient processing workflows:

   • Delta Processing: Process only new or changed files/records

   • Retry Processing: Automatically reprocess records with errors or missing results

   • Combined Approach: Process new data and fix errors in a single pipeline

pgmpy is a Python library for causal and probabilistic modeling using graphical models. It provides a uniform API for building, learning, and analyzing models, such as Bayesian Networks, Dynamic Bayesian Networks, Directed Acyclic Graphs (DAGs), and Structural Equation Models (SEMs). By integrating tools from both probabilistic inference and causal inference, pgmpy enables users to seamlessly transition between predictive and causal analyses.

Below The Fold

ClickHouse® is an open-source column-oriented database management system that allows generating analytical data reports in real-time

Magnitude uses vision AI to enable you to control your browser with natural language.

• 🧭 Navigate - Sees and understands any interface to plan out actions

• 🖱️ Interact - Executes precise actions using mouse and keyboard

• 🔍 Extract - Intelligently extracts useful structured data

• ✅ Verify - Built-in test runner with powerful visual assertions

You can use it to automate tasks on the web, integrate between apps without APIs, extract data, test your web apps, or as a building block for your own browser agents.

Uber wrote a good article on how they are raising the bar on the model deployment by implementing a number of different checks and balances for model quality through some of the MLOps practices.

Uber’s centralized ML platform called Michelangelo, which supports over 400 use cases, executes more than 20,000 training jobs monthly, and serves 15 million predictions per second during peak load. In such an environment, even minor degradations in model quality can ripple out rapidly, making “silent failures” or subtle regressions dangerous and hard to detect. Unlike conventional code, ML systems are deeply probabilistic and tightly coupled to their ever-changing data sources, rendering static code tests insufficient for reliability.

In order to provide high safety standards in such an environment, Uber adopts various principles and adopts methods to improve their operational efficiency and operate the systems in a reliable manner.

Main principles that they have adopted:

• Data and Code as Artifacts: Every ML model is fundamentally dependent on its source data and code — both are equally critical in determining system reliability and are targets for silent failures.

• End-to-end Safeguards: Effective safety requires embedding protections at every stage — feature engineering, model training, pre-production validation, and post-deployment monitoring.

• Automation and Actionability: Safeguards are coupled with automated mechanisms (alerts, rollout gates, instant rollbacks) that act instantly when anomalies or regressions are detected.

These principles provide both deployment safety and mitigation strategies in case something goes wrong post-deployment.

After the principles, the post is almost like a summary of what a good MLOps book should cover in terms of four categories with regards to ML Lifecycle:

1. Data and Feature Engineering

The earliest line of defense is ensuring feature-level consistency:

• Null Handling: Uber enforces consistent representation of nulls and requires identical imputation logic in both training and serving. This eliminates subtle sources of schema or value drift.

• Schema Validation: Michelangelo’s pipelines automatically check data schemas for type mismatches, distribution shifts, and cardinality changes before they affect downstream systems.

2. Model Training and Operational Robustness

Training is tightly linked with operational baselines:

• Distributional Statistics: Offline training computes and records key statistics (percentiles, averages, null rates) for every numerical feature.

• Latency Budgets: All model architectures are vetted against strict latency budgets to ensure serving environments remain performant.

• Standardized Reporting: Each run produces model reports containing data-quality metrics, feature-importance rankings, and core performance stats—these are the basis for “deployment readiness” reviews.

3. Robust pre-Production Validation

• Backtesting: Models are validated against historical production data to catch regressions invisible in aggregate metrics.

• Shadow Testing: Most critical use cases undergo “shadow” deployments, in which new models run in parallel with production, feeding on the same live input—outputs are compared in real time, but only the production model affects users. This is increasingly required by default for online models.

4. Safe and Controlled Rollout

• Phased Rollout: Models are first deployed to a “traffic slice” with continuous monitoring. Any breach of error, latency, or resource thresholds triggers auto-rollback to the last approved version.

• Fallback Plans: Every rollout includes predefined fallback and contingency measures to ensure business continuity.

Monitoring & Observability

For monitoring and observability, they use extensively Hue to both monitor and create alerts on the operational metrics such as availability, latency as well as prediction or model quality metrics such as calibration, entropy.

• Operational Metrics: Availability, latency, throughput.

• Prediction Metrics: Score distributions, calibration, entropy.

• Feature Health: Null rates, drift detection via statistical tests, online-offline feature parity checks.

• Automated Alerts and Promotion Gates: If thresholds are breached (especially in high-risk cases), promotions can be blocked autonomously until remediated.

• “Hue” System: Separates data profiling from monitoring so teams can compare live and historical feature distributions, supporting slicing and debugging at Uber scale (Flink and Pinot power this stream analytics).

• Automated data-quality checks, system monitoring, and gradual rollout with rollback are default for every model.

• Continuous “offline” and “real-time” validation, plugged seamlessly into Michelangelo, rapidly highlights regressions—this means issues that once took days to be detected now surface in hours or less.

• Model Safety Deployment Scoring: Uber tracks four key aspects for each model family:

  • Offline evaluation coverage

  • Shadow deployment coverage

  • Unit-test coverage (for both monorepo and new lines)

  • Active monitoring and alerting coverage

• These metrics form a transparent safety score showing a family’s overall readiness for deployment, driving continuous improvement and adoption.

Libraries

Skyvern automates browser-based workflows using LLMs and computer vision. It provides a simple API endpoint to fully automate manual workflows on a large number of websites, replacing brittle or unreliable automation solutions.

Skyvern was inspired by the Task-Driven autonomous agent design popularized by BabyAGI and AutoGPT -- with one major bonus: we give Skyvern the ability to interact with websites using browser automation libraries like Playwright.

This approach has the following advantages:

1. Skyvern can operate on websites it’s never seen before, as it’s able to map visual elements to actions necessary to complete a workflow, without any customized code

2. Skyvern is resistant to website layout changes, as there are no pre-determined XPaths or other selectors our system is looking for while trying to navigate

3. Skyvern is able to take a single workflow and apply it to a large number of websites, as it’s able to reason through the interactions necessary to complete the workflow

4. Skyvern leverages LLMs to reason through interactions to ensure we can cover complex situations. Examples include:

   1. If you wanted to get an auto insurance quote from Geico, the answer to a common question “Were you eligible to drive at 18?” could be inferred from the driver receiving their license at age 16

   2. If you were doing competitor analysis, it’s understanding that an Arnold Palmer 22 oz can at 7/11 is almost definitely the same product as a 23 oz can at Gopuff

Pico-Banana-400K is a large-scale dataset of ~400K text–image–edit triplets designed to advance research in text-guided image editing.
Each example contains:

• an original image (from Open Images),

• a human-like edit instruction, and

• the edited result generated by Nano-Banana and verified by Gemini-2.5-Pro.

The dataset spans 35 edit operations across 8 semantic categories, covering diverse transformations—from low-level color adjustments to high-level object, scene, and stylistic edits.

MSCCL is an inter-accelerator communication framework that is built on top of NCCL and uses its building blocks to execute custom-written collective communication algorithms. MSCCL vision is to provide a unified, efficient, and scalable framework for executing collective communication algorithms across multiple accelerators.

NVIDIA Inference Xfer Library (NIXL) is targeted for accelerating point to point communications in AI inference frameworks such as NVIDIA Dynamo, while providing an abstraction over various types of memory (e.g., CPU and GPU) and storage (e.g., file, block and object store) through a modular plug-in architecture.

DeepEP is a communication library tailored for Mixture-of-Experts (MoE) and expert parallelism (EP). It provides high-throughput and low-latency all-to-all GPU kernels, which are also known as MoE dispatch and combine. The library also supports low-precision operations, including FP8.

Below the Fold

LightlyStudio is an open-source tool designed to unify your data workflows from curation, annotation and management in a single tool. You can work with COCO and ImageNet on a Macbook Pro with M1 and 16GB of memory!

PyScripter is a free and open-source Python Integrated Development Environment (IDE) created with the ambition to become competitive in functionality with commercial Windows-based IDEs available for other languages.

Katakate aims to make it easy to create, manage and orchestrate lightweight safe VM sandboxes for executing untrusted code, at scale. It is built on battle-tested VM isolation with Kata, Firecracker and Kubernetes. It is orignally motivated by AI agents that need to run arbitrary code at scale but it is also great for:

• Custom serverless (like AWS Fargate, but yours)

• Hardened CI/CD runners (no Docker-in-Docker risks)

• Blockchain execution layers for AI dApps

Argilla is a collaboration tool for AI engineers and domain experts who need to build high-quality datasets for their projects.

Trailbase is an open, blazingly fast, single-executable Firebase alternative with type-safe REST & realtime APIs, built-in WebAssembly runtime, SSR, auth and admin UI built on Rust, SQLite & Wasmtime.

Simplify with fewer moving parts: an easy to self-host, single-executable, extensible backend for your mobile, web or desktop application. Sub-millisecond latencies eliminate the need for dedicated caches, no more stale or inconsistent data.

ThinkingMachines wrote a rather interesting and in depth piece on the Modular Manifolds.

The fundamental challenge in training large neural networks lies in maintaining tensor health—ensuring that weights, activations, and gradients neither grow excessively large nor become vanishingly small. Traditional approaches have focused extensively on normalizing activations through techniques like layer normalization and gradient normalization through optimizers like Muon, but weight matrix normalization remains under-explored despite its potential benefits for training stability and predictability.

This post specifically presents a framework “modular manifolds” that constrains weight matrices to specific geometric structures (manifolds) while co-designing optimization algorithms that respect these constraints. This approach promises several advantages:

1. simplified hyperparameter tuning by focusing on tensors whose size matters most,

2. prevention of weight norm explosions,

3. improved matrix conditioning for more predictable behavior

4. facilitation of Lipschitz guarantees for robustness.

Before explaining further, let’s look at some basic concepts especially what Manifold is.

Manifold Concepts

A manifold represents a curved surface that appears locally flat when examined closely, with the locally flat approximation at any point called the tangent space. The key insight is that optimization should occur within the tangent space rather than projecting updates back onto the manifold after each step, ensuring that the learning rate corresponds directly to the actual optimization step length.

The fundamental principle involves three sequential steps for any manifold optimizer:

• finding the unit-length tangent vector that maximizes progress in the gradient direction,

• scaling this direction by the learning rate and updating weights

• retracting the updated weights back onto the manifold.

The choice of distance measurement in the tangent space critically affects the optimal update direction, with non-Euclidean distance measures potentially enabling greater progress toward the gradient direction for fixed step length.

The optimization problem considers constraining a vector parameter to a hypersphere in d-dimensional space, where all points maintain unit Euclidean norm. The tangent space at any point consists of vectors orthogonal to the current position, and the problem turns into a constrained minimization seeking the update direction that minimizes linear change in loss while satisfying both size and tangent constraints.

Using Lagrange multipliers, the optimal update formula emerges as subtracting the radial component from the gradient, normalizing the result, and multiplying by the learning rate. The retraction map, derived through Pythagoras’ theorem, involves dividing updated weights by the square root of one plus the squared learning rate to maintain the spherical constraint.

Extending manifold optimization to matrix parameters requires careful consideration of how weight matrices function within neural networks. Most transformer weight matrices act as “vector-multipliers,” transforming input vectors through matrix multiplication, and their behavior can be understood through singular value decomposition, which reveals how matrices stretch input vectors along different axes.

The Stiefel manifold becomes as the natural choice for matrix constraints, defined as the set of matrices where all singular values equal one, ensuring that the matrix neither excessively amplifies nor diminishes input vectors. For tall matrices (m ≥ n), this manifold can be characterized by the constraint W^T W = I_n, directly generalizing the hyperspherical constraint from vector parameters.

The manifold Muon algorithm combines Stiefel manifold constraints with spectral norm distance measurement to limit the maximum stretching effect of weight updates. This formulation leads to a constrained optimization problem that can be solved through dual ascent methodology, converting the original constrained minimization into an unconstrained maximization problem.

The solution involves introducing Lagrange multipliers and applying a series of mathematical transformations, ultimately yielding a dual problem solvable by gradient ascent. The complete algorithm consists of four steps: running gradient ascent on dual variables to find optimal Lagrange multipliers, computing the optimal update using the matrix sign function applied to a combination of gradients and weighted matrices, applying this update to the weights, and retracting weights back to the manifold using the matrix sign function.

The manifold Muon problem generalizes the hyperspherical case to matrices, seeking update directions that minimize gradient alignment subject to spectral norm constraints and tangent space requirements. The matrix sign function, which normalizes singular values to one, plays a central role in both computing optimal updates and performing manifold retraction.

The dual ascent approach transforms the problem through multiple steps:

• reformulating as a saddle point problem where maximization over Lagrange multipliers penalizes tangent space violations,

• applying trace properties and Sion’s minimax theorem,

• solving the inner minimization to obtain optimal updates parameterized by Lagrange multipliers,

• deriving the dual function involving nuclear norm gradients

The modular manifolds framework addresses the crucial question of how manifold constraints interact when combining layers to build complete networks. Rather than requiring global coordination, the theory enables reasoning about individual layers in isolation while automatically handling inter-layer interactions through a principled abstraction.

Any neural network module is characterized by three mathematical attributes: a forward function mapping from parameter space and input space to output space, a submanifold constraining the weights, and a norm serving as a measuring stick on weight space. For example, a Stiefel-constrained linear module combines linear transformation with Stiefel manifold constraints and spectral norm measurement.

When composing two modules, specific rules ensure that Lipschitz properties are preserved and automatically compiled for the joint system. The new forward function results from composing existing forward functions, the new manifold constraint becomes the Cartesian product of existing manifolds, and the new norm function represents the maximum of existing norm functions weighted by special scalar coefficients.

These scalar coefficients serve as learning rate budgets across layers, emerging naturally from the mathematical structure rather than requiring manual tuning. The composite norm automatically derives separate optimizers for each layer while the coefficients handle learning rate allocation, ensuring that updates respect the overall network’s Lipschitz properties.

Lipschitz Analysis

Budgeting learning rates across layers directly relates to understanding network output sensitivity with respect to weights. Manifold constraints enable much tighter bounds on this sensitivity compared to unconstrained approaches. For instance, a Stiefel-constrained linear module with unit-norm inputs maintains Lipschitz properties with constant one, meaning output changes are bounded by weight perturbation magnitude measured in spectral norm. This Lipschitz statement extends automatically to composed modules following the composition rules, providing principled foundations for scaling weight updates throughout the network. Practical implementation of manifold optimization requires efficient computation of several mathematical operations, particularly the matrix sign function used in manifold Muon. Recent research like the Polar Express algorithm show promise for fast matrix sign computation on GPUs, but further algorithmic innovations may be necessary for large-scale deployment.

The dual ascent approach in manifold Muon involves iterative optimization to solve for Lagrange multipliers, potentially requiring multiple inner iterations per optimization step. This computational overhead must be weighed against benefits of improved conditioning and stability, particularly for large models where training stability is paramount.

Modularity and Architecture Design

The modular manifolds framework opens numerous questions about optimal manifold choices for different network components. Attention heads and embeddings may benefit from different constraint types, and the framework naturally accommodates mixed constraint patterns where some tensors remain unconstrained.

Architecture-optimizer co-design represents a promising direction, where manifold constraints exemplify tight integration between architectural components and optimization algorithms. While hard manifold constraints may not ultimately prove optimal, they demonstrate the potential for discovering new co-design opportunities.

Advanced Optimization Techniques

The dual ascent formulation of manifold Muon suggests opportunities for applying more sophisticated convex optimization techniques to solve dual problems faster and more reliably. Convergence analysis remains an open question, particularly regarding whether improved weight matrix conditioning accelerates overall convergence.

Regularization effects of manifold constraints deserve deeper investigation, including whether constraint radii can be tuned to improve generalization performance. The implicit regularization provided by manifold constraints may offer new approaches to controlling model complexity.

Most manifold optimization research operates within Riemannian geometry where distances derive from inner products. However, neural networks naturally involve operator norms like the spectral norm that do not emerge from inner products, creating non-Riemannian geometric structures with sharp-cornered norm balls and non-unique gradient flows.

This non-Riemannian world may harbor undiscovered optimization principles particularly relevant to neural network training. The geometric structure of neural network weight spaces may provide insights into more effective optimization approaches.

Challenges

Scaling manifold optimization techniques to contemporary large models requires addressing several practical considerations. Efficient GPU implementations of manifold operations remain crucial, particularly for operations like matrix sign computation that may not have optimized implementations in standard deep learning frameworks. Memory overhead associated with storing dual variables and performing additional computations must be balanced against training benefits.

Connection to Existing Optimization Theory

This framework provides a principled approach to understanding how local optimization decisions affect global network behavior through compositional Lipschitz analysis.

The relationship between manifold constraints and existing regularization techniques deserves further exploration, particularly understanding how geometric constraints relate to more familiar penalty-based approaches. The automatic learning rate budgeting emerging from modular manifold composition offers an alternative to manual hyperparameter tuning that may prove more principled and effective.

Impact on Training

By constraining weight matrices to well-conditioned manifolds, the approach may fundamentally alter training dynamics in ways that improve both stability and convergence. The elimination of weight norm explosions and the guarantee of bounded condition numbers create more predictable training behavior that may be particularly valuable for large-scale experiments.

The Lipschitz guarantees provided by manifold constraints offer robustness benefits that extend beyond optimization to model deployment, where bounded sensitivity to input perturbations provides security and reliability advantages.

Libraries

Llama-Factory is a library that provides Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024).

Features

• Various models: LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen, Qwen2-VL, DeepSeek, Yi, Gemma, ChatGLM, Phi, etc.

• Integrated methods: (Continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, etc.

• Scalable resources: 16-bit full-tuning, freeze-tuning, LoRA and 2/3/4/5/6/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ.

• Advanced algorithms: GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ and PiSSA.

• Practical tricks: FlashAttention-2, Unsloth, Liger Kernel, RoPE scaling, NEFTune and rsLoRA.

• Wide tasks: Multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding, etc.

• Experiment monitors: LlamaBoard, TensorBoard, Wandb, MLflow, SwanLab, etc.

• Faster inference: OpenAI-style API, Gradio UI and CLI with vLLM worker or SGLang worker.

Qwen3-Omni is the natively end-to-end multilingual omni-modal foundation models. It processes text, images, audio, and video, and delivers real-time streaming responses in both text and natural speech. We introduce several architectural upgrades to improve performance and efficiency.

SimpleFold, the first flow-matching based protein folding model that solely uses general purpose transformer layers. SimpleFold does not rely on expensive modules like triangle attention or pair representation biases, and is trained via a generative flow-matching objective. SimpleFold can scale to 3B parameters and train it on more than 8.6M distilled protein structures together with experimental PDB data. To the best of researchers’per knowledge, SimpleFold is the largest scale folding model ever developed. On standard folding benchmarks, SimpleFold-3B model achieves competitive performance compared to state-of-the-art baselines. Due to its generative training objective, SimpleFold also demonstrates strong performance in ensemble prediction. SimpleFold challenges the reliance on complex domain-specific architectures designs in folding, highlighting an alternative yet important avenue of progress in protein structure prediction.

Claude Code Router allows you to use Claude Code as the foundation for coding infrastructure, and you can decide how to interact with the model while enjoying updates from Anthropic.

spaCy is a library for advanced Natural Language Processing in Python and Cython. It’s built on the very latest research, and was designed from day one to be used in real products.

• spaCy comes with pretrained pipelines and currently supports tokenization and training for 70+ languages. It features state-of-the-art speed and neural network models for tagging, parsing, named entity recognition, text classification and more, multi-task learning with pretrained transformers like BERT, as well as a production-ready training system and easy model packaging, deployment and workflow management.

Features

• Model Routing: Route requests to different models based on your needs (e.g., background tasks, thinking, long context).

• Multi-Provider Support: Supports various model providers like OpenRouter, DeepSeek, Ollama, Gemini, Volcengine, and SiliconFlow.

• Request/Response Transformation: Customize requests and responses for different providers using transformers.

• Dynamic Model Switching: Switch models on-the-fly within Claude Code using the /model command.

Below the Fold

zoxide is a smarter cd command, inspired by z and autojump.

It remembers which directories you use most frequently, so you can “jump” to them in just a few keystrokes.
zoxide works on all major shells.

• GitHub Actions Integration: Trigger Claude Code tasks in your GitHub workflows.

• Plugin System: Extend functionality with custom transformers.

SSH3 is a complete revisit of the SSH protocol, mapping its semantics on top of the HTTP mechanisms. It comes from our research work and we (researchers) recently proposed it as an Internet-Draft (draft-michel-remote-terminal-http3-00).

In a nutshell, SSH3 uses QUIC+TLS1.3 for secure channel establishment and the HTTP Authorization mechanisms for user authentication. Among others, SSH3 allows the following improvements:

• Significantly faster session establishment

• New HTTP authentication methods such as OAuth 2.0 and OpenID Connect in addition to classical SSH authentication

• Robustness to port scanning attacks: your SSH3 server can be made invisible to other Internet users

• UDP port forwarding in addition to classical TCP port forwarding

• All the features allowed by the modern QUIC protocol: including connection migration (soon) and multipath connections

SWE-Bench Pro is a challenging benchmark evaluating LLMs/Agents on long-horizon software engineering tasks. Given a codebase and an issue, a language model is tasked with generating a patch that resolves the described problem.

OpenAI's recent release of its first open-weight models since GPT-2—gpt-oss-120b and gpt-oss-20b—represents far more than a return to the company's open-source roots. This move signals a defensive strategy that mirrors Google's Android strategy, demonstrating how open-sourcing can paradoxically strengthen competitive moats while appearing to give away the crown jewels for companies.

In this article, I will draw some parallels between OpenAI's pivot to OSS and Google's Android ecosystem reveals how companies can weaponize openness to maintain long-term dominance.

After Google acquired Android, its strategy established the definitive playbook for using open source as a defensive weapon. Google open-sourced Android and the move was anything but altruistic—it was strategic defensive play designed to prevent platform lock-in by competitors. The core was that by commoditizing the mobile operating system layer, Google could ensure its services remained accessible across the entire mobile ecosystem, regardless of hardware manufacturer.

Android's open-source nature created what appeared to be a fragmented market with multiple hardware vendors—Samsung, LG, HTC, and others—all competing on the same platform. However, one thing was same with this level of fragmentation: every Android device became a Google services delivery vehicle. While hardware manufacturers competed on margins, Google maintained control over the profitable layer—search, advertising and later other services. The company traded short-term licensing revenue for long-term platform dominance, ensuring that even as the mobile market exploded, Google's core business remained protected and its distribution of services are not disturbed in a fragmented HW vendors space.

The defensibility wasn't just about market access or distribution; it was about preventing competitive platform lock-in. Had Apple's iOS or Microsoft's Windows Mobile achieved total dominance, Google could have found itself locked out of mobile entirely, with competitors controlling the gateway to billions of users. Android's open architecture made such exclusion impossible while creating powerful network effects that grew stronger with each new device and developer.

Android actually won against iOS in the world with a large market share(Android: 72.13% and iOS: 27.48%) and competed well within USA(58.68% iOS and 41.03% Android).

OpenAI's Strategic Parallel

OpenAI's open-weight model release follows the same strategic logic, adapted for the AI era. Facing mounting pressure from competitors like DeepSeek, Alibaba's Qwen, and other open-source models that deliver "roughly 90% of the performance at a mere 10% of the cost", OpenAI found itself in a position eerily similar to Google in 2005—potentially being outplayed by more open alternatives.

The strategic calculus is clear: rather than lose market share to open-source competitors, OpenAI is commoditizing the foundation model layer while maintaining superiority in the premium tier. The gpt-oss models aren't OpenAI's best technology—that remains with GPT-5 and the closed-source premium offerings. Instead, they're strategic weapons designed to prevent the open-source ecosystem from entirely bypassing OpenAI's influence and defend OpenAI against such pressures.

This approach serves multiple defensive purposes simultaneously. First, it counters the narrative that OpenAI has abandoned its open-source roots, addressing criticism from who accused the company of betraying its founding principles. Second, it provides a competitive alternative to increasingly sophisticated other competitor models.

Most importantly, the open-weight models function as an ecosystem builder. Just as Android attracted developers to Google's platform, gpt-oss models draw AI researchers and developers into OpenAI's orbit, creating network effects around the company's tools, frameworks, and cloud infrastructure. I will expand this a bit more later but especially fine-tuning will become more important to build specialized models, this section might be more and more important as it will allow developers to use a particular operating system to build their applications.

Closed and Open as Complementary Weapons

Both strategies lies in how they balance closed and open components to maximize strategic advantage. Google didn't open-source everything about Android—Google Services, the most valuable components of the ecosystem, remained proprietary. This allowed Google to maintain control over critical functionality while benefiting from the innovation and adoption that open source enables.

OpenAI employs a similar dual strategy architecture. The open-weight gpt-oss models provide a foundation that developers can build upon, but they're explicitly positioned as complementary to OpenAI's premium closed-source offerings. The architecture also includes hybrid capabilities—if the open model encounters a task beyond its capabilities, it can seamlessly call OpenAI's closed-source models in the cloud where the better/more capable model can be used and marketed to the user if the OSS model cannot respond to the task at hand.

This creates a good funnel effect: developers start with the free open models, become embedded in OpenAI's ecosystem, and naturally graduate to premium services as their needs scale or capabilities that lie beyond OSS models. The open models serve as both competitive defense and customer acquisition vehicle, similar to how free Android devices drove users toward Google's profitable services.

Network Effects and Platform Lock-in

Both strategies leverage network effects to create sustainable competitive advantages. Android's success created a virtuous cycle: more devices meant more users, which attracted more developers, which created better applications, which attracted more users. The scale advantages became self-reinforcing, making it increasingly difficult for competitors to challenge Google's position.

OpenAI's open-weight strategy aims to replicate these dynamics in AI. By providing high-quality foundation models under permissive Apache 2.0 licensing, OpenAI encourages widespread adoption and customization. As developers build applications, fine-tune models, and create derivative works, they become increasingly embedded in OpenAI's technical ecosystem.

The network effects extend beyond direct usage. Open models enable OpenAI to benefit from crowdsourced research and development. The global community of developers becomes an unpaid R&D workforce, discovering new applications, identifying limitations, and contributing improvements that OpenAI can incorporate into future releases. This approach provides innovation velocity that even well-funded internal teams struggle to match.

The Strategic Duality

Both strategies lies in their simultaneous offensive and defensive capabilities. Defensively, Android protected Google from mobile platform exclusion while OpenAI's open models protect against commoditization by pure open-source alternatives. Offensively, both strategies actively undermine competitors' business models.

Android's open nature made it difficult for competitors to charge premium prices for mobile operating systems, effectively commoditizing Microsoft and other proprietary mobile platforms. Similarly, OpenAI's open-weight models directly challenge the business models of competitors like Anthropic and other AI companies that rely primarily on API access revenue.

The attack mechanism is particularly sophisticated because it converts competitors' advantages into liabilities. Companies that built their entire business around selling access to AI models suddenly face competition from free, high-quality alternatives. This would also prevent and make the new entry to be harder while OpenAI does not lose its competitive moat through its premium offering. Further, OpenAI—with its diversified revenue streams including enterprise contracts, cloud services, and premium features—can absorb the short-term revenue impact while competitors struggle with their core business model disruption.

Strengthening Through Apparent Weakness

Both strategies demonstrate the moat paradox: sometimes the best way to build defensibility is to appear to give it away. By open-sourcing core technology, both Google and OpenAI created deeper, more sustainable competitive advantages than traditional proprietary approaches could provide.

The key insight is that technology itself is not the ultimate moat. True defensibility comes from network effects, data advantages, talent concentration, and ecosystem lock-in. Open source can actually strengthen these deeper moats by accelerating adoption, creating developer loyalty, and establishing technology standards that competitors must follow rather than lead.

Google's Android strategy proved that controlling the platform matters more than controlling the code. OpenAI's approach suggests that in AI, the training infrastructure, and the path to premium capabilities matters more than protecting individual model weights.

Google open sourced a new Gemma 3 270M model, which is a family of open-source models from Google. The main model is a compact model engineered for hyper-efficient AI applications, mainly mobile and applications where power consumption is as important as model accuracy.

Further, the larger Gemma collection is enhanced with other more efficient models, including Gemma 3, Gemma 3 QAT (quantization-aware training), and Gemma 3n—a mobile-oriented, real-time multimodal AI model tailored for edge deployments rather than server based applications.

For LLMs, generally tendency to build and employ ever-larger models under the notion that size equates to capability. Therefore, the larger the model, the better model is. However, there is also an important dimension in LLM application and deployment where efficiency matters a lot, especially in mobile and edge deployments. Specialized, smaller models like Gemma 3 270M offer a leaner, cost-effective alternative, especially for highly defined tasks such as text classification or structured data extraction for applications that do not require a lot of capabilities that large models offer while still can do an excellent job on these applications that are deployed in mobile and edge.

Gemma 3 270M is designed as a fine-tuning-friendly, compact foundation model with a parameter count of 270 million. It is intentionally positioned as the starting point for building task-specific AI agents—so-called “small, specialized models”—each narrowly focused and highly optimized for its domain. The underlying philosophy is to enable developers to create modular AI systems, each with targeted abilities, rather than burdening a single, large model with many disparate tasks.

It is good for:

• Fast, accurate text classification

• Data extraction from documents

• Instruction-following and text structuring tasks

• Multimodal applications when paired with additional assets (e.g., for mobile or edge deployment)

Gemma 3 270M: Technical Overview

• Parameter Count: 270 million—this makes the model light compared to mainstream large language models (LLMs) that often contain billions or tens of billions of parameters. Compare this with GPT-OSS-120B from OpenAI which has about 117 Billion parameters.

• Fine-Tuning Paradigm Shift

  • Gemma 3 270M is specifically designed for rapid, efficient task-specific fine-tuning. Its size and robust pre-training make it ideal for organizations aiming to deploy multiple narrow models—sometimes called “expert models”—each optimized for a particular workflow.

  • This approach is consistent with growing trends in AI, where “model specialization” outperforms attempts to cover all needs with a single, general-purpose foundation model.

• Instruction Structuring and Out-of-Box Quality

  • Larger models are often cumbersome and require extensive re-training or adaptation to follow instructions accurately. In contrast, Gemma 3 270M offers high-fidelity instruction-following without extra overhead. The architecture incorporates prior learnings from instruction tuning, reinforcement with human feedback, and possibly synthetic data training strategies, ensuring reliable zero-shot and few-shot performance for common tasks.

• Efficiency and Cost-Effectiveness

  • Smaller parameter count translates into significantly lower compute, memory, and energy requirements—crucial for both enterprise-scale deployments (where inference costs scale rapidly with usage) and on-device AI scenarios.

  • Gemma 3 270M’s architecture likely employs advanced quantization and pruning techniques for further efficiency, although the article references QAT (in Gemma 3 QAT) but doesn’t specify whether this model is pre-quantized.

• Edge and Mobile Compatibility Through Shared Architecture

  • By building on the same architecture as Gemma 3 and Gemma 3n, Google ensures horizontal portability—models can be trained and then later adapted for smaller, even mobile-specific variants.

  • The efficient transformer backbone, fused attention mechanisms, and potential optimizations for ARM or edge hardware mean developers can expect real-time inference, even on non-datacenter hardware.

The models are available in HuggingFace, you can also experiment and try out the models in the Google Vertex as well. Fine-tuning on Gemma models is also available as a notebook.

Libraries

POML (Prompt Orchestration Markup Language) is a novel markup language designed to bring structure, maintainability, and versatility to advanced prompt engineering for Large Language Models (LLMs). It addresses common challenges in prompt development, such as lack of structure, complex data integration, format sensitivity, and inadequate tooling. POML provides a systematic way to organize prompt components, integrate diverse data types seamlessly, and manage presentation variations, empowering developers to create more sophisticated and reliable LLM applications.

LangExtract is a Python library that uses LLMs to extract structured information from unstructured text documents based on user-defined instructions. It processes materials such as clinical notes or reports, identifying and organizing key details while ensuring the extracted data corresponds to the source text.

1. Precise Source Grounding: Maps every extraction to its exact location in the source text, enabling visual highlighting for easy traceability and verification.

2. Reliable Structured Outputs: Enforces a consistent output schema based on your few-shot examples, leveraging controlled generation in supported models like Gemini to guarantee robust, structured results.

3. Optimized for Long Documents: Overcomes the "needle-in-a-haystack" challenge of large document extraction by using an optimized strategy of text chunking, parallel processing, and multiple passes for higher recall.

4. Interactive Visualization: Instantly generates a self-contained, interactive HTML file to visualize and review thousands of extracted entities in their original context.

5. Flexible LLM Support: Supports your preferred models, from cloud-based LLMs like the Google Gemini family to local open-source models via the built-in Ollama interface.

6. Adaptable to Any Domain: Define extraction tasks for any domain using just a few examples. LangExtract adapts to your needs without requiring any model fine-tuning.

7. Leverages LLM World Knowledge: Utilize precise prompt wording and few-shot examples to influence how the extraction task may utilize LLM knowledge. The accuracy of any inferred information and its adherence to the task specification are contingent upon the selected LLM, the complexity of the task, the clarity of the prompt instructions, and the nature of the prompt examples.

Meka Agent is an open-source, autonomous computer-using agent that delivers state-of-the-art browsing capabilities. The agent works and acts in the same way humans do, by purely using vision as its eyes and acting within a full computer context.

It is designed as a simple, extensible, and customizable framework, allowing flexibility in the choice of models, tools, and infrastructure providers.

The agent primarily focuses on web browsing today, and achieves state-of-the-art benchmark results in the WebArena Benchmark (72.7%).

TraceRoot is an open-source debugging platform that helps engineers fix production issues 10x faster by combining structured traces, logs, and source code context with AI-powered analysis.

Below The Fold

Optician Sans is a free typeface based on the 10 historical optotype letters seen on millions of eye charts around the world, finalising the work that was started decades ago.

The universal optometrist eye charts have been around for decades and seen by millions of people worldwide. The first standardised chart was created by Hermann Snellen in the Netherlands in 1862, before Louise Sloan designed a new set in 1959. These letters make up the now universal chart for testing visual acuity, also known as the LogMAR chart, which was developed by National Vision Research Institute of Australia. But ever since 1959, they have consisted of only 10 letters.

Optician Sans is a fully functional typeface based on the historical optotype letters, including numbers and special characters.

PR Quiz is a GitHub Action that uses AI to generate a quiz based on a pull request. It can help you, the human reviewer, test your understanding of your AI Agent's code. (And block you from deploying code you don't understand!)

copyparty turns almost any device into a file server with resumable uploads/downloads using any web browser

Kitten TTS 😻 is an open-source realistic text-to-speech model with just 15 million parameters, designed for lightweight deployment and high-quality voice synthesis.

Articles

Google wrote about a new agent framework for solving common machine learning tasks called MLE-STAR,(Machine Learning Engineering via Search and Targeted Refinement) which is a general multi-agent framework for building automated machine learning engineering agents that can generate complete ML solutions from task descriptions and datasets end to end.

The research mainly addresses critical limitations in existing Large Language Model (LLM)-based MLE agents, which rely heavily on internal LLM knowledge and employ coarse exploration strategies that modify entire code structures at once, limiting their ability to select effective task-specific models and perform deep exploration within specific components. This all or nothing approach does not build an iterative solution and sometimes it is ineffective of solving a subcomponent of a problem.

This research addresses some of the limitations through general(not task specific) multi-agent framework that distinguishes from AutoML type of approaches where the search space is much more limited and have to be very well defined(both in terms of problem statement and a possible solution among all of the possible paths).

Web Search

One of the main advantage of MLE-STAR's integration of web search as a tool for retrieving state-of-the-art models is ability to fetch the most of the recent information about a particular problem rather than solely depending on LLM's internal knowledge. This addresses the common problem where LLMs propose outdated models (like logistic regression for text classification tasks) due to bias toward familiar patterns from pre-training data. This also becomes common for reasoning tasks especially for research oriented and study oriented tasks where model does not have all of the information that is trained on, but can use the information at the “test time” to reason about and curate a good answer even though it does not necessarily know this information in the “training time”.

In order to do that, they have the following execution path:

• Uses Google Search to retrieve M effective models for given tasks

• Retrieves both model descriptions (T_model) and corresponding example code (T_code)

• A retriever agent (A_retriever) generates structured JSON outputs containing model information

• Retrieved models are evaluated by a candidate evaluation agent (A_init) that generates executable Python scripts

Refinement

Unlike existing approaches that modify entire code structures, MLE-STAR introduces a nested loop refinement system that targets specific ML pipeline components.

Outer Loop - Component Identification:

• Performs ablation studies using an ablation study agent (A_abl) to identify which code blocks have the most significant impact on performance

• A summarization module (A_summarize) processes ablation results

• An extractor module (A_extractor) identifies the critical code block (c_t) for refinement and generates initial plans (p_0)

Inner Loop - Iterative Refinement:

• A coding agent (A_coder) implements refinement plans

• A planning agent (A_planner) proposes new strategies based on performance feedback from previous attempts

• Process continues for K iterations, selecting the best-performing candidate

• Outer loop repeats T times targeting different critical code blocks

3. Orchestration of Ensemble Selection

MLE-STAR has an ensemble method that goes beyond simple voting or averaging. Instead of selecting the single best solution, it follows:

• Generates multiple candidate solutions from parallel runs

• Uses an ensemble strategy planner (A_ens_planner) to propose effective combination methods

• An ensembler agent (A_ensembler) implements these strategies

• Iteratively refines ensemble approaches based on performance feedback over R iterations

Through this, it can optimize the selection after multiple possible paths and solutions are proposed and still optimize these paths through the ensembler agent.

If we were to summarize this multi-agent system, we can outline 13 specialized agents that interact with each other with slightly different objective functions while optimizing the problem at the hand(problem definition is in the next section):

1. Retriever Agent (A_retriever): Searches web for relevant models

2. Candidate Evaluation Agent (A_init): Generates and evaluates initial solutions

3. Merging Agent (A_merger): Combines multiple models into initial solution

4. Ablation Study Agent (A_abl): Creates ablation study code for component analysis

5. Summarization Agent (A_summarize): Processes ablation study results

6. Extractor Agent (A_extractor): Identifies critical code blocks for refinement

7. Coder Agent (A_coder): Implements code refinements

8. Planner Agent (A_planner): Proposes refinement strategies

9. Ensemble Strategy Planner (A_ens_planner): Plans ensemble combinations

10. Ensembler Agent (A_ensembler): Implements ensemble strategies

11. Debugging Agent (A_debugger): Fixes code execution errors

12. Data Leakage Checker (A_leakage): Prevents test data contamination

13. Data Usage Checker (A_data): Ensures all provided data is utilized

Problem Formulation

The system formally defines the optimization problem as finding s* = argmax(h(s)), where S is the space of possible Python script solutions and h is a performance score function. The multi-agent framework takes datasets D and task descriptions T_task as input, working across any data modalities (tabular, image, text, audio) and task types (classification, regression, sequence-to-sequence).

Experiments were conducted on 22 Kaggle competitions from MLE-bench Lite, using Gemini-2.0-Flash and Gemini-2.5-Pro as base models. The system retrieves 4 model candidates, performs 4 inner refinement loops across 4 outer loops, and explores ensemble strategies for 5 rounds within a 24-hour time limit.

Medal Achievement Rates:

• MLE-STAR with Gemini-2.5-Pro: 63.6% medal achievement rate (36.4% gold medals)

• MLE-STAR with Gemini-2.0-Flash: 43.9% medal achievement rate (30.3% gold medals)

• AIDE baseline with Gemini-2.0-Flash: 25.8% medal achievement rate (12.1% gold medals)

This represents an 18+ percentage point improvement in overall medal achievement and a 147% improvement in gold medal achievement over the best baseline.

The paper also has a lot more details on agents as well as the benchmarks.

Libraries

TinyTroupe is an experimental Python library that allows the simulation of people with specific personalities, interests, and goals. These artificial agents - TinyPersons - can listen to us and one another, reply back, and go about their lives in simulated TinyWorld environments. This is achieved by leveraging the power of Large Language Models (LLMs), notably GPT-4, to generate realistic simulated behavior. This allows us to investigate a wide range of convincing interactions and consumer types, with highly customizable personas, under conditions of our choosing. The focus is thus on understanding human behavior and not on directly supporting it (like, say, AI assistants do) -- this results in, among other things, specialized mechanisms that make sense only in a simulation setting. Further, unlike other game-like LLM-based simulation approaches, TinyTroupe aims at enlightening productivity and business scenarios, thereby contributing to more successful projects and products. Here are some application ideas to enhance human imagination:

• Advertisement: TinyTroupe can evaluate digital ads (e.g., Bing Ads) offline with a simulated audience before spending money on them!

• Software Testing: TinyTroupe can provide test input to systems (e.g., search engines, chatbots or copilots) and then evaluate the results.

• Training and exploratory data: TinyTroupe can generate realistic synthetic data that can be later used to train models or be subject to opportunity analyses.

• Product and project management: TinyTroupe can read project or product proposals and give feedback from the perspective of specific personas (e.g., physicians, lawyers, and knowledge workers in general).

• Brainstorming: TinyTroupe can simulate focus groups and deliver great product feedback at a fraction of the cost!

Torchleet is a leetcode for Pytorch. TorchLeet is broken into two sets of questions:

1. Question Set: A collection of PyTorch practice problems, ranging from basic to hard, designed to enhance your skills in deep learning and PyTorch.

2. LLM Set: A new set of questions focused on understanding and implementing Large Language Models (LLMs) from scratch, including attention mechanisms, embeddings, and more.

Positron is:

• A next-generation data science IDE built by Posit PBC

• An extensible, polyglot tool for writing code and exploring data

• A familiar environment for reproducible authoring and publishing

Arch handles the pesky low-level work in building agentic apps — like applying guardrails, clarifying vague user input, routing prompts to the right agent, and unifying access to any LLM. It’s a language and framework friendly infrastructure layer designed to help you build and ship agentic apps faster.

🌐 MCP-Use is the open source way to connect any LLM to any MCP server and build custom MCP agents that have tool access, without using closed source or application clients.

💡 Let developers easily connect any LLM to tools like web browsing, file operations, and more.

• If you want to get started quickly check out mcp-use.com website to build and deploy agents with your favorite MCP servers.

• Visit the mcp-use docs to get started with mcp-use library

• For the TypeScript version, visit mcp-use-ts

Below The Fold

any-llm is a single interface to use different llm providers from Mozilla, which offers:

• Simple, unified interface - one function for all providers, switch models with just a string change

• Developer friendly - full type hints for better IDE support and clear, actionable error messages

• Leverages official provider SDKs when available, reducing maintenance burden and ensuring compatibility

• Stays framework-agnostic so it can be used across different projects and use cases

• Actively maintained - we use this in our own product (any-agent) ensuring continued support

• No Proxy or Gateway server required so you don't need to deal with setting up any other service to talk to whichever LLM provider you need.

RisingWave is a stream processing and management platform designed to offer the simplest and most cost-effective way to process, analyze, and manage real-time event data — with built-in support for the Apache Iceberg™ open table format. It provides both a Postgres-compatible SQL interface and a DataFrame-style Python interface.

RisingWave can ingest millions of events per second, continuously join and analyze live streams with historical data, serve ad-hoc queries at low latency, and persist fresh, consistent results to Apache Iceberg™ or any other downstream system.

Manticore Search is an easy-to-use, open-source, and fast database designed for search. It is a great alternative to Elasticsearch.

vet is a command-line tool that acts as a safety net for the common but risky curl | bash pattern. It lets you inspect remote scripts for changes, run them through a linter, and require your explicit approval before they can execute.

libpostal is a C library for parsing/normalizing street addresses around the world using statistical NLP and open data. The goal of this project is to understand location-based strings in every language, everywhere. For a more comprehensive overview of the research behind libpostal, be sure to check out the (lengthy) introductory blog posts:

• Original post: Statistical NLP on OpenStreetMap

• Follow-up for 1.0 release: Statistical NLP on OpenStreetMap: Part 2

Jujutsu is a powerful version control system for software projects. You use it to get a copy of your code, track changes to the code, and finally publish those changes for others to see and use. It is designed from the ground up to be easy to use—whether you're new or experienced, working on brand new projects alone, or large scale software projects with large histories and teams.

Jujutsu is unlike most other systems, because internally it abstracts the user interface and version control algorithms from the storage systems used to serve your content. This allows it to serve as a VCS with many possible physical backends, that may have their own data or networking models—like Mercurial or Breezy, or hybrid systems like Google's cloud-based design, Piper/CitC.

Notesnook is a free (as in speech) & open-source note-taking app focused on user privacy & ease of use.

Notesnook is our proof that privacy does not (always) have to come at the cost of convenience. We aim to provide users peace of mind & 100% confidence that their notes are safe and secure. The decision to go fully open source is one of the most crucial steps towards that.

ZUSE is a sleek, minimal IRC client for your terminal. Built with Go and powered by the elegant Bubble Tea framework. Chat faster, cleaner, and without distractions right from your terminal.

Google wrote an article on LSM-2, a foundation model designed for wearable sensor data, leveraging a novel self-supervised learning (SSL) framework called Adaptive and Inherited Masking (AIM). This approach addresses the challenge of incomplete data that is common in real-world wearable devices, due to device removal, charging, motion artifacts, and other disruptions.

Unlike traditional SSL methods that require complete data or rely on imputation and filtering—which introduce biases or waste valuable data—AIM treats missing data as a natural, informative aspect of sensor streams, enabling the model to learn directly from incomplete data without explicit imputation.

AIM extends masked autoencoder (MAE) pre-training by introducing a dual masking strategy that combines token dropout and attention masking. Token dropout improves computational efficiency by dropping a fixed number of masked tokens (segments of missing or deliberately masked data) before the encoder stage, while attention masking handles the variable and unpredictable additional tokens that remain masked based on real-world missingness. This technique effectively balances reducing input sequence length with maintaining model capacity to handle naturally fragmented data streams. During fine-tuning and evaluation phases, AIM applies attention masking exclusively to naturally missing tokens, allowing the model to generalize and robustly process variable data fragmentation.

Adaptive and Inherited Masking (AIM) Capabilities:

• Dual Masking: AIM distinguishes between inherited (naturally missing) tokens, reflecting real sensor gaps, and artificial mask tokens, deliberately applied for self-supervised learning objectives.

• Token Dropout: During pre-training, a fixed number of masked tokens (from both inherited and artificial types) are dropped before the encoder, reducing computational burden by shortening the effective sequence length.

• Attention Masking: Remaining masked tokens are handled within the encoder's transformer blocks via explicit attention masks—ensuring the attention mechanism ignores these tokens during representation learning. This allows the model to process any configuration of missing data.

• Unified Treatment: AIM treats inherited and artificial mask tokens equivalently at the transformer level, teaching the model to reason about and reconstruct both types, enabling robustness to random and structured data fragmentation.

Pre-training and Fine-tuning:

• During SSL pre-training, AIM intersperses reconstructed and naturally missing tokens. The model learns to generate latent representations that encode both the underlying data structure and the distribution of missingness.

• At fine-tuning and deployment, only inherited masks (i.e., the real-world missing entries) are present; attention masking ensures seamless adaptation to any fragmentation pattern, a vast improvement over imputation-based or aggressive filtering techniques.

Head Layer Probes:
After the transformer backbone, task-specific linear probes are used for evaluations:

• Classification Tasks: Embeddings are averaged across observed tokens as the input to a linear classifier.

• Regression Tasks: A similar approach supplies inputs to a linear regressor.

• Generative Tasks: The model reconstructs missing input regions by generating output directly for masked locations.

Google had an extensive multimodal wearable dataset:

• Size: 40M hours of data from 60,000+ participants over three months (March–May 2024).

• Device Heterogeneity: Devices include Fitbit, Google Pixel watches, and trackers with multiple sensor channels (e.g., heart rate, accelerometry, EDA, temperature).

• Labeling: All data are anonymized. Meta-data (user annotations for 20 activities, self-reported physical/mental health, demographics) collected in parallel.

• Partitioning: Data from each user appears in either training, fine-tuning, or evaluation splits, but never more than one, strictly preventing data leakage and ensuring reliable generalization tests.

Data Preprocessing and Ingestion Scheme:

• Tokenization: Raw continuous time-series from various sensors are discretized into non-overlapping time-channel patches, permitting efficient batching and masking within the transformer—standardizing sequences with variable sampling rates and occasional dropout artifacts.

• Missingness Quantification: Every input sequence is annotated with a precise missingness map, denoting which patches result from real-world sensor gaps. This map governs inherited masking throughout the pipeline.

• No Imputation: Critically, AIM and LSM-2 do not perform statistical or model-based imputation; instead, every instance of missingness (whether due to device-off, motion artifacts, or signal drop) is treated as valuable, information-bearing structure.

Efficient Large-scale Ingestion:

• Incoming data are validated and indexed with hash-based deduplication to prevent redundant storage.

• Meta-data and sensor streams are stored in a format optimized for parallel readout and random-access, crucial for efficient construction of masked batches for SSL training.

• Custom ingestion pipelines ensure batch construction correctly respects subject splits and missing annotations.

It performs relatively well in a number of tasks:

• Generative Tasks: LSM-2 achieves up to 77% better mean squared error (MSE) reconstruction of missing sensor signals than LSM-1 under high (80%) missingness, excelling in random imputation, interpolation, extrapolation, and channel-missing scenarios.

• Classification and Regression: LSM-2 outperforms not just LSM-1 but also supervised baselines in 20-class activity classification and binary health conditions (hypertension, anxiety), with robust feature generalization to new tasks via linear probes.

• Scaling: Unlike LSM-1, LSM-2’s performance scales almost linearly with increases in data, subjects, compute, and model capacity, showing no evidence of saturation

Libraries

Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model with 32 billion activated parameters and 1 trillion total parameters. Trained with the Muon optimizer, Kimi K2 achieves exceptional performance across frontier knowledge, reasoning, and coding tasks while being meticulously optimized for agentic capabilities.

Key Features

• Large-Scale Training: Pre-trained a 1T parameter MoE model on 15.5T tokens with zero training instability.

• MuonClip Optimizer: We apply the Muon optimizer to an unprecedented scale, and develop novel optimization techniques to resolve instabilities while scaling up.

• Agentic Intelligence: Specifically designed for tool use, reasoning, and autonomous problem-solving.

Model Variants

• Kimi-K2-Base: The foundation model, a strong start for researchers and builders who want full control for fine-tuning and custom solutions.

• Kimi-K2-Instruct: The post-trained model best for drop-in, general-purpose chat and agentic experiences. It is a reflex-grade model without long thinking.

Contextual Observation & Recall Engine(C.O.R.E) is a portable memory graph built from your llm interactions and personal data, making all your context and workflow history accessible to any AI tool, just like a digital brain. This eliminates the need for repeated context sharing . The aim is to provide:

• Unified, Portable Memory: Add and recall context seamlessly, and connect your memory across apps like Claude, Cursor, Windsurf and more.

• Relational, Not just Flat Facts: CORE organizes your knowledge, storing both facts and relationships for a deeper richer memory like a real brain.

• User Owned: You decide what to keep, update or delete and share your memory across the tool you want and be freed from vendor lock-in.

SymbolicAI is a neuro-symbolic framework, combining classical Python programming with the differentiable, programmable nature of LLMs in a way that actually feels natural in Python. It's built to not stand in the way of your ambitions. It's easily extensible and customizable to your needs by virtue of its modular design. It's quite easy to write your own engine, host locally an engine of your choice, or interface with tools like web search or image generation. To keep things concise in this README, we'll introduce two key concepts that define SymbolicAI: primitives and contracts.

Arch handles the pesky low-level work in building agentic apps — like applying guardrails, clarifying vague user input, routing prompts to the right agent, and unifying access to any LLM. It’s a language and framework friendly infrastructure layer designed to help you build and ship agentic apps faster.

Metaflow is a human-centric framework designed to help scientists and engineers build and manage real-life AI and ML systems. Serving teams of all sizes and scale, Metaflow streamlines the entire development lifecycle—from rapid prototyping in notebooks to reliable, maintainable production deployments—enabling teams to iterate quickly and deliver robust systems efficiently.

Originally developed at Netflix and now supported by Outerbounds, Metaflow is designed to boost the productivity for research and engineering teams working on a wide variety of projects, from classical statistics to state-of-the-art deep learning and foundation models. By unifying code, data, and compute at every stage, Metaflow ensures seamless, end-to-end management of real-world AI and ML systems.

The code is available in GitHub.

MedGemma is a collection of Gemma 3 variants that are trained for performance on medical text and image comprehension. Developers can use MedGemma to accelerate building healthcare-based AI applications. MedGemma comes in two variants: a 4B multimodal version and a 27B text-only version.

MedGemma 4B utilizes a SigLIP image encoder that has been specifically pre-trained on a variety of de-identified medical data, including chest X-rays, dermatology images, ophthalmology images, and histopathology slides. Its LLM component is trained on a diverse set of medical data, including radiology images, histopathology patches, ophthalmology images, dermatology images, and medical text.

goose is your on-machine AI agent, capable of automating complex development tasks from start to finish. More than just code suggestions, goose can build entire projects from scratch, write and execute code, debug failures, orchestrate workflows, and interact with external APIs - autonomously.

Whether you're prototyping an idea, refining existing code, or managing intricate engineering pipelines, goose adapts to your workflow and executes tasks with precision.

Designed for maximum flexibility, goose works with any LLM and supports multi-model configuration to optimize performance and cost, seamlessly integrates with MCP servers, and is available as both a desktop app as well as CLI - making it the ultimate AI assistant for developers who want to move faster and focus on innovation.

Below The Fold

Sirius is a GPU-native SQL engine. It plugs into existing databases such as DuckDB via the standard Substrait query format, requiring no query rewrites or major system changes.

BloomSearch provides extremely low memory usage and low cold-start searches through pluggable storage interfaces.

• Memory efficient: Bloom filters have constant size regardless of data volume

• Pluggable storage: DataStore and MetaStore interfaces for any backend (can be same or separate)

• Fast filtering: Hierarchical pruning via partitions, minmax indexes, and bloom filters

• Flexible queries: Search by field, token, or field:token with AND/OR combinators

• Disaggregated storage and compute: Unbound ingest and query throughput

atopile is a language, compiler and toolchain to design electronics with code.

Design circuit boards with the same powerful workflows that software developers use - version control, modularity, and automated validation. Instead of point-and-click schematics, use human-readable .ato files that can be version controlled and shared. Capture design intelligence and validation rules in code to ensure your hardware works as intended. QuickStart page is pretty good if you want to get started right away.

Containerization package allows applications to use Linux containers. Containerization is written in Swift and uses Virtualization.framework on Apple silicon.

Stress-Terminal UI, s-tui, monitors CPU temperature, frequency, power and utilization in a graphical way from the terminal.

Maestro is a general-purpose workflow orchestrator that provides a fully managed workflow-as-a-service (WAAS) to the data platform users at Netflix.

It serves thousands of users, including data scientists, data engineers, machine learning engineers, software engineers, content producers, and business analysts, for various use cases. It schedules hundreds of thousands of workflows, millions of jobs every day and operates with a strict SLO even when there are spikes in the traffic. Maestro is highly scalable and extensible to support existing and new use cases and offers enhanced usability to end users.

The library is available in GitHub.

More To Read

• From a previous employee on his time spent in OpenAI as reflections.

• A good blog post on going through some unknown “artisanal” git commands and shows basic prompts and commands in a funny way.

SmolLM3 is a fully open 3 billion-parameter transformer decoder optimized for efficiency, multilingual support, long-context reasoning, and dual-mode instruction from HuggingFace.

HuggingFace wrote a rather detailed a blog post on both some of the model architecture changes that they have done and some of the mid-training and post-training strategies.

Trained on 11.2 trillion tokens, it outperforms other 3B models (Llama-3.2-3B, Qwen2.5-3B) and competes with larger 4B models (Qwen3-4B, Gemma3-4B) while offering a complete “blueprint”—architecture, data recipes, and fine-tuning methodology—to empower researchers and engineers.

Model Overview

Over 2000 feet, the model would look like something like this in terms of properties and capabilities.

• Scale & Performance

  • 3 B parameters; trained on 11.2 T tokens with a three-stage web/code/math curriculum.

  • Win-rate leadership on 12 benchmarks (HellaSwag, ARC, Winogrande, CommonsenseQA, MMLU-CF/Pro, PIQA, OpenBookQA, GSM8K, MATH, HumanEval+, MBPP+) and competitive parity with 4 B models in knowledge, reasoning, math, and coding tasks.

• Multilingual

  • Supports six languages (English, French, Spanish, German, Italian, Portuguese).

  • Evaluated on Global MMLU, MLMM HellaSwag, Flores-200, Belebele.

• Long-Context

  • Native context window extended to 64 K tokens via staged RoPE theta adjustments.

  • Extrapolation to 128 K tokens at inference using YARN.

• Dual-Mode Instruct Model

  • /think (reasoning) vs. /no_think (direct answer) flags enable explicit or collapsed reasoning traces.

  • Supports XML and Python tool-calling interfaces.

Architectural Enhancements

There are interesting multiple adjustments to the model architecture and changes to the traditional methods that are being used and outlined below:

• Grouped Query Attention (GQA): Replaces standard multi-head attention with four query groups, matching task performance while reducing KV cache memory during inference.

• NoPE (No Position Embeddings): Adopts a hybrid attention strategy by omitting rotary position embeddings in every fourth layer, significantly boosting long-context fidelity without degrading short-context capabilities.

• Intra-Document Masking: Ensures tokens from distinct documents within the same batch don’t attend to each other, improving training stability for long sequences.

• Training Stability Adjustments: Removes weight decay from embedding layers (inspired by OLMo 2), stabilizing embedding norms and overall training dynamics.

Mid-Training and Post-Training Strategies

Long-Context Extension

Additional 100 B tokens in two phases:

• 4 K → 32 K (RoPE θ = 1.5 M)

• 32 K → 64 K (RoPE θ = 5 M)
  Both phases upsampled long-document code, books, and web data, with ablations confirming sufficient long-context gains without explicit long-document upsampling.

General Reasoning Injection

Trained on 35 B reasoning-trace tokens (OpenThoughts3, Llama-Nemotron-Post-Training) via ChatML template and packing for 4 epochs (140 B tokens), producing a reasoning-capable mid-training checkpoint for fine-tuning.

Supervised Fine-Tuning (SFT)

1.8 B token mixture (1 B non-reasoning, 0.8 B reasoning) across 22 datasets, balancing domains: math, code, general reasoning, instruction, multilinguality, tool calling. Synthetic reasoning traces generated by Qwen3-32B for underrepresented domains. Trained 4 epochs (~8 B tokens) with BFD packing; aligned with user-turn masking and tool-output masking.

Alignment via Anchored Preference Optimization (APO)

APO (variant of Direct Preference Optimization) leverages Tulu3 and synthetic reasoning preference pairs (chosen: Qwen3-32B; rejected: Qwen3-0.6B) to stabilize optimization and improve downstream performance. Ablations revealed reasoning training slightly degraded long-context scores, prompting mitigation via model merging.

Model Merging

Using MergeKit, a linear blend (0.9 reasoning-aligned “soup” + 0.1 long-context mid-training checkpoint) restored RULER 128 K performance while retaining reasoning gains.

MUVERA (Multi-Vector Retrieval Algorithm) from Google Research is a method in information retrieval (IR) that enables the speed and efficiency of single-vector search while preserving the accuracy and expressiveness of multi-vector models. This advance is critical for search engines, recommender systems, and natural language processing, where capturing nuanced semantic relationships is essential but computational cost has historically limited practical deployment of multi-vector approaches. Below is a detailed, technical summary of the MUVERA system, its theoretical foundations, empirical results, and implications for large-scale IR.

Single-Vector Embeddings and MIPS

Traditional neural embedding models (e.g., BERT-based bi-encoders) represent each data point—such as a document or query—as a single vector in a high-dimensional space. Retrieval is performed by measuring the inner product (dot product) between the query vector and document vectors. This allows for Maximum Inner Product Search (MIPS), a well-studied problem with highly optimized algorithms and infrastructure. The main advantages of this approach are:

• Speed: Single-vector MIPS can be performed extremely fast, even at very large scale of document counts(billions to trillions).

• Memory efficiency: Each item is represented by one vector.

• Simplicity: Off-the-shelf search systems can be used.

However, single-vector representations often lose fine-grained semantic information, limiting retrieval accuracy, especially for complex queries or long documents.

Multi-Vector Models

To address these limitations, multi-vector models (notably ColBERT) represent each item as a set of vectors—for example, one vector per token or phrase. Similarity between a query and a document is computed using more expressive functions, such as the Chamfer similarity (sum of maximal similarities between query and document vectors). This approach:

• Greatly improves retrieval accuracy and expressiveness.

• Captures nuanced relationships (e.g., matching specific query terms to document segments).

But, it comes with also major drawbacks:

• Computational cost: For each query-document pair, all pairs of query and document vectors must be compared.

• Latency: Multi-vector similarity is much slower than single-vector MIPS.

• Scalability: Not suitable for web-scale retrieval without major engineering compromises.

MUVERA: To Solve them All

MUVERA bridges the gap by reducing multi-vector retrieval to single-vector MIPS through a mathematically principled approach.

The central innovation is the construction of Fixed Dimensional Encodings (FDEs). For each query and document (which are originally sets of vectors), MUVERA computes a single vector such that the inner product of these FDEs closely approximates the true multi-vector similarity (e.g., Chamfer similarity).

This transformation is data-oblivious (does not require learning from data), fast, and provably accurate.

MUVERA further provides provable guarantees on the quality of the approximation:

• For any query/document pair, the FDE dot product is an εε-approximation to the true multi-vector similarity.

• This is the first method with provable guarantees for reducing multi-vector similarity search to single-vector MIPS, ensuring that retrieval quality is not sacrificed for speed.

The MUVERA retrieval pipeline consists of two stages:

1. Initial Retrieval: Use the FDEs to perform single-vector MIPS over the corpus, leveraging existing fast and scalable infrastructure.

2. Re-ranking: For the top-k candidates, compute the exact multi-vector similarity (e.g., Chamfer) to ensure final accuracy.

This hybrid approach combines the best of both worlds: fast candidate generation and accurate final ranking.

MUVERA constructs FDEs such that their inner product closely approximates this sum of maximum inner products. The construction uses random projections and aggregation techniques to ensure that the mapping is efficient and preserves the necessary similarity structure.

To further reduce memory and storage requirements, MUVERA incorporates product quantization:

• FDEs are compressed by up to 32× (e.g., a 10,240-dimensional FDE stored in just 1,280 bytes).

• This compression incurs negligible loss in retrieval quality.

• The result is a dramatic reduction in memory footprint, making large-scale deployment feasible.

MUVERA supports asymmetric FDE construction, where queries and documents can be encoded differently to further optimize retrieval performance. This flexibility allows for tuning the system to specific application needs or hardware constraints.

Benchmark

MUVERA was evaluated on several standard IR benchmarks, including the BEIR suite and MS MARCO. Key results:

• Recall: MUVERA achieves the same or better recall as prior state-of-the-art multi-vector heuristics (e.g., PLAID, ColBERT).

• Efficiency: MUVERA retrieves 2–5× fewer candidates for the same recall, reducing computational cost.

• Latency: End-to-end retrieval is up to 90% faster than prior multi-vector systems.

• Memory: With product quantization, MUVERA’s memory usage is a fraction of baseline methods

MUVERA is open-source with implementations available in C++ and Python.

Libraries

Biomni is a general-purpose biomedical AI agent designed to autonomously execute a wide range of research tasks across diverse biomedical subfields. By integrating cutting-edge large language model (LLM) reasoning with retrieval-augmented planning and code-based execution, Biomni helps scientists dramatically enhance research productivity and generate testable hypotheses.

MiniMax-M1 is powered by a hybrid Mixture-of-Experts (MoE) architecture combined with a lightning attention mechanism. The model is developed based on our previous MiniMax-Text-01 model, which contains a total of 456 billion parameters with 45.9 billion parameters activated per token. Consistent with MiniMax-Text-01, the M1 model natively supports a context length of 1 million tokens, 8x the context size of DeepSeek R1.

MiniCPM-o is the latest series of end-side multimodal LLMs (MLLMs) ungraded from MiniCPM-V. The models can now take images, video, text, and audio as inputs and provide high-quality text and speech outputs in an end-to-end fashion. Since February 2024, there have been released 6 versions of the model, aiming to achieve strong performance and efficient deployment. The most notable models in the series currently include:

• MiniCPM-o 2.6: 🔥🔥🔥 The latest and most capable model in the MiniCPM-o series. With a total of 8B parameters, this end-to-end model achieves comparable performance to GPT-4o-202405 in vision, speech, and multimodal live streaming, making it one of the most versatile and performant models in the open-source community. For the new voice mode, MiniCPM-o 2.6 supports bilingual real-time speech conversation with configurable voices, and also allows for fun capabilities such as emotion/speed/style control, end-to-end voice cloning, role play, etc. It also advances MiniCPM-V 2.6's visual capabilities such strong OCR capability, trustworthy behavior, multilingual support, and video understanding. Due to its superior token density, MiniCPM-o 2.6 can for the first time support multimodal live streaming on end-side devices such as iPad.

• MiniCPM-V 2.6: The most capable model in the MiniCPM-V series. With a total of 8B parameters, the model surpasses GPT-4V in single-image, multi-image and video understanding. It outperforms GPT-4o mini, Gemini 1.5 Pro and Claude 3.5 Sonnet in single image understanding, and can for the first time support real-time video understanding on iPad.

Agenticseek is voice-enabled AI assistant autonomously browses the web, writes code, and plans tasks while keeping all data on your device. Tailored for local reasoning models, it runs entirely on your hardware, ensuring complete privacy and zero cloud dependency.

OmniSealBench provides a comprehensive benchmark for evaluating the performance of neural watermarking techniques. The benchmark includes a variety of datasets, evaluation metrics, and tools for training and testing neural networks for watermarking.

flow_matching is a PyTorch library for Flow Matching algorithms, featuring continuous and discrete implementations. It includes examples for both text and image modalities. This repository is part of Flow Matching Guide and Codebase.

Clojure MCP connects AI models to your Clojure development environment, enabling a remarkable REPL-driven development experience powered by large language models (LLMs).

Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model with 32 billion activated parameters and 1 trillion total parameters. Trained with the Muon optimizer, Kimi K2 achieves exceptional performance across frontier knowledge, reasoning, and coding tasks while being meticulously optimized for agentic capabilities.

Key Features

• Large-Scale Training: Pre-trained a 1T parameter MoE model on 15.5T tokens with zero training instability.

• MuonClip Optimizer: We apply the Muon optimizer to an unprecedented scale, and develop novel optimization techniques to resolve instabilities while scaling up.

• Agentic Intelligence: Specifically designed for tool use, reasoning, and autonomous problem-solving.

Model Variants

• Kimi-K2-Base: The foundation model, a strong start for researchers and builders who want full control for fine-tuning and custom solutions.

• Kimi-K2-Instruct: The post-trained model best for drop-in, general-purpose chat and agentic experiences. It is a reflex-grade model without long thinking.

More details and benchmarks are in the post.

Below the Fold

Gophish is an open-source phishing toolkit designed for businesses and penetration testers. It provides the ability to quickly and easily setup and execute phishing engagements and security awareness training.

Pangolin is a self-hosted tunneled reverse proxy server with identity and access control, designed to securely expose private resources on distributed networks. Acting as a central hub, it connects isolated networks — even those behind restrictive firewalls — through encrypted tunnels, enabling easy access to remote services without opening ports.

Cactus is a cross-platform framework for deploying LLM/VLM/TTS models locally in your app.

• Available in Flutter and React-Native for cross-platform developers.

• Supports any GGUF model you can find on Huggingface; Qwen, Gemma, Llama, DeepSeek etc.

• Run LLMs, VLMs, Embedding Models, TTS models and more.

• Accommodates from FP32 to as low as 2-bit quantized models, for efficiency and less device strain.

• MCP tool-calls to make AI performant and helpful (set reminder, gallery search, reply messages) etc.

• Fallback to massive cloud models for complex tasks and upon device failures.

• Chat templates with Jinja2 support and token streaming.

Xenharmlib is a generalized music theory library that supports traditional Western and non-western harmonic systems, unconventional microtonal and macrotonal tunings, diatonic and posttonal set theory and non-standard notations.

Flix is a principled effect-oriented functional, imperative, and logic programming language developed at Aarhus University

XAI has published a video on how they trained their new model Grok 4 against a very large training cluster.

Stanford Scaling Intelligence group wrote a blog post on Tokasaurus, a new LLM inference engine optimized for throughput-intensive workloads.

It can do batch processing tasks such as codebase scanning, synthetic data generation, and reinforcement learning pipelines. The main motivation comes from that unlike traditional chatbot applications where individual response latency is important, modern AI workflows often require processing thousands of sequences efficiently, making throughput optimization the primary concern where Tokasaurus is aiming to achieve fill this need.

Comparing to vLLM and SGLang across various throughput-focused benchmarks, researchers claim that Tokasaurus shows 3x throughput. This performance gain stems from optimizations tailored for both small and large models, addressing different bottlenecks that emerge at various scales. The project's influence extends beyond raw performance metrics, as it demonstrates how specialized design choices can dramatically improve efficiency for specific use cases which is one of the main theme in the blog post.

Tokasaurus adopts significantly different approach to LLM inference by prioritizing throughput over latency, leading to architectural decisions that is different from general-purpose inference engines. Further, it is implemented in pure Python, leveraging PyTorch's compilation capabilities and the FlashInfer library for attention operations. This design choice prioritizes flexibility and extensibility, following the philosophy of projects like GPT-fast.

The engine's architecture centers around an adaptive CPU manager that maintains deep input queues for GPU computation. This manager operates asynchronously and adaptively, monitoring queue depths and automatically adjusting its behavior to prevent GPU starvation. When the input queue approaches depletion, the manager dynamically skips optional operations like stop string checking and new sequence onboarding until sufficient queue depth is restored.

The system supports flexible parallelization strategies within single nodes, accommodating any combination of data, tensor, and pipeline parallelism. Currently, Tokasaurus supports models from the Llama-3 and Qwen-2 families, with plans for broader model support.

Small Model Optimizations

For small models, CPU overhead becomes a critical bottleneck that can significantly impact overall throughput. Traditional inference engines attempt to mitigate this through asynchronous processing, where CPU tasks for batch N+1 are prepared while the GPU processes batch N. Tokasaurus extends this concept with an adaptive management system that goes beyond simple asynchrony.

The adaptive manager continuously monitors the depth of the input queue and implements dynamic prioritization of tasks. When the system detects potential GPU starvation, it automatically reduces CPU overhead by temporarily suspending non-critical operations. This approach ensures that GPU utilization remains high even under varying workload conditions, a crucial factor for maximizing throughput in small model deployments.

The effectiveness of this optimization is demonstrated through benchmarks using the ShareGPT dataset, where Tokasaurus consistently outperforms competing engines. The adaptive nature of the system allows it to maintain high performance across different request patterns and batch sizes, making it particularly suitable for variable workloads.

Dynamic Prefix Identification and Hydragen Integration

One of Tokasaurus's features is its dynamic implementation of Hydragen (also known as cascade attention or bifurcated attention), which optimizes attention computation for sequences with shared prefixes. Prefix sharing occurs frequently in practical LLM applications, including repeated sampling for mathematical problems, document-based question answering, and system prompt reuse across different conversations.

The challenge in implementing Hydragen in a production inference engine lies in detecting shared prefixes dynamically as sequences continuously start and finish. Tokasaurus solves this through a greedy depth-first search algorithm executed before each model forward pass. This algorithm iteratively identifies the longest possible shared prefixes among active sequences, maximizing the efficiency gains from shared computation.

The impact of this optimization is particularly pronounced for small models, which dedicate a relatively larger fraction of their computational budget to attention mechanisms compared to larger models. In benchmarks reproducing the Large Language Monkeys experiment—where 128 GSM8K problems each receive 1024 answer attempts—Tokasaurus achieves over 2x throughput improvement compared to other engines. This dramatic improvement demonstrates the practical value of dynamic prefix detection in real-world scenarios with high degrees of sequence overlap.

Large Model Optimizations

Tokasaurus was originally motivated by the need to efficiently run large model inference on L40S GPUs, which lack high-bandwidth NVLink connections between GPUs. Without NVLink, the communication costs of tensor parallelism across multiple GPUs become prohibitive, making pipeline parallelism a more attractive option. Pipeline parallelism requires much less inter-GPU communication by partitioning the model across different stages rather than distributing each layer.

The engine's pipeline parallelism implementation excels in throughput-focused scenarios because it naturally requires large batch sizes to achieve efficiency. Batches from the manager are subdivided into microbatches distributed across pipeline stages, and since throughput optimization typically involves using the largest batch size that fits in memory, this aligns perfectly with pipeline parallelism requirements.

Benchmarking results using Llama-3.1-70B on eight L40S GPUs demonstrate the effectiveness of this approach, with Tokasaurus achieving over 3x throughput improvement compared to vLLM and SGLang's pipeline parallel implementations. This significant performance gain makes high-end inference accessible to organizations with more modest GPU infrastructure, democratizing access to large model capabilities.

Asynchronous Tensor Parallelism for High-End Hardware

For users with access to NVLink-enabled GPUs such as B200s, H100s, and A100s, Tokasaurus offers a different optimization strategy through Asynchronous Tensor Parallelism (Async-TP). This feature leverages PyTorch's relatively new capability to overlap inter-GPU communication with other computations, effectively hiding communication costs.

The implementation maintains both torch-compiled versions of models with and without Async-TP enabled, allowing the system to automatically switch between configurations based on batch size. Through benchmarking, the team discovered that Async-TP introduces significant CPU overhead and only provides benefits at very large batch sizes (6,000+ tokens).The automatic switching mechanism ensures optimal performance across the full range of operational conditions.

This dual-optimization approach demonstrates Tokasaurus's adaptability to different hardware configurations, maximizing performance whether users have access to premium interconnects or more standard GPU setups. The torch compilation integration also enables end-to-end optimization, further improving performance through advanced compiler optimizations.

FlashInfer Integration and Attention Optimization

Tokasaurus leverages the FlashInfer library for attention and sampling operations, providing state-of-the-art kernel performance across diverse inference scenarios. FlashInfer implements efficient attention kernels for both sparse and dense KV-cache storage formats, supporting single-request and batch operations across prefill, decode, and append stages.

The integration with FlashInfer enables Tokasaurus to achieve significant performance improvements in attention computation, which often represents a substantial portion of inference time. FlashInfer's block-sparse format for KV cache management optimizes memory access patterns and reduces redundancy. The library's customizable attention templates and Just-In-Time compilation capabilities allow adaptation to various attention variants and hardware configurations.

FlashInfer's load-balanced scheduling algorithm adjusts to the dynamic nature of user requests while maintaining compatibility with CUDA Graph requirements for static configuration. Comprehensive evaluations demonstrate FlashInfer's ability to achieve 29-69% inter-token latency reduction compared to compiler backends, 28-30% latency reduction for long-context inference, and 13-17% speedup for parallel generation scenarios.

Benchmarking Methodology and Results

The Tokasaurus team implemented benchmarking protocols to ensure fair comparisons with existing engines.All engines were configured with identical KV cache sizes and maximum running request limits, with careful tuning of remaining parameters for each system. The benchmarks report average throughput across multiple runs after completing warmup phases, and all experiments were conducted on identical hardware configurations.

Two primary workload types were used for evaluation: ShareGPT dataset completion (a standard benchmark for inference engines) and reproduction of the Large Language Monkeys experiment using GSM8K mathematical problems. The latter workload particularly highlights the benefits of prefix sharing optimization, as it involves sampling 1024 answers for each of 128 problems.

All experiments utilized the OpenAI API interface to standardize interactions across different engines, though additional testing with vLLM's Python API showed modest additional improvements. The team made benchmarking scripts and commands publicly available, enabling reproducibility and independent verification of results.

The benchmark results demonstrate Tokasaurus's superior performance across multiple scenarios. For small models using the ShareGPT dataset, Tokasaurus consistently outperformed both vLLM and SGLang. The most significant improvements appeared in the Large Language Monkeys benchmark, where the prefix sharing optimization enabled over 2x throughput gains.

For large model evaluation using Llama-3.1-70B on eight L40S GPUs, Tokasaurus's pipeline parallelism implementation achieved over 3x throughput improvement compared to existing solutions. These results demonstrate the engine's effectiveness across different model sizes and hardware configurations.

The benchmarking also revealed interesting insights about the relationship between batch size and optimization effectiveness. Async-TP only becomes beneficial at very large batch sizes, while the adaptive CPU management provides consistent benefits across varying load conditions. These findings inform optimal deployment strategies for different use cases.

Competitive Landscape

The LLM inference engine landscape includes several very good libraries, each with different strengths and optimization focuses. vLLM pioneered continuous batching and PagedAttention for memory-efficient serving, achieving dramatic improvements over naive implementations. SGLang focuses on structured generation and efficient request scheduling, often achieving superior throughput in specific scenarios.

Tokasaurus differentiates itself through its specialized focus on throughput-intensive workloads and dynamic optimization capabilities. While vLLM excels in general-purpose serving and SGLang provides strong structured generation support, Tokasaurus's adaptive management and dynamic prefix detection offer unique advantages for batch processing scenarios.

The comparison reveals that no single engine dominates all scenarios, with performance varying based on model size, hardware configuration, and workload characteristics. Tokasaurus' contribution lies in pushing the boundaries of throughput optimization for specific use cases where batch processing efficiency is very important.

Henry H.M. Ko provides a comprehensive, exploration of TPUs, focusing on their architectural philosophy, hardware-software co-design, and scalability from single-chip to multi-host deployments in his TPU Deep Dive blog post. It compares different aspects of TPU with GPUs.

• Early Considerations: Google began exploring specialized hardware for machine learning in 2006, initially weighing GPUs, FPGAs, and custom ASICs. Early workloads didn’t justify the investment, so CPUs sufficed.

• Turning Point: By 2013, neural networks powered features like voice search, and internal projections showed that scaling with CPUs alone would be infeasible. This spurred the development of a custom ASIC: the TPU.

• Current Role: Today, TPUs underpin Google’s AI services, powering both training and inference for models like Gemini and Veo, as well as high-throughput recommendation models (DLRMs).

The post specifically focuses on the TPUv4 architecture, which is representative of the latest generations (e.g., TPUv6p "Trillium", TPUv7 "Ironwood"). Each TPUv4 chip contains:

• Two TensorCores: Responsible for all computation (inference-specialized chips have one).

• Shared Memory Units:

  • CMEM (128 MiB): General on-chip memory.

  • HBM (32 GiB): High Bandwidth Memory for large data storage.

Each TensorCore comprises several specialized units:

1. Matrix Multiply Unit (MXU):

   • 128x128 Systolic Array: The heart of the TPU’s compute capability, optimized for dense matrix multiplications.

2. Vector Processing Unit (VPU):

   • Handles elementwise operations (e.g., ReLU, add, multiply, reductions).

3. Vector Memory (VMEM, 32 MiB):

   • Acts as a buffer for data fetched from HBM, staging it for computation.

4. Scalar Unit and Scalar Memory (SMEM, 10 MiB):

   • Manages control flow, scalar ops, and memory address generation.

5. Instruction and Control Logic:

   • Directs the operation of the VPU and MXU.

How does it compare with GPUs?

• Memory: TPUs have much larger on-chip memory (CMEM, VMEM, SMEM) but smaller HBM compared to GPUs (e.g., NVIDIA H100: 256 KB L1, 50 MB L2, 80 GB HBM).

• Compute Cores: TPUs have fewer, but much more specialized compute cores comparing to GPUs.

• Throughput: TPU v5p achieves 500 TFLOPs/sec per chip; a full pod (8960 chips) reaches ~4.45 ExaFLOPs/sec. TPUv7 "Ironwood" claims up to 42.5 ExaFLOPS/sec per pod (9216 chips).

  

Systolic Array:

• A grid of interconnected processing elements (PEs), each performing simple operations (e.g., multiply-accumulate) and passing results to neighbors.

• Advantages: Once data enters the array, no further control logic is needed. There are minimal memory reads/writes except for input and output, maximizing efficiency.

• Ideal Use Cases: Matrix multiplications and convolutions, which dominate deep learning workloads, map perfectly onto systolic arrays.

Downsides

• Sparsity Handling: Systolic arrays excel with dense matrices but don’t benefit from sparsity—PEs still process zeros, leading to wasted cycles for sparse workloads. This is a big downside and limitation as models like Mixture-of-Experts (MoE) become more common and ubiquitous especially in GenAI workflows.

Memory Hierarchy Comparison with GPUs

• GPUs: Rely on caches to handle unpredictable memory access patterns, enhancing flexibility but increasing energy consumption.

• TPUs:

  • Predictable Access Patterns: Most ML workloads have regular, predictable access patterns.

  • Scratchpad Memory: By using ahead-of-time compilation (via the XLA compiler), TPUs can precompute memory accesses and use scratchpad buffers instead of caches.

  • Energy Efficiency: Arithmetic operations are much cheaper than memory accesses. By minimizing memory operations, TPUs achieve both speed and substantial energy savings.

XLA Compiler

• Role: The XLA compiler analyzes computation graphs ahead of time, generating highly optimized binaries that minimize memory traffic and maximize data reuse.

• JAX Integration: JAX uses XLA under the hood. When a function is jitted, JAX traces it to a static computation graph, which XLA compiles for the TPU. However, different input shapes require recompilation, making JAX less suitable for dynamic workloads.

• Flexibility: The heavy reliance on AoT compilation sacrifices flexibility—dynamic models or variable-length loops are less efficient.

• Compiler Dependency: The system’s performance hinges on compiler quality and static analysis.

• Memory vs. Compute: On modern chips, memory operations consume orders of magnitude more energy than arithmetic. HBM3 (used in TPUs) is more efficient than older DDR3/4 DRAM, but memory remains the dominant energy cost.

• Scaling Laws: Increasing FLOPs (compute) is preferable to increasing memory operations, as it improves both speed and energy efficiency.

Multi-Chip and System-Level Architecture

Tray (Board) Level

• Composition: A tray contains 4 TPU chips (8 TensorCores), each with its own CPU host (for inference, one host per two trays).

• Interconnects:

  • Host to Chip: PCIe.

  • Chip to Chip: Inter-Core Interconnect (ICI), offering higher bandwidth than PCIe.

Rack Level

• Structure: 64 TPUs per rack, connected in a 4x4x4 3D torus topology.

• Interconnects: ICI and Optical Circuit Switching (OCS).

• Scalability: The modular design allows racks to be combined into larger systems.

Terminology

• TPU Rack: Physical unit with 64 chips ("cube").

• TPU Pod: Maximum number of TPUs connected via ICI and OCS (e.g., 4096 chips for TPUv4).

• TPU Slice: Any subset of TPUs within a pod, not necessarily contiguous.

Topology and Communication

3D Torus and OCS

• 3D Torus: Each chip connects to neighbors in three dimensions, reducing hops and latency.

• OCS (Optical Circuit Switching): Used for wraparound connections, turning each axis into a ring (1D torus). This reduces worst-case communication hops from N−1N−1 to (N−1)/2(N−1)/2 per axis, crucial for scaling.

Benefits of OCS

1. Wraparound: Faster communication and reduced latency.

2. Noncontiguous Slices: OCS allows logical grouping of non-adjacent chips, enabling flexible resource allocation and higher utilization.

3. Scalability: Fewer physical wires and more flexible configurations support larger pod sizes.

Topology Choices

• Cube (e.g., 8x8x8): Maximizes bisection bandwidth, ideal for all-to-all communication (data/tensor parallelism).

• Cigar (e.g., 4x4x32): Favors pipeline parallelism, reducing sequential layer communication time.

• Rectangle (e.g., 4x8x16): Intermediate trade-offs.

Superpod

• Definition: The largest configuration of interconnected chips via ICI and OCS (e.g., 4096 chips for TPUv4, 9216 for TPUv7).

• Physical Layout: Multiple racks interconnected, forming a massive, unified compute resource.

Slices and Flexibility

• Dynamic Slicing: Thanks to OCS, slices can be noncontiguous, allowing for flexible job scheduling and resource allocation.

• Performance Impact: The chosen slice topology directly affects communication bandwidth and parallelism efficiency.

Beyond the Pod

• Multi-Pod: Connecting multiple pods requires slower interconnects, but enables even larger clusters for the most demanding workloads.

TPU Limitations

While TPUs are great for a variety of different use cases, it still has a number of limitations that are in following:

• Sparsity: Systolic arrays are inefficient for sparse workloads, a growing concern as models evolve.

• Flexibility: AoT compilation and static memory layouts hinder dynamic model architectures.

• Compiler Reliance: Performance and efficiency depend heavily on the XLA compiler’s ability to optimize computation graphs.

• Topology Tuning: Selecting the optimal slice topology is non-trivial and model-dependent.

Libraries

MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.

MCP helps you build agents and complex workflows on top of LLMs. LLMs frequently need to integrate with data and tools, and MCP provides:

• A growing list of pre-built integrations that your LLM can directly plug into

• The flexibility to switch between LLM providers and vendors

• Best practices for securing your data within your infrastructure

Meta Lingua is a minimal and fast LLM training and inference library designed for research. Meta Lingua uses easy-to-modify PyTorch components in order to try new architectures, losses, data, etc. We aim for this code to enable end to end training, inference and evaluation as well as provide tools to better understand speed and stability.

Protenix — a trainable, open-source PyTorch reproduction of AlphaFold 3.

Protenix is built for high-accuracy structure prediction. It serves as an initial step in our journey toward advancing accessible and extensible research tools for the computational biology community.

dattri is a PyTorch library for developing, benchmarking, and deploying efficient data attribution algorithms. You may use dattri to

• Deploy existing data attribution methods to PyTorch models

  • e.g., Influence Function, TracIn, RPS, TRAK, ...

• Develop new data attribution methods with efficient implementation of low-level utility functions

  • e.g., Hessian (HVP/IHVP), Fisher Information Matrix (IFVP), random projection, dropout ensembling, ...

• Benchmark data attribution methods with standard benchmark settings

  • e.g., MNIST-10+LR/MLP, CIFAR-10/2+ResNet-9, MAESTRO + Music Transformer, Shakespeare + nanoGPT, ...

If you have time to watch one video last week, make sure it is this one:

Below The Fold(BTF)

Bun is an all-in-one toolkit for JavaScript and TypeScript apps. It ships as a single executable called bun.

At its core is the Bun runtime, a fast JavaScript runtime designed as a drop-in replacement for Node.js. It's written in Zig and powered by JavaScriptCore under the hood, dramatically reducing startup times and memory usage.

Foam is a personal knowledge management and sharing system inspired by Roam Research, built on Visual Studio Code and GitHub.

You can use Foam for organising your research, keeping re-discoverable notes, writing long-form content and, optionally, publishing it to the web.

Foam is free, open source, and extremely extensible to suit your personal workflow. You own the information you create with Foam, and you're free to share it, and collaborate on it with anyone you want.

Hurl is a command line tool that runs HTTP requests defined in a simple plain text format.

It can chain requests, capture values and evaluate queries on headers and body response. Hurl is very versatile: it can be used for both fetching data and testing HTTP sessions.

Hurl makes it easy to work with HTML content, REST / SOAP / GraphQL APIs, or any other XML / JSON based APIs.

Kan is the open-source project management alternative to Trello.

• 👁️ Board Visibility: Control who can view and edit your boards

• 🤝 Workspace Members: Invite members and collaborate with your team

• 🚀 Trello Imports: Easily import your Trello boards

• 🔍 Labels & Filters: Organise and find cards quickly

• 💬 Comments: Discuss and collaborate with your team

• 📝 Activity Log: Track all card changes with detailed activity history

• 🎨 Templates (coming soon) : Save time with reusable board templates

• ⚡️ Integrations (coming soon) : Connect your favourite tools

Quarkdown is a modern Markdown-based typesetting system, designed around the key concept of versatility, by seamlessly compiling a project into a print-ready book or an interactive presentation. All through an incredibly powerful Turing-complete extension of Markdown, ensuring your ideas flow automatically into paper.

PGQueuer is a minimalist, high-performance job queue library for Python, leveraging PostgreSQL's robustness. Designed with simplicity and efficiency in mind, PGQueuer offers real-time, high-throughput processing for background jobs using PostgreSQL's LISTEN/NOTIFY and FOR UPDATE SKIP LOCKED mechanisms.

Istio is an open source service mesh that layers transparently onto existing distributed applications. Istio’s powerful features provide a uniform and more efficient way to secure, connect, and monitor services. Istio is the path to load balancing, service-to-service authentication, and monitoring – with few or no service code changes.

Stanford's Hazy Research group recently explored how to maximize the speed of open-source models on modern GPUs, particularly in the challenging scenario of generating a single sequence with Llama-3.2-1B. Their findings reveal significant performance limitations in popular LLM inference engines and introduce a "megakernel" approach that delivers better speed improvements comparing to other approaches.

They identified an opportunity when running a single sequence with Llama-1B, performance is strongly memory-bound, meaning that execution speed is primarily limited by how quickly model weights can be loaded from GPU global memory. Popular LLM inference engines like vLLM and SGLang are only able to utilize at most 50% of the available GPU bandwidth when running this workload on high-performance accelerators like NVIDIA's H100.

The core problem is that existing systems break down a model's forward pass into approximately one hundred separate kernels, each implementing only a few operations such as RMS normalization, attention mechanisms, MLP layers with activations, and rotary position embeddings. Each of these kernels comes with setup and teardown periods during which no useful work is performed and creates overhead, resulting in inefficiencies.

Three key problems with the current kernel-based approach that create what they call "memory pipeline bubbles"—periods when the GPU is not actively loading from memory:

1. Strict Ordering of GPU Kernels: GPU kernels are launched in a strict sequence where thread blocks in a new kernel cannot start until all thread blocks from previous kernels have completely finished. This creates waiting periods as stragglers finish their work.

2. Kernel Launch and Teardown Costs: Each kernel launch incurs overhead. Even with NVIDIA's CUDA graphs (designed to hide these costs), measurements on an H100 showed launch costs of about 1.3 microseconds per kernel—time during which the GPU performs no useful work.

3. Weight and Activation Loading Delays: After a new kernel starts, the GPU must wait to load weights and activations before computation can begin. These latencies result in thousands of idle cycles.

For a model like Llama-1B with 16 layers and approximately 7 kernel launches per layer, even with an optimistic 5 microseconds of stalling per kernel, these inefficiencies limit performance. The theoretical memory limit on an H100 would allow approximately 1,350 forward passes per second, but pipeline bubbles reduce this to around 770 forward passes per second or even lower with further inefficiencies.

To address these limitations, one approach could be to merging the entire Llama-1B forward pass into a single "megakernel" that eliminates kernel boundaries altogether. This approach achieves remarkable performance, utilizing 78% of memory bandwidth on an H100 and outperforming existing systems by over 1.5x.

However, there are many challenges building and pipelining all of the kernel operations into a single kernel.

Challenge 1: Fusing Numerous Operations

Traditional kernel fusion typically merges just two or three operations, but the Llama-1B forward pass requires fusing approximately one hundred operations. To manage this complexity, one needs to build an on-GPU interpreter where each streaming multiprocessor (SM) receives and executes a sequence of instructions. In order to do so, there are several key instructions for their Llama forward pass megakernel:

• Fused RMS norm & QKV & RoPE instruction

• Attention computation instruction

• Attention reduction instruction (for ThunderGQA on long sequences)

• O-projection + residual instruction

• Fused RMS norm & up-gate & SiLU instruction

• Down-projection + residual instruction

• RMS norm & language modeling head instruction

These instructions were implemented using a common CUDA template with standardized load, store, and compute functions to ensure interoperability within the interpreter framework.

Challenge 2: Managing Shared Memory Resources

The megakernel architecture allows for pipelining memory loads across instructions, starting to load weights for an upcoming instruction even while a previous instruction is still finishing. However, this creates a resource allocation challenge: the loaded weights need somewhere to be stored.

The solution was to implement a shared memory paging system. On an H100, the authors of the blog post divided the first 213KB of shared memory into thirteen 16KB pages, with remaining shared memory reserved for instruction parameters. Instructions explicitly request and release these pages from the interpreter, which automatically passes released pages to subsequent instructions. This allows new instructions to begin loading data as soon as shared memory becomes available, which also helps increasing utilization of the memory bandwidth.

Challenge 3: Explicit Synchronization

In traditional multi-kernel execution, CUDA ensures that all input tensors for a kernel are fully produced and available before the kernel launches. With megakernels, this guarantee doesn't exist—when an SM starts executing a new instruction, its inputs might not be ready.

To address this synchronization challenge, the authors implemented a counter system in GPU global memory. When an instruction completes, it increments a specific counter; when a new instruction starts, it waits for relevant counters to reach target values, confirming that all dependencies have been satisfied.

This approach enabled advanced optimizations like chunking the intermediate state in multi-layer perceptrons (MLPs). Rather than waiting for the entire hidden state to complete before beginning the down projection matrix multiply (as would be necessary with PDL), the megakernel produces and consumes the intermediate state in four chunks, each with its own counter. This allows the down projection to start processing a chunk as soon as it's ready, without waiting for the entire hidden state.

Performance Results

The megakernel approach improved the existing implementation through speed quite a bit:

• On an H100, it runs almost 2.5x faster than vLLM and over 1.5x faster than SGLang.

• On a B200, the performance gap with vLLM increases to over 3.5x, while maintaining more than 1.5x advantage over SGLang.

Other Approaches

While the megakernel approach demonstrates superior performance, the authors also acknowledge other technical approaches have their own merits:

1. CUDA Graphs: This NVIDIA feature helps hide kernel launch costs, reducing launch overhead from about 2.1 microseconds to 1.3 microseconds on an H100. While this represents a 38% improvement, it still leaves significant performance on the table compared to the megakernel approach.

2. Programmatic Dependent Launch (PDL): This NVIDIA mechanism allows the next kernel to start preparing while the previous kernel is running, potentially improving pipelining.However, the authors found that PDL's synchronization mechanism (cudaGridDependencySynchronize) is too coarse-grained. For example, it requires waiting for all queries, keys, and values to complete before starting attention calculations, rather than allowing heads to start as soon as they're ready. The megakernel's fine-grained counter-based synchronization offers more flexibility and efficiency.

3. Traditional Kernel-Based Execution: The standard approach of dividing operations into separate kernels does provide automatic synchronization guarantees and simpler programming models. However, these benefits come at the cost of significant performance penalties for low-latency, memory-bound workloads.

Libraries

garak checks if an LLM can be made to fail in a way we don't want. garak probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses. If you know nmap or msf / Metasploit Framework, garak does somewhat similar things to them, but for LLMs.

garak focuses on ways of making an LLM or dialog system fail. It combines static, dynamic, and adaptive probes to explore this.

garak's a free tool. We love developing it and are always interested in adding functionality to support applications.

miniDiffusion is a reimplementation of the Stable Diffusion 3.5 model in pure PyTorch with minimal dependencies. It's designed for educational, experimenting, and hacking purposes. It's made with the mindset of having the least amount of code necessary to recreate Stable Diffusion 3.5 from scratch, with only ~2800 spanning from VAE to DiT to the Train and Dataset scripts.

Swift Scribe is a privacy-first, AI-enhanced transcription application built exclusively for iOS 26/macOS 26+ that transforms spoken words into organized, searchable notes. Using Apple's latest SpeechAnalyzer and SpeechTranscriber frameworks (available only in iOS 26/macOS 26+) combined with on-device Foundation Models, it delivers real-time speech recognition, intelligent content analysis, and advanced text editing capabilities.

Below the Fold

container is a tool that you can use to create and run Linux containers as lightweight virtual machines on your Mac. It's written in Swift, and optimized for Apple silicon.

The tool consumes and produces OCI-compliant container images, so you can pull and run images from any standard container registry. You can push images that you build to those registries as well, and run the images in any other OCI-compliant application.

container uses the Containerization Swift package for low level container, image, and process management.

Zellij is a workspace aimed at developers, ops-oriented people and anyone who loves the terminal. Similar programs are sometimes called "Terminal Multiplexers".

Zellij is designed around the philosophy that one must not sacrifice simplicity for power, taking pride in its great experience out of the box as well as the advanced features it places at its users' fingertips.

Zellij is geared toward beginner and power users alike - allowing deep customizability, personal automation through layouts, true multiplayer collaboration, unique UX features such as floating and stacked panes, and a plugin system allowing one to create plugins in any language that compiles to WebAssembly.

pgroll is an open source command-line tool that offers safe and reversible schema migrations for PostgreSQL by serving multiple schema versions simultaneously. It takes care of the complex migration operations to ensure that client applications continue working while the database schema is being updated. This includes ensuring changes are applied without locking the database, and that both old and new schema versions work simultaneously (even when breaking changes are being made!). This removes risks related to schema migrations, and greatly simplifies client application rollout, also allowing for instant rollbacks.

Ephe is an ephemeral markdown paper to organize your daily todos and thoughts.

Traditional todo apps can be overwhelming.
Ephe is designed to organize your tasks with plain Markdown.
Ephe gives you just one clean page to focus your day.

patolette is a C / Python color quantization and dithering library.

At its core, it implements a weighted variant of Xiaolin Wu's PCA-based quantizer (not to be confused with the popular one from Graphics Gems vol. II, which is already available here).

Some of its key features are:

• Avoids axis-aligned subdivisions

• Supports the CIEL*u*v* and ICtCp color spaces

• Optional use of saliency maps to give higher weight to areas that stand out visually

• Optional KMeans refinement

RomM (ROM Manager) allows you to scan, enrich, browse and play your game collection with a clean and responsive interface. With support for multiple platforms, various naming schemes, and custom tags, RomM is a must-have for anyone who plays on emulators.

NVIDIA introduces CosAE (Cosine Autoencoder), a novel, generic Autoencoder that seamlessly leverages the classic Fourier series with a feed-forward neural network. CosAE represents an input image as a series of 2D Cosine time series, each defined by a tuple of learnable frequency and Fourier coefficients. This method stands in contrast to a conventional Autoencoder that often sacrifices detail in their reduced-resolution bottleneck latent spaces. CosAE, however, encodes frequency coefficients, i.e., the amplitudes and phases, in its bottleneck. This encoding enables extreme spatial compression, e.g., 64x downsampled feature maps in the bottleneck, without losing detail upon decoding.

I think there is a strong argument on why this type of architecture can be better than traditional vanilla AE:

1. Fourier coefficients exploit information that occurs in the images in a more straightforward manner.

2. Fourier coefficients can provide compression capability in a much better than the traditional vanilla auto encoder architectures.

There is also strong argument on why this type of architecture can be worse than traditional vanilla AE:

1. Fourier coefficients would be better for natural images and images that are generally good for visual elements of human eye and from the real world, but may not work as well for synthetically generated images.

2. While Fourier coefficients can provide better compression capabilities, it might also prevent other types of relationships that a traditional autoencoder might uncover.

Attention Wasn’t All We needed

Stephen Diehl wrote an excellent article around attention and subsequent developments around attention and how crucial they are for optimizing the architecture and hence the cheeky title “Attention Wasn't All We Needed”.

"Attention Is All You Need" paper established the foundation for transformer architectures, but subsequent research has revealed significant opportunities for optimization and enhancement of this architecture both from efficiency and better accuracy point of view.

Dielh looks at three critical attention mechanism innovations that address fundamental limitations in the original transformer design: 1. Group Query Attention (GQA), 2. Multi-head Latent Attention, and 3. Flash Attention.

These mechanisms tackle distinct but somehow interconnected challenges including memory bandwidth bottlenecks during inference, quadratic computational complexity for long sequences, and inefficient GPU memory utilization patterns.

Group Query Attention: Optimizing Memory Bandwidth for Inference

Group Query Attention(GQA) represents a shift in how attention mechanisms balance computational efficiency with representational capacity during inference. It specifically targets the key-value (KV) cache bottleneck that emerges during autoregressive generation, where previously computed keys and values must be stored and repeatedly accessed for each new token prediction. The main idea of GQA is that the computational bottleneck and memory footprint in multi-head attention are heavily influenced by the size of the K and V projections and their corresponding caches, rather than the query projections themselves.

Traditional multi-head attention maintains N_h distinct heads for queries, keys, and values, resulting in substantial memory requirements during inference when the KV cache must store all previous computations. GQA introduces an asymmetric design where N_h query heads operate with only N_kv key-value heads, where N_kv < N_h and N_h is typically a multiple of N_kv. This architectural modification directly addresses the observation that key and value representations can be effectively shared across multiple query heads without significant performance degradation.

The mathematical foundation of GQA builds upon standard attention mechanisms while introducing parameter sharing. Given an input sequence representation X ∈ R^(L×d_model), where L represents sequence length and d_model denotes embedding dimension, the mechanism first projects X into queries, keys, and values using distinct linear transformations: Q = XW_Q, K = XW_K, and V = XW_V. The distinction lies in the dimensionality of these projection matrices, where W_Q ∈ R^(d_model×(N_h d_k)) maintains full query capacity while W_K, W_V ∈ R^(d_model×(N_kv d_k)) operate with reduced key-value dimensions.

The grouping mechanism divides the N_h query heads into N_kv groups, with each group containing g = N_h/N_kv query heads that share the same key and value representations. For the i-th query head, the corresponding key and value heads are determined by K_⌈i/g⌉ and V_⌈i/g⌉, where the ceiling function ensures proper group assignment. The attention computation for each head follows the standard scaled dot-product attention formula, but with shared key-value pairs:

Through this, we can keep the full expressiveness of query representations while dramatically reducing the memory requirements for key-value storage.

Implementation Efficiency and Practical Considerations

The practical implementation of GQA leverages efficient tensor operations to minimize computational overhead while maximizing memory savings. The technique employs repetition and interleaving operations to align the reduced set of key-value heads with the full set of query heads before performing batched matrix multiplication for attention scores. This approach, implemented through operations like repeat_interleave, ensures that the computational pattern remains highly parallelizable and compatible with existing GPU kernels.

The memory bandwidth benefits of GQA become particularly important during autoregressive generation, where the KV cache size directly impacts inference speed. By reducing N_kv, GQA significantly decreases the memory bandwidth required to load the KV cache at each decoding step, which represents the primary performance bottleneck in large model deployment. Empirical evaluations demonstrate that GQA achieves favorable trade-offs, maintaining most of the representational quality of full multi-head attention while delivering substantial speedups and memory savings.

Multi-query attention (MQA) represents an extreme case of GQA where N_kv = 1, providing maximum memory efficiency at the potential cost of some representational capacity. The flexible framework allows practitioners to tune the N_h/N_kv ratio based on specific deployment constraints and performance requirements.

Multi-head Latent Attention: Breaking Quadratic Complexity Barriers

Computational Complexity Revolution

Multi-head Latent Attention addresses one of the most fundamental limitations of transformer architectures: the quadratic computational complexity O(L²) with respect to sequence length L that emerges from the need for every input element to attend to every other element. This quadratic scaling becomes prohibitive for applications involving very long sequences, such as document processing, high-resolution image analysis, or extended audio processing. The mechanism introduces a sophisticated intermediary approach that decouples direct sequence interactions through a fixed-size set of learnable latent vectors.

The core architectural change replacing direct self-attention with a two-stage cross-attention process mediated by N_latents learnable vectors. This design assumes that essential information from long input sequences can be effectively compressed and summarized through these latent representations, thus maintaining representational power while achieving linear scaling with sequence length. The approach transforms the computational complexity from O(L²) to O(L × N_latents), enabling transformer-like architectures to handle previously intractable sequence lengths when N_latents << L.

Two-Stage Attention Mechanism Architecture

The mathematical formulation of Multi-head Latent Attention operates through two distinct cross-attention stages, each serving a specific information aggregation purpose. Let the input sequence be represented as X ∈ R^(L×d) and the learnable latent array as L ∈ R^(N_latents×d), where both undergo projection into query, key, and value representations using either shared or separate projection matrices.

The first cross-attention stage focuses on information extraction from the input sequence to the latent space. Latent queries Q_L attend to input keys K_X and aggregate information from input values V_X through the standard attention mechanism. This operation enables the latent vectors to selectively extract and compress relevant information from the entire input sequence, effectively creating a condensed representation that captures global context.

The second cross-attention stage reverses the information flow, allowing input queries Q_X to attend to the updated latent representations. Input queries attend to latent keys K_L and aggregate information from the processed latent values H_L. This bidirectional information flow ensures that each input element can access the globally-relevant information captured by the latent vectors while maintaining the ability to produce element-specific outputs.

Applications and Representational Trade-offs

Multi-head Latent Attention is very important in domains where traditional self-attention becomes computationally infeasible due to sequence length constraints. Applications include processing long documents where maintaining global coherence is crucial, high-resolution image processing where pixel patches are treated as sequence elements, extended audio signal analysis, and video processing where temporal dependencies span many frames. The fixed number of latent vectors provides a scalable approach that remains computationally tractable regardless of input sequence length. For recommendation systems, you can imagine that for each activity, user interactions, you want to preserve a strong understanding and connection between them even though they might occur not in close temporal proximity of each other.

The learnable latent vectors, initialized randomly and updated through standard backpropagation, adapt during training to function as a compressed memory bank relevant to the specific task. This adaptive behavior allows the mechanism to automatically discover the most relevant global patterns and dependencies for the target application. However, the approach does introduce an information bottleneck that may limit fine-grained local interactions compared to full self-attention, representing a trade-off between computational efficiency and representational completeness.

Despite this limitation, Multi-head Latent Attention is very efficient at capturing global context, making it particularly valuable for tasks where long-range dependencies are more important than detailed local interactions. The mechanism has proven effective across various modalities, enabling transformer-like architectures to be applied to domains previously constrained by computational limitations.

Flash Attention: Revolutionizing Memory Hierarchy Utilization

Memory Bandwidth Optimization Strategy

Flash Attention, particularly in its latest FlashAttention-3 implementation, changes how attention mechanisms interact with modern GPU memory hierarchies. The standard attention computation requires materializing and storing the full attention score matrix S = QK^T, where Q, K ∈ R^(N×d) represent query and key matrices for a sequence of length N. This conventional approach necessitates storing the complete N × N matrix S, resulting in O(N²) memory complexity that becomes prohibitive for long sequences due to the limitations of GPU high bandwidth memory.

The approach of Flash Attention circumvents this memory bottleneck by avoiding the materialization of the full attention matrix in the GPU's slower high bandwidth memory. Instead, the mechanism leverages tiling and recomputation techniques that process attention computations in smaller blocks specifically sized to fit within the much faster on-chip SRAM. This architectural awareness represents a crucial shift from algorithm-centric optimization to hardware-aware design that maximizes the efficiency of modern accelerator architectures.

Tiling and Recomputation Framework

The core computational strategy of Flash Attention involves decomposing the Q, K, and V matrices into manageable blocks that can be processed within the constraints of on-chip memory. This tiling approach ensures that each computational step operates on data that can be efficiently cached in SRAM, minimizing the expensive memory transfers to and from high bandwidth memory that typically dominate attention computation costs.

The recomputation aspect of Flash Attention represents a sophisticated trade-off between memory usage and computational redundancy. Rather than storing intermediate attention scores that would exceed SRAM capacity, the mechanism strategically recomputes certain values when needed, leveraging the superior computational throughput of modern GPUs compared to their memory bandwidth limitations. This approach aligns with the evolving hardware landscape where computational resources often exceed memory bandwidth capacity.

Convergence of Optimization Strategies

The three attention mechanisms outlined above represent complementary approaches to addressing distinct bottlenecks in transformer scaling and deployment.

1. Group Query Attention targets inference-time memory bandwidth,

2. Multi-head Latent Attention addresses training-time computational complexity for long sequences,

3. Flash Attention optimizes hardware utilization patterns.

   This convergence suggests that future transformer architectures will likely integrate multiple optimization strategies rather than relying on single-point solutions.

Libraries

SGLang is a fast serving framework for large language models and vision language models. It makes your interaction with models faster and more controllable by co-designing the backend runtime and frontend language. The core features include:

• Fast Backend Runtime: Provides efficient serving with RadixAttention for prefix caching, zero-overhead CPU scheduler, continuous batching, token attention (paged attention), speculative decoding, tensor parallelism, chunked prefill, structured outputs, quantization (FP8/INT4/AWQ/GPTQ), and multi-lora batching.

• Flexible Frontend Language: Offers an intuitive interface for programming LLM applications, including chained generation calls, advanced prompting, control flow, multi-modal inputs, parallelism, and external interactions.

• Extensive Model Support: Supports a wide range of generative models (Llama, Gemma, Mistral, Qwen, DeepSeek, LLaVA, etc.), embedding models (e5-mistral, gte, mcdse) and reward models (Skywork), with easy extensibility for integrating new models.

• Active Community: SGLang is open-source and backed by an active community with industry adoption.

BLIP3-o is a unified multimodal model that combines the reasoning and instruction following strength of autoregressive models with the generative power of diffusion models. Unlike prior works that diffuse VAE features or raw pixels, BLIP3-o diffuses semantically rich CLIP image features, enabling a powerful and efficient architecture for both image understanding and generation.

• Fully Open-Source:

  • Pretraining Data: 24 Million Detailed Captions, 5 Million Short Captions

  • Instruction Tuning Data: 60 k GPT-4o Distilled Instruction Tuning Data

  • Model Weights: 4 B, 8 B

• Unified Architecture: for both image understanding and generation.

• CLIP Feature Diffusion: Directly diffuses semantic vision features for stronger alignment and performance.

• State-of-the-art performance: across a wide range of image understanding and generation benchmarks.

CHOP is an optimization library based on PyTorch, with applications to adversarial examples and structured neural network training.

Below The Fold

Tabloid is as minimal but Turing complete programming language inspired, nay, supercharged by clickbait headlines that rule the Internet today. You can try Tabloid on the Tabloid website.

Mermaid is a JavaScript-based diagramming and charting tool that uses Markdown-inspired text definitions and a renderer to create and modify complex diagrams. The main purpose of Mermaid is to help documentation catch up with development.

Querybook is a Big Data IDE that allows you to discover, create, and share data analyses, queries, and tables.

Features​

• 📚 Organize analyses with rich text, queries, and charts

• ✏️ Compose queries with autocompletion and hovering tooltip

• 📈 Use scheduling + charting in DataDocs to build dashboards

• 🙌 Live query collaborations with others

• 📝 Add additional documentation to your tables

• 🧮 Get lineage, sample queries, frequent user, search ranking based on past query runs

Lyft has written about their real-time spatial-temporal forecasting system that provides predictions for their dynamic pricing and driver incentives across millions of geohashes in North America. This system addresses the main challenge of balancing supply (driver availability) and demand (ride requests) at hyper-local levels, even in scenarios influenced by unexpected events like concerts or protests.

The system uses a three-layered architecture:

1. Time-Series Foundation Layer

   • ARIMA variants handle baseline periodicity (daily/weekly cycles)

   • Exponential smoothing state space models (ETS) adapt to short-term trends

   • Integrated with causal factors via PyTorch-based IndexTensors that enable differentiable operations over spatiotemporal indices

2. Graph Neural Network Layer

   • Captures spatial dependencies using adjacency matrices based on:

     • Geohash centroids (H3 grid system)

     • Road network connectivity

     • Historical ride flow patterns

   • Implements attention mechanisms to weight influential neighboring geohashes

3. Online Learning Layer

   • Continuously updates model weights

   • Uses model slicing to retain long-term patterns while adapting to real-time signals

In order to build predictions against temporal and spatial domain; they adopt a geo hash based spatial processing where they create spatial grid and uses hierarchical forecasting to do coarse to finer granular level of predictions in spatial domain.

• H3 Hexagonal Grid (resolution 10 ~ 0.15 km² cells)

  • Enables efficient neighbor lookups via built-in hierarchy

• Hierarchical Forecasting:

  • Aggregates predictions from fine to coarse resolutions (geohash 10 → 9 → 8)

  • Uses Sparse Tensor Representations to handle 4M+ geohashes efficiently

They adopt the following model optimization and latency reduction techniques to make the inference flow to be real-time and low latency:

• Model Quantization: FP16 → INT8 conversion using NVIDIA TensorRT framework under the hood

• Hotspot Prediction: Pre-warms Redis cache for geohashes with >100 requests/min

• Dynamic Batching: Groups geo-hash requests using Flink's session windows

The system accounts for external factors through a bayesian structural time-series structure:

Forecast = BaseModel(History) × CausalImpact(Events) + RealTimeAdjustment

where:

• Causal Impact uses Bayesian structural time-series to measure:

  • Weather changes (precipitation, temperature)

  • Local events (concerts, protests)

  • Road closures

• Implemented as PyTorch modules with custom backward passes

Libraries

VectorVFS is a lightweight Python package that transforms your Linux filesystem into a vector database by leveraging the native VFS (Virtual File System) extended attributes. Rather than maintaining a separate index or external database, VectorVFS stores vector embeddings directly alongside each file—turning your existing directory structure into an efficient and semantically searchable embedding store.

VectorVFS currently uses Meta's Perception Encoders (PE) [arxiv] which includes image/video encoders for vision language understanding, it outperforms InternVL3, Qwen2.5VL and SigLIP2 for zero-shot image tasks.

• Zero-overhead indexing
  Embeddings are stored as extended attributes (xattrs) on each file, eliminating the need for external index files or services.

• Seamless retrieval
  Perform searches across your filesystem, retrieving files by embedding similarity.

• Flexible embedding support
  Plug in any embedding model—from pre-trained transformers to custom feature extractors—and let VectorVFS handle storage and lookup.

• Lightweight and portable
  Built on native Linux VFS functionality, VectorVFS requires no additional daemons, background processes, or databases.

AXLearn is a library built on top of JAX and XLA to support the development of large-scale deep learning models.

AXLearn takes an object-oriented approach to the software engineering challenges that arise from building, iterating, and maintaining models. The configuration system of the library lets users compose models from reusable building blocks and integrate with other libraries such as Flax and Hugging Face transformers.

AXLearn is built to scale. It supports the training of models with up to hundreds of billions of parameters across thousands of accelerators at high utilization. It is also designed to run on public clouds and provides tools to deploy and manage jobs and data. Built on top of GSPMD, AXLearn adopts a global computation paradigm to allow users to describe computation on a virtual global computer rather than on a per-accelerator basis.

Agent S is an open-source framework designed to enable autonomous interaction with computers through Agent-Computer Interface. Our mission is to build intelligent GUI agents that can learn from past experiences and perform complex tasks autonomously on your computer.

Vision Transformer (ViT) extends the application range of transformers from language processing to computer vision tasks as being an alternative architecture against the existing convolutional neural networks (CNN). Since the transformer-based architecture has been innovative for computer vision modeling, the design convention towards an effective architecture has been less studied yet. From the successful design principles of CNN, we investigate the role of the spatial dimension conversion and its effectiveness on the transformer-based architecture. If articularly attend the dimension reduction principle of CNNs; as the depth increases, a conventional CNN increases channel dimension and decreases spatial dimensions. Such a spatial dimension reduction is beneficial to a transformer architecture as well, and proposed a novel Pooling-based Vision Transformer (PiT) upon the original ViT model. The paper accompanied by the library is also available in Arxiv.

Guardrails is a Python framework that helps build reliable AI applications by performing two key functions:

1. Guardrails runs Input/Output Guards in your application that detect, quantify and mitigate the presence of specific types of risks. To look at the full suite of risks, check out Guardrails Hub.

2. Guardrails help you generate structured data from LLMs.

   

GPTSwarm is a graph-based framework for LLM-based agents, providing two high-level features:

• It lets you build LLM-based agents from graphs.

• It enables the customized and automatic self-organization of agent swarms with self-improvement capabilities.

Seed-Coder (previously known as Doubao-Coder) is a family of lightweight yet powerful open-source code LLMs comprising base, instruct and reasoning models of 8B size.

Seed-Coder demonstrates that, with minimal human effort, LLMs can effectively curate code training data by themselves to drastically enhance coding capabilities.

Seed-Coder represents our initial step towards contributing to the open-source LLM ecosystem. We look forward to seeing Seed-Coder drive advances in code intelligence and empower broader applications in the open-source LLM community!

Kuzu is an embedded graph database built for query speed and scalability. Kuzu is optimized for handling complex analytical workloads on very large databases and provides a set of retrieval features, such as a full text search and vector indices. Our core feature set includes:

• Flexible Property Graph Data Model and Cypher query language

• Embeddable, serverless integration into applications

• Native full text search and vector index

• Columnar disk-based storage

• Columnar sparse row-based (CSR) adjacency list/join indices

• Vectorized and factorized query processor

• Novel and very fast join algorithms

• Multi-core query parallelism

• Serializable ACID transactions

• Wasm (WebAssembly) bindings for fast, secure execution in the browser

Below The Fold

Scrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. For both beginners and experts, Scrapling provides powerful features while maintaining simplicity.

PgDog is a transaction pooler and logical replication manager that can shard PostgreSQL. Written in Rust, PgDog is fast, secure and can manage hundreds of databases and hundreds of thousands of connections.

CoreControl is the only dashboard you'll ever need to manage your entire server infrastructure. Keep all your server data organized in one central place, easily add your self-hosted applications with quick access links, and monitor their availability in real-time with built-in uptime tracking. Designed for simplicity and control, it gives you a clear overview of your entire self-hosted setup at a glance.

Articles

Stanford's Hazy Research wrote a post a while back. It compares attention structures with newer efficient and possibly attention-free alternatives, particularly focusing on associative recall(AR) capabilities in the comparison.

There has been a lot of interest in attention-free, efficient LLM architectures that aim to challenge the dominance of Transformer-based models as attention is really expensive and hard to scale due to how much compute that it requires for scaling. These new architectures include various Approximate Attention Methods(AAN), S4, Liquid S4, MEGA, GSS, H3, BIGS, Hyena, RWKV, RetNet, Monarch Mixer, Mamba, etc. While recent work suggested some of these architectures could match attention in benchmark quality, this post aims to investigate their performance more thoroughly, particularly focusing on a class of sub-quadratic Transformer alternatives called "gated-convolutions" (e.g., Hyena, H3, and RWKV).

The blog post reveals a significant performance gap in associative recall (AR) - the ability to recall information previously mentioned in the prompt. This capability has substantial implications for in-context learning and practical applications and how LLMs are being used in the real-world applications.

The evaluation setting uses pretrained 17 language models from scratch spanning four parameter scales (70M, 150M, 360M, and 1.4Bn) across various architectures to study the scaling properties of the various model architectures as well:

1. H3: Uses a short convolution followed by a long convolution, sandwiched by element-wise gating.

2. Hyena: Similar to H3 but differs in the parametrization and placement of convolutions.

3. RWKV: Often described as an RNN but can be rewritten as a gated convolution, differing in parametrization for convolution filters, ordering of gating, convolutions, and projections.

4. Pure long convolutions (S4-like): Uses a global/long convolution without gating.

5. Llama-style Transformers: Modern Transformer baseline with rotary embeddings.

All models were trained on uniform infrastructure and data (The Pile for 10B tokens, 50B for 1.4Bn parameter models) using the EleutherAI GPT-NeoX codebase.

They have found a consistent perplexity gap between state-of-the-art attention-free models and Transformer based architectures(attention-based). More significantly, they have identified that a single issue was responsible for more than 82% of this gap: poor performance on tokens requiring associative recall. Remarkably, a 1.4 billion parameter Hyena model underperformed a 70 million parameter attention model by over a perplexity point on the AR slice.

To measure AR capability in real LLMs, the post proposes a simple proxy: evaluating the model's perplexity on the subset of next-token-predictions that form completions to repeated bigrams, termed "AR Hits". For example, in phrases like "crosscut complex" and "Maison Bergey," the second occurrences of "complex" and "Bergey" are considered AR Hits because they can be predicted by recalling the prior occurrences of these bigrams in the context.

Critically, the researchers distinguished between bigrams that might be memorized during training (like "Barack Obama") and those that require genuine in-context recall. When plotting perplexity against the frequency of bigrams in training data, they observed a pattern: for AR hits appearing infrequently in training data, there was a large gap between gated convolution models and attention models. On other hand, tokens and frequently appearing AR hits, there was virtually no gap between the models.

This finding suggested that in-context recall is the fundamental issue separating gated convolution models from attention-based ones and why attention based models are superior for applications that require in-content recall in the model properties. The researchers further confirmed this by comparing RWKV-Raven 7B and Llama 2 7B, finding that the gap persists at larger scales and increases as models need to conduct more recalls per input sequence.

When examining the AR capabilities though, the evaluation setup had a big problem: gated convolutions could solve synthetic AR tasks perfectly in previous work, yet showed significant gaps on real-world language data. To resolve this gap in the evaluation methods, they developed a new synthetic task called Multiple Query Associative Recall (MQAR).

The key insight was that prior synthetic formulations assumed one query per input at a fixed position, with tokens from a small vocabulary (less than model dimension). However, real language modeling often requires performing multiple recalls at varying positions with tokens from large vocabularies (larger than model dimension).

MQAR better reflects these real-world demands by:

1. Having multiple keys that need to be recalled in a single sequence

2. Placing keys at random positions in the sequence

3. Using a larger vocabulary size

Through theoretical analysis and experiments, post mentions that:

Even though gated convolutions are sub-quadratic in sequence length, the architecture requires larger model widths (dimensionality, hidden size) than attention to solve MQAR effectively.

They validated these theoretical results by constructing BaseConv, a canonical, simplified gated convolution model that can provably simulate all other models built from gating and convolution primitives including H3, Hyena, and RWKV. The experiments among these architectures, they observed that model dimension needs to grow with sequence length to solve MQAR as well as attention does.

While all the architectures can solve the AR task, the scaling required to do so becomes increasingly problematic with longer sequences, negating some of the efficiency benefits of gated convolutions. Interestingly, another way to look at this is that transformer based or attention based model architectures might be more efficient on learning for a given/finite sequence length than the ones that are attention-free.

After empirical results, the post also tries to build a hypothesis on a theoretical ground to explain the results, especially on scaling properties of attention-free methods. In order to do so, they manually set weights of gated convolution models to solve associative recall and demonstrated why poor scaling emerges from these architectures.

They define a minimal representation of gated convolution called BaseConv, which can simulate all other architectures built from gating and convolution primitives. In code, BaseConv is elegantly simple:

v = conv(u) // convolution
w = linear(u) // projection
y = v * w // gating

• A convolution between two discrete sequences can be viewed as a polynomial operation

• Gating (Hadamard product) between two sequences can also be viewed as a polynomial operation

• Therefore, an overall gated convolution model can be analyzed as a complex polynomial

In here, main idea is to compare operation in the attention function and operation in the BaseConv.

Both attention and convolutions take input sequences of embeddings and output sequences of the same shape by applying a linear transform that "mixes" the embeddings together. However, they differ from one main dimension is that:

• In Attention: The mixing matrix is a function of the input

• In Gated Convolutions: The mixing matrix is defined by model parameters and is NOT a function of the input

How Attention solves MQAR

Attention can solve MQAR using two layers:

• Layer 1 (Shift-by-one): Shifts each key by one position so the output embedding contains both the MQAR key and value (e.g., by storing the MQAR key in the first dimensions and the MQAR value in the rest).

• Layer 2 (Lookup): Performs long-range lookups using the MQAR key part of the embedding as attention keys and the MQAR value part as attention queries and values. This makes it easy to find matching MQAR keys by computing pairwise similarity of attention query and key embeddings.

Crucially, attention can solve this task with model dimension independent of sequence length (it just needs to be large enough to store two token representations).

How Gated Convolution solves MQAR

In contrast, gated convolutions require more complex solutions that scale with sequence length. Their two-layer solution works as follows:

• Layer 1 (Finding Matching MQAR Keys): Uses convolution and gating to compare each token to all other tokens to find matching MQAR keys.

• Layer 2 (Outputting MQAR Values): Uses the matches identified in the first layer to output the appropriate MQAR values.

  

In the literature, an architecture's efficiency is typically measured in terms of the asymptotic cost of a layer. If there's one takeaway lesson from our work, it's this: taken alone, the complexity of a layer is an inadequate measure of efficiency

In here, the finding is that both the evaluation method and how the model architecture responds to scaling need to be considered when we measure efficiency of model architecture.

What if we measured an architecture's efficiency in terms of the FLOPs required to solve a specific task, instead of the FLOPs required per layer?

In order to solve that, they come up with a new testbed for this synthetics called “Zoology” to measure efficiency.

Libraries

Zoology provides machine learning researchers with a simple playground for understanding and testing language model architectures on synthetic tasks. This repository can be used to reproduce the results in our paper Zoology: Measuring and Improving Recall in Efficient Language Models. See the section on reproducing paper experiments for details.

Mega: Moving Average Equipped Gated Attention is the PyTorch implementation of the Mega paper.

StripedHyena is the first alternative model architecture competitive with the best open-source Transformersof similar sizes in short and long-context evaluations.

StripedHyena is a deep signal processing, hybrid architecture composed of rotary (grouped) attention and gated convolutions arranged in Hyena blocks, with improved scaling over decoder-only Transformers. StripedHyena is designed to leverage the specialization of each of its layer classes, with Hyena layers implementing the bulk of the computation rjequired for sequence processing and attention layers supplementing the ability to perform targeted pattern recall.

• Efficient autoregressive generation via a recurrent mode (>500k generation with a single 80GB GPU)

• Low latency, faster decoding and higher throughput than Transformers.

• Significantly faster training and finetuning at long context (>3x at 131k)

• Improved scaling laws over state-of-the-art architectures (e.g., Transformer++) on both natural language and biological sequences.

• Robust to training beyond the compute-optimal frontier e.g., training way beyond Chinchilla-optimal token amounts

👾 Letta is an open source framework for building stateful agents with advanced reasoning capabilities and transparent long-term memory. The Letta framework is white box and model-agnostic.

vLLM is a fast and easy-to-use library for LLM inference and serving.

vLLM is fast with:

• State-of-the-art serving throughput

• Efficient management of attention key and value memory with PagedAttention

• Continuous batching of incoming requests

• Fast model execution with CUDA/HIP graph

• Quantizations: GPTQ, AWQ, INT4, INT8, and FP8.

• Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.

• Speculative decoding

• Chunked prefill

BAML is a simple prompting language for building reliable AI workflows and agents.

BAML makes prompt engineering easy by turning it into schema engineering -- where you mostly focus on the models of your prompt -- to get more reliable outputs. You don't need to write your whole app in BAML, only the prompts! You can wire-up your LLM Functions in any language of your choice! See our quickstarts for Python, TypeScript, Ruby and Go, and more.

Guidance is an efficient programming paradigm for steering language models. With Guidance, you can control how output is structured and get high-quality output for your use case—while reducing latency and cost vs. conventional prompting or fine-tuning. It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditionals, loops, tool use) and generation seamlessly.

Stanford Alpaca project aims to build and share an instruction-following LLaMA model. The repo contains:

• The 52K data used for fine-tuning the model.

• The code for generating the data.

• The code for fine-tuning the model.

• The code for recovering Alpaca-7B weights from our released weight diff.

Below The Fold

E2M is a Python library that can parse and convert various file types into Markdown format. By utilizing a parser-converter architecture, it supports the conversion of multiple file formats, including doc, docx, epub, html, htm, url, pdf, ppt, pptx, mp3, and m4a.

✨The ultimate goal of the E2M project is to provide high-quality data for Retrieval-Augmented Generation (RAG) and model training or fine-tuning.

Core Architecture of the Project:

• Parser: Responsible for parsing various file types into text or image data.

• Converter: Responsible for converting text or image data into Markdown format.

Generally, for any type of file, the parser is run first to extract internal data such as text and images. Then, the converter is used to transform this data into Markdown format.

Yazi (means "duck") is a terminal file manager written in Rust, based on non-blocking async I/O. It aims to provide an efficient, user-friendly, and customizable file management experience.

Teable uses a simple, spreadsheet-like interface to create powerful database applications.

A shell compiler that converts shell scripts into secure, portable and static binaries. Unlike other tools (ie. shc), Bunster does not just wrap your script within a binary. It literally compiles them to standalone shell-independent programs.

Under the hood, Bunster transpiles shell scripts into Go code. Then uses the Go Toolchain to compile the code to an executable.

Bunster aims to be compatible with bash as a starting move. Expecting that most bash scripts will just work with bunster. Additional shells will be supported as soon as we release v1.

ty is an extremely fast Python type checker and language server, written in Rust.

Articles

Samuel Bowman has a rather interesting paper to talk about some of the properties of LLMs and the future direction on this area. He argues the following 8 things in the paper for LLM to be true:

1. LLMs predictably get more capable with increasing investment, even without targeted innovation.

2. Many important LLM behaviors emerge unpredictably as a byproduct of increasing investment.

3. LLMs often appear to learn and use representations of the outside world.

4. There are no reliable techniques for steering the behavior of LLMs.

5. Experts are not yet able to interpret the inner workings of LLMs.

6. Human performance on a task isn’t an upper bound on LLM performance.

7. LLMs need not express the values of their creators nor the values encoded in web text.

8. Brief interactions with LLMs are often misleading

I will try to expand these areas in the following parts:

1. LLMs Predictably Get More Capable With Increasing Investment, Even Without Targeted Innovation

A foundational insight is that the capabilities of LLMs improve predictably as a function of scale-measured in three primary dimensions:

• Model size (number of parameters)

• Training data volume

• Compute used for training (FLOPs)

This relationship is formalized in scaling laws paper, which show that as these dimensions increase, the model’s performance on a broad range of language tasks improves smoothly and predictably, often following power-law trends.

For example, the progression from GPT to GPT-2 to GPT-3 involved relatively minor architectural changes but massive increases in training compute (up to 20,000× more for GPT-3) and data, resulting in qualitative leaps in capability. GPT-4 continued this trend, outperforming humans on many professional exams.

This enables LLMs to demonstrate the following properties when it comes to scaling:

• Predictability: Scaling laws allow researchers to estimate performance improvements before training expensive models, reducing trial-and-error.

• Economic justification: The ability to forecast returns on investment has driven multi-billion-dollar funding rounds.

• Limited innovation needed: Most gains come from investing more compute/data rather than fundamentally new architectures or training algorithms.

2. Many Important LLM Behaviors Emerge Unpredictably as a Byproduct of Increasing Investment

While overall performance improves predictably, specific capabilities often emerge abruptly and unpredictably once the model crosses certain scale thresholds. This phenomenon is called emergent abilities. Some of these emergent abilities demonstrated in LLMs are:

• Few-shot learning: The ability to perform new tasks from just a few examples in the prompt, which was not present in smaller models.

• Chain-of-thought reasoning: The capacity to generate step-by-step reasoning improving performance on complex tasks.

These enable the following properties for LLMs:

• Uncertainty: Developers know that larger models will be better overall but cannot reliably predict which new skills will appear or when.

• “Mystery box” effect: Investing in larger models is akin to buying a black box with unknown but potentially valuable new capabilities.

• Planning challenges: Responsible deployment and preparation for novel capabilities require flexibility and ongoing monitoring.

3. LLMs Often Appear to Learn and Use Representations of the Outside World

Despite being trained solely on text prediction through training datasets compiled through various resources, LLMs develop internal representations that correspond to real-world concepts and abstractions:

• Semantic representations: Models encode color concepts in ways that align with human perception.

• Theory of mind: LLMs can infer what an author knows or believes and use this to predict text continuation.

• Spatial and object representations: Models track properties and locations of objects in stories, sometimes representing spatial layouts.

• Visual reasoning: Even without direct visual training, models like GPT-4 can generate instructions in graphics languages to draw objects.

• Game state tracking: Models trained on textual game move descriptions learn internal representations of board states.

• Common sense and fact-checking: LLMs can distinguish misconceptions from facts and estimate claim plausibility.

• Passing reasoning tests: LLMs perform well on benchmarks like the Winograd Schema Challenge, which require commonsense reasoning beyond surface text cues.

These capabilities enables the following properties for LLMs to demonstrate:

• Beyond next-word prediction: While technically LLMs predict text, their learned representations enable abstract reasoning and world modeling.

• Weak but growing: These abilities are currently imperfect and sporadic but improve with scale and training innovations.

• Augmentation: Integration with vision models and external tools further enhances world understanding.

4. There Are No Reliable Techniques for Steering the Behavior of LLMs

LLMs are pretrained to predict text continuations, but practical applications require them to follow instructions or behave in desired ways. Steering model behavior is challenging:

• Fine-tuning and instruction tuning: Adjusting model weights on specialized datasets can improve alignment but is costly and imperfect.

• Prompt engineering: Carefully crafting input prompts can guide outputs but is brittle and often unreliable.

• Reinforcement learning from human feedback (RLHF): Human preferences guide model outputs but cannot guarantee consistent behavior.

• Lack of interpretability: Without understanding internal mechanisms, it is hard to predict or guarantee model responses.

These properties create the following issues/problems for the LLMs:

• Unpredictability: Models can produce harmful, biased, or nonsensical outputs despite steering attempts.

• Safety concerns: Deploying LLMs safely requires extensive monitoring and fallback mechanisms.

• Research gap: Developing robust, reliable steering methods remains a major open challenge.

5. Experts Are Not Yet Able to Interpret the Inner Workings of LLMs

LLMs are deep neural networks with billions of parameters and highly distributed representations:

• Opaque internals: The learned weights and activations do not correspond to human-understandable concepts.

• Lack of interpretability tools: Current methods (e.g., attention visualization, neuron activation analysis) provide limited insight.

• Complex emergent phenomena: Behaviors arise from interactions of many components, not isolated modules.

• Ongoing research: Efforts in mechanistic interpretability aim to reverse-engineer model reasoning but are nascent.

These properties create the following issues/problems for the LLMs:

• Black-box nature: Understanding why a model produces a certain output is difficult.

• Trust and accountability: Lack of interpretability complicates debugging, auditing, and regulatory compliance.

• Safety risks: Hidden failure modes and biases may go undetected.

6. Human Performance on a Task Isn’t an Upper Bound on LLM Performance

LLMs have demonstrated superhuman performance on many benchmarks:

• Standardized exams: GPT-4 outperforms average qualified humans on the bar exam, SAT, and other professional tests.

• Speed and scale: LLMs can process and generate text orders of magnitude faster than humans.

• Novel capabilities: Some tasks, like large-scale code generation or multi-document synthesis, exceed typical human abilities.

These properties create the following issues/problems for the LLMs:

• Revising expectations: Human benchmarks are not ceilings; LLMs can surpass human performance in many domains.

• New applications: This opens possibilities for automating complex cognitive tasks.

• Ethical considerations: Superhuman performance raises questions about job displacement, decision-making authority, and AI governance.

7. LLMs Need Not Express the Values of Their Creators Nor the Values Encoded in Web Text

LLMs are trained on vast datasets scraped from the internet, which contain diverse and often conflicting viewpoints, biases, and cultural values:

• Value misalignment: Models may generate outputs that do not reflect the ethical or normative values of developers or society.

• Bias amplification: Models can reproduce or amplify harmful stereotypes present in training data.

• Steering attempts: Efforts to align models with human values (e.g., RLHF) are imperfect and context-dependent.

• Value pluralism: The multiplicity of values in training data means models may express contradictory or incoherent value systems.

These properties create the following issues/problems for the LLMs:

• Ethical challenges: Ensuring that LLMs behave in socially acceptable ways is complex.

• Governance needs: Transparency, auditing, and stakeholder engagement are critical.

• Customization: Tailoring models to domain- or community-specific values may be necessary.

8. Brief Interactions with LLMs Are Often Misleading

Users often form impressions of LLMs based on short, anecdotal interactions, which can be deceptive:

• Overestimation: Early impressive outputs may lead users to overestimate model understanding or reliability.

• Inconsistency: Models can produce contradictory or erroneous answers on repeated queries.

• Context sensitivity: Small changes in prompts or conversation history can drastically alter outputs.

• Evaluation difficulty: Measuring true model capabilities requires systematic, large-scale testing rather than isolated demos.

These properties create the following issues/problems for the LLMs:

• Caution in interpretation: Users and policymakers should avoid drawing conclusions from limited interactions.

• Need for rigorous evaluation: Benchmarking and adversarial testing provide more reliable assessments.

• User education: Training users to understand model limitations is essential for responsible use.

Overall, the paper talks about very interesting observations and properties for LLMs in 8 different dimensions that are both positive and negative aspects to pay attention to.

Libraries

SELFRec is a Python framework for self-supervised recommendation (SSR) which integrates commonly used datasets and metrics, and implements many state-of-the-art SSR models. SELFRec has a lightweight architecture and provides user-friendly interfaces. It can facilitate model implementation and evaluation.

Pipeline RL is a scalable asynchronous reinforcement learning implementation with in-flight weight updates. Designed to maximize GPU utilization while staying as on-policy as possible.

TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Proximal Policy Optimization (PPO), and Direct Preference Optimization (DPO). Built on top of the 🤗 Transformers ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.

FramePack is a next-frame (next-frame-section) prediction neural network structure that generates videos progressively.

FramePack compresses input contexts to a constant length so that the generation workload is invariant to video length.

FramePack can process a very large number of frames with 13B models even on laptop GPUs.

FramePack can be trained with a much larger batch size, similar to the batch size for image diffusion training.

Knowledge Graph Attention Network (KGAT) is a new recommendation framework tailored to knowledge-aware personalized recommendation. Built upon the graph neural network framework, KGAT explicitly models the high-order relations in collaborative knowledge graph to provide better recommendation with item side information.

Dataherald is a natural language-to-SQL engine built for enterprise-level question answering over relational data. It allows you to set up an API from your database that can answer questions in plain English. You can use Dataherald to:

• Allow business users to get insights from the data warehouse without going through a data analyst

• Enable Q+A from your production DBs inside your SaaS application

• Create a ChatGPT plug-in from your proprietary data

NeuralNote is the audio plugin that brings state-of-the-art Audio to MIDI conversion into your favorite Digital Audio Workstation.

• Works with any tonal instrument (voice included)

• Supports polyphonic transcription

• Supports pitch bend detection

• Lightweight and very fast transcription

• Allows to adjust the parameters while listening to the transcription

• Allows to scale and time quantize transcribed MIDI directly in the plugin

AudioCraft is a PyTorch library for deep learning research on audio generation. AudioCraft contains inference and training code for two state-of-the-art AI generative models producing high-quality audio: AudioGen and MusicGen.

mlx-audio is a text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon.

Below The Fold

Posting is an HTTP client, not unlike Postman and Insomnia. As a TUI application, it can be used over SSH and enables efficient keyboard-centric workflows. Your requests are stored locally in simple YAML files, so they're easy to read and version control.

With CubeCL, you can program your GPU using Rust, taking advantage of zero-cost abstractions to develop maintainable, flexible, and efficient compute kernels. CubeCL currently fully supports functions, generics, and structs, with partial support for traits, methods and type inference. As the project evolves, we anticipate even broader support for Rust language primitives, all while maintaining optimal performance.

LazyDocker is a simple terminal UI for both docker and docker-compose, written in Go with the gocui library.

Prefect is a workflow orchestration framework for building data pipelines in Python. It's the simplest way to elevate a script into a production workflow. With Prefect, you can build resilient, dynamic data pipelines that react to the world around them and recover from unexpected changes.

With just a few lines of code, data teams can confidently automate any data process with features such as scheduling, caching, retries, and event-based automations.

DrawDB is a robust and user-friendly database entity relationship (DBER) editor right in your browser. Build diagrams with a few clicks, export sql scripts, customize your editor, and more without creating an account. See the full set of features here.

Index is the SOTA open-source browser agent for autonomously executing complex tasks on the web.

UIT is a library for performant, modular, low-memory file processing at scale, in the Cloud. It works by offering a 4-step process to gather a file hierarchy from any desired modality, apply filters and transformations, and output it in any desired modality.

• performance: speed is of essence when navigating and searching through large amounts of data

• low-memory by applying streaming and parallelization we can run this in low-memory environments such as Cloudflare workers

• modular: modularity is beneficial because by making it composable we get a clear high-level overview of all building blocks. also, not all building blocks can be ran in the same runtime or location.

Articles

Traditional search engines on Pinterest relied heavily on user engagement signals and keyword-based retrieval, which often failed to capture nuanced relevance between a user's query and the diverse, multimedia content as they cannot be described or interpreted through the keywords. Pinterest wrote a great blog post on their approach to solve this problem through leveraging large language models (LLMs) to directly model and improve search relevance, resulting in good gains in user experience and engagement.

The article talks about main 4 different directions:

1. LLM Based X-Encoder

2. Pin Data Representation for LLM

3. Knowledge Distillation

4. Semi-Supervised Learning

which I will expand a bit more and technical details in the following sections:

1. LLM Based X-Encoder

Pinterest introduced a cross-encoder(X-Encoder) LLM as a "teacher" to predict the relevance of Pins to search queries. This model takes both the query and detailed Pin text features as input and outputs a multiclass relevance score, trained using human-annotated data and cross-entropy loss. This approach makes the search to be better than traditional keyword matching, allowing for a deeper semantic understanding of both queries and content of the pins.

LLM-Based X-Encoder Architecture

• Input: Both the search query and enriched Pin text features are concatenated and fed into the model.

• Model: Fine-tuned transformer-based architectures (e.g., BERT, T5, mDeBERTa, XLM-RoBERTa, Llama-3–8B) are used as cross-encoders.

• Task: Multiclass classification, predicting a 5-level relevance score (from "not relevant" to "highly relevant").

• Training: Uses human-annotated data, minimizing cross-entropy loss.

2. Pin Data Representation for LLM

To maximize the LLM’s effectiveness on both content and query pairs, Pinterest engineered a comprehensive set of text features for each Pin:

• Pin titles/descriptions: Direct user input.

• Synthetic captions: Generated using BLIP model, providing descriptive text for images.

• High-engagement query tokens: Captures search terms that historically led to high engagement with the Pin.

• Board titles: Context from user curation.

• Link titles/descriptions: Additional context from external sources.

This multi-source representation of Pins ensure high coverage for queries and quality, enabling the model to better represent and understand and match the intent behind user queries. Ablation studies also show that adding each feature incrementally improves the model’s predictive accuracy, highlighting the importance of feature engineering.

3. Knowledge Distillation

While the LLM-based teacher model is highly accurate, it is computationally expensive and therefore not feasible for real-time production use at Pinterest’s scale. To address this, Pinterest uses knowledge distillation: the teacher model generates relevance labels for billions of query-Pin pairs, which are then used to train a lightweight "student" model. This student model is further optimized for speed and efficiency can be deployed in production to serve search results in real time.

More details with regards to teach and student model and their training paradigm are in the following:

• Teacher model (LLM X-encoder): Used offline to generate relevance labels for billions of query-Pin pairs.

• Student model: A lightweight feed-forward neural network that ingests:

  • Query-level features (interest embeddings, SearchSAGE query embeddings)

  • Pin-level features (PinSAGE embeddings, visual/image embeddings, SearchSAGE Pin embeddings)

  • Query-Pin interaction features (BM25/text match scores, historical engagement rates)

4. Semi-Supervised Learning

By leveraging the teacher model to label massive amounts of previously unlabeled data, Pinterest significantly expands its training set beyond what is feasible with manual annotation. This semi-supervised learning approach not only increases the volume of training data but also enhances generalization to new languages and concepts, as the teacher model is multilingual and can adapt to seasonal and global trends.

• The teacher model, being multilingual, enables the system to generalize to new languages and concepts not present in the original human-labeled data.

• This approach allows Pinterest to scale relevance modeling globally, adapting to new trends and seasonal content without requiring manual annotation for every locale.

Results are pretty impressive:

• LLM-based models (especially larger ones like Llama-3–8B) significantly outperform both traditional embedding-based models and smaller language models in predicting relevance.

• Online: A/B tests show over 1% improvement in search feed relevance (nDCG@20) and over 1.5% improvement in search fulfillment rates globally, including in countries and languages not represented in the human-annotated training data, which makes the existing approach to be very generalizable and scalable outside of the training datasets of the Pins.

InstructPipe is a research prototype developed by Google to streamline the creation of machine learning (ML) pipelines in visual programming environments. By leveraging large language models (LLMs) and natural language instructions, InstructPipe automates the selection and connection of nodes in a visual editor, making the process accessible to both novices and experts.

Visual programming frameworks, such as Visual Blocks for ML, allow users to construct computational workflows by connecting modular blocks in a node-graph editor. This low-code approach is designed to lower the barrier for ML prototyping, enabling users to focus on high-level logic rather than intricate coding details.

However, even with visual programming, new users often struggle to set up pipelines from scratch. They must identify, select, and connect appropriate nodes from a blank workspace, which can be daunting and time-consuming, especially for those unfamiliar with ML concepts or the available node types.

InstructPipe addresses these challenges by introducing an AI assistant that translates human instructions into functional visual programming pipelines. Users can describe the desired pipeline in natural language, and InstructPipe automates much of the pipeline construction process.

In order to build such a system, it uses the following components:

• Node Selector (LLM Module 1): Given a user instruction and a pipeline tag (e.g., “multimodal”), this module identifies a list of potentially relevant nodes. It uses brief node descriptions to filter out unrelated options.

• Code Writer (LLM Module 2): Receives the selected nodes and user input, then generates pseudocode that defines the structure and connections of the pipeline. This module is provided with detailed node descriptions and examples for context.

• Code Interpreter: Parses the generated pseudocode and renders the pipeline as a directed acyclic graph (DAG) in the visual editor, enabling further human-AI collaboration.

Pipelines in Visual Blocks are typically represented as verbose JSON files. InstructPipe introduces a pseudocode format that is highly token-efficient, compressing a 2,800-token JSON pipeline into a 123-token representation. This concise format maintains essential structural information while sacrificing some fine-grained annotations.

Libraries

TritonAcademy has a number of resources for Triton and tooling around Triton. Triton is an open-source programming language and compiler designed specifically for GPU programming. It aims to simplify the development of efficient GPU kernels by providing a higher-level abstraction than CUDA or other low-level GPU programming models.

Triton enables developers to write high-performance GPU code with Python-like syntax while automatically handling many low-level optimizations that would otherwise require significant expertise in GPU architecture. It was developed by OpenAI and is now widely used in machine learning and scientific computing applications.

Tutorials for Triton can also be a good accompanying site to provide complementary resources.

LightlyTrain brings self-supervised pretraining to real-world computer vision pipelines, using your unlabeled data to reduce labeling costs and speed up model deployment. Leveraging the state-of-the-art from research, it pretrains your model on your unlabeled, domain-specific data, significantly reducing the amount of labeling needed to reach a high model performance.

This allows you to focus on new features and domains instead of managing your labeling cycles. LightlyTrain is designed for simple integration into existing training pipelines and supports a wide range of model architectures and use-cases out of the box.

Why LightlyTrain

• 💸 No Labels Required: Speed up development by pretraining models on your unlabeled image and video data.

• 🔄 Domain Adaptation: Improve models by pretraining on your domain-specific data (e.g. video analytics, agriculture, automotive, healthcare, manufacturing, retail, and more).

• 🏗️ Model & Task Agnostic: Compatible with any architecture and task, including detection, classification, and segmentation.

• 🚀 Industrial-Scale Support: LightlyTrain scales from thousands to millions of images. Supports on-prem, cloud, single, and multi-GPU setups.

Chain-of-Experts (CoE) changes sparse Large Language Model (LLM) processing by implementing sequential communication between intra-layer experts within Mixture-of-Experts (MoE) models.

Mixture-of-Experts (MoE) models process information independently in parallel between experts and have high memory requirements. CoE introduces an iterative mechanism enabling experts to "communicate" by processing tokens on top of outputs from other experts.

Experiments show that CoE significantly outperforms previous MoE models in multiple aspects:

• Performance: CoE with 2x iterations reduces Math validation loss from 1.20 to 1.12

• Scaling: 2x iterations matches performance of 3x expert selections, outperforming layer scaling

• Efficiency: 17.6% lower memory usage with equivalent performance

• Flexibility: 823x increase in expert combinations, improving utilization, communication, and specialization

These advantages constitute a "free lunch" effect, enabling efficient scaling of LLMs.

Mem0 (pronounced as "mem-zero") enhances AI assistants and agents with an intelligent memory layer, enabling personalized AI interactions. Mem0 remembers user preferences, adapts to individual needs, and continuously improves over time, making it ideal for customer support chatbots, AI assistants, and autonomous systems.

DiffMM is a new multi-modal recommendation model that enriches the probabilistic diffusion paradigm by incorporating modality awareness. It utilizes a multi-modal graph diffusion model to reconstruct a comprehensive user-item graph, while harnessing the advantages of a cross-modal data augmentation module that provides valuable self-supervision signals.

Reinforcement Learning (RL) with rule-based rewards has shown promise in enhancing reasoning capabilities of large language models (LLMs). However, existing approaches have primarily focused on static, single-turn tasks like math reasoning and coding. Extending these methods to agent scenarios introduces two fundamental challenges:

1. Multi-turn Interactions: Agents must perform sequential decision-making and react to environment feedback

2. Stochastic Environments: Uncertainty where identical actions can lead to different outcomes

RAGEN addresses these challenges through:

• A Markov Decision Process (MDP) formulation for agent tasks

• Reason-Interaction Chain Optimization (RICO) algorithm that optimizes entire trajectory distributions

• Progressive reward normalization strategies to handle diverse, complex environments

STORM: Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking is an LLM-powered knowledge curation system that researches a topic and generates a full-length report with citations.

marimo is a reactive Python notebook: run a cell or interact with a UI element, and marimo automatically runs dependent cells (or marks them as stale), keeping code and outputs consistent. marimo notebooks are stored as pure Python, executable as scripts, and deployable as apps.

VILA is a family of open VLMs designed to optimize both efficiency and accuracy for efficient video understanding and multi-image understanding.

Recoder is a fast implementation for training collaborative filtering latent factor models with mini-batch based negative sampling following recent work:

Tutorials

• Simons Institute and SLMath Joint Workshop: AI for Mathematics and Theoretical Computer Science

  • Bigram model:

    • A very simple language model based on counting consecutive tokens.

  • Transformer:

    • Implement and train a simple Transformer language model.

  • Addition:

    • Train a model for four-digit addition.

  • Triangle-free graphs:

    • Train a model to generate triangle-free graphs.

Below the Fold

• cobalt is a media downloader that doesn't piss you off. it's friendly, efficient, and doesn't have ads, trackers, paywalls or other nonsense.

  paste the link, get the file, move on. that simple, just how it should be.

Upscayl lets you enlarge and enhance low-resolution images using advanced AI algorithms. Enlarge images without losing quality.

Apple Section

I have recently started looking into CoreML and found some good resources in the intersection of ML and CoreML(machine learning framework by Apple):

• iOS-learning-materials is a resource for web-resources, tutorials, Stack Overflow and Quora Q&A, GitHubcode repositories and useful resources that may help you dig a little bit deeper into iOS. All the resources are split into sub-categories which simlifies navigation and management.

• neural-engine is a comprehensive repo that goes over neural engine aspects. Apple-Silicon-Guide is another resource for Apple’s specific chips and how to use them. CoreML Kit covers the software library that builds on top of the apple silicon.

• NSFWDetector is a small (17 kB) CoreML Model to scan images for nudity. It was trained using CreateML to distinguish between porn/nudity and appropriate pictures. With the main focus on distinguishing between instagram model like pictures and porn.

Lumina gives you an opportunity to skip having to write AVFoundation code, and gives you the tools you need to do anything you need with a camera you've already built.

This repository has a collection of Open Source machine learning models which work with Apples Core ML standard.

Apple has published some of their own models. They can be downloaded here. Those published models are: SqueezeNet, Places205-GoogLeNet, ResNet50, Inception v3, VGG16 and will not be republished in this repository.

SimilaritySearchKit is a Swift package enabling on-device text embeddings and semantic search functionality for iOS and macOS applications in just a few lines. Emphasizing speed, extensibility, and privacy, it supports a variety of built-in state-of-the-art NLP models and similarity metrics, in addition to seamless integration for bring-your-own options.

CoreML Model Zoo has a lot of models published by Apple and other companies.

Articles

Infini-gram is a new approach in language modeling, offering a modern revival of n-gram language models (LMs) at scale. This system processes n-gram queries with unbounded context length across trillion-token corpora with remarkable efficiency. The project scales traditional n-gram approaches to 5 trillion tokens—containing approximately 5 quadrillion unique n-grams—making it the largest n-gram language model ever created. Infini-gram achieves millisecond-level query processing, demonstrating that classical statistical language modeling approaches remain relevant and they are complementary to neural methods in the era of large language models(LLMs).

Infini-gram modernizes traditional n-gram language models in two fundamental ways:

1. massive scaling of training data

2. removal of context length constraints.

The system processes n-gram queries across an unprecedented volume of 5 trillion tokens, combining several major open-source text corpora including Dolma (3T tokens), RedPajama (1.4T tokens), Pile (380B tokens), and C4 (200B tokens). This represents the largest n-gram language model ever created, surpassing previous implementations by orders of magnitude.

The most significant technical innovation is the expansion of "n" from traditionally small fixed values (typically ≤5) to an unbounded approach, and therefore the name of "∞-gram LM." Traditional n-gram models were constrained to small context windows because the computational requirements grew nearly exponentially with increasing n-values. Infini-gram overcomes this limitation through a variant of the backoff approach, where the system resorts to smaller n-values only when longer n-grams have zero counts. This enables the model to utilize the maximum possible context, significantly improving prediction accuracy compared to fixed-length n-gram models.

The technical architecture powering Infini-gram is based on suffix arrays, a data structure that stores the ranking of all suffixes of a byte array. The byte array represents the concatenation of all tokenized documents in the training corpora. The suffix array occupies O(N) space and can be constructed in O(N) time, making it remarkably efficient for the scale involved. This approach eliminates the need to precompute and store massive n-gram count tables, which would be prohibitively expensive for unbounded n-values.

Performance metrics demonstrate extraordinary efficiency: n-gram counting operations complete in approximately 20 milliseconds regardless of n-gram length when querying the RedPajama corpus (1.4T tokens). N-gram language model probability estimation and decoding functions remain under 40 milliseconds per query, while the ∞-gram functionality takes slightly longer (under 200 milliseconds) as it must determine the longest possible n. Perhaps most impressively, these operations require minimal computational resources, functioning effectively with only CPU and RAM, as the index can remain on disk during inference. The system requires "0 GPU for both training and inference," distinguishing it from resource-intensive neural approaches.

The ∞-gram framework enabled novel analyses of both human-written and machine-generated text. ∞-gram language model achieves surprisingly high accuracy for next-token prediction at 47%, outperforming traditional 5-gram models which achieve only 29%. Prediction accuracy increases significantly when utilizing larger context windows and when ∞-gram estimates are sparse.

More details about the approach and paper are available in here and code is also available in GitHub.

Eugene Yan wrote an excellent piece on how LLM can be used in recommendations systems by reviewing a number of different papers from companies that work on or does research in recommendations space.

I want to categorize these into 4 main areas:

1. Augmented Model Architecture → LLM for Recsys

2. LLM for Data

3. LLM for Scale on a budget

4. Unified Model Architecture → LLM as Recsys

1. LLM for Recsys

Recommender systems are evolving beyond traditional ID-based approaches by integrating semantic understanding through LLMs and multimodal fusion. These architectures address cold-start challenges while improving interpretability.

Semantic IDs (YouTube)

Innovation: Replaces arbitrary item IDs with content-derived identifiers that preserve semantic relationships.
How:

1. Multimodal Encoding: A transformer model processes video frames and audio tracks into dense embeddings, capturing temporal relationships through cross-attention layers.

2. Hierarchical Compression: The Residual Quantization VAE progressively compresses embeddings into 8 discrete codes, where each layer refines the reconstruction error from previous steps. This allows efficient nearest-neighbor searches while maintaining item similarity.

3. Adaptive Tokenization:

   • N-gram Hashing splits IDs into fixed-length segments for embedding table lookup, enabling partial matches.

   • SentencePiece learns variable-length subword units from ID distributions, better handling long-tail items through adaptive grouping.

Why It Works: By encoding content features directly into IDs, the system preserves item relationships even for new entries. The hierarchical compression balances reconstruction accuracy with storage efficiency.