Inspiration

This post is dedicated to Dwarkesh Patel. He continuously keep bringing up the topic of continual learning with his guests. As he rightly points out, we can not say we have arrived at AGI unless we solve continual learning. There have been many mechanism for continual learning. Some involve building external memories, but until models can not learn to update their weights with every single interaction - just the way human re-wire neural pathway with every interaction - we can not really say we have arrived at AGI.

Preface

Continual learning is, in my view too, the holy grail for AGI. A model that cannot learn after deployment is frozen at its training cutoff - every hour it is used produces no durable improvement, and every new interaction starts from the same blank slate. An LLM model that accumulates the model of its world from its own experience, without a retraining run would be a massive leap over today’s models and help us arrive at AGI.

The observation that started this essay are simple facts about Kimi' K3: No ROPE and fixed state.
1. First of all, no ROPE means practically infinite context (in theory).
2. Of K3’s 93 attention layers, 69 i.e. about 75% are KDA layers whose entire memory is a fixed: 128 × 128 state matrix per head, however long the context gets. Only 24 layers i.e. 25% are Gated MLA layers with a KV cache that grows linearly with context. What is so special about K3 is despite having 75% KDA or linear layers, it performs near the frontier - it is the first model to do that. In other words, three quarters of the model’s attention runs on fixed-size, lossy, compressed memory (KDA) without giving up much capability; exact token-level cache in every layer is apparently not required for the frontier performance!

There is a familiar analogy. The human brain is also a fixed-size container, and we plainly do not remember everything that happens to us: we keep the important ideas, let the rest go, and what we retain is less a recording than an updated world model. For the details we cannot afford to keep, we rely on external media — phonebooks are a classic example. Our intelligence lives mostly in a compressed model, not in an archive. KDA works the same way: bounded capacity, error-driven updates, decay that keeps what is rehearsed and fades what is not, generalisation instead of verbatim storage.

Having known Yang Zhilin’s work (CEO Moonshot), he is not making choices these choices (like KDA, no ROPE) lightly!!! He could have simply followed DeepSeek’s MLA + DSA and scaled the model, but he did not…

While Kimi K3’s KDA shows a pathway towards continual learning with its fixed state that we will explore in detail, none of the core idea is specific to KDA only: any delta-rule linear attention has the same structure and same potential.. The idea of Linear transformers (Kimi K3 is Hybrid Linear - we will see more details soon) acting as fast weight programmers was articulated by Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber in their paper: Linear Transformers Are Secretly Fast Weight Programmers (2021), building on Schmidhuber’s fast-weight idea from 1992. While the idea may seem old, it was not taken as a serious idea, as Linear Transformers themselves were not mainstream. What Kimi’s KDA did was to give a proof that you can achieve frontier performance, making the idea mainstream now. This essay takes that 2021 observation, and asks concretely what it would take to turn that idea into a full blown continual learning architecture to take us to AGI.

Trivia: did you recognise who is the third person mentioned on the above paper? Yep!

What follows is a deep dive essay about a significant aspect of Kimi K3 i.e. its KDA state. It is not just a cache-able state, but a set of fast weights, trained by gradient descent on every token the model processes. Making it persistent is a plausible path to continual learning.

We see topics in the below order: the mechanism (§3), the exact loss and gradient that make the update a learning rule (§4), why the absence of RoPE helps (§5), three design proposals toward ‘real’ continual learning (§7), and the capacity and systems limits I see (§8).

Contents

Introduction

1. The problem: deployed models do not learn

2. Kimi K3 in brief

3. The KDA state, mechanically

4. The update is gradient descent on a loss

5. Why the absence of RoPE matters here

6. The thesis: the state is a world model in fast weights

7. Three proposals toward continual learning

8. Should we have a bigger state? How big a state we could have?

9. Summary

1. The problem: deployed models do not learn

A deployed language model has exactly two places where information about the current interaction can live. The first is the weights, which are frozen after training: they encode everything the model learned from its training corpus, and they do not change when you talk to the model. The second is the context — the token history in the prompt, together with the per-layer KV cache built from it. The context is writable: every new token adds to it, and the model can attend back to anything in it. But it is temporary. When the session ends, the cache is discarded, and the next session starts from the frozen weights plus nothing.

Continual learning means closing the gap between these two stores: the model should keep something it learned during deployment, across session boundaries, without a retraining run. Fine-tuning on user data is one answer, but it is offline, expensive, and risks catastrophic forgetting of the base weights. The interesting question is whether the architecture itself already contains a writable, persistent memory that is more compact and more “weight-like” than a raw token history.

Kimi K3’s attention design contains exactly such an object: the KDA state. It is a fixed-size matrix per attention head that is updated on every token by an explicit learning rule — one that is mathematically identical to an SGD step on a per-token loss.

2. Kimi K3 in brief (you may skip)

Kimi K3 is a 2.8-trillion-parameter mixture-of-experts model with 104B active parameters per token, 93 layers, and a 1M-token context window. Its attention is hybrid: 69 KDA (Kimi Delta Attention) layers interleaved with 24 Gated MLA layers in a 3:1 pattern — the ratio that Kimi Linear’s ablations found optimal. The MoE backbone (Stable LatentMoE, 896 routed experts with 16 active per token) and the cross-depth Attention Residuals are important to K3 but mostly orthogonal to this article’s argument; the two diagrams below are included for orientation.

The two attention types maintain very different runtime memories. Each Gated MLA layer keeps a standard KV cache (in latent form) that grows by a fixed amount per token. Each KDA layer keeps a fixed-size state matrix per head — d_k × d_v = 128 × 128 — no matter how long the sequence gets. The total memory picture per request:

So K3 already splits its runtime memory into a growing part (MLA cache) and a constant part (KDA state). The proposal developed later is, in essence, to make that split semantic as well: session-specific detail in the growing part, long-term learned structure in the constant part — and to carry the constant part across sessions.

3. Understanding the KDA state update

Each KDA head maintains a state matrix S_t ∈ ℝ^d_k×d_v. For each incoming token the head produces four vectors from its projections: a key k_t, a value v_t, a per-channel decay α_t ∈ (0,1)^d_k, and a write strength β_t. The state then advances by

S_t= (I− β_tk_tk_t^⊤) Diag(α_t)S_t−1+ β_tk_tv_t^⊤(1)

Reading (1) right to left, three things happen per token:

Reading from the state is a matrix–vector product. A query vector q_t produces the output o_t = S_t^⊤q_t: a query-weighted mixture of the state’s rows. The same operation written as a row vector, o_t^⊤ = q_t^⊤S_t, is how the chunkwise kernel computes it for many tokens at once:

The readout is what the rest of the network consumes at inference time. The diagram below shows the full read path, and contrasts it with the write path:

Two properties matter for everything that follows.

1. First, the state has fixed size: whether the model has seen a hundred tokens or a million, the memory is the same 128 × 128 matrix per head — all past experience is compressed into it, lossily.

2. Second, the update is local in time: S_t depends only on S_t−1 and the current token’s vectors. There is no replay buffer, no attention over history, no second pass. These are exactly the properties of an online learning system, and the next section shows that this is not an analogy but an identity.

4. Re-interpretation: The KDA state update is gradient descent on a loss

This section is very technical, you can directly jump to the last diagram in this section to see the results.

Define the decayed weights A = Diag(α_t)S_t−1 — the state after the forget step, before the new token is written. Before writing, the head predicts the current token’s value from its key using those weights: v̂ = A^⊤k_t. The prediction error is e_t = v̂ − v_t. Now score that prediction with a squared loss on the decayed weights:

L_t= ½‖A^⊤k_t−v_t‖² = ½‖e_t‖²(2)

Differentiating (2) with respect to the matrix A gives a rank-1 gradient (derived entry by entry below):

∇_AL_t=k_te_t^⊤(3)

and one SGD step from the decayed weights, S_t = A − β_t∇_AL_t, is exactly the update (1): the erase term is the part of the gradient step that removes the stale prediction along k_t, and the write term is the part that installs the corrected value.

The KDA recurrence is online gradient descent — specifically the classical delta rule (Widrow–Hoff LMS, 1960) with a per-channel weight-decay regularizer — run on the model’s own activations, at inference time, on every token. The following diagrams unpack each piece of that claim.

The loss involves matrices and vectors but is itself a scalar — it is just a sum of squared entry-wise errors, and the batch version replaces the vector error with an error matrix and the norm with the Frobenius norm:

Differentiating a scalar with respect to a matrix is defined entry-wise: the gradient is the matrix of partial derivatives, equivalently the unique matrix G satisfying dL = ⟨G, dA⟩_F under the Frobenius inner product. Both views, and a finite-difference check on all 64 entries of a toy state:

For completeness, the full derivation from the loss equation to k_te_t^⊤, by two independent routes - index calculus with the Kronecker delta, and matrix differentials with the trace trick:

Finally: why this loss and not another? Because any per-entry penalty ρ(e) produces a gradient of the same rank-1 form, k ψ(e)^⊤ with ψ = ρ′ — and the quadratic is the unique choice whose derivative is the identity, making the update linear in the error, exactly the delta rule:

5. Why the absence of RoPE matters here?

K3’s attention layers use no rotary position embedding. In the Kimi Linear design this is deliberate: the MLA layers are NoPE, and positional/recency information is carried entirely by the KDA layers’ data-dependent decay gates — ablations found NoPE with KDA outperforming RoPE, and RoPE base-frequency sensitivity is a known complication for context extension. This has a specific consequence for continual learning.

A RoPE-based memory encodes position into every cached key: a fact learned at position 50,000 of session 1 is stored rotated by 50,000 steps. Resuming such a cache in session 2 either continues the position counter (positions drift ever upward, into regimes the model may extrapolate poorly to) or resets it (every cached key is now mis-rotated). The KDA state has no such coupling. Recency in (1) is expressed by how many decay steps an association has survived, not by where it occurred. An association written a million tokens ago and never refreshed has simply faded by the product of the α’s along the way — the same fading whether those tokens were in one session or ten. The state is therefore a position-free checkpoint: it can be saved at the end of one session and loaded at the start of another with no re-indexing, no counter bookkeeping, and no rotational inconsistency. Whatever the state’s other limitations (§8), position handling is not one of them — an accidental but real enabler for the proposals below.

6. The thesis: the state is a world model in fast weights

Sections 3–5 establish the mechanism. Now we make the claim.

The frozen weights of K3 are a world model in the usual sense: compressed statistics of the training corpus, distilled by gradient descent over months of compute. The KDA state is the same kind of object at a different scale and speed. It is a set of fast weights: matrices that are not trained offline but are trained by the forward pass itself, one SGD step per token, on a prediction loss defined over the model’s own activations. Over a session, the state matrices across all heads in 69 layers accumulate a compressed, associative model of that session’s world: the documents read, the entities discussed, the corrections the user issued, the structure of the task.

Three properties make this look like a continual-learning substrate rather than just a cache:

• It learns by gradient descent. The update is not bookkeeping (append, evict); it is error-driven. The state moves downhill on a loss, which is what “learning” means everywhere else in the model.

• It generalizes by construction. The state does not store tokens; it stores directions in representation space. A probe key similar — not identical — to a written key retrieves a similar value, because the memory is a linear operator, not a table. This is the difference between a world model and a transcript.

• It is bounded and position-free. Fixed size means the cost of carrying it never grows; NoPE means carrying it across sessions is well-defined (§5).

There is one obvious gap between this and continual learning: today the state is thrown away at the end of a session. While the state is maintained across several interactions within a session, at the first token/interaction of a new session, every KDA head starts from S₀ = 0 and relearns the world from scratch. The machinery for accumulation exists; the persistence does not. The next section is about closing that gap.

7. Three proposals toward continual learning

Proposal 1 — the state-handoff experiment

Retain the KDA state across sessions while discarding the MLA KV cache, and measure what survives. The current serving arrangements reset both: for new session, we start with zero state and no MLA KV cache. The experiment changes one bit: at a session boundary, checkpoint all 69 (layers) × 96 (heads) state matrices (~0.22 GB total today, proposed to increase at least 10 times in the following text), drop the MLA cache entirely, and initialise the next session from the checkpoint instead of zeros.

Construct session 1 to teach the model things it cannot know from pre-training — a synthetic project’s conventions, a new codebase’s layout, a user’s preferences, fictional entities with defined properties. Then in session 2 initiated with KDA state from the first session, but with no MLA KV cache (25% of layers) and no repeated context, probe: does the model behave as if it knows these things? It is likely to remember a few details from the KDA state. However, in deep learning we can always train the model what we want, provided we have a dataset for tasks and know how to compute loss. So, we will need to create tasks that use KDA state from past sessions, and prompts that do not repeat knowledge that is likely to be in that KDA state.

Another direction is to eliminate MLA completely, as it complicates such adaptations. The model could only have large KDA states (discussed more in section 8.2) across 100% of the layers, with a tool to retrieve specific details from the past context.

Proposal 2 — disentangle the two memories by role

The KDA state should - ideally - hold the world model; the MLA KV cache should hold the specifics. Today the two stores are differentiated only by mechanism (compressed vs. exact). Can you explicitly encourage them to focus on different things? Below statements represent an action of multiplication was performed by the speaker. The only difference in them is specifics. When such prompts are processed by the model, the KDA states should be relatively the same, while MLA state can be very different.
1. “I multiplied 5 and 4”

2.”I multiplied 20 and 3”
One can achieve it by giving model a proper training signal for the same. You can design a loss that penalizes the model, if KDA states differ too much in such cases. Basically, we want to dis-entangle the presentation provided by KDA state and MLA state as much as possible.

However, this leads to philosophical question of what is knowledge and what is information! For example, are a project’s conventions knowledge or information? Hence, I believe, eliminating MLA completely from the model with a much larger KDA state could be the way to go (discussed more in section 8.2)

Proposal 3 — specifics that outlive a session go to external memory

The MLA cache still grows linearly, and over an indefinite continual-learning horizon it too becomes too big. At K3’s 27.6 KB per token for the 24 Gated MLA layers, a never-ending session reaches 29 GB at 1M tokens - and continual learning means there is no “end of session” at which to drop it. However, we can not store infinitely long KV MLA.

The resolution is to take Proposal 2 one step further for long-lived specifics: a phone number, an account ID, a deadline — these should live in an external store (a database, a key–value memory, a document store) and come back through tool calls. The model’s job is not to remember the phone number; it is to remember that there is a phone number, where it lives, and how to query for it. That meta-knowledge - the schema of one’s own external memory - is exactly the kind of compressed, relational fact the KDA state can hold, and hold it efficiently.

The split becomes three-tiered: 1) frozen weights for universal knowledge, 2) KDA state for the learned world model, 3) external memory for arbitrary specifics at arbitrary scale, with the MLA cache reduced to a true working buffer for the current session (or completely eliminated in the design itself).

8. Should we have a bigger state? How big a state we could have?

8.1 The state is small — and that bounds the thinking it can hold

The entire writable, learnable memory of K3 at inference is ≈ 0.22 GB (69 [layers] × 96 [heads] × 128 [dk] × 128 [dv] × 2 bytes). Compare: the frozen weights are 2.8T parameters, ≈ 1.4 TB at MXFP4: four orders of magnitude more. Everything the model can learn from an interaction, everything it can “think with” beyond the context window, must fit in that 0.22 GB. This is the hard ceiling on the world model of §6, and it impacts in three ways:

• Interference. When two experiences share key directions in representation space, the second write partially overwrites the first. Finite d² entries per head means collisions are inevitable over long horizons.

• Decay. State can be roughly viewed as a stack of memories. Every channel (row in the state) is multiplied by its channel’s α at every step. Knowledge that is never rehearsed has a shorter life as the state is biased towards the recent knowledge by design. We want a world model that is really long horizon. During the training the model learns to remember/to forget by controlling α. So there has to be ‘very long context learning’ training that will teach it to remember for longer horizon.

• Compression loss. 0.22 GB cannot hold the specifics of a long collaboration - which is precisely why Proposals 2 and 3 assign specifics elsewhere. But it also bounds how much relational structure the state can carry before newer structure crowds out older.

8.2 A model this size could afford a bigger state — what would it cost?

d_k = d_v = 128 is a design choice for efficiency. A 2.8T-parameter model could plausibly afford larger state: wider head dimensions, more heads, or a higher KDA:MLA ratio. Suppose the state were scaled up by an order of magnitude, to ~2.2 GB. Three consequences, in order of certainty:

• Forward-pass traffic. Decode is memory-bandwidth-bound, and the state is read (o = S^⊤q) and written (the rank-1 update) on every token, in every KDA layer. Today’s state traffic is 2 × 0.22 GB ≈ 0.43 GB per token — small next to the ~50+ GB of weight reads for 104B active parameters. At 10× state, state traffic becomes ~4.3 GB per token: still second-order, but no longer negligible, and it scales linearly while weight traffic stays flat (in this example, we are holding active parameters constant). On a device with ~2 TB/s memory bandwidth, for example, that is ~2 ms added to time-per-output-token on account of memory read-write alone - a direct latency hit on every generated token. Still not too bad…

• Compute. The per-token update is O(d_k·d_v) per head — cheap today, but quadratic in head dimension. Doubling d_k, d_v quadruples both state size and update FLOPs. Prefill kernels (chun-kwise, DPLR-form) scale the same way, so prefill cost per token rises too. But, at least in decode, FLOPs is not a big concern anyways.

• Serving resources. Persistent per-user states turn the state from a tiny object that you can choose to discard to a large object that you must carefully retain for months: 0.22 GB per user today, 2.2 GB at 10×, times concurrent users, with checkpoint/restore on every request and privacy/isolation requirements that KV caches don’t have because they are disposable (you can totally imagine user suing you to get back their KDA state :)!)

That said nothing we have discussed so far is difficult with today’s accelerators, HBM and networking technology: model weights stores fixed common knowledge, state stores learnable knowledge (per user), and state bytes are currently 1/6000th of the budget. Even a 10–50× larger state would be a rounding error against 1.4 TB of weights (at MFPP4).

The real question is whether training dynamics (decay schedules, β, the 3:1 ratio) still behave at that capacity, which is an empirical question, not an arithmetic one.

8.3 Other open risks

• Unverified accumulation. The state was trained to operate within one context. Whether a state accumulated over weeks of carried sessions remains well-conditioned — or drifts, saturates, or amplifies errors through the AttnRes path — is unknown without the Proposal-1 experiment.

• No protection against stale knowledge. The frozen weights at least benefit from training-time curation. A carried state learns from whatever the user said, including mistakes, with no validation pass.

• Evaluation is unsolved. Continual-learning benchmarks measure fact retention; measuring the quality of a carried world model — better priors, better calibration about a user’s environment — needs new evaluations, not just recall probes.

9. Summary

Kimi K3’s KDA layers maintain a fixed-size state matrix per head, updated per token by a rule that is exactly one SGD step on a squared prediction loss over the model’s own activations — loss ½‖e‖², gradient k e^⊤, step from the decayed weights.

The state is therefore a fast-weight world model: compressed, associative, generalising, and - because the architecture uses no RoPE - free of absolute-position coupling, so it can be checkpointed and resumed across sessions without inconsistency. What is missing is persistence: today the state resets to zero at every session boundary.

The path to continual learning is: (1) run the handoff experiment — carry the state, drop the MLA cache, and measure what survives; (2) disentangle the two memories — world model in the KDA state, session specifics in the MLA cache; alternatively experiment with dropping MLA completely (3) move long-lived specifics to external memory reached by tools, since even the MLA cache grows without bound on an indefinite horizon. The main limitation is capacity: 0.22 GB of state against 1.4 TB of frozen weights bounds how much the model can think with and remember, with interference and decay as the potential failure modes.

I believe, a model of K3’s scale could carry a substantially larger state; the cost is linear growth in per-token memory traffic (latency), quadratic growth in update compute per head dimension (but decode is not compute bound), and a new persistent-storage tier in the serving stack. Whether the learning dynamics scale with the capacity is the open question.

Let me remind you: Having known Yang Zhilin's work (CEO of Moonshot), he is not making choices these choices (like KDA, no ROPE) lightly!!! He could have simply followed DeepSeek's MLA + DSA and scaled the model, but he did not. They have been investigating this architecture for long, and there is a reason…

Sources

1. Schlag, I., Irie, K., Schmidhuber, J., Linear Transformers Are Secretly Fast Weight Programmers (ICML 2021) — linear attention as a fast-weight programmer, the delta rule as the state’s learning rule: arXiv:2102.11174. The fast-weight idea itself: Schmidhuber, J., Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks (Neural Computation, 1992).

2. Moonshot AI — Kimi K3 model announcement and architecture summary (2.8T parameters, 104B active, 69 KDA + 24 Gated MLA layers, 1M context): openlm.ai/kimi-k3

3. Kim et al., Kimi Linear: An Expressive, Efficient Attention Architecture — KDA update rule, SGD-on-decayed-state view, NoPE ablations, 3:1 hybrid ratio, 75% KV-cache reduction: arXiv:2510.26692 · code: github.com/MoonshotAI/Kimi-Linear

4. vLLM — A Preview of Production-Scale Kimi K3 Support (KDA-aware prefix caching, fused decode kernels, serving implications): vllm.ai/blog/2026-07-22-kimi-k3-preview

All diagrams were generated for this article with matplotlib; the worked numbers in the mechanism figures come from a small toy instance (d = 8, fixed seed), and the architecture figures (memory, MoE, AttnRes) from the public K3 docs.

The recent progress of AI in mathematics an OpenAI model’s disproof of the Erdős unit-distance conjecture in May 2026 [1], and the August 2026 announcement of ten results on long-open problems in mathematics and theoretical computer science produced by an internal version of OpenAI’s next model family, Astra [2][3] does NOT require an explanation in terms of a new kind of intelligence!

Don’t get me wrong, this is exceptional. May be this is the point at which we all will believe we entered the superintelligence era, when history books are written. But the mechanism that enable these models to perform maths is same that allows them to create powerful code. Also, it is likely to be a very simple overall system. “The better lesson” guarantees it. OpenAI takes that lesson very seriously.

Then why it took so long for models to become great at maths, compared to coding? The delay has three identifiable causes: less training data, the later arrival of verification infrastructure, and weaker economic incentives. This article states the mechanism, then examines each cause, then speculates what comes next for us as the society.

The mechanism is the same

The training recipe behind these results is definitely not new: scale, data, and back-propagation, followed by reinforcement learning with verifiable rewards (RLVR).

• The model generates candidate solutions token by token.

• The harness is the infrastructure around the model. It runs the problem-solving agent for a long time, fetches the data the agent needs (definitions, prior lemmata, tools), conducts verification of candidate outputs, and returns feedback to the agent so it can revise its work.

• The reward engine assigns a reward to a completed trajectory — for example, 1 if the proof checks, 0 otherwise. Sometimes you may choose to assign rewards to individual steps as well (’process reward’) if your reward engine is granular enough.

• RL then adjusts the model: it increases the probability of trajectories that earned high reward and decreases the probability of trajectories that earned low reward.

Nothing in this mechanism is specific to math (or coding). It transfers to any domain that satisfies two conditions: enough material to learn from, a large bank of tasks and a reliable way to score completed attempts. The question is therefore not “why could AI do mathematics at all” but “why did mathematics satisfy these two conditions later than coding did.”

CAVEAT - there is also possibility that there was no weight updating of general purpose LLM as in case of Google DeepMind’s AlphaProof Nexus (May 2026). They discovered the general purpose models are so much efficient that they do not need to update their weights - they could do their magic just with an agentic loop. They however couple the large model with a small model and use evolutionary techniques for superior results.

Quote from that paper that I recommend review: “To this end, we developed a framework, AlphaProof Nexus, for LLM-aided proof generation and used it to build a basic agent in which a set of subagents independently searches for proofs with feedback from the Lean compiler. We also developed a “full-featured” agent in which subagents are coordinated using an evolutionary algorithm and can use AlphaProof, a system for olympiad-level Lean theorem-proving based on reinforcement learning, as a focused proof tool.

Our full-featured agent autonomously solved 9 Erdős problems out of 353 attempted, including two questions that had been open for 56 years, at the inference cost of a few hundred dollars per problem. It also proved 44/492 open conjectures from the Online Encyclopedia of Integer Sequences (OEIS), resolved a 15-year-old open question on Hilbert functions in algebraic geometry, improved an open bound in convex optimization by discovering a novel algorithmic parameter schedule, identified several misformalizations in the literature, helped resolve an open problem from Ben Green’s well-known list and is aiding ongoing research efforts on quantum optics and graph theory.

To understand the impact of the agent design on these results, we did a post-hoc analysis of the performance of the full-featured and basic agents, as well as two agents with intermediate capabilities, on the 9 Erdős problems solved by the full-featured agent. Remarkably, the basic agent solved all 9 problems, though at a higher cost on the harder problems.”

1. Cause one: data

Coding had GitHub: hundreds of millions of repositories of working, reviewed, machine-readable code. Mathematics has no equivalent corpus. The largest library of formalized mathematics, Lean’s mathlib, contains roughly two million lines of code accumulated over nine years [9][13] — large for formal mathematics, but a small fraction of the available code corpus. Informal mathematical writing is additionally scarce, dense, and often paywalled.

A model can only learn what it can read, so in my opinion, strong mathematical capability had to wait for models that generalize better from less domain-specific data, and for techniques that manufacture training signal where natural data is thin (auto-formalization, self-generated problems).

2. Cause two: verification infrastructure likely arrived late

RL needs a reward signal, and a reward signal needs a cheap, reliable judge of completed attempts. Coding had one early: compilers, test suites, and CI pipelines. Mathematics acquired its equivalent only recently. The sequence of OpenAI’s three math announcements illustrates the infrastructure being built in stages (indirectly):

• October 2025. OpenAI researchers claimed GPT-5 had “found solutions to previously unsolved Erdős problems.” Within days it was clear the model had retrieved existing papers that the maintainer of

  erdosproblems.com

  was unaware of; the claims were withdrawn after public criticism [4][5]. There was no verification step at all, the failure was precisely a failure of verification.

• May 2026. The unit-distance disproof was validated by a panel of external mathematicians who read the argument and published a companion paper [1][6]. Verification existed, but it was human and slow.

• August 2026. The ten Astra results ship with Lean 4 formalizations: machine-checkable proofs with no unfinished goals and only standard axioms, published openly alongside the manuscripts [2][3][10]. Verification is now automated. Anyone can re-run the check without trusting the authors.

The August release also shows how seriously the verification layer is now engineered. The published repository pins the toolchain (Lean 4.32.0, mathlib) and includes “comparator challenges”: instructions for re-checking each proof by compiling it in a sandbox, exporting the proof terms, and replaying them through both Lean’s own kernel and an independently implemented external checker (nanoda, written in Rust), while confirming that the proved statement is identical to the claimed theorem and that no placeholder or hidden axiom was used [10][11]. This stack is the Lean community’s documented “gold standard” for validating proofs from untrusted sources — the documentation names “unaligned AI” as one of the scenarios it is designed for [12]. In other words, the verification infrastructure for machine-generated mathematics was built in anticipation of exactly this moment.

Big labs are ramping up their hiring of top talent in maths and science, so we can now expect the speed to accelerate further:

3. Cause three: economics

Labs allocate effort in rough proportion to expected return. Coding assistance is a market worth hundreds of billions of dollars; pure mathematics has little direct commercial value. This alone predicts that coding would be industrialized first, and it is sufficient to explain much of the sequencing without any claim that mathematics is intrinsically harder for these systems.

The August 2026 announcement adds a relevant data point: OpenAI reports that the inference used to produce all ten results would have cost roughly $2,000 at API prices [3]. Whatever the marketing value of that figure, the direction is clear: the cost of an attempt on a decades-old open problem has fallen from a research career to an API bill, and OpenAI’s own researchers state they did not spend heavily per problem and that test-time compute could be pushed much further [8].

When attempts are this cheap, the economic argument for ignoring mathematics disappears, and the field’s progress accelerates for the same reason coding’s did: many cheap, verifiable attempts.

Did they use a harness and tools?

The verifier was almost certainly inside the search loop, not only after it.

OpenAI has disclosed the verification artifacts but not the architecture of the system that searched for the proofs. I assert, as a matter of inference, that the search itself ran inside a harness with the verifier available as a tool, the same arrangement used in coding agents, and that not doing so would have been irrational. The reasons:

1. It is the standard pattern. In coding, no serious agent works one-shot: it writes code, runs the compiler and tests, reads the errors, and revises. The Lean compiler plays exactly the role of a compiler and test suite for proofs, and its error messages are free, immediate, and precise feedback. Discarding that signal would mean working with one hand tied.

2. Other labs say so explicitly. Describing their own prover agent’s formalization of the unit-distance disproof, Logical Intelligence states that “most issues can be checked using the compiler, checkers, and a comparator. This allows us to run many iterations of code improvement without human involvement” [14]. Harmonic’s Aristotle and DeepMind’s AlphaProof use the same generate-verify-revise loop. There is no known competitive approach that omits it.

3. The disclosed facts are consistent with it. OpenAI describes Astra’s work as long-running search in which candidate approaches were tried, unproductive branches abandoned, and arguments assembled and checked [2][3]. This would imply some sort of harness with tools.

4. The economics demand it. OpenAI reports roughly $2,000 of inference for all ten results [3], and its researchers say they “didn’t spend a lot on each problem” [8]. Cheap per-problem budgets are only achievable if failures are detected early and automatically — which is what a verifier-as-tool within a harness provides.

The assertion is an inference/speculation, but high confidence one.

Implication for other branches

If the delay is explained by data, verification, and economics rather than by any intrinsic difficulty of mathematics, then the same logic applies to the other quantitative sciences — physics, biology, medicine, weather, seismology — as each acquires its own version of the verification tool + harness (simulators, assays, formal models). OpenAI’s announcement frames the results in the same terms: open math problems are being used as a proving ground for scientific reasoning in general [2]. Mathematics is the first science (if you allow my equating maths to science) to get a complete verification loop. It is unlikely to be the last...

Conclusion & where we go from here as a society

AI progress in mathematics was late for mundane reasons: thin training data, the late arrival of automated verification, and weak commercial pull. All three constraints have now relaxed, and the results followed. The recent breakthroughs are evidence for the generality of the existing recipe - scale, data, back-propagation, a harness that runs the agent and checks its work, RLVR with a reward engine that scores completed trajectories properly - not evidence that mathematics required something more.

Furthermore, AI is always will outshine human mathematician (the way it has started outclassing human coders), as models are trained on vaster amount of data compared to what any single mathematician can consume across their life. It can connect far away dots (that is what was credited for success than OpenAI reported in May). However, that said, I do not think this is death of Mathematicians, as those gentlefolks are most likely to ask the models the right questions that will bring out model’s brilliance. We will continue to learn and teach maths for generations to come.

In fact, I expect far more people to take interest in mathematics now that they have great teachers in form of LLMs. They will further apply maths to far more problems. I expect massive progress in many domains including molecular biology, material science, astrology and many others. Golden age of maths has arrived! There is far more mathematical structure in the universe than what human mind alone can figure out. Now we have the machines of loving grace to help us. Digital camera, particularly phone camera, did not destroy the livelihood of great photographers. In fact, it created great appreciation for great photography, will many photographers making far more money that what they could in past. Likewise, I believe the golden era of mathematics is upon us.

Maths is in nature everywhere!

Construction of beautiful Museum of Future in Dubai. One of the most complex buildings in the world.

Complex molecular machines responsible processes on cellular levels.

References

1. OpenAI, “An OpenAI model has disproved a central conjecture in discrete geometry,” May 20, 2026.

   https://openai.com/index/model-disproves-discrete-geometry-conjecture/

2. OpenAI, “Ten advances in mathematics and theoretical computer science,” August 2026.

   https://openai.com/index/ten-advances-in-mathematics/

3. MLQ.ai

   , “OpenAI publishes ten claimed math advances, with formal peer review still pending,” August 2, 2026.

   https://mlq.ai/news/openai-publishes-ten-claimed-math-advances-with-formal-peer-review-still-pending/

4. TechCrunch, “OpenAI’s ‘embarrassing’ math,” October 19, 2025.

   https://techcrunch.com/2025/10/19/openais-embarrassing-math/

5. The Decoder, “Leading OpenAI researcher announced a GPT-5 math breakthrough that never happened,” October 18, 2025.

   https://the-decoder.com/leading-openai-researcher-announced-a-gpt-5-math-breakthrough-that-never-happened/

6. T. Samples, “OpenAI’s math breakthrough played to AI’s strengths,” Understanding AI, May 28, 2026.

7. S. Aaronson, “Dispatches from the possibly last days of human relevance,” Shtetl-Optimized, May 27, 2026. https://scottaaronson.blog/?p=9782

8. Hacker News discussion (including comments by OpenAI’s Noam Brown on cost per problem and failed attempts), “Ten advances in mathematics and theoretical computer science,” August 2026.

   https://news.ycombinator.com/item?id=49132058

9. “Growing Mathlib: maintenance of a large scale mathematical library,” arXiv:2508.21593, August 2025.

   https://arxiv.org/html/2508.21593v1

10. OpenAI, ten-proofs repository (Lean certificates for the ten results; build and independent-checking instructions).

    https://github.com/openai/ten-proofs

11. OpenAI, ten-proofs/ComparatorChallenges README (verification via landrun, lean4export, nanoda).

    https://github.com/openai/ten-proofs/blob/main/ComparatorChallenges/README.md

12. Lean Language, “Validating a Lean Proof” (comparator + external checkers as the gold standard for high-risk scenarios including “unaligned AI”).

    https://lean-lang.org/doc/reference/latest/ValidatingProofs/

13. K. Buzzard, “Human mathematicians are being outcounterexampled,” Xena, July 20, 2026 (Sol’s 1.2M-line Lean formalization in three weeks; mathlib at 2.3M lines over nine years).

    https://xenaproject.wordpress.com/2026/07/20/human-mathematicians-are-being-outcounterexampled/

14. Logical Intelligence, “Aleph Prover Uses Formal Methods to Disproof Erdős Theorem,” May 28, 2026 (iterative check-and-revise loops “without human involvement” using compiler, checkers, and comparator).

    https://logicalintelligence.com/blog/aleph-prover-erdos-disproof-lean-4-formal-methods

One of the details that has caught X community’s imagination is that Kimi K3 has constant state. In other words it does not have linearly growing KV cache as the context grows, or your conversation lengthens (for the most part - we will see details and caveats soon).

Long context use cases - e.g. agentic coding - has increased memory demand massively, resulting in 10 to 30 times increase in prices of memory stocks. If you don’t require as much memory to store KV cache, is this price increase justified? There is a vibrant debate going on this topic on X.

The detractors say Hybrid Linear attention used by Kimi K3 called Kimi Delta Attention (KDA) is around for quite some time. So it is already factored in prices of memory stocks.

However, they miss the point that - unlike previous attempts - Kimi K3 proved that Hybrid Linear could work so well at 1T+ parameter scale and deliver frontier performance. This is NET NEW information and deserves serious attention. Before Kimi K3 Hybrid Linear was believed not to be a technique for frontier but just a neat trick used by likes of
- Qwen (who never quite get to the frontier) or
- Nvidia (who tries everything under the sun, just because they have infinite GPUs).

Let us dive in!

(I want to make it clear - THIS IS NOT INVESTMENT ADVICE).

Brief history of Hybrid Linear:

Linear models always held great promise on paper just like Communism.

Instead of constantly growing KV cache, as context grew; they have a constant state that gets updated. The state size is constant whether you have thousand tokens or a million tokens.

It sounds to good to be true, because it is! Linear models have issues with recalling content from earlier parts of conversation perfectly - e.g. telephone number from earlier part of the conversation.

Why? This is because such models are constantly updating their internal ‘summary’ (called state) of the context. However, since this context/summary is fixed in size (e.g. 128*128 in case of Kimi K3, per head), its storage capacity is limited. It compresses information to be efficient (that is what the model learns during its training), but it has to drop some information for sure. In other words, the compression is lossy.

One way to solve the problem is to go for ‘hybrid linear’ where you combine linear layers (typically 75%) with full attention layers (25%). Hybrid Linear can have efficient version of full attention as well (e.g. DeepSeek style MLA in case of Kimi K3). While this was attempted, the earlier hybrid linear models did not have performance at the frontier level!

What Kimi K3 showed is that - with their variation called Kimi Delta Linear Attention (KDA) - you can achieve near perfect recall, as good as, that of models using full attention in all layer. This is a massive breakthrough, whatever you have been given to believe about it....

How did Kimi K3 achieve it?

Moonshot are not first to use hybrid linear, but they came up with a variation of it that is highly effective. It allows fine grained control over memory decay.

Off course they did many other things, involving data engineering, kernel development etc., use of attention residuals, stable latent MoE etc. (we will understand them towards the end). But they did something seemingly simple that resulted in exceptional performance gain on the attention part.

The model’s KDA state can essentially be thought of as a stack of memories (over simplified version). At every new token position a diagonal matrix is computed and the state is multiplied with it. This results in forgetting some memories more and some memories less (jargon: per channel decay). In earlier versions you would forget all the memories to equal extent. This trick of selectively and judiciously forgetting significantly improved the performance!

This is a beautiful visualisation that shows how state matrix updates happen. I will have another blog coming that explains, how this leads to the holy grail “continuous learning”.

How much memory saving is there with Kimi K3?

K3 has 93 layers and 96 heads. KDA has 128 dv and dk dimensions.

Let us see what it takes to process 128K tokens and 1 million tokens in terms of state and KV storage. Let us see two cases:

1. 93 MLA full attention layers (All-MLA) vs

2. 69 KDA + 24 MLA full attention (actual Kimi K3 model as depicted above)

At the bottom of the article I have posted the calculations.

Key observations:

• For All-MLA, KV cache storage grows from 13.7 GB to 107 GB from 128K to 1M tokens.

• As you can see, KDA state for Kimi K3 is constant at 0.22 GB regardless of the context length. Only the MLA part grows from 3.5 GB to 29 GB from 128K to 1M tokens context.

• In both cases i.e 128K and 1M context, there is saving of about 73%. The beauty is KDA does not compromise on frontier performance while offering these saving. MLA is already a massive saving over MHA and GQA (almost 90%), KDA save 73%!

Some will argue savings with DeepSeek V4 architecture at Kimi K3 scale would be even higher. True, DeepSeek V4 is not at the frontier yet. And even if it reaches the frontier, it will improve in exactly same direction: less storage.

This beautiful visualisation produced by Kimi K3 itself shows the savings:

Is Kimi K3 unique in the industry when it comes to such dramatic memory saving?

DeepSeek V4 (https://arxiv.org/pdf/2606.19348v1) has done exceptional work on cutting KV cache by 98%, but nobody considers them near frontier. When DeepSeek V4 was launched did you observe the same level of activity on X?

Also, for all their brilliance all the layers in DeepSeek V4 are either CSA (Compressed Sparse Attention) of HCA (Highly Compressed Attention) - both types are compressed along the sequence dimension and it is lossy. There is not a single layer that has uncompressed information, that is all the layers are lossy - recall in Kimi K3 25% of layers have full attention.

My hypothesis is that Kimi K3 will have better ability to retrieve specific information from earlier context in long context scenarios. Hence, I foresee Kimi K3 architecture will be widely adopted by other labs as well (e.g. ZAI, MiniMax, Qwen, Tencent, Xiomi etc.), just like DeepSeek V3 (MLA) and later DeepSeek V3.2 (MLA + DSA) became widespread in the industry.

What about Funda AI’s point that you need to cache KDA state multiple times, as it is not additive like KV cache? Also, will subagents not need KV cache storage?

Recall the previous example, total KDA state across 69 layers is 0.22 GB. So, even when you cache/snap-shot it at every 20K tokens during a long session, it is 50 times across 1M context.

So theses 50 spanshots will take 50*0.22 GB. It all adds up to just 11GB - total l (29.22 GB + 11 GB) stays well below 107 GB required for All MLA at 1M tokens context.

Second, Funda AI’s point was that main agents can spawn subagents. People will spawn many subagents. But typically, once a subagent returns the control to main again, its job is done. Its KV + state is deleted. So even if people launch agents that launch hundreds of subagents, KV + state needs are not long term. The main session / main agent’s state needs to be preserved much longer; and this where KDA saves a lot!!!

There are patterns where there are long running async agents, in those cases their KV + state would need to be maintained. But I see those patterns implemented rarely. This is not to say it won’t change in future.

Synchronous subagents (a very common pattern):

Asynchronous subagents (a relatively rare pattern):

Will KDA’s adoption then will have downward impact on memory demand?

I am not a fortune teller, but there is a small and non-zero probability of it busting memory bubble.

First why it may not have any impact on memory demand:

• While savings are real, if open source AI gets widely adopted, most deployments are not going to be as efficient as they are at closed frontier labs. Hence, any gains at the model level will be lost at industry level. We will have large number of small deployments that are not very efficient. E.g. for most enterprises & governments use happens across 12 hrs in a day. What their open source model infra and its memory going to do at night? Infra at large labs is used 24*7 due to global customer base.

• Also, we are making a big assumption that leading labs already don’t have similar approaches. OpenAI, Anthropic, DeepMind employ some of the smartest people in the world. So there is quite good chances that such innovations are already in use and have been factored into the memory demand.

That said every once in a while there is a SpaceX that comes and shows the industry leaders how their approaches were inefficient. They think from the first principals. Incumbents - due to organization structure issues, or due to not being hungry enough, do not innovate on key dimensions. History of capitalism is filled with thousands of such examples....

So it is an unlikely scenario - but still with some non-zero probability - that if large labs were not as starved for resources, they may not have explored the entire design space to reduce memory demand to such an extent.

In the AI industry itself, it has happened many times before where big labs have shown gross incompetence:

1. XAI’s training pipeline had poor MFU (GPU utilisation) - due to habit of throwing compute at the problem, plus other cluster misconfiguration and organization level issues.

2. Despite all the resources at their disposal, Meta did train a sh*tty version of DeepSeek’s MoE for Llama 4.

3. Google took 1.5 years since ChatGPT launch to bring Gemini to the forefront and it is still no one’s favourite model for the most popular use case: AI coding.

Constraints force innovation, so it is not unlikely that small highly well organised teams outshine large resource rich teams. And now that K3 is out and its architecture is well know, if KDA like technique - that is now successful at the frontier scale - is adopted by closed labs too; it means 75% reduction in memory demand at least for a few labs, and they are quite big individually, so the impact could be in tens of billions.

These memory savings could be eaten up with increased use due to reduction in cost for sure (Jevon’s paradox), but it is not guaranteed that consumption growth will always outstrips the savings....

Diffusion of other innovations: Kimi K3’s major innovation is not KDA alone!

Kimi K3 gets rid of positional encoding completely: Positional encodings inform what is sequence number of each token in the context. Its KDA layers, due to their recurrent nature do not need position encodings. They get rid of them even for MLA layers and it does not hurt. Benefits? It does not any special effort to extend model context length compared to earlier models. Elegant.

Kimi K3 implements attention residuals at 2.8T scale: It ensures, effectively, that higher layers are able to pay selective attention to inputs from lower layers. In earlier approaches higher layers could not distinguish between inputs from lower layers at a given token position, as all the values used to be aggregated. This new approach gives the model much higher discernibility. (https://arxiv.org/pdf/2603.15031)

Kimi K3 implements Nvidia’s Stable Latent MoE: The main benefit of a stable latent Mixture-of-Experts (LatentMoE) architecture is higher model accuracy and more expert capacity at the same or lower inference cost. It achieves this by projecting token representations into a smaller latent space before running expert routing and computation. They picked if from Nvidia’s Jan 2026 paper (https://arxiv.org/pdf/2601.18089v1)

With all these innovations put together, MoonShot team achieve 2.5 training efficiency over their previous architecture Kimi K2. This is to say, they achieve same validation loss at 2.5 times less FLOPs. That is massive saving even for training compute!!!! This will allow many more labs to train such high scale models with limited compute.....

MoonShot has also open sourced their FlashKDA GPU kernel for fast computation of KDA state and MoonMoE for dispatch and combine of tokens. They are pretty confident in sharing so much knowledge, because, they are already onto the next set of innovations that will take then even futher (at the end of day, they too are capitalists).

Scaling laws are not laws of nature, we can always discover better!

Furthermore, we may discover newer scaling laws with newer architectures and training paradigms, offering even denser intelligence. Scaling laws are statistical observations, they are not laws of nature that can not be changed. What this means is that with newer models, for smaller size (and lower KV and state) we can have better performance. So demand explosion does not mean running the same large models forever. Models will continue to get denser in terms of intelligence.

We have an ecosystem in form of Chinese labs that is highly constrained, and highly capable - a very potent mix, as discussed. They are discovering corners of the design space that have not been discovered by more well funded labs. They make rapid progress due to open sharing of knowledge. This saves others from wasting their efforts and reinventing the wheel. The evolution of architecture happen much more rapidly than otherwise....

Also, this success by MoonShot team is sure to inspire other equally capable teams across DeepSeek, ZAI, MiniMax, Tencent, Alibaba, ByteDance SEED and even emerging players like Baidu, Ant Ling, Xiomi Mimo, Meituan and many others to be the best, not just as an open source model, but to be the best overall.

Lot of new innovations will happen across emerging labs in the west, as well, whether it is at xAI or MSL now they have gotten their act together and started delivering great models.

That said, as I keep pointing out, China has most number of young AI researchers based on NEURIPS statistics. Contributions in this field come disproportionately more from younger researchers (the source thread), so many more breakthrough are likely to emerge from that part of the world.

China also has 3 major urban centers with AI talent density rivaling that in the Silicon Valley, they act as idea mixing crucibles (the source thread).

Banning open source AI will not help

An unfortunate conclusion of this discussion may be that we must ban open source AI to protect the western markets. But that would be highly counter productive.

Whether open source AI is banned or not in the western world, these design spaces will get explored, and papers will get published. West also has lot of new great labs emerging. They will discover lot of new ideas too, now that it has been realised there could be lot of efficiency gains that are still possible.
The best thing is to encourage and adopt these innovation and be responsible with investments and avoid wild speculation.

The future is very very bright due to this technology, but we have to be careful not be crazy speculators who get margin called.
We haven’t been using as much AI as we could be: whether it is in corporate world, education, research or entertainment.
1. While coding has been the the mainstay of GenAI token use, applications like CoWorker have just begun.
2. I would love for every student in India to have unlimited access to Opus 4.8 or Kimi K3 level models. They do have access, but to weaker models only.
3. Furthermore, we could have much higher value use cases of AI in Science, Engineering and Medicine.
4. Also, we have gotten an aging world that we need to care for. Ai is going to be greatly beneficial there.
There exist so many possibilities across entertainment and gaming as well. As Jensen Huang mentioned: “every pixel we see on the screen for entertainment will be generated”.

Conclusion

Stock market crash of 1929- the Great Depression - was not a result of demand destruction. It was a result of speculative excesses. The whole world sufferred for two decades. It arguably led to rise of Fascism and WW2.

While the future is uncertain, what is clear in my observations is that, AI bulls and memory bulls, in particular, do not factor in algorithmic innovations and discovering of newer scaling laws. For them everything gets eaten by Jevon’s paradox, hence not interesting. This attitude has percolated in retail investors as well and has led to insane behaviour that has potential to bring down economies of several countries. I want to introduce new angles into their analysis. They have far better information on supply chain situations and corporate earnings. I want to arm them with more technical perspective.

To crash memory & AI stocks, we don’t need massive reduction in demand, as Jaya Gupta has pointed out. Even small reduction in demand can start a game theoretical competition wherein existing bag holders start booking profits with intention to buy stocks later, this can crash the whole market. And this seems to have begun.....

It is important to not let such large bubbles form and burst catastrophically for the society. Overall I stay very optimistic about potential of this technology (and even memory demand) and future infra build out. It is the most exciting time to be alive!

(NOT INVESTMENT ADVICE)

Appendix

Detailed calculation of memory savings:

To show how good Kimi K3 is, I got K3 itself to do these calculation I have verified they are correct. As mentioned earlier we consider two cases:

1) Kimi K3: 69 KDA + 24 Gated MLA full attention vs

2) Full MLA: 93 MLA full attention

We can not have a world divided between Pax Silica led by United states and WAICO led by China. It is impossible to ensure AI’s benefits while avoiding any potential risks in that scenario.

Background:

Earlier this week Jensen set X on fire by joining and gaining nearly 1 million followers after posting his letter in support of Open Source AI.

Organization whose market cap is 15T USD plus have signed it:

This was resistance to seeming regulatory capture by closed labs to ban open source AI. X community and wider society is rightfully concerned of capture of this important technology by couple of labs and it got massively positive response. After OpenAI too signed the letter, Anthropic became the company that got cursed the most. They have been explicit about the concern about open source, and they have been quite careless about their communication….

Debate set off by OpenAI’s Dean Ball

Open AI’s Policy Advisor Dean Ball set off a vigorous debate after Kimi K3 was launched. He mentioned how different agencies of US government could create regulatory risk around using open source and discourage companies from using it.

Per him open source is dangerous to the business model of closed labs and prevents ‘innovation’ (really?). Dean Ball - as anyone can guess - came under intense fire as people believed he laid bare the game of regulatory capture. The tweet got 11 million views and got lot of negative attention on OpenAI (that said - Dean was probably stating what is likely to happen rather than what should happen. However, being OpenAI employee, he did not get benefit of the doubt).

Will Manidis provided a great rebuttal to Dean and argued such regulatory risk creation will be anti-constitutional and a malpractice. Per him, open source may hurt closed lab’s business model, but will be beneficial to the wider western economy. He passionately argued, ensuring closed lab’s profits is not the government’s job and they need to work harder for their profits. He got widespread support from the X community.

That said, OpenAI has made their stance clear by signing Jensen’s proposed letter.

Now the intense scrutiny shifted to Anthropic. Careless tweeting by Anthropic employee’s did not help Anthropic’s position…

Today Dario published his take on the matter, which is very balanced and very welcome. This to an extent should set the record straight. Thanks Dario. Before I write where I somewhat disagree and greatly agree with Dario, let me share the comments I liked the most from his letter:

Quote 1: “I agree with much of it (Jensen’s letter): open weights expand access to the AI economy, they strengthen competition at least for some use cases, and they give customers greater control. Concerns about distillation should be addressed through targeted legal and commercial frameworks—the same measure I described above.”

Quote 2: “Note that to be effective, testing would need to be global, which means even the CCP would need to be on board. I think this may actually be possible: as I wrote in The Adolescence of Technology, limited cooperation around preventing AI biological weapons may be possible because it is in China’s interest too.”

My comments on 4 key aspects of this letter:

1)Whose responsibility is it to prevent “Industrial scale distillation”,

2)Who gets to say which models are safe

3)Co-operation with CCP.

1.) Whose responsibility is it to prevent “Industrial scale distillation”:

First of all, failing to prevent distillation - if it is happening at an “industrial scale” - shows failures of security mechanism at big labs. Industrial scale means easy to detect by definition.

Big labs should work with law makers to treat model outputs as IP. - it should be inshrined in law. Logically, they would be required to treat all the material they train on as IP as well. I believe this will lead to a fairer world.

Treating model outputs as IP will allow FBI investigations regarding IP theft. If it is happening in other countries, USG’s diplomatic relationships should be used to restraint and punish (to the extent possible by relevant local law there). After all you use credit cards for Claude accounts it should be easy to trace people involved. Existing law enforcement and court system is sufficient of punishing individuals and labs involved.

I also believe since geopolitical competitor is involved, it should be possible to involve FBI as of today, even when model outputs are not treated as an IP.

What this discussion reveals, is big labs have a lot to learn and invest in security. It is understandable, as they are still new organizations. They should consult seasoned security organizations. Implementing KYC - which may impact revenue in short term - may be a good long term solution. I hope they do.

If distillation without logits is as effective as claimed, failing to prevent that is handing over advantage to geo-political adversary, for sure - and it is dangerous and should be a matter of concern to the whole western world and concerned decision makers in the big labs should be held responsible.

When an individual feels cold, they should wear a coat or stay inside a warm place, they/anyone should not set nearby forrest on fire for warmth. That is an analogy that must be kept in mind by policy makers when in-acting policy on this matter.

There is a detailed discussion on the distillation debate on the All-In-Podcast from last week. Take a look.

Another funny and interesting take on the distillation debate was from Parker Conrad (@parkerconrad). If distillation is not preventable and help reach frontier so easily, then probably you don’t need to worry about China pulling ahead, because you can distill them back :).

2.) Who gets to say which model is safe:

Yes, for industry body for testing - no lab, however well meaning, can be trusted to test their own models and declare them safe. But it should not be regulator that can be captured through lobbying by any side. It should have industry and academia stakeholders.

Just as in Air Travel industry, where every incident reveals some weakness of the system and makes system as a whole secure. Just like that every incident should make overall AI industry safer for humanity. Hence knowledge sharing is important.

In the interest of humanity, big labs need to publish in details of their interpretability, safety and guardrail techniques. It is for common social good.

Jensen is not stopping - he understand the need for AI security for AI to be widespread in out society being a responsible actor. He launched Open Secure AI Alliance today in collaboration with very credible names ins security business like PaloAlto Network, CrowdStrike and CloudFlare! It is also supported by large corporations like Nvidia and Microsoft. We need to have such industry body for AI safety testing.

3.) Co-operation with CCP:

Also, co-operation with CCP is a must. They are a force to reckon with whether you like it or not. No scheme will work without them. This is a welcome stance by Dario.

CCP does care about their legitimacy. It got threatened during Covid lockdowns and there were nationwide protests due to excesses of lockdowns.

Chinese labs are led by very capable leaders and they have exceptional talent whether it is at DeepSeek or Moonshot or ZAI or Xiomi. They too deeply care about humanity. There should be dialogue across the labs too on this matter. If they are not taking into confidence and on board, any AI security efforts are guaranteed to fail.

What about biological security risk posed by open models? Do open models alone pose such risks?

Dario is very vocal about it, as he thinks biological hazards are hard to prevent.

Quote: “But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true. For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case (as we saw with Operation Warp Speed). Questions like this should be empirically answered by rigorous pre-release testing, not assumed in advance.”

This is a clear and present danger. AI security is hard. Despite their best attempts, users have been able to jailbreak Anthropic models from time to time - recently Fable 5 was pulled off due to security failures, as per WSJ.

Anthropic’s classifiers also fail from time to time. Pliny the Liberator X account has been able to Jailbreak many closed source LLMs from time to time including Fable-5. So, the danger is not from open models alone and it can also be from closed models.

Many ‘black hat hackers’ use Claude for their hacking attempts due to it strong coding abilities, as noticed by Noah Lebovic (@NoanLevovic) - who works on dual use AI. He used to work at Anthropic before. Hackers succeed by not letting Claude know what it is being used for. Claude believes it is doing regular trouble shooting and is too happy to help you find bugs, which then are exploited by these hackers.

This is not to say Anthropic’s models are uniquely vulnerable. On the contrary, they get the maximum attention due to their popularity. Open models will likely have more of such vulnerabilities.

Now coming back to the biological risk, prevention of biological risk will need a global framework. It is not limited to having access to LLMs alone. For example, if you believe that Covid was made in a lab and escape accidentally, you will have to also note that we did not have frontier LLMs in 2019 and 2020 (yeah GPT-3 was launched around the same time as the pandemic start, but by today’s standards it was a toy LLM). We need something similar to Treaty on the Non-Proliferation of Nuclear Weapons for prevention of biological risks. Given every authoritarian country - including China, Russia, Iran, North Korea suffered due to Covid, there will be universal collaboration on this one.

The nature of biological weapons is such that you can not only attack your enemy without you and your society getting impacted too, so nation states are far less likely to attempt it - however authoritarian. That said - there are ISIS style attackers who do not care about wider society - may try it. Hence, we need to be extra careful. However, we must note that LLMs are not a pre-requisite to make dangerous viruses and bacteria. The knowledge already exists in the public domain.

Is a dialogue with China and CPP necessary, if there are export control?

Every national and nation state alliance like Pax Silica is well within rigths to deny technology that it deems critical to its geopolitical competitors. United states has strict export controls on export of advanced chips and semiconductor manufacturing equipment to China - particularly for advanced nodes. Dario argues strengthening of those controls. He also brings attention to potential smuggling of chips, as well as, access to chips by Chinese players in international data centers.

Fair. However, China has the largest scientific and engineering talent base in the whole world. It is impossible that they do not overcome semiconductor export bans on their own. It will take time, but they will get there. Huawei is already going to ship their Ascend 950 SuperPoDs from Q4 2026, and there are reports of trillion parameter models being trained on Huawei’s Ascend chips (e.g. LongCat 2.0 by Meituan). Huawei has figured out a way to build massive clusters of chips capable of training trillions of parameters of models through what they call Tau Scaling Laws.
So, China is surely - though slowly - likely to get there. Hence, no amount of export control invalidate the need for dialogue with China. I admire Dario proposing a dialogue with CCP.

Conclusion:

Absolute power corrupts absolutely, let us not go there!

Free competition is a key virtue of the western world. It has resulted in freedom and prosperity. AI Security Concerns are very important, but we should not allow them to lead to killing of competition and concentration of power. Absolute power corrupts absolutely. We need to have an well developed mechanism for AI safety testing and it can not be left to individual labs only. Jensen’s attempt to Open Secure AI Alliance, is a welcome step in that direction.

A scenario where China has free competition between hundreds of labs and research teams, and makes rapid progress and western world does not have such a competition will be dangerous. Any efforts for safety - which is very very important - can not scuttle competition.

Co-operation with China is not optional:

CCP for their rational self interest will co-operate on AI security. Believe it or not, they care about their people too. If you find it hard to believe, you can at least believe they care to preserve their power and legitimacy. Also, leaders of Chinese labs care equally about the humanity - there is no point in dehumanising them. Co-operation should be easy. After all we are all humans made up of bone and flesh. Any AI security effort that do not involve them as key stakeholders are doomed to fail.

Appendix

A note to Dario and Anthropic

Thanks for giving this world Claude - the machine of loving grace - and constant championing of AI security. So many lives have been made better because of Claude, and AI security is a very important topic. We do believe you truely care. We also thank you for resistance use of AI on surveillance on citizens even when you got hurt in the process (Anthropic’s letter to the Department of War). We may not agree with you every single time on the approach, but we are in awe of you and your contributions to humanity. Love you!!!

A note to maximalists on AI security

There are some maximalists (well-meaning) like Roon (@tszzl) of OpenAI who argue that this technology is so dangerous that under no circumstance we can have it open.
You can read my detailed reply to him here.

If AI is as dangerous as nuclear weapons, we can not have it in the hands of private players. There is no precedent anywhere in the world of private corporations possessing weapons of mass destruction - including nuclear weapons. This naturally leads to nationationalization of labs (e.g. having NASA like organization who are mandated to develop this technology). I think nationalisation is too extreme - but it becomes the only natural conclusion, if we treat this technology as extremely dangerous that is impossible to safeguard, or safeguard against.

“Hardware failures won’t limit AI scaling.” - Epoch AI, i.e. no square-cube-law type limitations for AI training as AI clusters grow in size and number of components.

How Constraint Becomes a Design Philosophy in Nature and in GeoTechnical Competition

During his 1835 voyage on the HMS Beagle, Charles Darwin collected several small bird specimens from the Galápagos Islands, initially misidentifying them as unrelated wrens, blackbirds, and mockingbirds. It was only after returning to England that ornithologist John Gould revealed these birds were actually a collection of closely related, unique finch species. This critical realisation directly inspired Darwin's radical new idea of the "theory of evolution." He deduced that a single mainland ancestor had colonised the islands, with its descendants gradually branching out into entirely new species to survive. The most striking evidence of this was their highly specialised beak shapes, which had evolved via natural selection to match specific food sources, from crushing hard seeds to probing cactus flowers (Decades later, modern researchers Peter and Rosemary Grant confirmed this theory on the island of Daphne Major, documenting real-time evolution as droughts and rainy seasons rapidly altered the finches' food supply and drove measurable changes in beak sizes.).

This is what isolation does to evolution. It does not just weaken isolated species; it pushes them in new directions. Free from mainland competition and shaped by local constraints, island species evolve along paths the wider world does not take. The results can look strange or inferior by general standards, but they are well adapted to their specific environment.

Huawei is in a similar position. Export controls are its isolation: no EUV lithography, no leading-edge process nodes, no unrestricted HBM. Rather than building an inferior copy of Western chips, Huawei has developed in a different direction, optimising for what its environment allows. While Nvidia and AMD focus on transistor density because they have access to advanced silicon, Huawei has had to ponder over a different question: what can you build into a world-class AI system when you cannot have the best transistor density? The answer is quite a lot actually, if you are willing to use the resources you do have—power, space, mature manufacturing, and strong communications technology—to compensate for the one you lack: transistor density. That is the subject of this essay.

Because EUV lithography will remain unavailable for the foreseeable future, Huawei has focused on areas where transistor density is not the deciding factor. Its approach, described in what it calls the Tau scaling laws, is to scale along the time dimension: if you cannot keep making each die denser and faster at the historical rate, you improve connectivity to reduce time delays. Huawei is applying this idea not only to chip design, which has been covered widely in the media, but also to systems: networks and overall AI cluster design. That system-level work is the topic of this essay. The second version of the τ-scaling paper describes three coordinated layers that reduce system latency together:

• a unified system fabric (UnifiedBus, or UB),

• a near-packaged optical engine (Hi-ONE), and

• a reorganisation of the package topology (3D Folding, which turns a quadratic scaling problem into a linear one by moving edge-bound resources onto surfaces).

The goal of this stack is a large-scale AI cluster that works as a single logical unit, which Huawei calls “System-as-One-Chip.”

Evolution is unplanned, but Huawei’s adaptation is deliberate. The Galápagos finches did not choose their beaks; they stumbled into those shapes due to evolutionary pressure. Huawei’s engineers, on the other hand, are treating their constraints as design guidelines and rapidly trying out ideas that will meet or overcome those constraints. What nature reaches over millennia through trial and error, a focused company can engineer within a few product cycles. This essay is a story of that adaption…

One bus to rule them all

The instrument of this strategy is the Unified Bus (UB) now in its version two: it replaces need for having different protocols like NVLink, PCIe, Ethernet etc.
It is Huawei’s approach to networking AI data centers. They have two version of it: 1) Flat/CLOS: as implemented in CloudMatrix 384 (384 Ascend 910s), 2) UB-Mesh which is a 4D-FullMesh with CLOS for PoD level networking for training.

Various sources indicate Ascend 950 based SuperPoDs have multiple topologies possible flat/CLOS for inference, as well as, hybrid involving UB-Mesh for training. -

• The flat/CLOS need more switches and more optical interconnects (NPO, we will see more about it later), hence costs more for networking, but is ideal for implementing inference topologies like WideEP with large model.

• UB-Mesh (nD-FullMesh + CLOS) reduces need for switches and optical interconnect. It in-fact costs less than 1/3 of flat/CLOS topologies. It takes the advantage of the fact that most traffic during training is local/is within groups of nodes and NPUs.

Confirmations on Ascend 950 supporting both:

You can learn more here about UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecture published in March, 2025. In addition to UB-Mesh, which is a specific topology (nD-FullMesh + CLOS), you will learn about UB design philosophy in general. The Ascend 950 Architecture paper and Tau Scaling Laws V2 paper published in Jun 2026 add more details.

Deeper look into UB

Prevailing architectures move every byte through layered protocols - PCIe to the host, a proprietary fabric in the chassis/rack, Ethernet or InfiniBand between chassis/racks, software-managed remote memory on top - and each layer exacts a conversion, a serialization, a buffer, a handshake. Just count how many different network protocols you see around GB200 in the below diagram. Network protocols are color coded.

Here is a great diagram from the UB Mesh paper that explains distinction between the prevailing approach and that of Huawei’s UB:

The paper’s measured result is end-to-end remote access falling from the tens of microseconds of TCP/IP-class stacks to roughly 100 nanoseconds - about a 500-fold compression of system τ along the dominant communication axis. At rack scale the system approaches what the paper designates a System-as-One-Chip: a fabric-coherent machine in fact, not metaphor.

Chip Level Support for UB

As we saw, where the industry maintains a zoo of interconnects - PCIe between GPU, host & NICs, a proprietary scale-up fabric, InfiniBand or Ethernet for scale-out - UB is a single protocol that carries all of these roles: memory semantics, IO semantics, and network semantics on one wire, with ports flexibly assigned between chip-to-chip and node-to-node duty.

The Ascend 950 exposes eighteen ports (4 links each, totalling 72 total) from one SerDes pool; four can run PCIe, two can run Ethernet, and all of them speak UB natively. There is no seam in the protocol between “inside the rack” and “outside the rack.” Stay with me if it is not clear yet!

All-optical interconnect between racks/cabinets

Huawei is moving to near-package optics (NPO) and linear-drive optics that eliminate the power-hungry DSP of conventional pluggable modules - leaning on its heritage as a communications-equipment company to do so. From Ascend 950 generation, they use NPO for inter rack/cabinet networking.

Eliminating the boundary that must not be crossed lightly - the geometry copper imposes

Consider why an NVL72 looks the way it does. NVIDIA’s GB200 racks pack seventy-two GPUs into a single dense enclosure not to save floor space but because passive copper cannot carry NVLink-class bandwidth much beyond two meters. Inside that copper tether, each GPU enjoys eighteen NVLink connections at roughly 900 GB/s per direction. Outside it, the system falls off a cliff: scale-out runs over one or two 400 Gbps optical links through discrete NICs - the optical network is, ironically, the slow one.

A GB200 GPU speaks to its seventy-one NVLink neighbours at 900 GB/s per direction, and to everyone else at 50 GB/s through a 400 Gbps (50 GB/s) NIC - an eighteen-to-one drop. GB300 softens the fall to nine-to-one by doubling the NIC to 800 Gbps (100 GB/s), but a nine-fold penalty is still harsh. Every large-model parallelism strategy on NVIDIA hardware is, at bottom, an exercise in designing around this ratio: tensor parallelism must stay inside the 900, data parallelism is exiled to the 50, experts are placed with the seam in mind, and much of the framework engineer’s craft is the art of never letting chatty traffic wander onto the thin side of the wall. The result is a two-tier world with a hard seam: blistering bandwidth for seventy-two devices, an order of magnitude less with everyone else. Below diagram presents a simplified network diagram for understanding scale out and scale up, though it doesn’t show 72 GPUs per rack for simplicity.

UB: FLAT/CLOS topology

Huawei’s approach dissolves this geometry, particularly in FLAT/CLOS topology. An Ascend 950 presents 2,016 GB/s bidirectional, just over a terabyte per second each way, from one fungible pool of eighteen ports - whether the peer sits in the next slot or eight rows away. There is no eighteen-to-one cliff to design around, and the developer is spared most of the judiciousness NVIDIA’s ratio requires. The one boundary that remains is semantic still, not bandwidth: direct UB Memory (NVLink equivalent) load/store spans only 1,024 NPUs - the 128 TB address window - while URMA (RDMA equivalent), the asynchronous mechanism that collective communication (NCCL equivalent) actually uses, runs uniformly across all 8,192 NPUs of a SuperPoD (eight rows of sixteen racks/cabinets) and beyond it over UBoE (UB over Ethernet).

If implemented with flat/CLOS, the magic in Huawei’s numbers is this: its everywhere-bandwidth roughly equals NVIDIA’s inside-bandwidth (scale up with NVLink) - and exceeds NVIDIA’s outside-bandwidth (scale out with NICs) ten- to twenty-fold. In case of Nvidia, NVLink uses copper and scale out network (Ethernet of Infiniband) connections use optical transreceivers. For Flat/CLOS topologies, Huawei uses NPO optical links for Ascend 950 SuperPoDs, and optical transreceiver for Ascend 910 based CloudMatrix 384 (while within a rack, CPU and NPUs use copper UBLink to talk to each other via local UB-Switch. NPU-NPU communication happens through a FLAT/CLOS network through an optical link. It costs to use optical links, but that is the trade off you do for ‘flatness’).

SemiAnalysis does an exceptional job of describing such FLAT/CLOS network in context on CloudMatrix 384 for Ascend 910 NPUs. Do check it out.

UB: UB-Mesh topology:

In UB-Mesh for Ascend 950 SuperPoD (nD-FullMesh + CLOS), there are bandwidth cliffs, but it does not matter as it meant for training workloads whose high bandwidth needs are within ground of nodes & NPUs. This approach allows using more copper (direct attach copper, or active copper cable) over optics and requires fewer switches. For large clustered, saving are massive.

This is how network looks like. The network is organised to maximize use of copper. (red and yellow links), and optics only used to connect pods into SuperPoDs and connect to data center network. Also, lack of switches causes bandwidth cliff, but it is okay due to traffic patterns of LLM training as shown in the above table.

One of the downsides of complex networks like UB-Mesh is that maintenance can be hard. However, the white-paper discusses approaches to deal with them:

The failure-rate problem is answered at the protocol layer and above. UB-Mesh builds reliability into every layer. Link failure are rapidly detected and re-routing is arranged by propagating information/notification about failures.

UB-Mesh specifies a 64+1 high-availability design: every rack carries a sixty-fifth NPU as a standby, activated on failure so that an LLM training job continues uninterrupted. Two things about that design deserve notice. First, the granularity: the spare sits on the same fabric, load/store-reachable, so recovery is an allocation decision rather than a topology event - contrast an NVL72, whose seventy-two-GPU coherent island contains no in-domain spare, so one dead GPU degrades the whole domain until a technician arrives (or you stop using the Compute tray; reducing the capacity).

What is a SuperPoD? What is a SuperCluster?

(For this section, we focus on SuperPoD configuration for inference where racks/cabins are organised in rows and NPO is used for communication between NPUs through central cabinets dedicated for switching.)

“For reasons we all know, we don’t have access to advanced process nodes, so we’ve decided to focus our efforts on making breakthroughs by combining chips – essentially connecting more computing resources.” - Eric Xu, Rotating Chairman, Huawei at Huawei Connect 2025

• Each 950 SuperPoD consists of 8 rows of 16 NPU cabinets/racks with 64 NPUs in each cabinet/rack, totally 8,192 NPUs (950DT specifically)

• There are actually 20 cabinets/racks per row accounting for networking ones, as well. Thus a 950 SuperPoD has 160 racks spanning across whopping 1000 square meters.

• SuperCluster connects 64 such SuperPoDs with 8.192 NPUs each, totalling 524,288 NPUs.

• SuperNode Network: UnifiedBus optical interconnect: 16 PB/s and 2,1 μs latency.

• SuperNode Performance: 8 ExaFLOPS in FP8 and 16 ExaFLOPS in FP4

• SuperCluster Network: SuperNodes can be connected to one another with UBoE in datacenters with existing Ethernet investment

• SuperCluster Performance: 524 ExaFLOPS FP8 (for training) and 1 ZettaFLOPS FP4 (for inference).

SuperPoD and SuperCluster visualisation

I created this visualisation to help you understand scale of SuperPoD and SuperClusters. SuperPod and SuperClusters to be available from Q4 2026 will use Ascend 950DT chips not Ascend 950PR. They are nearly equivalent chips and differ only in HBM and HBM bandwidth: 144GB at 4TB/s vs 128GB at 1.6TB/s, respectively.

SuperPoD:

SuperCluster:

SuperCluster Data Center:

Oh, the irony!

If you think about it, Huawei has similar philosophy to design datacenters as that of planners of American suburbs: big beautiful home, far apart from one another; far apart from other suburbs and city centers, but connected with wide highways and fast cars.

On the other hand, Nvidia and AMD design for highest possible density of chips in one rack. Their planning reminds you of Hongkong in comparison! Many of their strengths and problems are also rooted in their density.

Scale-up domain comparison

While SuperPoD blurs distinction between scale up and scale out; let us define scale up on Nvidia terms for sake of comparison. Nvidia defines scale up as domain where GPUs can access each other’s HBM directly. In case of 950 SuperPoD with 8192 NPUs, only 1024 NPUs located in a row can access each other’s memory over UB Memory. So let us take a row as scale up domain for comparison.

16 Ascend 950 cabinets/racks, sixty-four NPUs per rack: 1,024 NPUs, each with 128 GB (for 950 PR) or 144GB (for 950DT) of high-bandwidth on-package memory. Every one of them can read and write every other’s memory synchronously, ordinary loads, stores, and atomics, as if it were local. The row’s total is 128 TB of directly addressable memory. The 950 white-paper caps the UB load/store address space at exactly 128 TB, 2^47 bytes, and 1,024 950PR chips at 128 GB lands on it to the byte. In case of 950DT with 144 GB HBM per NPU, there will be total of 144 TB of HBM across 1024 NPUs. However, going by the white paper, only 128 TB will be directly addressable, additional HBM could be used for other use cases that are local in nature.
The sixteen-rack row is not merely a configuration; it appears to be the design point around which the coherency window was sized. One row equal one flat memory machine (roughly equivalent of NVL72, but with more FLOPs, HBM and memory bandwidth, as we will see soon)

An NVL72 offers roughly 14 TB of NVLink-coherent memory with GB200, about 20 TB with GB300. The Huawei row offers nine times the former. On compute, dense against dense - NVIDIA’s 10 and 15 petaFLOPS of FP4 per GPU for GB200 and GB300 are dense ratings, with sparsity figures double that. Ascend publishes only dense - Ascend 950DT delivers roughly 2 PFLOPs per NPU, totalling approximately 2 exaFLOPs per row (1024 NPUs), against 0.72 exaFLOPs for a GB200 NVL72 rack and about 1.08 exaFLOPs for GB300 NVL72 rack.

Below I present easy to follow comparison between chips and systems.

GPU vs NPU:

Here is comparison of Nvidia Blackwell GPUs and Ascend 950 NPUs in inference topology (FLAT/CLOS). You can notice, while NPUs have significantly more scale out bandwidth, they have lower specifications in other areas.

GPU & NPU Scale Up Domain Comparison

Huawei blurs the distinction between scale up and scale out domain, as we saw. However, direct memory access by one NPU of another NPU is only possible across 1024 nodes (row of 16 cabinets/racks, with 64 NPUs each). Let us use it as unit of comparison with NVL72. NOTE: per Huawei documentation SuperPoDs and SuperClusters are made of Ascend 950DT and not 950PR. Still I present below numbers for scale up domain for 950PR as well, if those NPUs were put in equivalent cabinet & row configuration as for 950DT.

Nvidia GPUs Generation by Generation progress for review

I added this great table for quick comparison. You will notice massive improvements in HBM and FLOPs and very high densities compared to Huawei chips. It will allow you to appreciate ingenuity of Huawei’s design. Despite ‘weaker’ individual components, the system as a whole is arguably more capable.

UB Deep Dive: Three key primitives

UB gives software three communication primitives, and the radius of each is different.

UB Memory is the synchronous primitive (1024 NPUs in a row): this one makes the row of 16 racks/cabinets effectively a single machine. A compute core issues an ordinary load, store, or atomic; a decoder resolves which node owns the address; the operation crosses the fabric; and the peer chip’s memory-management unit translates and permission-checks it before touching memory directly. No queues, no software, no driver in the path - remote memory is simply memory (this is how NVLink domain behaves). Its span is the 128 TB address window on the 1,024-NPU row.

In Nvidia ecosystem, this idea is around for long. NVLink allows one GPU to access another GPU’s memory if they are connected with NVLink. Beauty of NVL72 (that is available from Blackwell generation) is that 72 GPUs of the rack can access each other’s memory. For Blackwell generation, they can do it at 900 GB/s per GPU (unidirectional) and for Rubin generation, they will be able to do it at 1800 GB/s per GPU (unidirectional). This allows NVL72 to host very large models effectively. But this scales only to 72 GPUs. Beyond 72 you have to use much slower scale out network and direct memory access is not possible. Please also note GPUs inside NVL72 domain can also access CPU LPDDR memory, called as host memory. Below is a diagram for Hopper to show such connectivity (for Blackwell/Rubin I did not get an equally nice diagram, but they also have similar direct access with their NVL domain).

URMA is the asynchronous primitive: A compute core (or the hardware scheduler itself) rings a doorbell on a Jetty queue; dedicated hardware executes the transfer - writes, reads, sends, fetch-add and compare-and-swap atomics - and posts a completion, while the core goes on computing. It runs over two transports: a reliable mode with end-to-end retransmission sustaining four ports of bandwidth, and a lightweight mode that trusts the fabric’s link-layer reliability and sustains nine - a full terabyte per second each way. URMA’s span is everything: the whole 8,192-NPU SuperPoD over native UB, and even past it over UBoE (UB over Ethernet). Recall 64 8,192-NPU SuperPoDs can be connected into a SuperCluster with close to half a million chips in total!

URMA is equivalent of GPUDirect RDMA of Nvidia world (diagram below), but at much higher bandwidth of 1008 GB/s unidirectional (if the network if FLAT/CLOS) compared to 50 GB/s for GB200 and 100 GB/s for GB300 scale out network.

The CCU is the collective primitive - this is equivalent of NCCL in Nvidia world and has hardware implementation. The Collective Communication Unit (CCU) receives whole collectives (AllReduce, ReduceScatter, AllGather, All2All and kin) from the on-chip scheduler, unrolls them into fine-grained parallel steps, and executes each step as one of exactly two things: a Reduce, computed in the CCU’s own Reduce Units against its own MemorySlice storage, or a URMA transfer. Read the block diagram closely: the CCU exposes precisely two call interfaces - Reduce and URMA (asynchronous that can reach entire SuperCluster). There is no UB Memory (synchronous) path. The collective engine does not use synchronous communication at all.

By building collectives exclusively on URMA, the CCU inherits URMA’s span which is much larger than that of UB Memory’s: it spans all eight rows of the SuperPoD (8192 NPUs), and because URMA rides UBoE extends in principle to any NPU reachable over Ethernet (entire SuperCluster in theory). CCU also inherits URMA’s bandwidth and in a FLAT/CLOS networks it is massive. Contrast the Nvidia’s software equivalent: NCCL must change transports at the NVLink boundary, switching from peer memory access to network verbs, with hierarchical tricks to hide the boundary crossing. The CCU uses one mechanism for the entire span, so a collective algorithm has the same shape at 64 chips and at 8,192 and beyond. Interestingly, movement and reduction both happen in dedicated silicon, off the AI cores and largely off the system bus.

For readers fluent in the NVIDIA stack, the mapping is worth making explicit - because it is diagonal, not one-to-one:

NVSHMEM’s job is presenting one put/get/atomic API that quietly uses NVLink load/store inside the NVLink domain (72 GPUs) and GPUDirect RDMA outside it. It is a software library that uses two mechanism/protocols. NVSHMEM hides difference in scale up and scale out networks from the software using it. In Huawei world, the network mechanism is URMA across all the NPUs for equivalent communication, so it simplifies the design of NVSHMEM type libraries.

NCCL, meanwhile, matches the CCU’s function: both execute collectives, but NCCL is kernels running on the GPU’s own compute units - paying the tax in SM cycles, and switching transports mid-algorithm at the NVLink boundary. NVIDIA’s true hardware collective offload, SHARP, sits in yet a third place: the switch. NCCL’s equivalent in Huawei world is CCU. It has silicon for compute in form of Reduce units and it uses URMA mechanism to talk to all the NPUs.

Connecting to existing data center networks

The world’s most datacenters run on Ethernet and a new fabric that cannot interoperate with Ethernet will have challenges in adoption. The Ascend 950 white-paper offers two integration paths.
Option 1: Conversion in the switch. In this approach, the SuperPoD’s interior stays pure UB - every chip keeps its full eighteen-port pool on the native fabric - and the UB Switch itself performs the UB-to-Ethernet conversion at the boundary, presenting standard Ethernet uplinks to the datacenter spine. The whitepaper’s framing is that the SuperPoD joins existing Ethernet networks seamlessly at no extra hardware cost, because the conversion happens in switch silicon the SuperPoD needed anyway.

Option 2: Conversion on the chip itself. The Ascend 950 chip can also join Ethernet networks directly via UBoE technology, networking over industry- standard Ethernet switches. Each Ascend 950 can dedicate two of its ports to UBoE - 2×400 Gbps of the UB protocol carried directly over industry-standard Ethernet switches.

The interior mesh remains native UB (the diagram is explicit: the chips stay green-wired to each other).

Note, NVIDIA offers no equivalent: NVLink never converts to Ethernet in a switch or on the GPU itself. You need discrete NICs for scale out with Ethernet. Scale up (NVLink) and Scale out (Ethernet/Infiniband) are totally isolated from one another with no port sharing.

Is it all too good to be true?

If is fair question. Writing white papers like Tau Scaling Law is easy, but will these approaches work in production? Let us tackle objections one by one (we continue to focus on Ascend 950 SuperPoD with FLAT/CLOS network):

Objection one: the space

Sixteen racks/cabinet in row of Ascend 950 SuperPod against one NVL72 is unflattering, to be clear! But the objection has cause and effect backwards. NVIDIA’s situation is not a virtue; it is a symptom - the result of copper’s two-meter max length and Nvidia’s approach of using NICs/Ethernet for scale - out. Since scale-out network is slow, so Nvidia’s focus is on compute and memory density within NVL72 domain.

Huawei’s 1,024 NPUs spread across a full row precisely because their special design of network permits it. The sprawl actually makes thermal design also easier. Further more you can have 8 such rows leading to SuperPoD of 8192 NPUs and 64 SuperPoDs in a ‘SuperCluster’.

It will take a lot of space, but so what? Density was never the goal; it was the constraint. If you are rich, you build a big house and live comfortably. You do not cram into a studio and call it discipline.

Objection two: the memory bandwidth

Ascend 950PR’s memory bandwidth at only 1.6 TB/s, and that of 950DT’s at 4 TB/s trails 8 TB/s of NVIDIA’s current HBM3e HBM in Blackwell generation (Rubin generation uses HBM4 reported to have bandwidth around 22 TB/s). Per chip basis, Huawei is behind on the memory bandwidth and will remain behind for considerable future… Are they doomed then?

No, per-chip bandwidth is the wrong consideration for the workloads that is the most common: Mixture of Expert (MoE)! In MoE models - an architecture where China based labs excel - some ninety percent of parameters are in Mixture of Expert (MoE) layers, and expert parallelism (EP) spreads those experts across as many devices. Each token in a given batch accesses only a few experts in each layer; so what governs aggregate decode throughput is total memory and total memory bandwidth across the pool of devices.

Wider you are able to spread your expert (WideEP), fewer experts each GPU/NPU needs to support. This results in higher overall throughput for token generation. For example EP=32 results in 80% higher throughput per GPU compared to EP=8. This is how you can work with GPUs/NPUs with lower HBM bandwidth.

Ascend 950 SuperPoD’s single row (16 racks with 64 NPUs each) that has 1024 NPUs that have 4,096 TB/s of aggregate bandwidth compared 576 TB/s for NVL72.
It has 144 TB of capacity (of which 128TB can be on UB Memory address space) against 14 TB of GB200 NVL72 and 20TB of GB300 (approximately). Huawei offers bigger domain across which experts can be spread, its total HBM and HBM Bandwidth is many times higher than NVL72 system. So they can host really large models like even Kimi 3 with nearly 3 trillion parameters.

Many chips works in parallel as a single chip is why Huawei could afford to regress on FLOPS even against its own 910C (at FP16). The philosophy is that the system-level performance, not the per-chip one, decides how well you can serve trillion-parameter models economically.

Objection three: the latency

Ascend 950 racks/cabinets are connected with NPO (Near Packaged Optics). In a FLAT/CLOS topology all the NPUs are connected to switches at center with NPO, that is why any NPU to any NPU latency is uniform.
UB-memory access (equivalent of NVLink) works across entire row of 16 racks/cabinets i.e. 1024 NPUs. Moreover all 8,192 NPUs of a SuperPod can communicate with one another at UB bandwidth with URMA in FLAT/CLOS topology.

The article quotes hops around 2.1 microseconds against NVLink’s sub-microsecond copper. However, as you would recall NVLink connects only 72 GPUs, beyond it you would use NIC + Ethernet/Infiniband, it would have much higher overall latency. So Huawei’s approach offers to connect up to 1024 NPUs with UB Memory (NVLink equivalent) and all 8,192 NPUs in a SuperPod with URMA (RDMA equivalent) at full UB bandwidth and uniform 2.1 microseconds latency. It is a pretty good latency for such large group of NPUs (note: latest Huawei material quotes 3 microseconds).

To put in another way, NVIDIA’s latency profile is not “fast”; it is bimodal - magnificent inside a small circle (NVL72 rack), worse than UB outside it. Huawei’s is flat: 2.1 microseconds to any of a thousand peers of a row (even to 8192 peers within a SuperPoD), no cliff! For any model too large to live inside seventy-two GPUs, which is to say, all the future models on path to AGI & ASI, Huawei’s uniform microseconds latency beats bimodal latency profile.

(A careful reader will now be holding two latency figures in apparent tension: the roughly 100 nanoseconds of remote access claimed by the τ-scaling paper, and the 2.1 microseconds quoted here from Huawei Connect 2025 event. My read is they measure different segments of the same journey.

The ~100 ns is the protocol’s toll - the cost of issuing and completing a memory-semantic operation through UB hardware, the segment that TCP/IP-class software stacks inflate to tens of microseconds and the segment to which the paper’s ~500× compression refers.

The 2.1 µs is likely the whole trip: SerDes, electro-optical conversion, fiber, and switch hops between cabinets. Light travels at nearly 200k km per second in fibre optics, so it takes about 1 µs to travel 200m; add switch traversals and the 2.1 µs budget is nearly accounted for.)

And the microseconds of UB protocol themselves are largely hidden by design. The Ascend 950 chips are built to keep cores blind to fabric latency: asynchronous URMA transfers overlap with compute, the CCU runs collectives without occupying a single AI core, and hardware scheduling dispatches at nanosecond granularity.

Objection four: the power bill

A thousand NPU packages offer for two and a half times the dense FLOPS (compared to GB200 NVL72). Surely the performance-per-watt based comparison makes Huawei look ridiculous, right?

Not necessarily. For Huawei’s actual customers, the binding constraint is not watts; it is access to silicon at all. Against that constraint, the design is ruthlessly rational. Mature DUV nodes with multi-die packaging yield far better than reticle-limited monoliths on bleeding-edge processes. The memory is Huawei’s own HiBL & HiZQ rather than scarce, sanction-priced HBM from two Korean suppliers. The optics are linear-drive, deleting the DSP that burns a large share of a pluggable module’s power. Also, the sprawl (large number of racks) that looked wasteful in Objection One offers a thermal dividend: heat spread across a row is cheaper to remove than heat crammed into copper’s two-meter radius domain (recall all the trouble Nvidia is having with Rubin Ultra and Kyber racks due to high density). And the whole stack - chips, optics, switches, racks - is vertically integrated, with no seventy-five-percent gross margin stacked on top by Nvidia. China is growing electricity production at historic rates. Huawei is spending the abundant resource (power, floor space, mature fab capacity) to deal with not being able to manufacture most advanced nodes that Nvidia has access to. That is a a smart trade off, I would say, and every generation of the Tau curve - e.g. the 950DT’s doubled memory bandwidth compared to 950 PR being the first instalment - reduces the per-package gap while the systems advantage compounds.

CANN (Huawei’s equivalent of CUDA) ecosystem must achieve what CUDA ecosystem has achieved over a much longer period. CUDA ecosystem is both older and larger. But even here the trajectory favours the challenger - the stack is open-source, developers must manage just one fabric instead of three, and the hardest orchestration problems (collectives, scheduling, data movement) have been pushed into hardware precisely, so software developers have less to worry about.

Objection five: NPO is no panacea

Optics is not a miracle medicine. Optical links fail more than copper - and no one says so more bluntly than Huawei’s own engineers, whose UB-Mesh presentation at Hot Chips 2025 put optical error rates two to three orders of magnitude above copper’s, and treated node failure at SuperPoD scale as a matter of when, not if. Conventional optical modules (trans-receivers) are power gluttons: a DSP-based 800G pluggable draws 14 to 17 watts.

All true. But Huawei is not deploying “optics” in the abstract; it is deploying one deliberately chosen point in the optics design space, with everything co-designed/co-engineered by them - and each of the three failure modes above is attacked by a specific choice.

The power and conversion-latency problems fall to linear-drive optics. The τ-scaling paper describes Hi-ONE, the High-density Optical-interconnect-Node Engine: a Near-Packaged Optical (NPO) engine delivering 8 Tb/s per module, enough to carry an entire AI chip’s UB bandwidth on a single optical link. Its signal path is exactly the linear architecture we commonly call LPO, though the paper never uses the term: an analog equalisation driver and trans-impedance amplifier in place of a DSP. Industry measurements of this approach show a 40 to 50 percent cut in module power - down to 7 to 8.5 watts at 800G - and per-hop latency under 3 nanoseconds, versus 8 to 10 for DSP modules. What makes DSP-deletion safer is a trick: the UB protocol deliberately tolerates a relaxed bit-error rate, letting the transport layer absorb the errors the physics layer causes - a cross-layer trade between protocol and photonics that is only available to a designer who owns both. That is the catch that has stalled LPO in the merchant market - it demands exquisite SerDes linearity and tight co-engineering of module, ASIC, channel, and protocol that multi-vendor ecosystems struggle to coordinate - and it is precisely Huawei’s moat. As designer of the HiLink SerDes, the optical engine, the switch silicon, and the protocol, it is one of the few companies on earth that can close the linear-optics loop by fiat rather than by committee.

Hi-ONE’s near-package placement cuts the required electrical SerDes reach from roughly 100 centi-meters to about 5 - collapsing the lossy copper run that DSPs existed to compensate - while extending link reach from under a couple meter max to 100s of meters, which is what renders a gigawatt-scale, datacenter physically buildable at all. Interestingly they do not choose Co-packaged Optics (CPO), as it is complex to manufacture even though it saves a little bit more power. NPO is the pragmatist’s point on the curve: most of CPO’s electrical benefit, at pluggable-class manufacturability and cost.

Objection six: a cluster too big to keep alive

One objection remains, and it follows arithmetically from everything above: if Huawei needs more chips per FLOP, it inherits more failures per FLOP. Match a full SuperPoD’s 16.4 dense exaFLOPS with GB200s and NVIDIA needs about 1,640 GPUs to Huawei’s 8,192.

Per Epoch AI, for example, a single H100 SXM GPU fails approximately once per 50,000 hours,¹ but this rate scales linearly with hardware count. Consequently, a system with 16,384 GPUs experiences a failure every 3 hours on average, while a larger system with 1 million GPUs faces a failure every 3 minutes.^2”

Huawei has nearly 5 times more NPUs for the same flops (2 PF FP4 for Ascend 950 NPU vs 10 PF FP4 for GB200 GPU), so failures are higher in absolute count and so are cluster interruption.Add unproven silicon, proprietary memory, and a hall full of optics and picture starts getting even more complicated: the fleet seems too large just to keep alive long enough to train anything. Astute readers will remember “Square Cube Law” at this point.

Epoch AI, analysing exactly this question for NVIDIA-class clusters, showed that failure overhead does not scale with cluster size - it scales with its square root. The optimal checkpointing (weights, optimiser states) cadence balances checkpoint cost against expected loss, and at that optimum the fraction of training time lost grows as the root of (checkpoint cost × chip count × failure rate). Five times the chips costs roughly 2.2 times the overhead, not five; two and a half times costs about 1.6. And the level of that overhead is set almost entirely by one variable: how long a checkpoint takes. Write checkpoints to storage (like AWS S3) and the overhead sits near a couple of percent; write them into the spare memory of peer accelerators, a few replicas each, and recover by reading them straight back from those peers over higher speed network, and the checkpoint cost collapses to sub-second. Run the numbers at 8,192 chips, a chip failing every six hours with peer-memory checkpointing and the training time lost lands under half a percent. Epoch’s conclusion for NVIDIA - hardware failures will not limit scaling - applies to Huawei chips as well, as they do not assume any specific characteristic of Nvidia chips in their analysis.

Interestingly, nearly every mitigation Epoch prescribes as a technique on NVIDIA hardware exists as a primitive on this architecture.

1. Peer-memory checkpointing presumes spare memory - the 950DR carries roughly 72 GB per dense petaFLOP against NVIDIA’s 19 GB per dense petaFLOP (recall GB200 has 192GB HBRAM and 10 petaFLOPs for FLP4), pretty good room for replicas beside the optimiser state!

2. It presumes fast paths to peers - the fungible terabyte-per-second pool, with SDMA engines moving checkpoint bytes while the AI cores keep training. It presumes recovery can read a dead node’s state from its peers - which, inside the 128 TB window, is not an RDMA choreography at lower bandwidth, but a load instruction more like NVLink access at high bandwidth. This allows the replacement NPU to gain access to predecessor’s replicated state faster.

3. It presumes spares that can rejoin quickly - and as described in response to Objection Five: the sixty-fifth NPU standing by in every rack, on the same fabric, activation being an allocation decision rather than a repair event, where an NVL72 contains no in-rack spare at all. And beneath the checkpoint layer entirely, the memory’s own RAS machinery - online ECC, patrol scrubbing, transparent row-sparing - absorbs a whole class of faults before they ever register as a failure.

The overall summary of Huawei’s bet

In a post-Moore world, one can say, the fabric is the computer. NVIDIA built the world’s best small circle (NVL72, and NVL144 for Kyber racks) and must now fight physics to widen it; Huawei, barred from the transistor desnity race, built a world without a circle at all - and aimed it squarely at the place the Nvidia’s architecture is weakest: the boundary where copper ends (scale up) and optics begins (scale out). Huawei has simply decided there shouldn’t be a distinction between scale up and scale out network capabilities/bandwidth: they design their topology, chips, connecters and switches for that. They will have lot of friction initially, but once they cross to the other side they would have achieved the escape velocity!

It is long known that clouds and enterprises in China use Huawei clusters for inferencing. Now we also have evidence that 1.6T parameters models are being trained on Huawei cluster.

DeepSeek CEO Liang Wenfeng gives Huawei’s strategy a vote of confidence, in his leaked conversation with his investors.

References:

• Epoch AI’s Hardware failures won’t limit AI Scaling essay.

• UB-Mesh paper (64+1 high-availability design; direct-notification link-failure recovery) and Hot Chips 2025 presentation

• Serving Large Language Models on Huawei CloudMatrix384

• Eric Xu keynote, Huawei Connect 2025 (UnifiedBus reliability and fault-switching claims)

• SemiAnalysis for comparison of Nvidia GPU

• Ascend 950 NPU architecture whitepaper

• τ Scaling Law paper, v2 (adds UnifiedBus, Hi-ONE, and 3D Folding; network-scale τ analysis)

• Meituan Trains 1.6T parameters model on Ascend cluster

Appendix:

Positive impact of design of UB extends to many other areas such as KV caching. Also Ascend 950 design includes CPU cores. This bonus material talks about that.

The memory pooling for economic KV Cache storing

Agentic AI is a memory problem due to long context that long agentic trajectories need. Ultra-long contexts, multi-turn sessions, long-horizon plans - every token a model has attended to or generated leaves a key-value residue that must be kept warm for the next turn, and the whitepaper understands that this demand is growing exponentially, beyond what any accelerator’s on-package memory can hold. HBM is the most expensive real estate in the datacenter (have you seen memory stocks lately?); saving a dormant chat session’s cache there is like renting a Manhattan apartment to store winter clothes…

The conventional remedy is offload: spill KV blocks to the host CPU’s DRAM over PCIe and pull them back on resume (and to SSD if it is not going to be used soon enough). On standard systems that path is narrow and crowded - 64 GB/s per direction of PCIe 5.0, shared with everything else, with the host CPU’s software stack managing every transfer. NVIDIA’s answer connect CPUs and GPUs with ultra fast NVLink CC. For example, GraceBlackwell superchip, connects a CPU and its half-terabyte LPDDR to each of the two GPU over a 900 GB/s NVLink CC link. This is great. Also, GPUs can access other CPU’s LPDDR as well, as long as, they are on the same NVLink domain. But NVLink domain is only 72 GPUs!

Huawei’s answer is the ultra-large CPU memory pools over UB. Racks of CPUs fronting memory pools attach to the UB fabric as first-class citizens, and - per the white-paper - the compute chips of a rack/cabinet access that pool directly through their UB ports, at high bandwidth and low latency, making full use of CPU memory resources (more details on it are not available though).

The CPUs hiding among the NPUs

If you look closely at the chip’s floor plan and something unusual can be seen. Inside each AI die - not beside the package, not across a PCIe slot, but embedded at the corners of the AI Core array itself - sit general-purpose Linx816 CPU blocks, sharing the die with the compute they serve. The full chip carries four clusters of two cores each: eight ARMv8-A cores, sixteen hardware threads (the spec sheet’s “8C16T” bin is exactly this), each cluster with its own L3, all of it hardware-cache-coherent with the AI cores’ L2 and the shared on-package memory. The NPU has a CPU living inside it!!! This is quite unlike any design by Nvidia, AMD or Amazon.

Why plant CPUs in an NPU’s ? Because the control plane has a τ problem too. In the conventional Host–Device arrangement, every piece of housekeeping - page-table maintenance, operator compilation, performance monitoring, the scheduling of accelerators and IO - lives on the host, tens of microseconds away across PCIe and a driver stack. As kernels shrink toward the microsecond scale, that round-trip starts being significant. Huawei’s answer parallels its data-plane answer: collapse the distance. The embedded AI CPU runs an operating system on the device, absorbing the housekeeping locally; awkward operators that fit neither Cube nor Vector - control-heavy logic, general scalar work - execute in place instead of bouncing to the host - this saves lot of time! Interestingly, the CPU blocks sit adjacent to the L2 slices of the core arrays they manage (more on it in one of future essays)

Beyond UB (network): Inevitability of 3D folding

The latest Tau scaling law paper is very eloquent on this. So providing its explanation, as it is. Highly recommended to read it

This is the story of how China’s AI ecosystem is working together to overcome technological limitations imposed by US export controls - using ‘weaker’ components, reducing the need for HBM and inventing new architectures. This is also a story of how a food delivery giant Meituan ended up building a leading frontier lab destined for greatness. A feat unimaginable for western equivalents like Uber, DoorDash and InstaCart.

While X community has taken due note of Meituan’s LongCat Lab’ LongCat 2.0 model, they have vastly missed the ingenuity of LongCat Lab , as well as, that of the Huawei’s engineering. This article is an attempt to bridge the gap. At times it gets bit technical, but I promise that it will be very rewarding if you are curious to understand how the AI competition between United States and China is playing out and system architecture and model architecture level.

Background:

Meituna’s LongCat Lab’s LongCat 2.0 has been talk of the town since it was released this week. This is the first model publicly known to have been trained on Huawei 910Cs, most likely using CloudMatrix 384 Superpods that have 384 ASICS/NPUs each (compared to 72 of Nvidia NNVL72 racks). LongCat’s blog claims to have used 50K total ASICs. Two things stand out from their literature, as well as, that from Huawei:

• LongCat team have shown they are willing to make bold architectural bets and come on the top. 1. They have mastered sparse MoE that are hard to train. 2. They have invented better transformer attention. 3. They have further developed new axis of sparsity by developing n-gram based approach that trades of ASIC FLOPs & HBM with cheap CPU DRAM. It is intriguing as LongCat Lab belongs to China’s food delivery giant Meituan. Imagine Uber, DoorDash, Insta-cart of doing such feats! How outstanding it would be!

• Huawei has shown they can find their way around export controls and innovate. They have also shown how well they understand LLM pre-training and inferencing, and have designed their Superpod architecture with UB-Mesh for the same with memory pooling and many other innovations to make training stable against hardware failures. Huawei doesn’t just provide AI servers, but “AI Factory” scale solutions. It would be prudent to review it at this stage and it is going to be a globally important player.

First the Story of Meituan - why build a foundation model lab at all?

Imagine taking Uber Eats, DoorDash, Yelp, Groupon, TripAdvisor, and Lime, and melting them down into a single smartphone application. That is the everyday reality of Meituan, China’s leading lifestyle “super app”. While Western consumers hop between dozens of fragmented apps to handle separate errands, Meituan operates as a unified digital infrastructure for daily living. Within its single interface, a user can order a 30-minute lunch delivery, unlock a commuter bike, buy movie tickets, book a hotel stay, and secure a group discount for a hair salon. By leveraging ultra-high-frequency interactions like food delivery to anchor user habits, the tech giant cross-sells higher-margin lifestyle services seamlessly. It handles millions of daily transactions and acts as a primary digital gatekeeper for physical commerce.

What truly secures Meituan’s dominance as a super app is its aggressive transition into an AI-driven “lifestyle gateway,” powered by its massive, proprietary LongCat large language model. While Western tech companies use chatbots primarily for text generation or coding, Meituan leverages conversational AI agents, like its native assistant Xiaomei, to fundamentally reshape how consumers interact with the physical world. Instead of manually typing “ramen” and scrolling through pages of listings, a user simply tells the AI agent their budget, location, and plan. The AI then instantly cross-references the request against Meituan’s database of over 700 million real-time merchant inventories and 1.3 billion user reviews to book tables or order deliveries autonomously. Faced with narrowing profit margins due to fierce domestic e-commerce competition, Meituan is investing billions into applied commerce AI to act as an indispensable infrastructure layer, not just processing clicks, but dynamically executing daily real-world decisions.

AI is not optional for Meituan, it is essential for its survival. They have exactly same logic as META to own their LLM layer. It is too important to be left to other companies to provide. Expect them to keep releasing kicka** models and research papers.

‘Light Years Beyond’ to ‘LongCat Lab’

Meituan’s LongCat lab (LongCat AI) is the ambitious corporate research initiative established by Meituan’s co-founder and CEO Wang Xing. The lab was heavily formed through Meituan’s mid-2023 acquisition of Light Years Beyond (Guangnian Zhi Wai), an AI startup originally founded by Meituan co-founder Wang Huiwen.

There is a bit of heartwarming story there (quoting@kevinsxu): “In early 2023, soon after ChatGPT launched Wang Huiwen (Meituan co-founder) launched his own lab to build “China’s OpenAI”. He threw in $50 mil of his own money, attracted investments from other Meituan co-founders including CEO Wang Xing, and built a decent team. You can say he started the wave of Chinese AI labs looking to compete with OpenAI, Anthropic, that now includes DeepSeek, Moonshot, Z, etc. The lab was called ‘Light Year Beyond’. It didn’t last long unfortunately. By June 2023, Wang was suffering from mental health issues, the pressure was too much, and Meituan bailed him out and acquired it.” (Sources :Kevin’s tweet, SCMP)

Today Wang Huiwen would be pretty pleased with how his foundation turned out. Sure, the team working on LongCat now is not the exact same team, but he led the foundation that led to building of LongCat 2.0! A true visionary!

Does Meituan have money to sustain?

Meituan is a publicly traded company listed on the Hong Kong Stock Exchange under the ticker symbol 3690. It operates with a market capitalization of approximately $53.5 billion to $65 billion USD, and generates a trailing twelve-month (TTM) revenue of roughly $50 billion USD.

While the revenue seems huge, the company is currently unprofitable on a trailing twelve-month net basis. While its core food delivery and in-store operations are established and generate high transaction volumes, heavy spending on subsidies to fend off competitors (such as Alibaba and JD.com) and losses from expanding into new markets (such as the Middle East under the brand Keeta) have squeezed its net margins. That said this a company has massive cash flows and technology skills and they know how to raise. These are big boys of China’s tech industry.

The era of Trillion parameter models

LongCat 2.0 is a 1.6 trillion parameter models. For comparison, GLM 5.2 is half its size at near 700 billion. Training large models gets increasingly harder. - and this lab from the food delivery vendor Meituan have shown that they are more than capable of doing that. Not just that this model also is pretty good based on the benchmark numbers, including 70.8% score on Terminal bench 2.!

To be sure, Longcat team is not the first one in China to reach 1T mark. DeepSeek V4 Pro is 1.6T; Moonshot team is also rumoured to have 1T parameter model based on linear attention. So China based labs know how to train massive models. What is new is we now have model that is fully trained from scratch on China’s domestic chips Ascend 910C - and in doing so the lab as well as the hardware vendor have shown lot of innovative thinking. Longcat team also have taken bold architectural bets like n-gram and has come on top - we will discuss more about it.

Geo-Technological Implications

As per SemiAnalysis, Huawei can make up to 1.6 million Ascend 910C chips before they run out of stockpiled components. Quite a few to help reach AGI given Longcat 2.0 was trained with only 50K of these chips!

Also, SMIC of China has been manufacturing new dies for Huawei based on 7nm process node. So they may not run out logic chips ever. HBM is still a bottleneck, but 1.6 million chips are quite a lot to reach AGI. CXMT is also working aggressively to manufacture at least HBM3.

At the same time labs like DeepSeek are singularly focussed on reducing HBM and HBM bandwidth requirement. Their techniques like MLA, DSA, CSA, HCA are adopted by wider Chinese labs ecosystem, including LongCat Lab. In past I have covered those technique in depth in this essay: DeepSeek’s 10 trillion USD strategy

Longcat Lab uses DeepSeek’s MLA and a variant of DSA that is arguably more efficient. Likely inspired by DeepSeek’s Engram approach (that trades of CPU memory for ASIC compute & HBRAM), they also add an N-gram Embedding module that expands the embedding space by roughly 100× through N-gram token combinations, capturing richer local context and strengthening token-level representations. This expends more CPU attached memory but saves on ASIC compute and HBM Bandwidth - both of which are at a premium for Chinese ecosystem. To be fair to LongCat team - their paper LongCat Flash was published within 2 weeks of DeepSeek’s Engram and their approach has key variations. So they probably invented it independently. But idea is the same - trade off bit of DRAM memory for saving more expensive resources: ASIC FLOPs and HBM bandwidth. I highly recommend you to review LongCat Flash paper, as well as, DeepSeek’s Engrampaper. I have included key diagrams from LongCat and DeepSeek below.

On their end, Huawei also implements memory pooling to make DRAM memory usage efficient for critical use cases like model loading and KV Caching.

What is more, Huawei has designed the system to be fault resistant. Huawei’s approach is basically large number of low performance components working in parallel. Huawei is well aware that having large number of components for a given system capacity can interrupt training more frequently as some of the components are bound to fail, and their system has far more components compared to Nvidia’s equivalent.

Nvidia Blackwell Quick Review

Before, we look deeper into Ascend 910C it is useful to review Nvidia’s black-well. Blackwell provides most training and inference flops across the world. It comes in two variants B200 and B300 (ultra). The main difference in B200 and B300 is HBM (192GB per GPU vs 288GB) and FP4 FLOPs (10PF vs 15PF) and scale out bandwidth (400 Gbps vs 800 Gbps). Everything else is pretty much the same. Note that there are two dies per GPU.

Below diagram lists various critical bandwidth metrics. Note them carefully, as we will compare it with Ascend 910C.

Huawei Ascend 910C in comparison to Blackwell

Huawei 910C consists of two dies as well, just like Blackwell. It is the next version of 910B which had only one die. Nothing special / different here.

Where is the major difference is in various bandwidths as can be seen from the below diagram (Source: Huawei: Serving Large Language Models on Huawei CloudMatrix384). Nvidia typically quotes bidirectional bandwidth when the industry standard is unidirectional bandwidth reporting (”Jensen Math”). Hence I converted the Nvidia numbers to unidirectional. Also, Nvidia reports bandwidth for the both dies together. I have split it for 1 to 1 comparison with Ascend dies. The number in Red are for Blackwell GPUs.

Die to Die bandwidth: While other bandwidths are on similar order of magnitude, I can't help but notice that die to die bandwidth for 910C is pretty low (270GB/s compared to 5TB/s for Blackwells).

CPU-GPU bandwidth: Also, there is no dedicated NVLink CC type dedicated network for communication between GPUs and CPUs. This type of connectivity exists from Grace-Hopper generation. For both Grace-Hopper and Grace-Blackwell is 900GB/s (450GB/s unidirectional) and for Vera-Rubin it is 1800GB/s (900GB/s unidirectional). Below is a diagram showing it for Vera Rubin (sorry did not get similar diagram for Grace Blackwell, but I added numbers to the diagram).

In case of Huawei servers - to provide high bandwidth between CPUs and GPUs - CPUs and GPUs are connected using UB Switches and UB protocol. Huawei uses its UnifiedBus (UB) and UB-Mesh protocols as a universal fabric to scale up and connect CPUs, GPUs, and NPUs. Designed to replace legacy interconnects like PCIe, NVLink, and TCP/IP, the protocol pools massive arrays of processors into a single logical “SuperNode” memory space. In the below diagram Ascend 910C are referred to as ‘NPUs’. It shows a single node of CloudMatrix 384 - there are 48 such nodes with 8 NPUs each, thus total count is 384. There are 4 CPUs per node, so there are total 192CPUs in a Superpod.

Nvidia’s innovation: Rack-scale systems

NVL72 is a rack-scale system that has 18 compute trays, each with 2 CPUs and 4 GPUs, so total 72GPUs. These 72GPUs are connected to one another using NVLink and NVSwitchs (Switches are placed at the center of the racks). This allows those 72GPUs work like a single massive GPU. Any GPU can access any other GPUs HBM. The NVLink interconnect is based on copper (cheap) and high bandwidth. This is brilliance of Nvidia’s design:

Huawei’s answer: Multi-rack system CloudMatrix 384

Grace Blackwell NVL72 racks of Nvidia put 72GPUs in a single scale up domain where any GPU can talk to any GPU and access each other’s HBM memory with ease at extremely high bandwidth (900GB/s for Grace Blackwell and 1800GB/s for Vera Rubin unidirectional). Each Huawei 910C has lower FLOPs, HBM and various bandwidths. Despite that Huawei’s multi-rack system CloudMatrix 384 manages to beat NVL72 on capacity: FLOPs and Memory and various bandwidths. How?

By putting 384 NPUs in a single scape up domain. It is a gigantic feat.
This is where Huawei’s expertise as a gigantic networking company helps.

Quote from Huawei: “A defining feature of CloudMatrix384 is its peer-to-peer, fully interconnected, ultra-high-bandwidth network that links all NPUs and CPUs via the UB protocol. CloudMatrix384’s UB design is a precursor to the UB-Mesh. Each of the 384 NPUs and 192 CPUs connects through UB switches, enabling inter-node communication performance that closely approximates intranode levels. The inter-node bandwidth degradation is under 3%, and inter-node latency increase is less than 1 µs. Given that modern AI workloads are predominantly bandwidth-intensive rather than latency-sensitive, this marginal latency overhead has a negligible impact on the end-to-end performance of AI tasks. Overall, this design allows CloudMatrix384 to function as a tightly-coupled, large-scale logical node with globally addressable compute and memory, facilitating unified resource pooling and efficient workload orchestration.” (Source: Serving Large Language Models on Huawei CloudMatrix384)

So while Nvidia creates a giant server out of 72 GPUs of NVL72, Huawei manages to do it with 384 Ascend chips with help of UB protocol.

The Superpod looks rather large as it has multiple racks! What are the downsides? There is no free lunch in this world. Superpod requires much more power compared to NVL72 and its scale up networking is much more expensive. But the good thing is that China has lot of power, and the supply chain for Superpod is domestic from China’s perspective.

Even Huawei has AI Factory blueprint

Huawei doesn’t just provide Superpods, but they provide complete data center scale designs with reference architecture and networking solutions.
Multiple of Superpods can be connected into a gigantic network of up to 165K NPUs! Recall Longcat team used just 50K NPUs.

Question of the decade: Copper or Optics for CloudMatrix 384 scale-up?

A quick visual comparison with NVL72. It is a single slim rack with copper backplane connectivity for scale-up (NVLink, connected with NVSwitches). To be sure there is lot of copper (to be precise worth 100K plus USD). Look at the image. In Nvidia AI factory concept, copper is used for ‘scale up’ and optics is used for ‘scale-out’. Nvidia may use optics for scale up for Feynman GPUs in 2028-2029 time frame. Many optics stocks are up in that hope. Huawei uses optics for scale up as of today!

How are 384 NPUs connected with one another then for scale-up? NPUs within a given node can talk to each other via UB Switches and copper based connectivity, but to talk to other NPUs within a Superpod, they have to use optical connectivity as distances are longer than 2m, that copper can not handle.

SemiAnalysis believe they are using short range (SR8) optical transreceivers for such scale up networking. The other option is using LPO (Linear Pluggable optics). SemiAnalysis provides a great cost comparison for networking options (highly recommended to be a subscriber).

In comparison to NVL72, CloudMatrix 384 takes much more space as well (obviously as it is many racks next to one another) and requires more expense on networking and also on power.

Chip Level and system level Comparison between Nvidia and Huawei

This table from SemiAnalysis provides a great comparison between NVL72 based on GB200 and CloudMatrix 384 based on Ascend 910C.

Quote from SemiAnalysis: “A full CloudMatrix system can now deliver 300 PFLOPs of dense BF16 compute, almost double that of the GB200 NVL72. With more than 3.6x aggregate memory capacity and 2.1x more memory bandwidth, Huawei and China now have AI system capabilities that can beat Nvidia’s.

What’s more, is the CM384 is uniquely suited to China’s strengths, which is domestic networking production, infrastructure software to prevent network failures, and with further yield improvements, an ability to scale up to even larger domains.

The drawback here is that it takes 4.1x the power of a GB200 NVL72, with 2.5x worse power per FLOP, 1.9x worse power per TB/s memory bandwidth, and 1.2x worse power per TB HBM memory capacity.”

UB-Mesh: Huawei’s answer to NVLink

I bet we are going to hear a lot about UB-Mesh in coming days. UB-Mesh (built on Unified Bus, or UB) originated as Huawei’s proprietary technology, but Huawei has since open-sourced the specification. It is currently in a transition phase: while it is open-source and free to license, it is not yet an industry standard recognized by global bodies like the IEEE or IETF.

NVLink simplified networking a lot for Nvidia GPUs (they later developed NVLink-CC that works between Nvidia CPUs and GPUs). With UB-Mesh, Huawei is going a step further by unifying network between GPUs and CPUs as well.

UB-Mesh is explicitly designed to replace the PCIe/NVLink/IB “hybrid” stack with one unified, memory-semantic (load/store/atomic) fabric, and its ambition is specifically to extend NVLink-like peer memory access beyond a single scale-up node to a much larger mesh domain.

The UB’s peer-to-peer communication capabilities enable efficient pooling of hardware resources, including DDR DRAM, CPUs, NPUs and NICs. For instance, CPUs and NPUs are pooled via UB interconnects to enhance resource utilisation.

Different types of physical links proposed by UB-Mesh based on Distance:

Every KV Cacher’s Dream: Memory Pooling

Memory pooling in CloudMatrix384 works by aggregating the DRAM attached to all 192 Kunpeng CPUs across the supernode into a single, shared, high-performance memory pool, rather than leaving each CPU’s memory siloed to its own node.

This pool is made accessible to every one of the 384 Ascend 910 NPUs, whether they’re doing prefill or decode work, over the ultra-high-bandwidth Unified Bus (UB) network, with uniform bandwidth and latency regardless of where the data physically sits. This is a deliberate departure from conventional “KVCache-centric” designs, where a request has to be routed to the specific node holding its cached KV data because remote access is too slow; here, any NPU can pull from the shared pool directly, which decouples request scheduling from data locality entirely.

The practical payoff is threefold: it eliminates the under-utilised, siloed DRAM problem that plagues legacy architectures, it simplifies scheduling since requests no longer need cache-aware routing, and it raises overall cache hit rates and resilience under bursty or uneven workloads.

This pooled memory substrate is what powers services like Huawei’s elastic memory service (EMS), which specifically accelerates KV cache reuse, model weight loading, and checkpointing by giving them memory-class bandwidth and latency instead of falling back to slower, disk-bound I/O paths.

Such pooling has been a dream in the Nvidia ecosystem. It is implemented using software layer e.g. SGLang HiCache utilising Mooncake as its L3 storage backend provides distributed memory pooling. In Huawei CloudMatrix, necessary support is available at the hardware layer to pool CPU memory across entire of CloudMatrix 384 which has 192 Kunpeng CPUs! In next sections will take a deeper look.

Enabling extremely long horizon use cases on the way to AGI

We have entered the era of long horizon agents and 1M context has become commonplace. In such cases, you absolutely need to implement KV cache that will scale and cheap otherwise cost of serving grows rapidly. CloudMatrix 384 enables cheap and fast KV cache by providing Peer-to-Peer architecture over KVCache-centric architecture common in Nvidia ecosystem.

Existing systems like NVIDIA Dynamo and Mooncake are built around where the KV cache lives. When a request comes in, they send it to the specific node that already holds that request’s cached data from earlier. They have to do this because reaching memory on the same node is far faster than reaching another node’s memory — roughly 256 GB/s locally versus only about 25 GB/s across nodes. Fetching a cache from a remote node is slow, so it’s better to just send the request to wherever the cache already sits.

But this creates problems. The scheduler has to track where every cache is and route accordingly, which is complicated. It also throws off load balancing when traffic is uneven, since some nodes get overloaded just because they happen to hold the needed data. And it wastes resources: the DRAM on decode nodes mostly sits idle, unable to help with caching for anyone else.

CloudMatrix-Infer (a system proposed by Huawei) avoids all this by leaning on the UB interconnect, which is fast enough that reaching memory anywhere in the supernode costs about the same as reaching it locally. All the CPU memory is combined into one shared pool, and every NPU - whether it’s doing prefill or decode - can reach that pool directly. In effect, the old gap between “local” and “remote” memory disappears.

Because scheduling no longer has to care where the cache is stored, three things follow. First, the scheduler becomes simple: a request can go to any free NPU, which spreads the load evenly and keeps NPUs busy. Second, there’s no need for the complicated “send it to the node that has the data” logic, so the system is easier to build and maintain. Third, pooling all the DRAM together makes one big shared cache that uses memory more efficiently, gets more cache hits, and holds up better when traffic is bursty or lopsided.

Below diagram shows a common topology that separates prefill and decode nodes that can access CPU DRAM (and even model weights) over fast UB Plane.

To be clear similar memory access pattern is possible in Nvidia ecosystem. As can be seen in below diagram for GraceHopper superchips, a GPU can access memory of a CPU to which it is not attached. Extending this logic, you could logically pool memory of 36 CPUs on a NVL72 system. However, in CloudMatrix there are 192 CPUs and their memory is available to pool!

Dealing with the truth of life: component failures

Huawei’s UB-Mesh employs a N+1 high availability design: each rack consists of an additional backup NPU. In the event of unexpected failures in the NPUs within the system, the backup NPU is activated to restore functionality and ensure the uninterrupted continuation of LLM training jobs.

Recall CloudMatrix’ 384 ASICs are connected to one another using optics transreceivers for scale up. The routing system facilitates rapid failure recovery in the case of communication link failures through a novel direct-notification technique. Check out UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecture- it greatly describes how ‘auto-healing’ is implemented. This allows Huawei to have large number of lower performance components and still deliver reliably.

To be clear: CloudMatrix384’s UB design is a precursor to the UB-Mesh proposed in the above mentioned paper, so it may not have all the features. But Ascend 950 based upcoming system should have all these ‘auto-healing’ features.

Equivalent solution in Nvidia ecosystem is keeping spare nodes ready. So rather than just bringing spare GPU-NPU online, you take out the full node with a faulty GPU-NPU and replace it. The node is not removed physically during training - it is logically removed by removing it from the training flow.

What does the future hold for Huawei?

Logic dies is not an issue for Huawei. SMIC is scaling production of 7nm dies for Huawei - they are designed by HiSilicon for Huawei. Same with CPUs, Networking and even Optics. While Huawei does not have IP for DSPs in comparison to Broadcom and Marvell , HiSilicon designs ‘good enough’ DSPs for them. They can also make do with LPO (Linear Pluggable Optics) for transreceivers - LPO eliminates need for DSPs. ‘Average DSP’/LPO approach works as Huawei’s focus is on providing “complete AI factories” rather than individual components. They can ‘tune’ / ‘co-design’ different components together.

Huawei’s major bottleneck is HBM. For their Ascend 910C they have 128GB older generation HBRAM with 3.2TB/s bandwidth, compared to 192GB of B200 at enormous 8TB/s bandwidth. B300 has 288GB/s at 22TB/s, Ascend 910C looks pale in comparison. They have stockpiles for only 1.6M Ascend 910Cs (source:SemiAnalysis)

That said, as we saw, the entire Chinese labs ecosystem is working on developing techniques that reduce the need for HBM and HBM bandwidth and use more of DRAM/LPDDR and SSD. DeepSeek is the clear leader and most other labs adopt or adapt DeepSeek’s innovations. For example,

• DeepSeek V4’s CSA and HCA reduces HBM memory need for KV cache by 98% compared to GPQA baseline.

• DeepSeek’s DSpark reduces HBM Bandwidth by 66% for Decode stage which is memory bandwidth hungry.

• DeepSeek’s Engram/LongCat’s n-gram trades CPU memory for FLOPs and HBM RAM.

So HBM is not going to hold Chinese ecosystem back for long. CXMT is also working on HBM3. While yield may be low for now, given their track record, one can fully trust CXMT to overcome those challenges.

China now has home grown giants for DRAM/LPDDR and SSDs - those are CXMT and YMTC respectively. Even if they are not at the frontier of their respective technologies, they will be able to support China’s AI ecosystem as they have already reached ‘good enough’ capabilities and scale, and they are not stopping soon. SemiAnalysis has done excellent analysis on emergence of CXMT as a memory giant. It is clear, CXMT and YMTC may not help China build data centers at the same scale as the United States, but they can produce globally significant amounts of memory.

Further, what Huawei can always count on is ingenuity of Chinese labs for making do with what they have and work with less HBM and HBM bandwidth. Chinese labs like ZAI have further demonstrated their models like GLM 5.2 are RSI (recurrent self improvement) capable. DeepSeek, MoonShot, LongCat etc. will arrive there in couple of months. So entire AI ecosystem of China has reached what is often called as ‘the escape velocity’.

What does the future hold for Longcat Lab?

Behaviour of Chinese companies baffles westerners. If Uber, DoorDash or Insta-cart are not building AI (LLM) models, what is the need for their Chinese equivalents to build AI (LLM) models? If Ford and GM don’t build AI (LLM) models, why Xiomi is building models? - saving Xiomi AI story for another blog.

Here is the reason: China has multiple super apps: WeChat (Tencent), AliPay (Ant), Douyin (ByteDance), TaoBao (Alibaba) - West hasn’t seen anything like it yet. When westerners visit China they are always impressed how cool Chinese super apps are and how convenient daily life can be with help of these apps. With help of AI, these super apps are going to be even more powerful. They will help their users run their daily lives on auto-pilot. Whoever does it wins over the most users and makes the most revenue - which could be hundreds of billions annually. Hence, Meituan treats AI & LLM tech as the core competency - it is a matter of survival- and they will march ahead with even more investment.

Meituan has many challenges, including cut throat competition with Alibaba and JD.com for food delivery bleeding billions; but AI is an area where they would not be able to compromise. How could they? All their competitors have major AI labs: WeChat (Tencent), AliPay (Ant), Douyin (ByteDance), TaoBao (Alibaba) and using AI to win even more business!

Sources:

The new shape of intelligent experience

For last few years, every product team building on top of language models for voice experience has been forced to pick a corner of an awkward triangle: you could have fast, you could have smart or you could have cheap; but not all three at once. Real-time voice assistants were small and dumb because anything bigger could not respond inside a human’s patience window. Frontier reasoning models were astonishing in benchmark tables, but their real time voice versions felt lacking. If you found something that was intelligent and fast, it was also very expensive. In addition to experimenting with natively multimodal models, developers also tried cascading state of the art models together: ASR, LLM, TTS with each layer adding latency, each handoff losing context, each silence between turns reminding you that you were talking to a machine.

The interaction-model and background-model split, as proposed by Thinking Machines (TML), is the first credible answer to that triangle. By assigning the two halves of intelligence to different hardwares on different latency budgets, the architecture sidesteps the trade-off rather than negotiating with it. The half that has to be present is allowed to be small, fast, multimodal, and continuously attentive — five times a second, no exceptions. The half that has to be smart is free to be enormous, slow, deliberative, and tool-using, because it is never on the critical path of the 200ms cadence. Neither side is asked to compromise on what it is good at. The user experiences both, fused.

What does that unlock concretely? Consider the experiences that have been technically promised for a decade and have never quite worked:

1. A language tutor who can listen to you stumble through Spanish, gently correct your pronunciation mid-sentence, and simultaneously plan a six-month curriculum tailored to where you actually struggle. The interaction model handles the listening and the correcting in real time. The background model is somewhere else, watching your error patterns accumulate, deciding when to introduce subjunctive mood, drafting tomorrow’s lesson while you finish today’s.

2. A cooking companion that watches your hands through your phone camera, calls out that the onions are about to burn, and pulls up a substitute when you realise you’re out of cumin — without any of those interactions feeling like separate apps. The visual perception is handled in the 200ms loop. The recipe knowledge, the substitution reasoning, the meal-planning context lives in the background.

3. A therapy or coaching presence that can simply be there in long sessions, patient, responsive, never lagging; while the background model does the genuine work of remembering what was said three weeks ago, noticing a pattern across months, deciding when a gentle reframe is appropriate.

4. A workplace assistant that is actually in the meeting with you, not transcribing it for later. It can interject with the relevant figure when someone mis-cites a number, surface the Slack thread that supersedes a decision being re-debated, and do all of that without the awkward “OK assistant, are you there?” preamble that makes today’s tools feel like talking to a Roomba.

The pattern is the same in every case: the experience becomes natural when being there and being smart stop competing for the same compute budget. Once you stop forcing one model to do both, the entire UX problem dissolves. The model that’s there is allowed to be specialised for presence. The model that’s smart is allowed to be specialised for thought. They cooperate over a thin interface, and the user perceives a single attentive intelligence that, finally, does not punish them for talking the way humans talk.

This is what people mean when they say voice is the next interface. It was never really about voice. It was about whether intelligence could show up in the room without making you wait for it. The split architecture is the first proof that it can.

Finally we can have an ever present intelligent, responsive compassionate assistant like that showed in 2013 Science Fiction romance movie “Her”. This blog will discuss key technological milestones that led us to this point.

Welcome Thinking Machine’s Interaction Model

I have been mesmerised by the demos coming out of Thinking Machines interaction model. It is the same kind of mesmerisation I felt the first time I saw Kyutai’s Moshi, and then again when Sesame dropped (try the preview on their site — it is worth the click). We are getting closer to “Her”. Top tier intelligence, with high interactivity and compassion.

What is striking is that both Sesame and Thinking Machines’ new interaction model lean heavily on ideas pioneered by Kyutai’s Moshi. If you want to understand how Thinking Machines manages to combine frontier-grade intelligence with sub-second responsiveness, the most useful prerequisite is Moshi and its retrieval-augmented successor MoshiRAG.

The Thinking Machines (also referred to as TML) blog post is however sparse on how their interaction model interacts with their background model that does the asynchronous heavy lifting, and that is the gap this blog tries to fill.

I also call out differences between TML’s approach and that of Moshi’s on input side.

I also want to take a moment to thank Kyutai lab who build and open sourced Moshi and also to thank Thinking Machines team for taking the torch further and bringing us closer to “Her”. As discussed, the kind of applications it is going to enable are going to be profound. This one is one cool example:

Major milestones on the way

Most language models still treat conversation as a turn-based protocol: you finish, then I speak, then I finish, then you speak. That works for chat boxes. It falls apart the moment you want a model to translate live, count your push-ups, or stop you mid-sentence when you have made a factual error. Real conversation is full-duplex, because both parties produce and perceive at the same time, and silence and overlap carry as much information as the words.

Three lineages have been pushing at this problem from different angles. Moshi rebuilt the speech LM stack around a dual-channel architecture. Sesame scaled it to a level where it started feeling magical. MoshiRAG bolted on asynchronous retrieval to fix factuality without sacrificing real-time flow. And Thinking Machines’ interaction model takes both ideas and scales them to a 276B MoE backbone, to video, and to a deliberately tiny 200ms micro-turn cadence.

1. Moshi: the dual-channel foundation

Kyutai’s Moshi (2024) is a 7B parameter speech-text foundation model built on an RQ-Transformer (the paper). There are two transformers stacked together:

• A “temporal” Transformer running at 12.5 Hz, processing one frame every 80ms.

• A “depth” Transformer that, at each temporal step, predicts 8 audio codebook tokens — Moshi’s actual speech output, encoded by the Mimi neural codec.

The key trick is that Moshi is dual-channel: it ingests the user’s speech tokens and predicts its own speech tokens simultaneously, in interleaved streams. There is no voice-activity-detection harness deciding when it is “the model’s turn” — at every 80ms frame the model is free to speak, stay silent, backchannel, or interrupt. This is the architectural foundation that everyone in the modern full-duplex space, including Sesame builds on. Thinking Machines changes it a bit that we will see in detail.

2. Sesame: refining the experience

Sesame’s Conversational Speech Model takes the same full-duplex spirit and pushes hard on voice quality and emotional naturalness. The public technical details are slimmer than Moshi’s (which has an open paper), but the design lineage — continuous bidirectional audio streams, no separate VAD harness, low-latency codec output — is unmistakably Moshi-style. If you have tried the preview, the demo experience speaks for itself.

3. MoshiRAG: thinking while talking

Moshi has one weakness that is hard to engineer around at 7B parameters: factuality & intelligence. Speech LMs are trained on vastly less text (in word count) than text-only LMs, and it shows on QA benchmarks. The obvious fix is to make the model bigger, but a bigger model cannot run in real time. Kyutai’s answer, published in MoshiRAG in April 2026 (the paper), is to split the system in two:

• A front end that stays small and real-time (this is similar to TML’s interaction model) — Moshi 7B, a 1B streaming ASR, and a frozen reference text encoder (ARC-Encoder).

• A back end that handles knowledge retrieval asynchronously (this is similar to use of a background model by TML) — a local LLM like Gemma 3 27B or an API call to GPT-4.1.

The two are connected by a single new token. When Moshi sees a knowledge-intensive question coming, it predicts a <ret> token in its text channel. That fires off a retrieval call. Crucially, Moshi does not wait for the answer — it keeps speaking, producing a “pre-RAG” lead like “Let me think about that...” or “In the Netflix series...” that buys the back end roughly two seconds of slack. By the time Moshi reaches the body of its answer, the retrieved reference is ready.

An ASR model transcribes the user speech and that with useful context is provided to the back end model to do its work.

How the retrieved answer gets back into the stream

This is the part most people get wrong on first reading. The retrieved reference is not injected as user text, nor as model text, nor as audio. It comes back as a sequence of continuous embeddings — produced by ARC-Encoder, compressed 4× so the sequence is short enough to fit Moshi’s 12.5 Hz frame budget, then projected through a single trainable linear layer.

Those projected embeddings are then additively summed into Moshi’s input vector at the temporal Transformer level, over a short window. Mathematically, the input that was originally:

becomes:

for the duration of the reference’s compressed embedding sequence, and reverts to plain hi​ afterwards. The reference is a transient side-channel that biases Moshi toward grounded facts during the body of its turn — nothing more, nothing less.

The authors did try “insertive” injection (treating the reference as extra tokens spliced into the input sequence) and found it scored higher on accuracy. They rejected it anyway, because lengthening Moshi’s input sequence hurts the model’s ability to hold long conversations. Additive injection was the better engineering trade-off.

What changed in Moshi to make Moshi-RAG work

The architecture changes are surprisingly minimal from orginal Moshi. The paper says it outright: “the only modifications from the original Moshi model are the introduction of a special retrieval trigger token <ret> and a reference text encoder.” In practice there are four small additions:

1. A new <ret> token in the text vocabulary, placed in training data immediately before the “lead” portion of every knowledge-intensive turn (using TTS forced alignment).

2. A frozen ARC-Encoder that compresses retrieved text 4× into Moshi’s embedding space.

3. A one-layer trainable linear projection sitting between ARC-Encoder and Moshi.

4. A learnable h_dropout vector that replaces the reference 20% of training time, so Moshi learns to behave gracefully when retrieval fails or is late.

The base Moshi backbone — the RQ-Transformer, Mimi codec, dual-channel design — is untouched. MoshiRAG is fine-tuned from the original Moshi checkpoint with 100k updates at a tiny 2 × 10⁻⁶ learning rate.

4. Thinking Machines’ Interaction Model

The TML blog post is up-front about the lineage: “This approach builds upon prior work like Qwen-omni, KAME, MoshiRAG.” But the scale and the design choices have moved on substantially. The demoes are incredible, to say the least. Here is on more example. You can see more in the blog.

Time-aligned micro-turns. Where Moshi works at 12.5 Hz (80ms frames), TML works in 200ms chunks. Each chunk interleaves input and output across all modalities — audio, video, and text. There are no turn boundaries, no VAD, no harness. We will talk more about it in the next section.

Encoder-free early fusion. On the input side to Interactive Model, the blog has following details: Audio enters as dMel features through a light embedding layer. Video enters as 40×40 patches through a small hMLP. Audio output uses a flow head rather than a discrete codec. All of it is co-trained from scratch with the transformer. The 12B active parameters deal with all the modalities.

The background model split. This is the part that most directly echoes MoshiRAG. The interaction model handles real-time presence. When a task needs deeper reasoning, planning, or tool use, the interaction model delegates to a background model that runs asynchronously, then weaves results back into the conversation as they arrive — at a moment “appropriate to what the user is currently doing, rather than as an abrupt context switch.” That last phrase is exactly the MoshiRAG philosophy: hide the retrieval latency inside the natural rhythm of speech. One call out: where MoshiRAG bolts on a separate Whisper-class streaming ASR (1B parameters, frozen), TML does not explain whether they use an ASR model or not to provide input to the background model, but it is quite likely they do.

What the Thinking Machines blog does not tell us is how results come back into the interaction stream. MoshiRAG’s answer — additive embedding injection at the temporal Transformer level — is documented in the paper. TML’s mechanism is not. But the surface-level analogy is striking enough that it is reasonable to guess the implementation belongs to the same family.

Thinking Machine’s approach to model input:

Here there is a difference from Moshi

Moshi: combined at every frame. As we discussed earlier, Moshi’s temporal Transformer receives, at each 12.5 Hz time step, one input vector that is the element-wise sum of three streams: model text, model speech, and user speech embeddings (Equation 1 in the paper). MoshiRAG adds a fourth optional term for the reference embedding during the injection window. There is no notion of “input now, output later” within a frame. At every 80 ms tick the model simultaneously sees the user’s incoming audio and predicts its own next text and audio tokens. Three concurrent streams collapse into one combined vector per step.

TML Interaction Model: interleaved at chunk granularity, combined within a chunk: Thinking Machines runs at 200 ms chunks (5 Hz). The blog illustration is explicit about the macro structure — “the model receives a single interleaved token sequence” — and shows it as input_0 → output_0 → input_1 → output_1 → input_2 → output_2 → …. So at the between-chunk level, the model alternates: it consumes 200 ms of input, then emits 200 ms of output, then consumes the next 200 ms of input, and so on. That is sequence-based, not parallel.

But within a single 200 ms chunk, multiple modalities are still combined, just in a different way. The architecture diagram in the blog shows text frames, audio dMel features, and 40×40 video patches (encoded via hMLP) all flowing into model input after bring embedded.

Size comparison

TML Interaction Model’s Inference optimisations

The blog is pretty brief on this. Here is my read: the 200ms micro-turn design fundamentally changes what inference looks like. Standard LLM serving is optimised for big prefills followed by long decodes; TML produces a continuous stream of tiny prefill+decode rounds, five per second per active session. Per-turn overhead dominates the math, so most of the optimisation work is about killing that overhead.

Streaming sessions. Instead of treating each 200ms chunk as a fresh request that re-allocates buffers and rebuilds metadata, the inference server keeps a persistent sequence in GPU memory and appends incoming chunks to it. A version of this is upstreamed to SGLang.

Custom MoE kernels. TML-Interaction-Small is a 276B MoE with only 12B active. At interaction time the per-step batch is tiny, so the standard grouped-GEMM kernel that MoE inference usually relies on wastes most of its throughput. Thinking Machines replaced it with a GATHER+GEMV strategy — gather small per-token activations to the right experts, then run GEMV rather than GEMM — drawing on similar work in PyTorch’s gpt-fast and Cursor’s “warp decode.”

Latency-tuned kernel shapes. Beyond MoE, the kernels are explicitly tuned for the unusual shapes of bidirectional serving — where prefill and decode happen continuously at small sizes rather than in the canonical “big prefill, long decode” pattern.

The post is notably quiet on quantisation, speculative decoding, and KV-cache compression.

TML Background Model’s Inference optimisations & Hybrid Architecture

While the blog does not state what background model TML team used, it is probably a state of the art model. If you try to use an off the shelf LLM API like Sonnet 4.6 or Opus 4.7 you typically get 50-100 tokens per second per user. However, that is not enough for generating tool calls, executing them, preparing a response and handing it back to the interaction model. In such scenarios Cerebras like performance can make a material difference to serve SOTA intelligence in the real time. Cerebras can serve 1500+ tokens per second for SOTA open source models.

Furthermore, one could deploy a hybrid architecture wherein interaction model runs on an edge server or on a server in a datacenter, and it connects to the background model over a fast network (within the same location or over the internet). Even if the connection to the background model is lost, the system will continue to work as interaction model has enough intelligence to carry on a conversation. The TML blog is not 100% clear on how the models interact. If it is text out, text in, it is easy. But if there is tighter integration involving activations, then it may require higher bandwidth (though I don’t think this is what is happening. It is most likely text out, text in). Also, for now Cerebras does not have built in web search from what I know, so there will need to be an intermediate agent - which would also need to be fast.

Ambient AI

This is a privacy preserving setup and may be desired by many as interaction model is exposed to voice and video - which may haev sensitive content. This approach can safely enable “Ambient AI”. Ambient AI, often referred to as ambient intelligence (AmI) or ambient computing, represents a shift in artificial intelligence from reactive (waiting for prompts) to proactive and invisible (acting on context).

In my mental model if coding is 1X of token use, AI co-working is going to be 10X of token use, and Ambient AI is going to be 100X of token use. This two model architecture pattern and chipset like Cerebras are vital to enable it.

Trivia

One of my favourite blogger Stratecherry believes Cerebras main use case is not Coding, but such real time interaction use cases in voice (Ambient AI) and possibly robotics domain. I believe Cerebras - in addition to such performance sensitive applications - is also very important for coding, as well as, personal AI agents use cases when users are in driving seats. Stratcherry believe background agents will replace all the coding and personal agent use cases. I believe, users will want to be in the loop forever, not because they have to, but because they want to - just like people want to browse their social media themselves. It is immensely satisfying - plus it can compound the output! But, that is a debate for another time…
You can also read this excellent deep dive on Cerebras by Semianalysis.

Kyutai, take a bow!

Image Source: Techcrunch

Kyutai was launched in November 2023 at Station F in Paris, a non-profit AI lab seeded with roughly €300 million from Xavier Niel (Iliad), Rodolphe Saadé (CMA CGM), and Eric Schmidt's foundation, with a mandate that is still rare in the field: do frontier research, publish everything, release the weights, and treat open science as the default rather than a marketing posture. Six scientists took the stage that morning, and their CVs read like a quiet history of European generative AI.

1. Patrick Pérez, who became CEO, had spent decades at Inria, Microsoft Research, Technicolor, and Valeo, where he ran the valeo.ai lab.

2. Hervé Jégou had helped found Meta's FAIR Paris office and is the mind behind product quantization and the FAISS vector-search library that quietly underpins much of modern retrieval.

3. Edouard Grave came from FAIR's language modelling group.

4. Laurent Mazaré arrived from DeepMind by way of Jane Street.

5. Neil Zeghidour and Alexandre Défossez, both veterans of Google Brain, DeepMind, and Meta, had between them already invented many of the neural audio codecs the field now takes for granted — SoundStream, EnCodec, MusicGen, AudioLM.

When this team turned its attention to full-duplex speech, the result was Moshi: a system that did not just compete with the cascaded pipelines of the day but made them look like a category error. MoshiRAG followed in 2026, extending the same architecture with asynchronous retrieval while keeping the open-science commitment intact — every detail down to the training-time retrieval-delay sampling distribution is in the paper. In late 2025, Zeghidour, Mazaré, and Défossez spun out Gradium to commercialise the voice work while keeping the lab close, a quietly elegant solution to the hard economic question of how a non-profit funds frontier compute.

While Thinking Machine should get massive well deserved credit for bringing us this far, Moshi paved the way for this revolution. The lineage matters. So does the lab that chose to publish it. Many thanks to the team behind Kyutai!!!

Closing thoughts

Moshi proved you could ditch the cascaded ASR–LLM–TTS pipeline and get a real full-duplex speech model. MoshiRAG proved you could keep that real-time front end and still get factuality by farming knowledge work out asynchronously.

Thinking Machines is now proving you can scale the same two-tier idea to a 276B/12B Active MoE backbone, add video, and still hit sub-second turn-taking latency and basically build your HER. You could add more capable agents between the interaction model and the background model and create phenomenal applications that will feel much more pleasant to interact with. This will also create a market for Cerebras style inference accelerators. And it is a big market.

TML’s interaction model is incredibly impressive, and as they say: “this is the worst it will ever be”!

Also, if you haven’t watched the movie Her, let me tempt you by showing this trailer.

Moore’s Law — and why it’s over - and what comes after

In 1965, Intel co-founder Gordon Moore observed that the number of transistors on a chip doubles roughly every two years at constant cost, a trend that held for five decades and powered the entire digital revolution. But Moore’s Law was never a law of physics; it was an economic observation, and it depended on two things: transistors getting smaller, and smaller transistors being cheaper. Both have stopped. On the cost side, the price per transistor plateaued at the 28nm node around 2014 and is now rising: TSMC’s 2nm node, shipping in 2026, will be the first major node where cost per transistor goes up, not down. On the physics side, we’ve hit the atomic wall: at the 2nm process, critical transistor features are roughly 10 atoms wide. At TSMC’s upcoming 1.4nm node (2028), that shrinks to about 7 atoms. You cannot build a switch smaller than an atom; this isn’t an engineering problem to be solved, it’s a fundamental limit of matter (to be clear the node names are indicatively given, they do not indicate precise feature size; but even that being considered, it will be sufficiently clear that we can not go infinitely low on size.) Quantum tunneling causes electrons to leak through barriers that should be impenetrable, blurring the distinction between “on” and “off” that all digital computing depends on. The golden era where each generation was simultaneously denser, faster, and cheaper is definitively over.

That is why Jensen Huang (CEO Nvidia) has been saying in his every interview that Moore’s Law is dead! We can no longer shrink transistors at the pace Gordon Moore predicted in 1965. The doubling of transistor density every two years on a single chip has stalled. The number of transistors per GPU tile (we will talk more about tiles later) only increase slightly: 80 billion in Nvidia Hopper tiles, 104 billion in Nvidia Blackwell tiles, and 168 billion in Nvidia Rubin tiles. And yet, performance improved much more!

How? The answer isn’t one trick — it’s a system of engineering innovations across five axes (there is a name to it: extreme co-design):

· More tiles per GPU: tiling multiple dies into one GPU package

· Lower precision: FP8 → FP4 doubles effective FLOPS per tensor core

· More memory bandwidth: advancing HBM generations and adding more stacks

· Superchips: fusing CPU + dual GPUs with coherent chip-to-chip interconnects

· More GPU-to-GPU bandwidth: scaling NVLink and building rack-scale fabrics like NVL72

The Nvidia GPU Generations at a Glance

(To keep the table simple, I haven’t added H200, B300 and Rubin Ultra in the table. B300 offers for example much higher Peak AI TFLOPs for FP4 around 14,000 FP4. Rubin Ultra offers more performance and memory.)

For Rubin , Nvidia claims up to an effective 50,000 TFLOPS of FP4 performance can be achieved with an updated 3^rd generation Transformer Engine that replaces 2:4 structured sparsity from prior generations. Without use of it, it would be around 35,000 TFLOPs.

Rubin has another variant Rubin Ultra, which doubles the silicon to 4 reticle-limit chiplets, pairs them with 16 HBM4e stacks totalling 1 TB of memory, and delivers ~100,000 TFLOPS of FP4 compute with the 3^rd generation Transformer Engine .

Axis 1: More Silicon — Tiling Past the Reticle Limit

The Reticle Wall

Every chip is printed using a photolithography mask called a reticle. The maximum area a single reticle can expose on a wafer is roughly 800 mm² — a hard physical limit of the lithography equipment. The H100, at 814 mm², was already pushing right up against this boundary. You simply cannot make a single chip bigger. This is the wall that killed the naive version of Moore’s Law for GPUs.

Blackwell: Two Dies, One GPU

NVIDIA’s B200 broke through this wall with a dual-die design. Two reticle-limit dies (each ~104 billion transistors on TSMC 4NP) are placed side by side and connected via NV-HBI (NVIDIA High-Bandwidth Interface) — a custom die-to-die interconnect running at 10 TB/s. The two dies appear as a single GPU to software. This is how NVIDIA jumped from 80B to 208B transistors without a dramatic process shrink.

Rubin: Two Chiplets on 3nm

The Rubin R100 GPU also uses a 2-die design, but on TSMC’s N3P (3nm-class) process. Two reticle-limit compute chiplets are connected on a CoWoS-L advanced package, reaching approximately 336 billion transistors. The density improvement from 4nm → 3nm, combined with architectural improvements, is how NVIDIA achieves ~1.6× more transistors than Blackwell with the same number of tiles.

Rubin Ultra (2027): Four Chiplets

Rubin Ultra doubles the silicon again — four reticle-limit chiplets on a single package, using die-to-die bonding with a fast coherent interconnect. Combined with 16 HBM4e stacks (1 TB total), Rubin Ultra roughly doubles the compute of standard Rubin to ~100 PFLOPS FP4.

Key insight: NVIDIA isn’t relying on Moore’s Law. They’re using packaging innovation — putting more dies on a single substrate — to scale transistor count far beyond what any single die could hold.

Below images can help you understand this point visually!

For the Blackwell GPU, you can notice two tiles: one on the left and one on the right. Area is split approximately equally between the two.

For the Rubin GPU as well, you can notice two tiles: one on the left and one on the right. Area is split approximately equally between the two.

Axis 2: Lower Precision — How FP4 Doubles Effective FLOPS

One of the most powerful — and least understood — levers NVIDIA pulls each generation is reducing numerical precision. This is not about making calculations less accurate. It’s about doing more useful work per clock cycle by using smaller numbers where full precision isn’t needed.

The Precision Ladder

Neural networks are remarkably tolerant of reduced precision. The weights and activations in a trained model don’t need 32-bit or even 16-bit floating point to produce correct results. NVIDIA has exploited this insight aggressively:

Why Halving Precision Doubles FLOPS

The math is straightforward: when you halve the number of bits per value, the tensor cores can process twice as many values per clock cycle through the same silicon. A matrix multiply that operates on 8-bit values can crunch exactly twice as many elements per cycle as one operating on 16-bit values, using the same physical hardware.

NVFP4: Not Just Smaller Numbers

NVIDIA’s NVFP4 format isn’t a naive truncation. It uses a sophisticated dual-level scaling scheme:

· Values are grouped into small blocks (e.g., 32 elements)

· Each block gets its own FP8 scaling factor that captures the local dynamic range

· Individual values within the block are stored as 4-bit floats (1 sign bit, 2 exponent bits, 1 mantissa bit)

· The combination of per-block scaling + 4-bit values preserves accuracy within ~1% of FP8 on most LLM tasks

This is why NVFP4 works for both inference AND training — the scaling factors compensate for the limited range of 4-bit values.

The Double Benefit: FLOPS + Memory

FP4 doesn’t just double compute throughput — it also halves the memory footprint of model weights compared to FP8:

· A 70B parameter model in FP16: ~140 GB (doesn’t fit on one H100)

· Same model in FP8: ~70 GB (fits on one H100 with 10 GB headroom)

· Same model in FP4: ~35 GB (fits on one B200 with 157 GB free for KV-cache and batching)

Smaller weights mean more of the GPU’s HBM is available for KV-cache (critical for long-context inference) and larger batch sizes (critical for throughput). This is why NVIDIA claims up to 15× inference performance improvement from Blackwell over Hopper at system scale — it’s the compound effect of more FLOPS, more memory, more bandwidth, AND smaller model footprint.

Key insight: Hopper’s lowest native precision is FP8. Blackwell and Rubin add native FP4 support, which effectively doubles the useful FLOPS per tensor core without adding any transistors. This is a “free” 2× multiplier on top of all the other scaling axes.

Axis 3: More Memory Bandwidth — The HBM Scaling Playbook

AI workloads are memory-bandwidth-bound. A GPU can have all the FLOPS in the world, but if it can’t feed data to its compute cores fast enough, those cores sit idle. NVIDIA attacks this memory wall by advancing along three dimensions of HBM.

How HBM Works (30-Second Primer)

HBM (High Bandwidth Memory) is a 3D-stacked DRAM technology. Multiple DRAM dies are stacked vertically and connected using through-silicon vias (TSVs) — tiny vertical copper pillars that punch through each layer. The entire stack sits on a silicon interposer right next to the GPU die, connected via thousands of parallel wires.

Three Levers NVIDIA Pulls Each Generation

1. More stacks on the package

Each generation adds more HBM stacks, requiring a larger silicon interposer (CoWoS-L for Rubin).

2. More layers per stack (taller stacks)

· HBM3: 8 DRAM layers per stack

· HBM3e: 8–12 layers per stack

· HBM4: 12–16 layers per stack

· HBM4e: 16-20 (rumoured) layers per stack

3. Wider interface per stack

· HBM3: 1,024-bit interface → up to ~819 GB/s per stack

· HBM3e: 1,024-bit interface → up to ~1.2 TB/s per stack (higher data rate)

· HBM4: 2,048-bit interface → up to ~2.0 TB/s per stack (doubled bus width)

· HBM4e: 2,048-bit interface → up to ~3.0-4.0 TB/s per stack (doubled bus width and throughput)

HBM4 & HBM4e literally doubles the bus width — a fundamental architectural change in the JEDEC spec.

The Compound Effect

This is packaging engineering — more chips, taller stacks, wider bus. Not Moore’s Law.

Axis 4: The Superchip — Fusing CPU + Dual GPUs with NVLink-C2C

Starting with Grace Hopper, NVIDIA introduced the Superchip: a single module that fuses a CPU and GPU(s) together using NVLink-C2C (Chip-to-Chip) — a coherent, high-bandwidth interconnect fundamentally different from PCIe.

In this how Grace Hopper SuperChip looks. On the left you see Grace CPU and on the right you see Hopper GPU.

What is NVLink-C2C?

NVLink-C2C is a memory-coherent chip-to-chip interconnect that physically bonds the CPU and GPU dies on the same module. Unlike PCIe (which requires explicit data copies), NVLink-C2C creates a unified memory address space — CPU and GPU threads can access each other’s memory transparently.

Key properties of NVLink-C2C:

· Memory coherent — CPU and GPU see a single unified address space

· 900 GB/s bidirectional bandwidth (Grace Hopper and Grace Blackwell superchips) — 7× faster than PCIe Gen5

· 1.8 TB/s bidirectional bandwidth (Vera Rubin) — 7× faster than PCIe Gen6

· 5× more energy-efficient than PCIe per byte transferred (~1.3 pJ/bit)

· Enables KV-cache offloading from GPU HBM to CPU LPDDR without performance penalty

Why this matters: In LLM inference, the KV-cache can consume most of the GPU’s HBM. With NVLink-C2C, the KV-cache can spill into the CPU’s large LPDDR memory (480 GB–1.5 TB) at near-HBM speeds, dramatically increasing the effective context window without adding more GPUs. Let us see how these SuperChips look like.

Superchip Architecture for Grace Blackwell and Vera Rubin:
1 CPU + 2 GPUs

Grace Hopper SuperChip had 1 Grace CPU and 1 Hopper GPU, but now with Grace Blackwell and Vera Rubin Architectures there are 2 GPUs with 1 CPU.

Each Superchip connects one CPU to two GPUs via NVLink-C2C. The two GPUs also connect to each other via NVLink. This creates a tightly-coupled compute unit that serves as the building block for larger systems.

Here is a schematic diagram for Vera Rubin SuperChip. Following that we will see how the real SuperChips look like:

GB200 SuperChip: Notice two Blackwell (B200) GPUs in the upper part and one Grace CPU in the lower part.

GB300 SuperChip is a performance advancement over GB200: Notice two Blackwell (B300) GPUs in the upper part and one Grace CPU in the lower part.

Vera Rubin SuperChip: Notice two Rubin GPUs in the upper part and one Vera CPU in the lower part.

Axis 5: GPU-to-GPU Bandwidth — NVLink and the NVL72 Rack

A single GPU cannot train a trillion-parameter model alone. The speed at which GPUs exchange data — gradients during training, KV-cache during inference — directly determines system-level performance.

How NVLink Scales

NVLink bandwidth per GPU = number of links (ports) × bandwidth per link.

Each generation keeps the same 18 NVLink ports per GPU but increases the per-link speed through faster SerDes circuits. From H100 to B200: 2×. From B200 to Rubin: another 2×.

The NVL72: When 72 GPUs Become One

The Blackwell Generation Innovation

Perhaps the most architecturally significant innovation in the Blackwell generation isn’t the GPU itself — it’s the GB200 NVL72 rack system. It connects 72 B200 GPUs and 36 Grace CPUs (36 GB200 Superchips) into a single, liquid-cooled rack where all 72 GPUs communicate at full NVLink 5 speed. The entire rack forms a single NVLink domain with 130 TB/s of aggregate bisection bandwidth. There is a GB300 version of this NVL72 system.

Each rack has 18 trays with 2 SuperChips (4 GPUs and 2 CPUs), this total of 36 CPUs and 72 GPUs. Note however that 18 trays present you with 18 nodes/servers. You can split them for prefill and decode functions. This is how GB200 NVL72 racks looks. As an exercise can you find 9 NVSwitch trays and 18 compute trays?

How GPU-GPU Communication Works In NVL72 system

The rack contains 9 NVLink Switch trays, each housing 2 NVLink Switch ASICs. These 18 switch chips create a non-blocking, all-to-all fabric. Every GPU has 18 NVLink ports wired through the copper backplane to the switch trays, which route traffic to any destination GPU.

Comparing With H100 Based 8 GPU Servers Baseline

Consider this popular topology developed by SGLang team for DeepSeek inference on 12 servers with 8 H100 GPUs each. DeepSeek model has multiple layers (61 to be precise) and each has Attention and 58 have Mixture of Expert (MoE) experts. 90%+ model parameters are in MoE experts.

They reserve 3 nodes for Prefill (prompt processing and KV cache generation) and 9 for Decode (token by token generation). As you can see from the diagram, MoE experts are divided into subgroups across nodes, whereas each node (in fact each GPU) has full attention weights.

When tokens are processed by an attention layer, each of them gets sent to a bunch of experts for that layer. Some of those experts may be on the same server GPUs and some of them will be on GPUs on different servers. Communication for the former happens over NVLink within respective servers, communication for the latter happens over Infiniband which is much slower than NVLink. This introduces a performance penalty!

Source: LMSYS

How NVL72 Based Systems Solve This Problem?

Traditional GPU clusters connect 8 GPUs per node via NVLink, then connect nodes via InfiniBand. Inter-node bandwidth is 10–50× lower, creating a bandwidth cliff. The NVL72 eliminates this cliff for up to 72 GPUs. For mixture-of-experts LLM inference with GB200 based NVL72:

· Expert parallelism across 72 GPUs without bandwidth penalties

· 13.5 TB of unified GPU memory (72 × 192 GB) at NVLink speed

· 30× faster real-time inference on trillion-parameter models vs H100 (NVIDIA’s claim, we will see real world performance numbers later)

Grace Blackwell NVL72: Where It All Began!

If my memory serves right NVL72 was announced in GTC 2024. I highly recommend watching the following video:

Vera Rubin NVL72: The Next Step!

The Vera Rubin NVL72 packs 72 Rubin GPUs and 36 Vera CPUs (36 Vera Rubin Superchips) into a rack delivering 3.6 EFLOPS of FP4 inference. With NVLink 6 at 3.6 TB/s per GPU, the aggregate rack bandwidth reaches 260 TB/s — 2× the Blackwell NVL72. NVIDIA also introduces silicon photonics for rack-to-rack links, enabling POD configurations of up to 576 GPUs.

I highly recommend watching the following video:

Role of GPU-HBM & GPU-GPU Bandwidths

In both training and inference, GPU compute cores spend most of their time waiting for data — the bottleneck is almost never arithmetic, it’s how fast you can feed weights, activations, and gradients to the tensor cores. During training, each GPU must read the full model weights from HBM every forward and backward pass, exchange gradients with every other GPU after each step, and write updated weights back — so HBM bandwidth determines how fast a single GPU can churn through batches, while NVLink bandwidth determines how fast GPUs can synchronise with each other.

The H100’s 3.35 TB/s HBM bandwidth meant a 70B-parameter model in FP16 (~140 GB, H100 has only 80GB memory, but let us ignore that for now) took about 42 microseconds just to read once from memory; the B200’s 8 TB/s cuts that to 17.5 µs, and Rubin’s 22 TB/s brings it under 6.4 µs — directly translating to faster iteration.
NOTE: All weights are not read from HBM at once, but they are loaded by GPUs layer by layer. This exercise tells you cumulative time lost in loading weights.

During inference, the constraint is even starker: autoregressive token generation is almost entirely memory-bandwidth-bound because each token requires reading the full KV-cache and model weights but performs very little compute per byte read — this is why inference throughput scales nearly linearly with HBM bandwidth. On the NVLink side, large models that don’t fit on a single GPU must be split across multiple GPUs using tensor parallelism (splitting individual matrix multiplies) or pipeline parallelism (splitting layers), both of which require GPUs to exchange intermediate activations every few milliseconds — if NVLink bandwidth is 2× higher, you can split across 2× more GPUs before communication becomes the bottleneck, which is exactly why the GB200 NVL72’s 130 TB/s aggregate bandwidth enables mixture-of-experts models with hundreds of experts distributed across 72 GPUs without the communication penalty that would cripple the same workload on an 8-GPU H100 based node connected by InfiniBand. In short: HBM bandwidth sets the ceiling on single-GPU performance, and NVLink bandwidth sets the ceiling on how many GPUs you can scale to before returns diminish — every generation that raises both ceilings directly unlocks larger models, longer contexts, and lower latency.

Are These Innovations Helping?

One of the best places to see the performance difference caused by these innovations is checking out results of InferenceMax benchmark maintained by SemiAnalysis. Scaling to larger number of GPUs in a single domain and change of GPU generation can cause dramatic improvement in performance. Below we compare performance of H200 (8 GPU servers) with that of GB300 (NVL72 server with 72 GPUs) for inference of DeepSeek R1 model.

Quote by SemiAnalysis: At GTC 2024, Jensen said that GB200 NVL72 was 35x faster than Hopper. Nobody believed it and thought it was classic fake Jensen Math. When we tested the performance of it, it wasn't just 35x faster, it was over 50x times faster even against an strong Hopper baseline with all of the inference optimisation composed together like MTP, Disagg prefill, wideEP, etc. View the nuanced results at InferenceX dot com.

Source: SemiAnalysis

2026: The Era of Extreme Co-design

In the latest Nvidia GTC conference (GTC 2026), Nvidia announced five rack types. We will see in upcoming articles the roles of other racks. But for now remember Groq3 LPX helps accelerate decode performance, Vera CPU racks are useful for deploying large number of agents (during training with RL or inference), STX Storage racks are for KV Cache and Spectrum-6 SPX are for networking.

Nvidia truely understands that “the whole can be greater than the sum of parts”.

SUMMARY: Why Do GPUs Keep Getting Faster?

The era of “free” scaling from Moore’s Law is over. What we’re in now is the era of system-level scaling — where performance comes from co-designing the chip, the precision format, the package, the memory, the interconnect, and the rack as a single integrated system. And NVIDIA has mastered this playbook.

Moore’s Law gave us exponential gains for free. The post-Moore era demands exponential engineering effort for each generation. NVIDIA’s bet is that AI workloads are valuable enough to justify that effort — and so far, the market agrees.

APPENDIX

Approach that Huawei is following:

While Chinese chipmakers can not get EUV machine needed for chips below 7nm, they can still follow this extreme co-design approach and build efficient systems for AI training and inference that compensate for not having chips denser than 7nm. That is exactly what Huawei is doing with their Cloud Matrix multi rack solution. You can read more from this SemiAnalysis report. Off-course these systems would need more power, but power is abundant in China.

Nvidia’s western competition like AMD is following this approach as well. So this is the race where the fastest one wins. However, what benefits Nvidia is also being able to work with thousands of engineers from frontier labs that informs them how the next generation of systems need to be built. Otherwise they have no way of knowing. In return engineers build model architectures and inference engines that take advantage of Nvidia ecosystem capabilities. In Jensen’s view China has abundant AI talent. If Nvidia is not active in China, they miss out on all these learnings that are possible by closely working with that talent. Also labs from the region won’t build for Nvidia ecosystem - thus a great American company stands to lose its position.

Dedication:

Dedicating this article to the giant who inspired the industry at an exponential speed Gordon Moore. He has a true successor in form of Jensen Huang (Nvidia CEO) who continue to inspire the industry for exponential improvements.

We want to understand how much compute and data are needed to train the next generation of models which are going to be in 2T, 4T and 10T parameter count classes (rumours!). Anyways, Elon announced today that they are training their models in this class, so this would be a very timely exercise.

Methodology: We use formulas and approaches used in estimating training costs for DeepSeek V3, ARCEE, KIMI 2.5 etc. We will use GPU performance numbers from Nvidia Data-sheets for B200/B300 and NVL72 racks, and statistics on data availability from Epoch AI. When we say B200/B300 we are actually referring them in GB200/GB300 superchips, as a part of NVL72 racks. NVL72 has world size of 72GPUs, it makes it easier to train models in this class as those 72 GPUs can access each others HBM at high throughputs through NVLink/NVSwitch connectivity. That said data parallelism alone is not enough. You still need other parallelism (pipeline, tensor, sequence). To account for all the bottlenecks, we assume MFU (Model Flop Utilisation) of only 20%.

1. Assumptions

2. Training Data Requirements

Formula:

D = tokens_per_parameter × N_total

D = 40 × N_total

Token count is based on total parameters — all expert weights must be trained across the full run, even though only 5% activate per token.

The publicly crawlable internet provides ~100T tokens (Epoch AI analysis. I have highlighted 100T tokens mark below). Models requiring more must rely on synthetic data generation, rephrasing, or multi-epoch training. As we will see later, synthetic data generation becomes very costly as we approach 10T parameter class.

3. Sparsity Ratios of Frontier MoE Models

To justify the 5% active ratio, we surveyed the latest frontier models:

Why 5% Is a Sound Assumption

· DeepSeek V3/V3.2 (5.4%), GLM-5 (5.4%), and MiniMax M2.5 (4.3%) all cluster around 4–6%. 5% is the center of this range.

· Kimi K2.5 already achieves 3.1% — the trend is toward more sparsity. 5% is conservative for a next-gen model.

· Older models (Mixtral, DBRX) used ~27% active. In 18 months the field moved to 3–6%, a decisive shift.

· DeepSeek V3 trained successfully at 5.5% active on 14.8T tokens with no quality loss vs. dense baselines.

· 5% active yields practical sizes: 2T→100B, 4T→200B, 10T→500B active per token — all proven architecture scales.

4. Training Compute (FLOPs)

With MoE, only the active parameters compute per token:

N_active = N_total × 0.05

C = 6 × N_active × D

C = 6 × (N_total × 0.05) × (40 × N_total)

C = 12 × N_total²

5. GPU Specifications

B300 only outperforms B200 at FP4. At BF16, they are identical (2.25 PFLOPS).

6. Calculation Chain

Step 1: Effective_FLOPS = Peak_FLOPS × 0.20

Step 2: GPU_seconds = C ÷ Effective_FLOPS

Step 3: GPU_hours = GPU_seconds ÷ 3,600

Step 4: GPUs_needed = GPU_hours ÷ 2,160 (= 90 days × 24 hrs)

Step 5: NVL72_Racks = GPUs_needed ÷ 72

Step 6: Total_Cost = GPU_hours × $/GPU/hr

7. FP4 Scenario

Theoretical best case — all FLOPs at FP4 throughput.

From what I know it is not possible today, or not proven today at this scale. Nvidia has done it for smaller models like 12B with 4 bit precision training. But, even there they maintained many other things in higher precision. To quote: “Attention, embedding, non-linear layers, and other tensors: To ensure numerical stability during training, we retain the original precision (e.g., BF16 or FP32) for embeddings, the output projection head, normalization layers, non-linearities, and attention components, including softmax and the query-key and attention score-value batched GEMMs. The main weights (stored by the optimizer), weight gradients (used for gradient accumulation across microbatches and across data-parallel replicas), and optimizer states are also kept in FP32. Tensor parallel reductions are performed in BF16 precision.”

B200 (FP4) — $5/GPU/hr

B300 (FP4) — $6/GPU/hr

8. BF16 Scenario

This is a more realistic scenario that they did all their training in BF16.

B200 (BF16) — $5/GPU/hr

B300 (BF16) — $6/GPU/hr

9. Cost Comparison Summary

Total Training Cost

NVL72 Racks Required

10. Training Cost Visualisations

Figure 1: Training Cost by Model Size and GPU Configuration

Figure 2: NVL72 Racks Required

11. Synthetic Data Generation Cost

The 10T model requires 400T tokens but only ~100T are available from the internet. The remaining 300T must be generated synthetically. We analyze the cost using Claude Sonnet 4.6 as a representative frontier API model.

Sonnet 4.6 Pricing

Cost to Generate 300T Synthetic Tokens

Assuming a 1:10 input-to-output ratio (30T prompt tokens → 300T generated tokens):

Standard: (30T ÷ 1M × $3) + (300T ÷ 1M × $15) = $90M + $4,500M = $4.6B

Batch: (30T ÷ 1M × $1.50) + (300T ÷ 1M × $7.50) = $45M + $2,250M = $2.3B

(off-course in case of Anthropic - it would cost them much lesser than outsiders due to them not having to pay their own margins :) )

12. Total Cost: Training + Synthetic Data (10T class)

Figure 3: 10T class Cost Breakdown — Training vs Synthetic Data

Practical alternatives to save on synthetic data costs

· Self-hosted inference: Running a Llama-class (MiniMax etc would be good - as they have fewer active parameters and robust performance) model on owned GPUs costs ~$50-100M for 300T tokens — 20-40× cheaper than Sonnet 4.6 API.

· Multi-epoch training: Train on the same 100T tokens 4× with curriculum learning. Zero data generation cost, though diminishing returns after 2-3 epochs.

· Rephrasing/augmentation: Take 100T internet tokens and rephrase 3-4× using a cheap model. Much cheaper than generating from scratch.

· Hybrid approach: 100T internet + 50T high-quality synthetic + 3 epochs approximately 400T effective tokens at minimal generation cost.

13. What about post training?

We mostly focus on pre-training and synthetic data generation cost. Post training (SFT + RL) can cost additional 20% (Epoch AI’s analysis of DeepSeek R1). xAI have reported spent almost as much compute on post training for Grok 4 as on pre-training of the model.

Key Findings of this thought exercise

2T class is very achievable: 191 NVL72 racks, $148M at BF16. NeoLabs can still manage it. Data requirement (80T) is within internet availability.

4T class is feasible: 762 racks, $593M at BF16. Needs ~60T synthetic tokens to reach total of 160T tokens — manageable but starts getting very expensive.

10T class is within reach with hyper-scaler partners and truck load of cash: 4,763 racks, $3.7B at BF16. But 300T synthetic tokens add $2.3B (API) or ~$75M (self-hosted cheaper models).

Synthetic data can cost as much as training: At API pricing, 300T tokens via Sonnet 4.6 costs $2.3B — it is almost 60% of $3.7B training cost for 10T class model on B200 with BF16 precision. Self-hosted generation may become essential at this scale.

B300 only wins at FP4: At BF16, identical throughput to B200 but 20% more expensive. As I understand, Nvidia positions B300 for inference, rather than training.

Data is a major bottleneck: Not compute or GPUs alone— the 400T token requirement for 10T models is probably the hardest constraint to satisfy!!!!

Disclaimer

This is just a thought exercise to discuss order of magnitude compute needed for next generation of models. I have used all the public formulas and performance numbers. Thanks!