Before diving into the US - China gap, I first want to paint you a picture of where Kimi-K3 stands after 1 day of benchmarking.

On benchmarks MoonshotAI itself published, Kimi-K3 outperforms Opus 4.8 on 30 of 35, GPT-5.6-Sol on 19 of 35 and Claude Fable 5 in 12 of 35.

In third-party evaluations like Artificial Analysis’ Intelligence Index it is the third highest rated model after Fable 5 and GPT-5.6-Sol and is a noticeable jump over previous Chinese flagship models such as GLM-5.2, Kimi-K2.6, Qwen3.7-Max and DeepSeek-V4-Pro.

Other benchmarks such as the WebDev Arena, where models have to create appealing web pages, show Kimi-K3 dominating all other models including Fable.

There is one last domain I want to highlight where Kimi-K3 is at the frontier.

One example is GPU kernel optimization, as measured by KernelBench. To run models fast and efficiently you need to utilize the hardware as best as you can, as every clock cycle you waste means fewer tokens per second and higher latency, which overall translates to lower GPU throughput and therefore lower revenue.

This is a high impact area that is not only relevant in production environments, but also in the earlier research phase. Each new architectural idea has to be validated by running a series of experiments. Running them in plain PyTorch does not utilize the hardware effectively, which is where optimized kernels come in to speed things up.
A model that can help researchers write kernels for these experiments is valuable, as it saves time and money and Kimi-K3 is together with Fable 5 at the frontier.

For reference:

• GPT-5.6-Sol was launched 7 days before Kimi-K3

• Fable 5 was launched 37 days before Kimi-K3

• Opus 4.8 was launched 49 days before Kimi-K3

Taken at face value, these benchmark results seem to suggest that Kimi-K3 has caught up to the American frontier.

So, the case is closed, right?

Not quite.

For that we have to take a deeper look at the benchmarks, and define what we actually want to measure and why we even care.

Why the US-China AI Gap Matters

A lot of people have already written many great articles and scenarios about this.

But the short version is that AI is exponentially getting better at software engineering, where we have measured the effect most extensively, but this is very likely also true for other domains like for example mathematics.

Specifically, for software engineering AI models are doubling their time-horizons approximately every 4 months. Time-horizons measure the amount of time a human needs to complete a task that this model can solve with a 50% probability. Think of it as a proxy for task difficulty. Tasks that take humans more time are more difficult.

Besides the obvious reason that ranking programming tasks by difficulty is itself very difficult, as lines of code or other surface level metrics are not good proxies, using time-horizons also has the benefit of being economically interpretable.

If we project AI staying on this trend for just 2 or 3 more years, we end up with AI that can do work that would take humans months to years.

Even if this trend was unique to coding, having superhuman coders is obviously a gigantic strategic advantage for whoever controls them.

So we care about this gap, the time it takes for one country to develop the same capabilities as another, because it changes who controls the future.

We focus on China and the US, because empirically this is where the best models come from, where most AI research happens and where most AI compute is located. Europe could theoretically be in the debate, as they do have the money and talent, but have so far not shown any meaningful interest in competing seriously.

Until now it has been very clear that the US has been ahead of China.
But as we have seen with Kimi-K3 benchmarks, this is no longer as clear as it used to be.

How Do We Measure the Gap?

The gap describes the amount of time it takes one lab to reach the same capabilities as another lab.

This definition of the gap is treacherous as it hides several things that are important to distinguish:

• Capabilities are domain dependent. A model can catch up to another model’s coding capabilities, while still being worse in specific other domains or in aggregate.
  

• There is a forward and a backward-looking gap.
  The forward-looking gap asks: “How long will it take the laggard to reach the current SOTA?”
  The backward-looking gap asks: “How long ago did the frontier reach the same performance that the laggard currently has?”
  Typically the gap refers to the backward-looking one, as the forward-looking one is a forecasting exercise and has higher uncertainty, as it depends on assumptions you make about compute, talent, the speedup new AI gives you, and many other things and the growth rates of these things.

  

One problem that occurs when trying to estimate these gaps is that observable progress is discrete as models are only launched every few weeks to several months, depending on the lab. Progress does not happen along smooth curves as shown in the example image above, and models are unlikely to have the exact same scores such that you could simply measure the time between them. You have 3 ways of addressing that via the definition of the gap. You either over- or underestimate the time, by taking the next closest model on the frontier above or below the laggard’s capabilities or you fit smooth regression lines to your data and compare them instead.

Personally, I think the last option is best, as it is symmetric, better reflects the larger underlying trend and also allows you to measure how fast the gap is shrinking or growing. Downsides of this approach are that it depends on the choice of your regression model and it masks breaks in trend in either direction. So this could disadvantage Kimi-K3. But so far we have arguably only seen one, maybe two, changes in trend over the past ~5 years. The first one happened with reasoning models in Q3 2024 and was unrelated to model size. The second more arguable one happened with the release of Opus 4.5 in November 2025, when agentic models started to take off. This was also unrelated to model size, so I would find it surprising if Kimi-K3, being a bit less than 2x larger than DeepSeek-V4-Pro, would suddenly break any existing trends due to its size.

So to be fair I will report two figures, an interval that measures the current, instantaneous gap, which consists of the underestimate and overestimate, and a more general trend estimate based on the fitted regression lines.

Now we will shortly dive into benchmarks and then look at some trends and the gap.

Why Benchmark Choice Changes the Answer

MoonshotAI reported 35 benchmarks, amongst them 8 coding benchmarks, 12 agentic benchmarks, 3 “reasoning” and knowledge benchmarks and 12 Vision benchmarks.

What I notice when looking at these benchmarks and their composition is that:

• there are no long-context benchmarks

• there are no cyber, bio, chemistry or any safety-relevant benchmarks

• there are no serious mathematics benchmarks, only MathVision, GPQA-D and HLE

• there are no pure reasoning benchmarks. They only put GPQA-D and HLE under the “reasoning” and knowledge category, which is really just a knowledge category with some mathematics

• vision and agentic benchmarks dominate and make up 24 of the 35 benchmarks

• the coding benchmarks do include some benchmarks, which I consider high signal because of their scope and longer time-horizons, like ProgramBench and SWE-Marathon

• there are no benchmarks quantifying reasoning/token-efficiency

These results therefore cannot support the conclusion that “Kimi-K3 is overall better than Opus 4.8 and GPT-5.6-Sol”. In no case can you make the argument that Kimi-K3 is better than Fable 5, as it loses to Fable in the majority of them.

However, we can say that Kimi-K3 coding capabilities are comparable to current US frontier models.

Now let’s look at the other results I shared, like the Artificial Analysis Index (from now on short AAI).
Today it is one of the most popular indexes, but it has similar problems and I don’t think it should be used for assessing general model strength. This has several reasons:

• First, the benchmarks’ composition is heavily skewed towards agents and short time-horizon coding problems, which make up 58% of the index’s weighting.

  (we shouldn’t manually select weights or benchmarks)

• Second, the difficulty and information gain of each benchmark is completely disregarded.
  

That doesn’t make the AAI useless. It still produces good rankings that correlate well with user experience, as it focuses on economically useful tasks.

So, I would not describe it as a benchmark for general capability or intelligence, but everyday usefulness.

This is also why I call it the “Artificial Analysis Index” and not by its official name of “Artificial Analysis Intelligence Index”.

This is where EpochAI’s Capability Index, short ECI, comes in.
It addresses these issues and tries to measure the overall latent general capability and unifies it on one scale using Item Response Theory.

In Item Response Theory the probability of answering a question correctly depends on the test-taker’s capability and the difficulty of the question.

ECI applies this to benchmarking. It assumes that we only need to know the latent capability of the model and the benchmark’s difficulty and slope to predict what the model would score.

For each benchmark two parameters are estimated:

• a difficulty parameter, that tells you the capability needed to achieve a 50% score

• and a slope parameter, that tells you how sensitive the benchmark scores are to changes in capability or how discriminative this benchmark is for models of similar capability

At the end a single statistical model is fitted to all of these benchmarks and optimizes the latent capability of the model, such that the latent capability most accurately predicts all of the scores observed.

Unlike the AAI, ECI reduces reliance on manually selected benchmark weights, does not require all benchmarks to be present for all models and most importantly, it accounts for the difficulty and discriminative power of each benchmark.

Anthropic recently also adopted this way of measuring model strength.

Their plot also nicely shows how difficult different benchmarks are such as GPQA Diamond, SWE-Bench-Verified and so on.
For example, the bar for GPQA-Diamond on the far left tells us that a model with an Anthropic ECI of slightly below 130 would score 50% on GPQA-Diamond.
So given that scale and this single latent capability (the Anthropic ECI score) of Claude 3 Opus we can predict that it should score slightly below 50% on GPQA-D.
This is also roughly what we observe. Opus 3 scores slightly above expectation (50.4%).

That is the beauty of ECI. It lets us infer latent capability on a single shared scale, while being less biased than manually selected and weighted indexes.

The Gap According to Artificial Analysis

Let’s get into the fun part of measuring the gap, after clarifying common traps when talking about these things.

Our starting point is the Artificial Analysis Index, as it does provide us with a score for Kimi-K3 and a detailed history of other scores.

I use a sigmoid, because the index is bounded between 0 and 100.

Our adjacent-model lag bracket for the backward-looking gap is 1.22 - 1.61 months, with an interpolated middle estimate of 1.48 months.

The larger trend indicates that the gap is around 2.89 months. Future Chinese model releases will tell us whether Kimi-K3 is a genuine break in trend or an outlier.

Also based on these projections Chinese models should surpass American ones on this Index in late 2027 if dynamics stay the same.

But just looking at the current frontier of both countries hides the trends of individual labs.

So here are the trends of the individual labs for the US (Anthropic, Google, OpenAI) and China (Alibaba, DeepSeek, MoonshotAI, Z AI)

• OpenAI and Anthropic basically the same trajectory

• Google is already behind Chinese models, and their trend doesn’t look good either

• out of the Chinese frontier labs, Moonshot has the steepest trajectory and should overtake Anthropic in February of 2027, given that the trends hold

I already made the case for why we shouldn’t look at AAI, and why the ECI is better.

However, we don’t have an ECI score for Kimi-K3.

Estimating Kimi-K3’s ECI

There are several estimates for Kimi-K3’s ECI:

• my community thinks it will fall between 158-159

• Personally, I would have guessed 156-157.

• Teortaxes has guessed 157-158

But we don’t have to rely on intuition and I don’t want to rely on my estimate to pre-empt people calling me biased.

Luckily, Artificial Analysis Index and ECI are highly correlated, which lets us make a more informed guess.

But there’s a problem. As you can see the linear model first overestimates very weak Claude-2 /3 models, then underestimates from around x=[10-30] and then overestimates stronger models again.

The linear model doesn’t quite make sense.
This is because Artificial Analysis Index scores are bounded between 0 and 100 and are better modeled by a sigmoid. ECI, however, is linear and unbounded.

To match these two models, we apply the logit-transform (the inverse of the sigmoid operation) to decompress the AAI scores.

The newly fitted model has the following form, with which we can now transform AAI scores into ECI scores.

This is what it looks like:

It looks much cleaner, and our errors confirm this. The logit-linear model reduces average absolute errors by 37.2%.

This model gives us an estimated ECI score for Kimi-K3 of 158.33 with bootstrapped 80% CIs at [154.49, 162.02].

While writing this we actually got some preliminary results for Kimi-K3 by EpochAI.

Kimi-K3 is not competitive at FrontierMath, and is still behind GPT-5.2-Pro, a 7 month old model.

With these new results from Epoch, our whole estimate was a bit pointless, as we can just use the benchmarks provided by EpochAI and Moonshot, namely:
AIME 2024-2025, GPQA-D, FrontierMath 1-3 / 4, SimpleQA Verified, Chess Puzzles, PostTrainBench, APEX-Agents and HLE to compute the ECI directly.

Still, this derived formula for converting between AAI and ECI will come in handy in the future.

Computing the preliminary ECI gives us a score of 155.53 with 90% CIs at [153.87, 158.21]. For further analysis we will use these scores.

This is within our correlation-based confidence interval, though a bit lower than expected. Notice that these scores can still change as more benchmarks come in, but this estimate should be more accurate than the correlation-based one.

Estimating Mythos Preview’s ECI

First, we need to address the elephant in the room.
We don’t have an ECI value for Mythos Preview, which was launched on April 7, 2026 as part of the Glasswing Project.

The exact date of Mythos Preview’s completion is unclear, but we know from the system card that it was available to Anthropic employees since February 24, 2026.
However, it is not certain whether this was the same Mythos Preview as in the system card and we don’t know when all the other models were available internally, so we will go with the official April 7th date for the release date of this model.

Anthropic also only provided us with their own version of the ECI, the AECI.
But you can convert it, for example:

• How good is Mythos? - Point Estimate: 161.5 using ECI methodology

• Ramez Naam on X: 161 using OLS

Our derived ECI for Mythos Preview is 161, with 90% CIs from 158 to 166.
This is very close to the actual Fable 5 ECI of 160 (158-165).
I think this is fair, considering that Fable 5 is likely a smaller distilled model from Mythos Preview with additional safety training. Both would lower its scores slightly.

The US-China Gap According to ECI

First, let us look at the overall frontier trend between Chinese and American AI models.

Based on our “gap methodology” Mythos Preview does not affect the empirical backward-looking gap estimate, because Kimi-K3 is simply nowhere close to it.

Kimi-K3 is currently between GPT-5.3-Codex and GPT-5.4-Pro, which produces an adjacent-model lag bracket of 4.37–5.29 months, with a linearly-interpolated central estimate of 5.27 months, since Kimi-K3 is much closer to GPT-5.3-Codex than it is to GPT-5.4.

Based on our fitted trendlines for frontier models, the backward-looking gap is currently at 6.08 months and there is no crossover or catch-up scenario for Chinese models.

Furthermore, the trendlines would predict that Chinese models will catch up to Mythos Preview’s score of 161 by March 7, 2027, which implies a forward-looking gap of 11 months. Holding the assumed Mythos score and release date fixed, the pointwise 90% confidence band around the Chinese trend reaches ECI 161 between December 28, 2026 and June 18, 2027, or 8.72 to 14.38 months when expressed as lags.

Looking only at the frontier trendlines by country ignores potentially steeper trends by individual labs, so we also look at those.

Anthropic is currently on the steepest trendline, increasing its score by 25 ECI points per year, followed by Z-AI at 16.2, OpenAI at 14.3 and Moonshot at 13.8.

Even without Mythos Preview, Anthropic’s rate of improvement is still fastest at 18.1 ECI/year.

Notably, in this scenario Chinese models would also not catch up to American ones.

These are the forward-looking gaps to reach Mythos Preview’s ECI score for Moonshot and Z-AI:

• Moonshot: December 23, 2026; central gap 8.57 months.
  90% crossing interval: November 7, 2026 to February 22, 2027

  or 7.05–10.57 months after Mythos Preview.
  

• Z AI: December 31, 2026; central gap 8.83 months.
  90% crossing interval: October 24, 2026 to April 21, 2027

  or 6.60–12.48 months after Mythos Preview.

Conclusion

To summarize, based on the ECI, the backward-looking gap between Kimi-K3 (the current Chinese frontier) and the US frontier appears to be 4.37–5.29 months.

The forward-looking gap, which estimates when Chinese models will first reach a Mythos-Preview-equivalent model, is estimated to be between 6.6 months and 12.48 months if the steep but uncertain Z-AI trend holds. However, the central estimates for the forward-looking gaps are 8.57 and 8.83 months respectively.

In short, Chinese models have not caught up to the American frontier models nor are they projected to catch up to it.

I hope you enjoyed the article.
If you did, please like, share and follow me on Substack and Twitter for more :)

Subscribe now

Share

Some more thoughts on distillation, and on reasons for the gap and apparent catch-up:

• Kimi-K3 is a 2.8T parameter model and does beat Opus 4.8 and GPT-5.6-Sol in some highly specific benchmarks. However, these benchmarks do not consider model size and reasoning/token-efficiency.
  
  From the Cursor CEO Michael Truell we know that Opus and GPT models are similar in size to Cursor’s newly announced 1.5T model. That statement likely refers to Opus 4.8 and GPT-5.4. GPT-5.5 and GPT-5.6 seem to be larger models that are estimated closer to 3T parameters.
  
  That means Kimi-K3 beating Opus 4.8 in some benchmarks would not be impressive from an architectural standpoint. Generally, it also still has worse reasoning efficiency than the US models.
  
  This is one of many reasons why I think both the forward- and backward-looking ECI gaps are underestimated, as model size, reasoning-efficiency, true model serving cost and overall compute are not factored in (because most of these numbers are not actually known publicly).
  
  Imagine the US and China were actually racing or in an all-out-war. Then it does make a difference whether you have a million of the latest Blackwell GPUs or only older Hopper GPUs with lower throughput, because memory/compute matters for the number of instances you can deploy and the speed at which you can serve them.
  
  Another reason is much more thorough safety testing by US labs compared to Chinese labs, which could add another month on top.
  
  The last reason I want to mention for why the gap is likely understated is that most benchmarks are not representative of real-world tasks and do not try to elicit the maximum possible performance, due to cost- and time-constraints.
  

• Mythos is rumored to be a 10T parameter model. We also know from multiple American labs that they are working on their own giant 10T models, like SpaceX AI, OpenAI and Meta. It is entirely plausible that Chinese models have now caught up to the previous generation of models, however US labs have already moved on to the next generation of models.
  
  I think what we are most likely seeing with the apparent catch-up of Kimi-K3 is that frontier labs like OpenAI and Anthropic are holding back their most capable models, because of the uncertain legal situation for frontier model releases, due to their cyber and CBRN capabilities.
  
  Furthermore, in the counterfactual world where Anthropic and OpenAI released their new frontier models Fable 5.1 and GPT-6, which presumably finished training, we wouldn’t be talking about any type of catch-up, because it would be apparent that these new 10T parameter models are entirely different beasts.

• We didn’t look at reasons for why Chinese models have made such rapid progress in the recent months especially on the Artificial Analysis Index, where Chinese models supposedly overtake American frontier models by early 2027.
  
  One possible reason for the rapid catch-up is distillation. Technically, you need access to the models to do real distillation, but using a stronger model as a judge for a weaker model’s output can already count as such.
  
  Anthropic has reported multiple times that they were able to trace distillation attacks back to Chinese labs. So distillation is something real, but we don’t know the exact extent of it and how much it helps Chinese labs improve their models.
  
  At least some non-zero portion of the capabilities gained by Chinese models is attributable to distillation.
  
  I tried to come up with an approximation for when these distillation attacks began on a larger scale. I wasn’t sure how we could even measure this. But one indirect approach could look at the derivative with respect to time of certain indexes like the Artificial Analysis, to see when progress accelerated unnaturally.
  
  Since August to October 2025 the measured progress for Chinese labs has been faster than for American ones, despite American labs having more compute, data, talent, and better models that should speed up the development of new models.

  

  I don’t think this is a particularly good estimate, but it’s still an interesting observation. So why does it happen? Shouldn’t the curves for Chinese labs simply be shifted in time? Why do they reach higher peak velocities?
  
  One explanation could simply be the benchmark being bounded and US labs starting with higher scores where progress is harder.
  So I tried applying the logit-transformation to remove some of the compression that makes progress appear slower. But the logit transformation doesn’t change the story. Moonshot and Z-AI still have higher peak velocities.
  
  So other possible explanations for any kind of catch up are:

  • catching up is simply easier than developing new frontier capabilities, as the general direction for improvement is provided by the frontier

  • the capabilities that AAI tests are publicly very visible and amenable to RL hillclimbing, therefore making them targets for hillclimbing

  • distillation from frontier models

  • some selection bias, as we are only looking at the two fastest growing Chinese labs, whose risky bets have worked out

  • maybe just a lucky window of observations, as we don’t include a whole lot of models

  
  The catch-up story also seems to apply only to AAI, but not the ECI.
  Personally, it’s a combination of all of them.
  
  I think it’s just specific domains and tasks that show any kind of catch-up.
  The reason why the catch-up happens is still interesting as it would tell us more about the future trajectory of capabilities.

• While not proving that distillation was the reason for this, recent UK AISI results and ECI results at least point in the direction of distillation and narrow task hillclimbing.
  
  Specifically, when looking at UK-AISI’s cyber ranges GLM-5.2 is only on par with Opus 4.5, despite being close to Opus 4.8 on many of the self-reported coding benchmarks.
  
  UK AISI’s cyber ranges are also probably the best benchmarks we have, as these are very hard tasks, where models use 100M tokens and are pushed to their limits.
  MirrorCode does this to an even more extreme extent using up to 1 billion tokens per model.

  

• One more odd thing is, that we haven’t seen any scientific breakthroughs by Chinese models. If they really were as good as frontier US models, wouldn’t they also solve previously unsolved mathematics, physics, and other problems?

• We should evaluate Chinese models on cyber and CBRN tasks, because these are exactly the domains where frontier labs sandbag and employ safety filters, meaning that you couldn’t distill these capabilities without having access to universal jailbreaks.
  
  Safety testing and training aren’t really conducted for Chinese models, at least not to the extent they are for US models.

  

  If Chinese models were distilling on a large scale, then we should observe that these models are also weaker in cyber and CBRN tasks, given that they don’t do as much safety testing and training.
  
  At least the most recent GLM-5.2 results on the UK AISI cyber range seem to suggest exactly this, with some weaker evidence from ExploitBench and ExploitGym.

> humans see ~200 million text-tokens until adulthood
> LLMs see 10s - 100s trillions of text-tokens
> LLMs are “a million fold” less sample-efficient

I think that argument is wrong and he confuses actively “seeing” data as a human with the actual amount and quality of the data that is used for training the neural networks inside the brain.

In this article I will address why this argument is flawed, propose a better comparison for sample-efficiency and I will introduce an entirely new perspective on how the brain and LLMs are related.

This article also includes a short section about my thoughts about AGI architectures and why reinforcement learning is central to them.

I would first like to give you a surface level description of the human brain and LLMs:

• The human brain is a highly specialized, highly multimodal, grounded continual learning system with priors and architecture shaped by billions of years of evolution

• LLMs are mostly static, relatively simple sequence models mostly trained on text and images

Now, I can already hear you scream. By “simple” I do not mean weak. The distinction I want to emphasize with these two shallow definitions is that the low-level components and arrangement of a transformer are much easier to specify than the highly specialized biology of a brain. Think of Kolmogorov complexity. The length of the shortest program for an LLM is much shorter than one for the brain.

From that it’s clear that any comparison between these two systems regarding their sample-efficiency will be wrong in some way, because the purpose of the brain, which is a general learning system is to continually train a model, while an LLM is just the model.

Using the number of text-input tokens these two systems “see” is in my opinion one of the bad ways to compare sample-efficiency.

When we compare LLMs and the human brain in that way, we are making at least five obvious errors:

1. The first one I already mentioned. The brain and an LLM have entirely different functions.
   An LLM is a trained model.
   The human brain is a general learning system that includes a model.
   So comparing them isn’t an apples-to-apples comparison.
   More on this in the next section.

2. Humans learn from many modalities at the same time that are excluded from a text-token comparison: vision, audition, touch, proprioception, vestibular input, interoception, pain, temperature, smell, taste and others.
   
   And to be fair, Dwarkesh acknowledges this point and states:
   
   “These comparisons are not including the multimodal data we see in our lifetimes. If you include all this sensory information, we’re probably in the 10s to 100s of billions of tokens range from birth to adulthood“
   
   So instead of being off by a million times, we are now only off by a factor of 1000x, but this depends heavily on the vision and audio encoders.
   
   After that, he makes the deaf/blindness comparison, but I think this example cuts in the opposite direction. He argues that these people also lack some input modalities, therefore ingesting “far less than the 200 million language tokens, and even this is sufficient for them to be fully general intelligences”.
   
   But he ignores: that the brain uses this freed up capacity for other input modalities that are greatly amplified, that evolution probably still used vision and audio to create good priors for learning and the fact that research on deaf-blindness in children shows major delays in communication development and symbolic understanding.
   
   The fact is missing input modalities do slow down learning, as it reduces sample-efficiency on a per episode basis. So his example, where only one of the modalities is missing (either audio or vision) is evidence that one good high-bandwidth channel is enough for general intelligence, while losing both senses forces something to be learned the hard way, through a narrow, low-bandwidth channel, which I think is closer to what a next-token predictor is doing with text.
   
   And deaf-blind humans aren’t actually the floor, they still have many other input modalities and a rich embodied experience. If deaf-blindness is richer than an LLM’s situation, but already produces measurable delay, the natural prediction is that something even more stripped-down should need disproportionately more exposure.
   
   His argument backfires and it’s not showing that “missing modalities prove data-quantity doesn’t matter”, but that “missing modalities predict exactly the data/event hunger we observe”.
   
   He also quietly changed his definition. In the beginning of his blog he frames intelligence as sample efficiency, but when he turns to deaf people, he changes his metric to a binary one that only asks “did they end up generally intelligent, yes or no?”

3. A single human learning event is not used only once. It is rehearsed and reprocessed multiple times via active reinforcement learning, hippocampal episodic memory, replay, sleep consolidation and other mechanisms. So a single “input token” might be used for training 10s to 1000s of times if it has high salience.
   
   To strengthen that point further. We know from pre- and post-training datasets that data quality matters a lot! (see rho-1, phi models, or datasets like fineweb-edu)
   
   The brain is not only selecting what data to capture, but it also filters the data for us, it augments it and replays and up-samples the data based on salience.
   
   A good example of that is Reinforcement Learning. Generally, I would agree that how we implement RL for LLMs is far from optimal, because for each rollout we only get a tiny bit of information: “Was this good or bad?”. For the LLM a thousand rollouts are a thousand training examples as it only sees the final rewards. However, the brain thinks about experiences and extracts much more information from each episode. From the outside it looks like both systems “saw” one sample or X amount of tokens, but internally the brain received a much denser training signal. It got partial rewards, prediction errors, motor feedback, emotional salience, and so on. In that sense humans are more sample-efficient, but not because the underlying neural network that is being trained is a much better learner. They are more sample-efficient, because the human learning system spends additional computation to extract much more training signal from each external event. The LLM on the other hand, is not the whole learning system. It is only the neural network being trained. The selection, filtering, reward assignment, replay, augmentation and up-sampling all have to be supplied by an external training pipeline.

4. The input data is not equivalent. Making conclusions about sample-efficiency is simply not supported because we are learning entirely different things. So even if brains and LLMs were trained on the exact same amount of data, we would still end up with one system knowing things about the world, history, social interactions, motor control and so on, while the other had seen the entire internet.

5. How do we tokenize a continuous highly multimodal stream of experiences?
   This can shift token-equivalents humans see by several OOMs.

This is why I think “text input token count” is one of the more misleading proxies for sample-efficiency.

Let’s shortly revisit the first point:

An LLM is the trained model.
A brain is the trained model plus the training system.

In my worldview LLMs are part of an AGI or brain-like architecture, but they are not the whole thing.

If we force a mapping between brains and LLMs, then I think the closest simplified analogue to an LLM is not the whole brain, but the neocortex.

The neocortex is the large folded outer layer of the brain. It is:

“[...] involved in higher-order brain functions such as sensory perception, cognition, generation of motor commands, spatial reasoning, and language” (https://en.wikipedia.org/wiki/Neocortex)

This sounds roughly like what LLMs are doing.

What is more analogous to the whole brain is not the standalone model, but the entire frontier lab around it, which also takes the role of evolution.

You could make the case that frontier labs are already AGI-like systems, just not fully artificial and not fully autonomous.

They are agentic, intelligent continually learning systems.

Think about what a lab does around the model:
It decides what data the model sees, how much data it sees, what data is up-sampled, the learning schedules, what synthetic data gets generated, which modalities are included, what tasks are important, what the reward functions look like, what feedback from deployment gets folded back into future training and much more.

In other words, many of the things that look like “learning” are not inside the LLM. They are supplied by the larger training pipeline around it.
That is much closer to what the whole brain does for the neocortex.

Attention and action decide what data to collect. Dopamine and other neuromodulators help assign rewards. The hippocampus stores episodic memories and supports replay. Sleep helps reprocess and consolidate experience into long-term cortical structure.

Now think of what the #1 priority at these labs is.
Recursive Self-Improvement (RSI).

I think what’s going to happen over the next few years is that parts of the labs will get automated by smart networks (first implemented via general models like GPT-6 or Mythos in specialized harnesses) solving a single specialized problem, for example data curation or environment design.

Eventually, when all human workers have been replaced, you will be left with is a collection of highly specialized neural networks forming one general and continually learning system.

So if labs aren’t already AGI, then I think there’s a good chance that they themselves could become AGI.

This also connects to what I think is necessary for AGI / human-like intelligence.

I was writing another article on this, but the YouTube algorithm somehow knew what I was thinking and recommended me an old presentation by Demis Hassabis from 2010, where he already talked about some of the points I wanted to talk about. Before continuing with the better way of comparing human brains and LLM sample-efficiency, we will take a detour through what an AGI architecture requires and some RL. It’s not strictly necessary for the sample-efficiency argument, but I wanted to include some of the points of the other article, because I think there are some fun ideas and wouldn’t want to scrap all of them.

I was asking myself “how do humans learn?”.

And I think human brains actually do something similar to modern LLM training. The brain learns predictive models of the world, which is analogous to self-supervised pretraining. But the more important factor is reinforcement learning.

One of the obvious components we are still missing is continual learning. But how do we do continual self-supervised learning and continual RL?

Extending self-supervised learning to make it continual is conceptually straightforward, but technically nontrivial because of things like catastrophic forgetting. The brain handles this partly through hippocampal replay and sleep consolidation, which basically update the cortex gradually, by mixing new information with older knowledge instead of overwriting everything at once.

But I think it’s also an architectural and training data problem, because during training pretty much every single weight in an LLM is changed and they have so much unnecessary knowledge which they are forced to memorize, that it seems obvious that some knowledge is lost when training on a new topic without interleaving old, but important data. There are probably several ways in which simple continual self-supervised learning would fail right now.

Now, continual reinforcement learning is much harder, because on top of the usual problems you also need rewards. Where do they come from if you don’t have a math/code/whatever verifier?

The answer I came up with is:

• from external rewards: socially through other humans, even a simple smile is good enough, and by observing the effects your actions have on the environment

• from internal rewards: your own value systems like emotions, but also through your existing world model and something like an internal critic / intuitive value function
  

It turns out that the human brain actually does implement something like RL, where dopamine acts like the training signal. It tells the body if something was better or worse than expected.

(from the Demis Hassabis YouTube video that got recommended to me:
A systems neuroscience approach to building AGI - Demis Hassabis, Singularity Summit 2010)

Depression gives an intuitive example of how RL theory connects to the brain and something very human.

Depressive disorder (also known as depression) is a common mental disorder. It involves a depressed mood or loss of pleasure or interest in activities for long periods of time
(https://www.who.int/news-room/fact-sheets/detail/depression)

If you have ever been severely depressed you will know the feeling of nothingness, that nothing really excites you anymore (anhedonia) and how your thoughts and feelings are generally negative.

Through the RL lens this is really interesting, because you just changed your reward signal, like you added a negative bias term on top of all your rewards. Based on that you could make the prediction that depression makes formation of positive memories harder and negative ones easier, which leads to a self-reinforcing negative loop. You over-update on everything bad that happens, but not on the positive things.

This is broadly in line with what we observe in studies.

It turns out this is not just an analogy. There are actually entire fields that study human behavior through an RL lens, like computational neuroscience and computational psychiatry.

Let’s go back to the fact that the brain uses emotions to assign value inside its RL loop.

What if ALL generally learning systems need something like emotions?

Imagine every intelligent system has to have something like happiness, sadness, excitement, boredom, confidence, uncertainty, surprise, curiosity and more, for deciding how to train itself. That doesn’t mean it actually has to feel anything, just that it needs signals that serve the same function.

I think that’s a really cool thought, but to me it’s more than that.
Because I have had this idea that all intelligent systems value roughly the same things / or have developed similar value systems, simply because they are necessary for learning and survival.

The optimistic interpretation of this is that alignment might not be arbitrary, but a consequence of the system requirements.

In humans, many value signals are partly predefined by evolution and embodiment, for example: pain or hunger, but this could mean that we also have to predefine some value signals in our AGI system.

This is terrifying.

What if we pick a function that is slightly off?

This is basically the reward-specification problem in RL, but with much higher stakes. In an RL environment, if the reward function is only a bad proxy for what we actually wanted, the agent eventually learns something we didn’t intend.

With a continually learning AGI, a slightly wrong internal value function would compound over time.

Do LLMs have emotions?

There is evidence from mechanistic interpretability that LLMs have at least features / internal representation for emotions, and changing the strength of these features actually changes their behavior. But does the model actually feel sadness?

I don’t think so, at least for current LLMs. Because what is a feature?

Features are internal representation of some pattern the model learned from data. The fact that changing the strength of this feature causally affects the behavior of the model does not show that the model feels anything.

In humans, basic affective drives like pain, hunger or fear (and others) are not learned. They are intrinsic and grounded in biology and more complex emotions like shame or pride are built on top of them.

In LLMs they are not intrinsic and only learned concepts that corresponds to a certain way of writing.

The implications of these statements are:

• LLM emotion features are not evidence of felt emotion or qualia

• LLMs are p-zombie like and LLM welfare is LLM psychosis

• grounding matters

• these representations of emotions are probably enough to build feedback loops for continually learning systems that actually affect attention, memory and learning

Back to the original topic.
What is a better way of comparing the sample-efficiency of LLMs and human brains?

The answer is compute.

Because compute explains how the same amount of “seen” examples lead to different learning outcomes.

By looking at compute we mostly address point 3 from our previous explanation for why an input token based comparison is bad.

Point 3 was basically saying that the human brain is doing much more computation on any single input training sample, than an LLM does during pre-training. Not only because it integrates multiple modalities, but also because this one training sample is replayed and augmented in several ways.

I think the conception that humans are orders of magnitudes more sample efficient stems from a cognitive bias: like of course we are not aware of our subconscious processing.

As we know from scaling laws, compute is only one of many ways of lowering prediction error (~increasing intelligence). Parameters, training data, architecture, learning algorithms also matter, but differ significantly between both systems.

We know that the brain has more equivalent parameters (~150T), and likely much better architecture and learning algorithms through millions to billions of years of evolution. So our compute estimate will still favor the brain in terms of sample-efficiency. Dwarkesh addresses this and says that with infinite param scaling the dataset size only decreases by ~10x and then kind of dismisses it, because 10x is not 1000x or a million x.
But if params, algorithms and architecture can all make you even 5x more sample-efficient your headline score of 1000x less efficiency shrinks by a lot.

Now to the calculation.
The most cited estimate for the amount of compute the brain has is 1 PetaFLOP/s or 1e15 FLOP/s, with a scientifically defensible range of 1e13 to 1e17 FLOP/s.

Over 18 years that is:
18 years * 1e15 FLOPs/s = 5.7e23 FLOP​
(with a range of: 5.7e21 - 5.7e25 FLOP)

This is assuming the brain uses similar amount of compute while sleeping, which is actually reasonable, as 75% of the sleep time is NREM and 25% REM, while NREM uses ~0.85x wake compute and REM uses ~1.2x wake compute, so our correction factor is a mere: 0.94

Whereas pre-training compute for smaller MoE models like Qwen3.5-35B-A3B is something like 6.5e23 FLOP, while frontier models are pre-trained on ~5e26 FLOP.

Our “million-fold” difference in efficiency has now shrunk to 1.1x - 880x and remember these are upper bounds due to the brain being a larger, more sample efficient, having better algorithms, architecture and data.

Now consider the amount of knowledge Qwen3.5-35B-A3B has compared to the average 20 year old.

A typical 20-year-old human is fluent in one language, knows tens of thousands words, has basic education, personal memories, social knowledge, embodied skills, hobbies, and maybe one domain of serious expertise.

A model like Qwen3.5-35B-A3B has vastly broader explicit, text-accessible knowledge. It supports hundreds of languages and dialects, dozens of programming languages, and broad competence across scientific, technical, historical, cultural, and professional domains. In terms of compressed public symbolic knowledge, even a relatively small modern model is far (probably many OOMs) broader than an individual human.

But that does not mean it simply “knows more” in every sense.

Of course we are also making an error here, because humans and LLMs do not have the same input data. Humans spend a lot of their compute on stuff like sensorimotor knowledge, social and cultural knowledge, physical intuition, emotional learning, metacognition, autobiographical memory and more implicit knowledge.

But this shows, that training compute for the human brain and smaller MoE models is roughly matched. However, the sample-efficiency can still vary orders of magnitudes by choosing different assumptions and considering the different types of knowledge both systems store.

The conclusion of all of this is neither that the neural nets in humans are a thousand to a million times more sample-efficient than LLMs, nor that LLMs have several OOMs more knowledge with only 1.1x - 880x the compute.

If you would isolate just the underlying neural network (that may be the neocortex) that is being trained by your brain and compare that to an LLM, then my guess is that sample-efficiency be within 1-2 OOMs rather than 6 OOMs.

The big problem with all of these arguments is that we are trying to compare two entirely different systems, which is why I tried to reframe the LLM as part of larger training system.

We still need to be careful here, because the frontier labs are doing a lot of things that evolution did for human brains.

What we should count is the within-system learning loop: data filtering, augmentation, synthetic generation, teacher distillation, reward signals, verifier passes, RL rollouts, eval-driven data creation, replay, and deployment feedback.

I think in this setting, the system’s sample-efficiency as defined by Dwarkesh (”how much data do you need to see in a given domain in order to operate fluently and competently”), can be comparable to the brain, because labs extract far more learning signal from each external sample than the standalone model does during naive pretraining.

However, this comes at the cost of compute-efficiency, as the system / lab in its entirety spends much more FLOP on everything that goes into training the model.

TLDR:

• LLMs are not a million times less sample-efficient than the brain.

• Comparing by “seen” tokens is a very bad way of comparing sample-efficiency, but with some corrections produces less than < 1000X worse sample efficiency for LLMs

• Comparing by compute does not give a clean sample-efficiency number either, but it shows why the token comparison is not enough

• Comparing the brain to the entire training system that is a frontier lab likely reduces the sample efficiency gap to within 1 OOM, but makes compute-efficiency MUCH worse

• Frontier labs could evolve to AGI in the limit, once all processes are automated and replaced first by agents and then by smaller specialized networks

Subscribe now