But how good are these GGUFs, actually?

We don’t have a clear answer. This is a problem I’ve discussed many times before.

Metrics such as KLD and PPL are useful for measuring how much a quantized model deviates from the original model, but they are not anchored in the accuracy space. A KLD or PPL value may indicate that one quantization is worse than another, but it cannot tell you when the degradation becomes large enough that the model stops being useful.

By design, these metrics cannot answer that question.

The only reliable way around this is the expensive one: run real benchmarks and generate millions of tokens.

So that’s what I did.

For this article, I generated more than 150 million tokens, requiring roughly 8 days of continuous benchmarking on an NVIDIA RTX Pro 6000.

I evaluated 15 GGUF versions of Qwen3.8 27B, ranging from Q4 down to Q1, produced by several different providers.

The main goal was to measure two things: accuracy and token efficiency. In other words, how low can you go on a 12 GB, 16 GB, or 24 GB GPU before the accuracy loss becomes significant, or before the model starts generating substantially more tokens to solve the same tasks?

Acknowledgments

Verda provided the RTX Pro 6000s to run the experiments described in this article.

Verda is a full-stack AI cloud, built for high-performance inference, training, and agentic workloads, with data privacy and sustainability at its core.

You can check them out here. There is a $50 coupon that you can redeem in your Verda account, after provisioning it with $5, to try their GPUs.

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I don’t receive any form of compensation from Verda, or any usage information, related to this coupon.

List of the evaluated Qwen3.8 GGUF models

Note: The sizes listed below exclude the MTP and vision layers.

• unsloth/Qwen3.8-27B-GGUF

  • Qwen3.8-27B-UD-Q4_K_XL.gguf: 17.2 GB

  • Qwen3.8-27B-UD-Q3_K_XL.gguf: 12.8 GB

  • Qwen3.8-27B-UD-IQ3_XXS.gguf: 10.6 GB

  • Qwen3.8-27B-UD-Q2_K_XL.gguf: 9.5 GB

  • Qwen3.8-27B-UD-IQ2_XXS.gguf: 7.3 GB

  • Qwen3.8-27B-UD-IQ1_M.gguf: 6.7 GB

• huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF

  • Huihui-Qwen3.8-27B-abliterated-UD-Q4_K_XL.gguf: 17 GB

• AtomicChat/Qwen3.8-27B-GGUF

  • Qwen3.8-27B-AD-Q4_K_M.gguf: 16.8 GB

  • Qwen3.8-27B-AD-IQ3_S.gguf: 13.6 GB

  • Qwen3.8-27B-AD-IQ2_S.gguf: 10.8 GB

• bartowski/Qwen3.8-27B-GGUF

  • Qwen3.8-27B-IQ4_XS.gguf: 15.3 GB

  • Qwen3.8-27B-IQ3_XXS.gguf: 12.4 GB

• empero-ai/Qwen3.8-27B-Ridge-GGUF

  • Qwen3.8-27B-Ridge-3.7bpw.gguf: 12.3 GB

• ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

  • Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf: 10.1 GB

  • Qwen3.8-27B-GSQ-RCO-IQ2_S.gguf: 9.3 GB

Qwen3.8 27B GGUF Benchmark Methodology and Evaluation Setup

Each GGUF evaluation took roughly 10~15 hours on an NVIDIA RTX Pro 6000, so evaluating every available quantization was simply not practical.

All GGUFs were evaluated using the exact same set of 950 prompts, subsampled from MMLU-Pro, LiveCodeBench, and GPQA Diamond.

I used the inference hyperparameters recommended by Qwen rather than setting the temperature to 0. Greedy decoding would have made the results easier to compare, but it would also have produced significantly lower accuracy scores and substantially more generated tokens, increasing the overall evaluation cost. Note: That said, with a much larger budget, I think temperature = 0 is a very good setting for evaluating quantization damage. At temperature 0, even the BF16 model performs worse on these tasks, but it remains relatively stable and does not collapse. Poorly quantized models, on the other hand, may degrade much more severely under the same conditions. In that sense, greedy decoding can act as a harsher stress test, making quantization-induced degradation easier to expose.

The trade-off is that sampling introduces nondeterminism and fairly high score variance. To compensate for this, I ran each evaluation three times and averaged the results.

All GGUF models were served through a llama.cpp server. I set the maximum output length to 128K tokens and used low thinking mode. This still allows the model to generate sufficiently long reasoning traces to expose quantization damage, while keeping the total evaluation cost manageable.

How to read these results

The goal of this evaluation is not to produce a definitive ranking of GGUFs. Doing that properly across this many quantizations, with enough repetitions to confidently distinguish small differences, would easily cost $10,000+.

Instead, the goal is to identify which GGUFs show no clear signs of significant degradation compared with the original BF16 model.

For that purpose, I use a simple threshold: if a GGUF recovers more than 95% of the BF16 model’s accuracy, I consider it good enough.

In practice, a GGUF can occasionally score higher than the original BF16 model, but this should not be interpreted as the quantized model being better. It is simply a consequence of sampling and benchmark variance.

Below 95% accuracy recovery, I consider the degradation too large. At that point, you are probably better off using a smaller model at a healthier quantization level rather than forcing a heavily quantized 27B model to fit on your GPU.

There is also an important limitation to keep in mind.

All the benchmarks used here are single-turn, non-agentic tasks. Even though this evaluation is already expensive, it is still far too cheap to reliably tell us how well a given GGUF will perform on long-running agentic workloads.

For tasks such as agentic coding, even a seemingly small 1% accuracy loss can compound across many successive steps and tool calls. A model that looks nearly identical to BF16 on a single-turn benchmark may behave quite differently over a long agent trajectory.

If agentic workloads are your main use case, I would interpret these results asymmetrically: models near or below the 95% threshold are strong candidates to avoid, while models clearly above it should only be considered potentially good enough. How much quantization you can tolerate will ultimately depend on the length and complexity of your agentic tasks.

Note: I’m running more evaluations for UD-Q2_K_XL and UD-Q4_K_XL for agentic coding. It will take at least a week.

Qwen3.8 27B GGUFs: Token Efficiency and Accuracy

Below, I first present accuracy and then token efficiency before discussing the results together, because, as we will see, token efficiency is strongly correlated with accuracy degradation.

Accuracy

Token Efficiency

Read more

Hi everyone,

In this edition of The Weekly Kaitchup, I mainly discuss the evaluations I’m currently running and what I plan to publish next. I also share a few thoughts on the new GLM-5.3 Flash.

Qwen3.8 27B for Agentic Coding: Evaluations Are Still Running

I’m spending a lot of compute testing different configurations, and I’m starting to see some very interesting results. Note: Thanks to Verda for the compute sponsorship.

I’ll compile everything into a dedicated article, but since it will take some time before it’s ready, I thought I’d give you an early preview.

preserve_thinking is essential

Qwen puts it this way in the model card:

By default, Qwen3.8 retains thinking blocks from all historical messages, maintaining a complete reasoning trace across the conversation. This behavior, known as preserved thinking, ensures full context continuity and is especially beneficial for agent scenarios where decision consistency and reduced redundant reasoning are critical. It also improves KV cache utilization, optimizing inference efficiency in both thinking and non-thinking modes.

I can confirm this, but I’d go even further: if you disable preserve_thinking, your agent will generate many more tokens and fail much more often.

It’s not just beneficial. It’s necessary.

For example, if you disable preserve_thinking with xhigh, you get medium-level performance (or less) while consuming significantly more tokens.

Also, double-check your agent setup. Claude Code wasn’t preserving thinking traces in my initial runs, and it took quite a few retries and Codex’s help before I got the configuration right. I haven’t had issues with mini-swe-agent or Pi.

Pi Outperforms Qwen’s Published Score

I made small modifications to Pi to better handle NonZero errors. With that change, my xhigh runs on DeepSWE 1.1 reached a score of 46, four points higher than the result reported by Qwen with Claude Code.

I’ll publish the trajectories along with my modifications when I release the full article.

Maybe I should try OMP.

What’s Still Running?

I’m making a few final attempts to get a strong Claude Code run. So far, I haven’t managed to reproduce Qwen’s results.

My best DeepSWE 1.1 score with Claude Code is currently 35.2, about seven points below the score Qwen reported with their Claude Code setup.

Claude Code was dropping the reasoning traces between turns, but I’ve now fixed the preserve_thinking issue. Hopefully, that will help close the accuracy gap.

I’m also running a few baselines using my best Pi configuration:

• Qwen3.6 27B with thinking enabled and preserve_thinking on

• Qwen3.8 27B with low thinking

• Qwen3.8 27B with thinking turned off

These runs will probably be finished by Monday, and I expect to publish the article later in the week.

Muse Glimmer Quantization Analysis

All evaluations of the quantized Muse Glimmer models are now complete.

What remains is writing the analysis, but there’s so much going on right now that it’s been difficult to find the time. Still, I can already share a few results:

• Cyankiwi’s 4-bit AWQ quantization provides the best accuracy preservation.

• On Blackwell, NVFP4 is probably the better choice, especially for high-concurrency workloads.

• My mixed-precision INT version also performs well, but it slightly, and consistently, underperforms those two. That’s an acceptable cost to save 4 more GBs.

And since Muse Glimmer is already kind of forgotten, I don’t yet have a clear schedule for when I’ll publish the full analysis.

Meta was also planning to release the weights for Muse Spark 1.2, but I think it has missed its window. Several newer models now look considerably stronger. Hopefully, Meta will skip directly to a newer version rather than releasing the current 1.2 weights.

Ornith-1.5 Quantization

I also created a mixed-precision quantization of Ornith-1.5 9B.

• Quantized Ornith-1.5 9B

The quantized Ornith-1.5 9B models are compatible with vLLM 0.26+ and should be particularly well suited to high-concurrency workloads.

With the KV cache quantized to FP8, the model should run comfortably on a 12 GB GPU. Without KV-cache quantization, you’ll want 16 GB to use the full context window.

If there’s enough demand, I’ll also quantize the 35B version.

Qwen3.8 27B GGUF Evaluations

I’ve now evaluated around a dozen Qwen3.8 27B GGUFs from several providers. The full evaluation has taken more than a week on a single RTX Pro 6000, with each individual GGUF typically requiring 10–15 hours. The very low-bit, nearly broken quantizations can take even longer.

The main conclusion so far is fairly clear:

Below roughly 9–9.5 GB, you’re probably giving up too much accuracy.

As usual, the quantizations from Unsloth, LM Studio, AtomicChat, llama.cpp, and bartowski are very good. Down to UD-Q2_K_XL at 9.83 GB, they retain more than 95% of BF16 accuracy in my tests, including at context lengths of up to 128K tokens.

Below that, however, things deteriorate quickly.

UD-IQ1_M, for instance, loses more than half of the BF16 accuracy on some coding evaluations. It’s impressive that a model compressed this aggressively still works at all, but that doesn’t necessarily make it useful.

On LiveCodeBench, UD-IQ1_M performs slightly worse than a 4-bit Qwen3 4B, the first-generation Qwen3 model, while also requiring more memory and generating more tokens. On GPQA Diamond, its performance is not far from random.

So, in practical terms: I wouldn’t recommend Q1 quantizations of Qwen3.8 27B.

If your memory budget forces you below roughly 9.5 GB, I think it makes more sense to use a smaller model at a higher precision, for example, a Q4 Qwen3.5 9B, or Ornith-1.5 9B for agentic workloads.

UD-Q2_K_L looks more promising, although there’s an important limitation to my current results: I’ve only tested single-turn tasks, with context lengths up to 128K tokens.

For long-horizon agentic workloads, the picture could be different. Quantization errors can accumulate across many turns, tool calls, and reasoning steps, so a quantization that looks good on single-turn benchmarks may degrade more noticeably during an extended coding trajectory.

I’d like to test this properly with long-horizon agentic coding tasks.

For now, I expect the main Qwen3.8 27B GGUF evaluations to be completed this weekend. I’ll publish a dedicated article with the full accuracy and token-efficiency results, and I’m also evaluating several abliterated versions.

GLM-5.3-Flash: Smaller, but Also with a New Efficient Architecture

When the first rumors about GLM-5.3-Flash appeared, I assumed we were getting another model in the same category as GLM-4.7-Flash.

GLM-4.7-Flash was a lovely 30B-A3B MoE: 30 billion parameters in total, with only around 3 billion activated per token. It was large enough to be capable, yet still small enough to be genuinely interesting for local inference. But it was quickly overshadowed. Gemma 4 and the Qwen3.5 MoEs arrived soon after and appeared significantly stronger, pushing GLM-4.7-Flash out of the spotlight almost immediately.

GLM-5.3-Flash is very, very far from that size.

It has 320 billion parameters, with 18 billion active.

At this point, “Flash” describes the inference economics rather than the size of the model.

GLM-5.3-Flash is a 45-layer Mixture-of-Experts model with 288 routed experts, eight of which are selected per token, plus a shared expert.

More interestingly, Z.ai has significantly reworked the attention architecture. GLM-5 already replaced conventional full attention with DeepSeek Sparse Attention, but GLM-5.3-Flash goes a step further: for the first time in the GLM family, it combines linear attention with periodic sparse-attention layers. Linear attention handles cheaper local and state-based processing, while sparse attention is used periodically to retrieve relevant information from the full context. The result is much lower compute and KV-cache requirements at very long context lengths.

The model also introduces Manifold-Constrained Hyper-Connections (mHC) to the GLM architecture, replacing conventional residual connections with a more flexible routing mechanism designed to improve training stability and scaling efficiency.

There is even an additional optimization called IndexPool, which compresses the index used by the sparse-attention layers. Z.ai says that compared with the full GLM-5.3, the Flash architecture reduces attention compute by about 3× and average KV-cache size by about 4.4×.

Another major change: this is the first natively multimodal model in the GLM-5 family. Text and vision are part of the model from pre-training rather than vision being bolted on afterwards. Z.ai says the new base model was trained on a 30-trillion-token multimodal corpus.

And the performance looks extremely strong.

Z.ai also anonymously deployed the model before release under the name ox-alpha on OpenRouter and OpenCode. According to Z.ai, it became the most popular model of the week before anyone knew who had made it.

But the most interesting detail might be this one:

Z.ai says all of that traffic was served using Chinese AI chips.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisioning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

Only days after Qwen3.8 27B demonstrated just how much performance Qwen can now squeeze into a relatively conventional dense local model, Qwen is already showing us what Qwen4 will look like.

Qwen3.8-Flash-Next is a multimodal Mixture-of-Experts model with a 125B-parameter main model, only 6B parameters activated per token, another 51B parameters dedicated to a new n-gram embedding system, and a 4B MTP module for speculative decoding.

• Qwen/Qwen3.8-Flash-Next

But the most interesting detail is in its name.

Flash-Next is not just another member of the Qwen3.8 family. Qwen explicitly describes it as an experimental preview of the architecture that will underpin Qwen4. In fact, its configuration file in Transformers already identifies the architecture as qwen4_exp.

This is similar to what Qwen did with Qwen3-Next before Qwen3.5. Qwen3-Next introduced the hybrid Gated DeltaNet and Gated Attention architecture that was then reused throughout Qwen3.5, Qwen3.6, Qwen3.7, and Qwen3.8.

Now Qwen is doing it again.

In this article, I’ll first look at how Qwen3.8-Flash-Next compares with Qwen3.8 27B. Then I’ll dig into the new architecture, especially Qwen Sparse Attention, Gated Residual, and the unusual 51B-parameter n-gram embedding table.

Finally, I’ll estimate its actual memory consumption at 256K context, look at NVFP4 and Unsloth’s GGUF quantizations, and see what kind of hardware may realistically run it.

How good is Qwen3.8-Flash-Next?

Read more

In a previous article, I benchmarked Qwen3.8 27B using its highest reasoning effort, xhigh. That shows the model at its best, but it does not tell us whether xhigh is the most efficient way to run it. Qwen3.8 27B also supports medium and low reasoning efforts, while thinking can be disabled completely.

In this article, I compare these four configurations to see how reasoning effort affects both accuracy and token usage. I ran a large number of tasks and generated hundreds of millions of tokens. For each mode, I measure how well Qwen3.8 performs and how many tokens it uses to get there.

How much accuracy do we gain by letting Qwen3.8 think more, and how many extra tokens does that cost?

xhigh should give the best results, but it may also generate much longer reasoning traces. If medium or low can retain most of the accuracy while using significantly fewer tokens, they may be better choices in practice.

This is especially relevant for local inference. More reasoning tokens mean more generation time, more compute, and higher memory consumption. A small improvement in accuracy may not be worth it if the model requires two or three times as many tokens to achieve it.

I will first explain how Qwen3.8 reasoning effort works, how low, medium, and xhigh are implemented in the chat template, and how to set them correctly with vLLM and llama.cpp. I will then compare their accuracy and token usage across several benchmarks, before briefly putting the results in context against Qwen3.6 and Muse Glimmer.

Acknowledgments

Verda provided the RTX Pro 6000s to run the experiments described in this article.

Verda is a full-stack AI cloud, built for high-performance inference, training, and agentic workloads, with data privacy and sustainability at its core.

You can check them out here. There is a $50 coupon that you can redeem in your Verda account, after provisioning it with $5, to try their GPUs.

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I don’t receive any form of compensation from Verda, or any usage information, related to this coupon.

Note: These experiments focus on non-agentic tasks, where each prompt is evaluated as a standalone problem. I am testing long-horizon agentic coding separately, and the runs are almost complete, but there is still a large amount of data to analyze and verify. I plan to cover those results in a separate article, probably next week.

How Qwen3.8 Reasoning Effort Works

Qwen3.8 supports three reasoning_effort values: low, medium, and xhigh. Thinking can also be disabled with enable_thinking=false. The goal of these settings is to control how much reasoning the model does before producing its final answer. low asks the model to keep its reasoning short and focused, while xhigh encourages more careful and thorough reasoning. medium sits between the two.

Read more

Hi everyone,

Last week was mostly about new models. This week is a little different.

The three releases I found most interesting are more about getting substantially more out of existing models through better post-training and quantization:

• Ornith-1.5: a new generation of agentic models trained with a more complete self-improvement loop.

• LFM2.5 QAD: Liquid AI is using Quantization-Aware Distillation to make tiny Q4_0 GGUFs much more accurate.

• Escha-W2 for Qwen3.8 27B: a 10.15 GB version of Qwen3.8 27B that, according to Escha Labs, retains all of the FP8 model’s benchmark performance.

There is a common theme here.

We are getting increasingly sophisticated techniques for improving models after pre-training. Better RL can turn a relatively small base model into a much stronger agent. Training can compensate for quantization. And increasingly aggressive low-bit formats are starting to preserve much more accuracy than I would have expected a year ago.

Coming Soon in The Kaitchup

I’m nearly done with my article on inference speed and latency, covering Muse Glimmer, Qwen3.8 27B, and Dflash-2. It’s a direct follow-up to the article I published yesterday:

I also now have all the results I need to compare the accuracy and token efficiency of Qwen3.8 27B across its off, low, medium, and xhigh thinking modes. One thing I can already say: xhigh is much better.

As for my agentic coding experiments with Qwen3.8, comparing different agent harnesses across different thinking modes, I’m about halfway through.

Ornith-1.5: Self-Scaffolding Becomes Self-Improvement

I discussed Ornith-1.0 a few weeks ago.

For agentic coding, the model operates inside a scaffold: the prompts, tools, memory strategy, retry logic, and general orchestration used to solve a task. Normally, humans design this scaffold and then keep it fixed during reinforcement learning.

Ornith-1.0 instead allowed the model to generate a task-specific scaffold and rewarded both the scaffold and the resulting solution. And since it works very well, Ornith-1.5 pushes this idea further.

The model no longer only creates the scaffold. It can now generate:

• the task it will learn from,

• a scaffold for solving that task,

• and the solution rollout itself.

This effectively closes the loop.

Instead of humans providing a fixed collection of coding tasks and letting the model optimize how it solves them, the model can propose new tasks that are useful for training, build a strategy to solve them, execute that strategy, and then learn from the resulting reward.

I find this much more interesting than the benchmark tables.

Generating synthetic training tasks is obviously not new. What is interesting here is combining task generation with scaffold generation and reinforcement learning in one loop.

In principle, this allows the training distribution itself to evolve as the model improves.

If the model becomes good at one type of task, it can generate increasingly difficult variants rather than continuing to spend compute solving things it already knows how to do. What I’m curious about is the training cost. Because I assume finding new difficult tasks for the model that actually improve the reward is not that easy and could take many training steps

Three Models: 9B, 35B-A3B, and 397B

The Ornith-1.5 family currently contains:

• a 9B dense model

• a 35B-A3B MoE

• a 397B MoE

Note: Did you notice? No mention of an “Ornith 1.5 31B”. The release of Ornith 1.0 was supposed to include a model based on Gemma 4 31B. This model was never published. My assumption: it’s extremely difficult to fine-tune Gemma 4 for agentic coding, and the Ornith 1.0 31B was never good enough. We need a Gemma 4.5 that's good at agentic tasks.

The 9B is particularly impressive:

I’ll publish quantized versions next week!

It’s safe to assume they are also preparing a version, Ornith 2.0?, based on Qwen3.8.

LFM2.5 QAD

Liquid AI released new Q4_0 GGUFs for:

• LFM2.5-230M

• LFM2.5-350M

• LFM2.5-1.2B-Instruct

• LFM2.5-2.6B

They were trained with Quantization-Aware Distillation, or QAD.

I briefly discussed QAD earlier this year when NVIDIA used it for Nemotron 3 Nano.

QAD

With normal post-training quantization, we train the model at high precision and compress it afterward. If quantization changes the model’s outputs, we simply accept that loss.

QAD adds another training stage. You keep the high-precision model as the teacher and train the quantized model to reproduce its output distribution.

The loss can be viewed as:

KL(teacher || quantized student)

So instead of asking the quantized model to relearn the original tasks, we ask it to behave as similarly as possible to the unquantized model.

It only requires the original model, the quantized student, and suitable data for distillation.

Quantization is generally less forgiving as models get smaller.

A 30B model often has enough redundancy that reducing weight precision to four bits causes little damage. With models containing only a few hundred million parameters, every parameter is doing more work.

Liquid reports that the new QAD Q4_0 checkpoints retain approximately:

• 97.1% of BF16 performance for the 230M

• 96.5% for the 350M

• 97.4% for the 1.2B

• 96.6% for the 2.6B

They significantly improve over Liquid’s previous PTQ Q4_0 checkpoints that I evaluated a few weeks ago:

I ran the same evaluation settings on these QAD versions and independently confirmed LiquidAI’s conclusions: the new QAD Q4_0 versions are significantly better.

But why focus on Q4_0?

There are now many sophisticated GGUF quantization formats. Q4_0 is not one of them. It is old, simple, and widely supported. But that simplicity is also an advantage.

Hardware vendors and inference runtimes already know how to execute Q4_0 efficiently, including on CPUs and Arm devices.

The QAD versions match the quality of larger quantizations while decoding faster.

For the 230M and 350M models, the QAD Q4_0 checkpoints are between 4% and 33% faster, depending on the hardware.

Escha-W2: Qwen3.8 27B in 10.15 GB

Two weeks ago, I wrote about Escha Labs’ 2-bit version of Qwen3.6-35B-A3B.

Now, Escha Labs (about which we don’t know much) has done the same thing with Qwen3.8 27B.

Escha reports:

• 10.15 GB for the entire model on disk

• 82.6 tokens/s single-stream generation on an RTX 5090

• approximately 100% of FP8 benchmark performance averaged over the eight evaluations they have completed so far

Like their Qwen3.6 release, the model runs through a custom SGLang-based runtime.

They also say the model and runtime are Apache-2.0 licensed and that GGUF versions are coming.

10.15 GB for a 27B dense model is extremely small.

Almost No Benchmark Loss?

This is the part I want to test carefully. Escha says the new W2 checkpoint averages around 100% of the FP8 model across the eight benchmarks they have evaluated.

There is even a surprising LiveCodeBench v6 result:

• Escha W2: 86.81

• FP8: 85.16

Is the low-bit model better than FP8? Obviously not. The problem is that their evaluation was too cheap: a 28K max output-token limit, and only ~17% of the benchmark was actually run. Note: and most of the other benchmarks they used don’t even generate any tokens and rely on the model’s logits.

For Qwen3.8, evaluating quantization properly requires the full 262K-token context, because the model uses that context for reasoning. Quantization error compounds at every decoding step, so seeing little or no degradation early in the generation is encouraging, but it’s not enough to draw a conclusion.

LiveCodeBench and the other benchmarks they used, like GPQA, also have high variance. I run them four times before publishing my numbers. Yes, that’s not cheap, but it’s necessary.

My own evaluation of Escha-W2 is in the queue. I expect it to be significantly better than the other 2–3-bit versions of Qwen3.8, but certainly not as good as FP8. Hopefully, I’m wrong.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisioning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

The difference in accuracy between the two models is striking, especially given their similar parameter counts and dense architectures. Muse Glimmer struggles in particular with long-horizon agentic coding, while Qwen3.8 27B performs remarkably well, even across different agents and with thinking effort set to medium.

Still, as we saw in a previous article, Muse Glimmer may retain a few important advantages:

• It has a relatively short maximum context length. That is a limitation in itself, but it also suggests that the model was not trained to produce the extremely long reasoning traces Qwen3.8 can generate.

• Its KV-cache memory consumption is very low: it uses roughly 4 times less memory than Qwen3.8.

Taken together, these characteristics make Muse Glimmer considerably more memory-efficient than Qwen3.8 27B, despite being slightly larger.

In this article, I first present my own accuracy results for Muse Glimmer and Qwen3.8 27B across a broad range of tasks, with both models evaluated at xhigh thinking effort. The benchmarks cover coding, world knowledge, difficult mathematics, instruction following, and more.

I then look beyond raw accuracy to examine token efficiency: how many tokens each model typically needs to solve a problem, and how much memory each consumes once the length of its reasoning traces is taken into account. This gives a more complete picture of the trade-off between the two models than parameter count or benchmark accuracy alone.

Acknowledgments

Verda provided the RTX Pro 6000s to run the experiments described in this article.

Verda is a full-stack AI cloud, built for high-performance inference, training, and agentic workloads, with data privacy and sustainability at its core.

You can check them out here. There is a $50 coupon that you can redeem in your Verda account, after provisioning it with $5, to try their GPUs.

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I don’t receive any form of compensation from Verda, or any usage information, related to this coupon.

Note: For this article, I evaluated Qwen3.8 only in xhigh thinking mode. A follow-up will compare its low, medium, xhigh, and disabled thinking modes. I also plan to publish a separate article focused entirely on agentic coding with Qwen3.8, as there is a great deal to explore there. Results for quantized versions of both Muse Glimmer and Qwen3.8 will come later, as will a comparison of their inference speeds.

Accuracy: Qwen3.8 Comes Out on Top, While Muse Glimmer Behaves Very Differently

My results confirm that Qwen3.8 27B at xhigh thinking effort is a substantial improvement over Qwen3.6 27B:

Read more

Hi everyone,

This week was unusually rich in open-weight releases.

Meta released Muse Glimmer. Qwen finally released the weights of Qwen3.8 2.4T. NVIDIA released Nemotron 3.5 Lightning. DeepSeek just pushed a major V4 Pro update.

And, just as I was finishing this article, Qwen3.8 27B was released. It was worth the wait. According to Qwen’s evaluations, it improves very significantly over Qwen3.6 27B essentially everywhere, with particularly spectacular gains on long-horizon agentic coding: DeepSWE jumps from 13.3 to 42.2, while QwenSWEBench goes from 49.3 to 79.0. The 27B model also beats Qwen3.7-Plus on many coding and agentic benchmarks, and even beats Opus4.6 Max on QwenSWEBench, CoWorkBench, and LiveCodeBench. At this point, Qwen seems so far ahead that catching up is becoming difficult for everyone else.

There are also a few interesting changes compared with Qwen3.6. preserve_thinking, which Qwen3.6 introduced as an option for keeping reasoning traces across turns, is now enabled by default. Qwen3.8 also introduces reasoning_effort, with low, medium, and xhigh levels to trade reasoning depth against cost. Qwen now recommends temperature=1.0 and top_p=0.95 for thinking mode generally, and, for very long agentic runs, recommends allowing up to 262K tokens for reasoning and 131K for the final answer. I’ll publish a full analysis of Qwen3.8 27B within the next few days, with a separate look at its quantized versions later.

Note: I think this “preserve_thinking” (which is a feature we can find in other models) should be carefully evaluated, especially its impact on inference cost. Qwen3.8 is a crazy thinker (they recommend allowing “up to 262K tokens for reasoning”). So if you preserve reasoning for each turn, the context may grow to millions of tokens. In practice though, for tool calls, reasoning is often much shorter and more/better reasoning may yield fewer turns, so I guess it can work. But I’m really interested to know how important it is, in terms of accuracy/efficiency, to preserve the reasoning traces. KV cache quantization will likely be very important.

Finding enough time to cover all of this properly is hard. I don’t want to publish articles that simply repeat benchmark tables and model cards. A proper analysis means testing accuracy, token efficiency, memory, speed, and then doing it again for several quantized versions.

For Glimmer, I already published an article covering the model and its architecture:

• Muse Glimmer: Meta’s 30B Model Built for Efficient Inference

I also already have speed measurements for all the non-GGUF quantized versions I’m testing.

At the moment, essentially all my compute capacity, several RTX Pro 6000 GPUs provided by Verda,1 is being used to evaluate the quantized versions that run through vLLM: NVFP4, INT4, and my own mixed-precision versions.

You can already find those here:

• My quantized Muse Glimmer collection (they all run, but consider all of them very bad until I publish some official evaluations)

I’ll probably publish a short article focused only on speed first, since those results are already ready.

Then, expect perhaps two deeper articles comparing Glimmer with Gemma 4 and Qwen3.6/3.8, and another article focused specifically on the quantized versions. I may include GGUF versions in that one as well, but I’m not sure yet. There is only so much GPU time, and writing time, in a week.

So, for this Weekly Kaitchup, I’ll focus on the three releases that I won’t cover in more detail:

• Qwen3.8 2.4T: we finally know what is inside it, so I can check how well my predictions held up.

• Nemotron 3.5 Lightning: essentially the new Nano-sized Nemotron.

• DeepSeek V4 Pro 0813: a very large update that makes the word “Preview” much more meaningful.

Qwen3.8 2.4T: How Good Were My Predictions?

A few weeks ago, before Alibaba disclosed the architecture, I tried to estimate what hardware would be required to run Qwen3.8 2.4T:

• Qwen3.8: What Hardware Will You Need to Run Alibaba’s 2.4T Model?

At the time, we knew only one particularly important number: 2.4 trillion parameters.

Everything else had to be inferred.

Now we have the weights and architecture, so it’s a good opportunity to see what I got right and what should be corrected.

Qwen3.8-2.4T-A95B has 2.4T total parameters and 95B activated parameters. It has 92 layers and uses the hybrid architecture introduced with Qwen3.5, alternating Gated DeltaNet blocks with periodic full attention. Its MoE has 512 experts, with 10 routed experts plus one shared expert activated per token. Native context is 262K, extendable to just over one million tokens.

The downloadable 2.4T checkpoint is also more limited than the hosted Qwen3.8-Max: it is text-only and requires thinking mode, while the hosted Max version adds vision, non-thinking mode, built-in tools, and a 1M default context.

So, how did my predictions go?

✅ It is an extremely sparse MoE

This one was the most important and easiest assumption.

I wrote that a dense 2.4T model would make very little sense for inference and that Qwen3.8 was almost certainly going to be a highly sparse MoE.

That was correct.

Only 95B of 2.4T parameters are active for each token, or roughly 4% of the entire model.

✅ Nearly 99% of the parameters are in the routed experts

I assumed that approximately 99% of Qwen3.8’s parameters would live inside routed experts, based largely on Kimi K2 and other very large MoEs.

This turned out to be close.

Using the released dimensions, 512 experts, 92 MoE layers, hidden size 8192, and expert intermediate size 2048, the routed expert matrices account for roughly 2.371 trillion parameters, or about 98.8% of the entire 2.4T model.

My estimate in the original article was 2.374T routed-expert parameters.

✅ The routing sparsity

I used Kimi K2 as a proxy.

Kimi routes each token through 8 of 384 experts, or about 2.08% of its routed experts.

Qwen3.8 uses 10 of 512, or about 1.95%.

So, while the architectures are different, the routing sparsity I used for the estimate was close enough.

❌ I estimated ~75B active parameters

This is the main miss.

Using Kimi K2 as a proxy, I estimated that Qwen3.8 might require compute roughly comparable to a 75B dense model per token, before routing and communication overhead.

The actual number is 95B activated parameters.

That is around 27% higher than my estimate.

So, the general prediction, a 2.4T model with only a small fraction active, was correct, but Qwen3.8 is somewhat more computationally expensive per token than I expected.

❌ Around 1.4–1.5 TB with experts-only NVFP4

Wrong but close.

I estimated 1.387 TB for a version where the routed experts are stored in NVFP4 while the rest of the network remains at higher precision.

A community NVFP4 checkpoint from RadixArk now does exactly that: only the routed-expert linear layers are quantized to NVFP4 while the other components retain their original precision.

Its repository is 1.48 TB.

So my estimate was around 6–7% too low, but the overall hardware conclusion was correct.

A single 8xB300 node is enough to deploy this model.

Nemotron 3.5 Lightning: Nano, Updated

NVIDIA also released Nemotron 3.5 Lightning 30B-A3B this week.

If the naming feels slightly confusing, Nemotron 3.5 Lightning is effectively the updated model occupying Nemotron 3 Nano’s position in the family.

I previously tested the original Nano here:

• Nemotron 3 Nano: A Very Fast Model That Doesn’t Think Too Much

And I covered the much larger Super model here:

• Nemotron 3 Super: 1M Tokens, Small KV Cache

Lightning still uses the same general idea of interleaving Mamba-2 processing with selected attention layers and sparse experts, and it supports context lengths up to 1M tokens.

So this is not a new size class. It is much closer to a refreshed Nano.

But NVIDIA has brought several ideas developed across the rest of the Nemotron 3 family back into this smaller model.

The biggest additions are around inference and agents.

Multi-Token Prediction is now part of the model training, followed by an additional MTP-boosting phase. NVIDIA also releases DFlash and DSpark draft models for speculative decoding, so there are multiple ways to accelerate generation depending on concurrency and hardware.

This is interesting because MTP was already one of the important additions in Nemotron 3 Super.

NVIDIA also says that 3.5 Lightning received harness-optimized training for agent workloads. Large Nemotron models can do the expensive planning and reasoning, while Lightning is supposed to execute the many smaller calls produced by long-running agents.

There are BF16 and NVFP4 checkpoints, and NVIDIA says the model can reach up to 4x the output speed of similarly sized models. On its PinchBench test, NVIDIA reports roughly 86% accuracy while completing 10,000 tasks around 30% faster than Qwen3.6 35B at comparable accuracy. Those are NVIDIA’s numbers, so I would still like to reproduce the speed/accuracy trade-off independently.

DeepSeek V4 Pro 0813: “Preview” Really Meant Preview

Finally, DeepSeek (quietly) released DeepSeek-V4-Pro-0813.

When DeepSeek introduced V4 in April, both V4 Pro and V4 Flash were explicitly called Preview models. The architecture was extremely innovative.

V4 Pro has around 1.6T parameters with 49B active and combines Compressed Sparse Attention with Heavily Compressed Attention. DeepSeek also introduced Manifold-Constrained Hyper-Connections and trained with the Muon optimizer. At a one-million-token context, DeepSeek reported that V4 Pro required only 27% of the single-token inference FLOPs and 10% of the KV-cache memory of DeepSeek V3.2.

Accuracy was the problem.

The original V4 Pro was rather underwhelming for such a huge model, particularly on agentic tasks. Then DeepSeek released Flash-0731, and the Flash model was suddenly beating Pro Preview.

I wrote about that two weeks ago:

• DeepSeek-V4-Flash-0731 and Inkling Small: Smaller, but Better?

At the time, I wrote that Flash-0731 was “a very promising sign for the next V4 Pro update.” That update is here now. The model now also exposes three reasoning-effort settings: low, high, and max.

The accuracy jump is much more important.

A few examples from DeepSeek’s own evaluation table:

• DeepSWE: 12.8 (it was lower than Qwen3.8 27B!) → 62.7
  Current Flash-0731: 54.4

• Terminal Bench 2.1: 72.1 → 87.9
  Current Flash-0731: 82.7

• NL2Repo: 38.5 → 61.5
  Current Flash-0731: 54.2

• Cybergym: 52.7 → 83.3
  Current Flash-0731: 76.7

• Toolathlon Verified: 55.9 → 74.1
  Current Flash-0731: 70.3

• AutomationBench: 12.8 → 31.8
  Current Flash-0731: 25.1

Pro-0813 is now ahead of the current Flash-0731 model.

DeepSWE is the most spectacular improvement: 12.8 to 62.7.

As I discussed this week, agentic evaluations can move significantly depending on the harness, configuration, allowed steps, tools, and other details. Independent testing remains necessary.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisioning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

Coding benchmarks increasingly test more than whether a model can write a correct function. They ask an agent to explore an unfamiliar repository, run commands, edit files, interpret failures, test its work, recover from mistakes, and decide when the job is actually finished.

That makes benchmark performance a product of at least three things: the underlying model, the agent harness driving it, and the amount of time and context the system is allowed to consume.

I evaluated Laguna S 2.1 on two long-horizon benchmarks, DeepSWE and Terminal-Bench 2.1, to understand how much these surrounding conditions affect both accuracy and cost. In particular, I wanted to see how the model behaves outside Poolside’s native agent harness, how much additional inference-time compute improves results, and what happens when an agent is given enough time to keep working after it gets stuck.

The results show a capable model whose accuracy can improve substantially when it is given more room to work. They also expose the other side of long-horizon agents: unsuccessful trajectories can consume enormous amounts of computation without getting any closer to a correct solution.

Perhaps most importantly, the experiments reinforce that model configuration is only part of the story. The agent wrapped around the model matters enormously.

In this article, I compare Laguna S 2.1 across DeepSWE and Terminal-Bench 2.1, examine how turn limits, timeouts, and token budgets affect accuracy and cost, analyze the main failure modes, and explain why the agent harness itself has become a critical part of coding-model evaluation.

Acknowledgments

Verda provided the B300 to run the experiments described in this article.

Verda is the full-stack frontier AI cloud, built for high-performance inference, training, and agentic workloads with data privacy and sustainability at its core.

You can check them out here. There is a $50 coupon that you can redeem in your Verda account, after provisioning it with $5, to try their GPUs.

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I don’t receive any form of compensation from Verda, or any usage information, related to this coupon.

Agentic coding evaluation setup

I deliberately evaluated Laguna S 2.1 outside Poolside’s native pool harness.

For DeepSWE, I used mini-swe-agent.

For Terminal-Bench 2.1, I used the terminus-2 agent.

Poolside’s own Laguna S 2.1 evaluations use its pool agent harness, while the public DeepSWE leaderboard normally uses mini-swe-agent. My numbers should be read as measurements of Laguna operating inside these particular third-party agent setups, not as direct reproductions of Poolside’s published benchmark results.

Read more

Meta is back, and it’s great.

For a while, the center of gravity in open-weight AI moved toward Qwen, DeepSeek, Google’s Gemma family, and a growing number of Chinese labs. Muse Glimmer gives Meta a genuinely interesting entry again.

Muse Glimmer is a ~30B-parameter multimodal reasoning model from Meta Superintelligence Labs, released under Apache 2.0.

• Muse Glimmer

Glimmer is distilled from Meta’s much larger Muse Spark model, supports tool use and multimodal inputs, has a 131K context window, ships with official 4-bit quantization, and targets consumer hardware. It’s clearly positioned as a competitor to Gemma 4 and Qwen3.6 open dense models.

How good is Glimmer compared with Gemma 4 and Qwen3.6?

The benchmark results put Glimmer surprisingly close to Qwen3.6. But benchmark scores alone tell us nothing about efficiency.

Glimmer uses an exceptionally lightweight attention design, and its relatively short 131K context window suggests it was not trained around the extremely long reasoning traces Qwen3.6 can produce, sometimes stretching beyond 100K tokens.

That opens up an interesting possibility: Glimmer may deliver performance close to Qwen3.6 while approaching Gemma 4’s token efficiency and using significantly less memory overall.

In this article, I’ll first look at what makes Muse Glimmer interesting as a local model, then break down its architecture in detail, including its hybrid attention pattern, extreme GQA, gated attention, positional encoding, multimodal stack, and speculative decoding. I’ll then dig into its KV-cache efficiency, with a direct 131K-context comparison against Gemma 4 31B and Qwen3.6-27B.

Finally, I’ll look at the benchmark results, where Glimmer gets surprisingly close to Qwen3.6, and discuss why token efficiency may turn out to be an even more important metric for agentic workloads.

Note: I’m also preparing a full analysis detailing the model accuracy, token efficiency, memory consumption, and inference speed for the original and quantized versions. It should take about a week to gather enough data.

Read more

Hi everyone,

We have a countdown for the Qwen3.8 releases:

Next week is going to be a busy one.

Here’s my plan:

• Start my own evaluation of Qwen3.8 27B as soon as the model is released. This time, I’ll also cover agentic coding. I plan to publish the full analysis the following week.

• Run and report on evaluations of the GGUF versions by the end of the week.

• Publish a deeper analysis of the quantized models within 14 days of release.

This will be similar to what I have done for Qwen3.6.

I’ll also release my own quantized versions using my new quantization pipeline that brings together AutoRound (for high-quality quantization), LLM Compressor (for packing with a vLLM-friendly format), and a custom repacker (to correct bugs with some layers). The goal is to finally make mixed-precision models, with different layers quantized anywhere from 2-bit to 8-bit, work well with vLLM.

Conceptually, it brings some of the flexibility that GGUF provides for llama.cpp to the vLLM ecosystem, while targeting much higher throughput under heavy concurrency, for example, when running multiple sub-agents in parallel, by leveraging the Humming kernel.

I’ve already published Qwen3.6 27B variants built with this pipeline.

• Qwen3.6 WNA16

The 3.8 and 4.3 bit versions work well. I’m still evaluating the 3.5 and 3.0 bit versions. I’ll update the model cards once I’m sure they all work well. I’ll also add versions with a quantized language modeling head later this week. I thought about quantizing the token embeddings, but this isn't well supported by vLLM for the Qwen3.5 architecture (it doesn’t crash, but only generates “!!!!!”; that’s a bug, not a quantization quality issue).

Now I’m ready to do the same for Qwen3.8.

The Harness Can Matter Almost as Much as the Model

A very interesting test from Composio showed how much the agent harness can change the results, even when the model stays the same.

Yes, it sounds obvious when stated like this, but I increasingly come across model evaluations claiming superiority over other models while relying on numbers produced with suboptimal agent harnesses for the evaluated models, or with harnesses configured in completely different ways (more turns, higher timeouts, …).

At best, these comparisons are useless. More often, they are actively misleading.

Composio ran DeepSeek V4 Flash through four different harnesses, Claude Code, Codex, OpenCode, and Oh My Pi, across 30 multi-step agentic tasks. The tasks involved live tools such as Gmail, Google Sheets, GitHub, Slack, Notion, Calendar, Airtable, and PagerDuty. A task was considered successful only if it passed every predefined check.

Observations

• Oh My Pi achieved the highest success rate, completing 17 out of 30 tasks. Claude Code and Codex each completed 16, while OpenCode completed 14.

• OpenCode had the lowest estimated cost per successful task at $0.073, followed by Codex at $0.081. Oh My Pi came in at $0.103, while Claude Code was the most expensive at $0.195.

• Speed produced yet another ranking. Claude Code had the lowest median completion time at 122.7 seconds, with OpenCode close behind at 129.7 seconds. Codex took 245 seconds, while Oh My Pi was the slowest at 272.4 seconds.

So, it seems we have different trade-offs:

• Claude Code was the fastest, but also the most expensive.

• Oh My Pi completed the most tasks, but was the slowest.

• OpenCode was the cheapest, but also had the lowest success rate.

But we also have another angle not tackled here: inference-time sampling. Rerun the same experiments, and you may get a very different picture. I agree with Composio’s general conclusion, but we need many more runs to conclude which is the cheapest, fastest, and most accurate harness.

This is also why evaluating models for agentic coding is extremely difficult, and extremely expensive to do properly.

I’m currently preparing an article focused on Laguna S2.1 and how I benchmarked it for agentic coding. I ran into far more issues than I would have liked to admit, but the process made one thing very clear to me: a large number of the agentic coding benchmark tables I see online should be treated with extreme caution.

Without controlling for the harness, its configuration, tools, prompts, retry behavior, and execution environment, comparing model scores can quickly become meaningless.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

The community has tried to address this issue, but with limited success. One approach is to introduce a “thinking cap” that stops the reasoning process once the model reaches a predefined token budget. Another is to fine-tune the model on shorter, more efficient reasoning traces.

I examined both approaches in a previous article about Qwen3.5 and Qwen3.6. These approaches either performed poorly or required substantially more expensive fine-tuning.

Hopefully, Qwen3.8 27B, which is expected to be released this week, will be a more efficient reasoner.

In the meantime, BottleCap AI has released a promising custom alternative that significantly reduces the reasoning cost of Qwen3.6: ThinkingCap-Qwen3.6-27B.

ThinkingCap-Qwen3.6-27B is a post-trained version of Qwen3.6-27B. Its main improvement is behavioral: it has been trained to produce shorter reasoning traces before delivering an answer.

The model also tends to generate shorter final responses. According to BottleCap AI, this behavior emerged during training alongside the reduction in reasoning length.

In this article, I evaluate the model’s accuracy and token efficiency across several tasks. I compare it with the original Qwen3.6 and another model known for its efficient reasoning, Gemma 4 31B IT.

All three models are evaluated using exactly the same evaluation settings, with their recommended hyperparameters, making the results directly comparable. Every number reported here comes from my own testing.

I also examine the model’s reasoning stability. In particular, is ThinkingCap-Qwen3.6-27B less prone to excessively long or seemingly endless reasoning than the original Qwen3.6?

Read more

Hi everyone,

In this edition of The Weekly Kaitchup:

• DeepSeek V4 Flash: Now Better than DeepSeek V4 Pro (Preview)

• Inkling-Small: Better than Inkling (?)

• Escha-W2: Qwen3.6 35B Compressed to 12.3 GB and Faster

DeepSeek released an update for its V4 Flash model. As usual, DeepSeek released the weights for this update immediately, rather than imposing the waiting period we have seen from companies such as MiniMax and Qwen.

• deepseek-ai/DeepSeek-V4-Flash-0731

It now outperforms the much larger V4 Pro (Preview), with particularly strong gains in agentic coding:

They did not publish results for other task categories. I checked Artificial Analysis’ additional results and found that it also improved on other benchmarks, including GPQA Diamond, which is a very different task from agentic coding. In fact, none of the reported benchmarks appear to show a regression. This is not merely a more specialized model, it is a substantial improvement over the preview and a very promising sign for the next V4 Pro update.

On a related note, Qwen also released Qwen3.7 Flash this week. It is currently available only through APIs and is very inexpensive. The community has speculated that it may be a Qwen3.7 35B-A3B model and that Qwen will eventually release the weights. I am not certain this is correct, but it would make sense.

Inkling-Small: Smaller, but Better on Most Benchmarks

Despite the name, Inkling-Small is not a lightweight model in the usual sense. It is still an open-weight, decoder-only multimodal model that accepts:

• Text

• Images

• Audio

It follows the same general architecture as the larger Inkling. Both models use a sparse mixture-of-experts design, routing each token through six of 256 experts, along with two shared experts. Both also support context windows of up to one million tokens.

So Inkling-Small is mostly the same design, scaled down.

How much smaller is it?

The larger Inkling has:

• 66 transformer layers

• 975 billion total parameters

• About 41 billion active parameters per token

Inkling-Small reduces that to:

• 42 transformer layers

• 276 billion total parameters

• About 12 billion active parameters per token

That makes Inkling-Small a little over one-quarter the size of the larger model by total parameter count. Its active parameter count is reduced by a similar amount.

The NVFP4 version is much easier to fit

Inkling-Small requires around 600 GB of aggregate VRAM in BF16.

The NVFP4 checkpoint brings that down to roughly 180 GB.

For comparison, the larger Inkling’s NVFP4 checkpoint is around 592–600 GB. In other words, the quantized version of the large model takes roughly as much memory as Inkling-Small does in BF16.

The word “small” is doing some work here, but 180 GB is at least in the range of a compact multi-GPU server rather than a large GPU cluster.

The smaller model is weirdly better at a lot of things

The surprising part is that Inkling-Small beats the larger Inkling across most published head-to-head benchmarks.

A few examples:

• SWE-bench Verified: 80.2% vs. 77.6%

• Humanity’s Last Exam: 31.6% vs. 29.7%

• IFBench: 82.2% vs. 79.8%

The smaller model appears particularly strong in coding, tool use, instruction following, and several reasoning evaluations.

Thinking Machines attributes the improvement to a revised data mix, an updated training recipe, distillation from the larger Inkling model, and additional reinforcement learning focused on agentic coding.

The larger model still performs better in some areas, especially factual recall, broad knowledge coverage, and most audio evaluations.

Still, it is an unusual result: the smaller model uses far fewer parameters, needs much less memory, and yet comes out ahead on a majority of the reported benchmarks.

Another unexpected result is that it significantly outperforms Qwen3.5 397B despite being much smaller, especially on benchmarks where Qwen3.5 has always been exceptionally good, like GPQA Diamond.

Is it benchmaxxed?? I’m waiting for the community feedback!

Escha-W2: Qwen3.6 35B Compressed to 12.3 GB

If you primarily use GGUF models locally, a 12.3 GB version of Qwen3.6 35B may not sound especially impressive. After all, that is roughly the size of a heavily quantized GGUF model, like a Q2_K_XL, and we already know those can perform well, as shown by the results I published a few months ago.

The key difference with Escha-W2 is that it uses (presumably) a more advanced quantization method and comes with an optimized inference runtime. As a result, it should run faster than the typical GGUF model, while already supporting SGLang for significantly better throughput under concurrent workloads. The developers have also said that vLLM support is coming later.

How They Did It?

We don’t really know yet.

The original model is a mixture-of-experts model with:

• 35 billion total parameters

• Around 3 billion active parameters per token

• 256 experts

Escha Labs keeps that underlying architecture but compresses most of the expert weights to roughly two bits. More precisely, it uses a mixture of two- and three-bit quantization for the expert projections, while keeping the dense layers in INT8.

The whole model is 12.3 GB

The resulting checkpoint takes up 12.3 GB on disk.

It can run on:

• A single 24 GB GPU under the recommended configuration

• A 16 GB GPU with reduced context length or fewer concurrent requests

On an RTX 4090, Escha reports around 225 tokens per second for a single stream. The same setup reaches about 1,321 tokens per second when serving 32 requests at once.

Two bits, but roughly FP8-level benchmark scores

The more interesting part is how little the compression appears to affect most of the reported evaluations.

Across six benchmark categories, Escha-W2 averages 100.2% of the FP8 model’s score. That does not mean the quantized model is genuinely better, the small gains are mostly within normal evaluation variance, but it does suggest that the overall quality loss is limited.

A few examples:

• MMLU-Pro: 80.9 versus 82.3 for FP8

• MATH-500: 93.8 versus 91.2 (ignore this one; this is too old and too easy for Qwen3.6)

• GPQA-Diamond: 77.8 versus 74.7

• BFCL tool use: 88.9 versus 88.2

• RULER long-context retrieval: 89.9 versus 89.4

Coding is the main place where it loses ground

The clearest regression is on longer coding tasks, where quantization always does more damage.

On LiveCodeBench v6, Escha-W2 scores 62.6, compared with 67.0 for the FP8 baseline. Note: I don’t know which framework they used to get a 67.0, but these scores seem very low. It should be around 85.0. This suggests that they ran it with a limited context length, like 32K max tokens. Not great. This means that the accuracy gap could be greater at longer context length, as quantization tends to be worse as sequence length increases.

It is not a normal drop-in quantization

There is one practical catch: Escha-W2 needs a custom runtime.

Its weights use a custom packed format, so the model does not simply load into standard inference software as a regular GPTQ, AWQ, or GGUF checkpoint. Escha currently provides two options:

• An SGLang-based runtime for concurrency, tool calling and structured output

• A standalone ZML runtime aimed at single-user inference

The problem with specialized runtimes is that it’s very hard work to maintain them.

The checkpoint is also text-only, even though the original Qwen architecture includes vision components.

When they release the vLLM runtime, I’ll double-check the accuracy on longer context and compare it with quantized versions of similar size.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

Prism ML has released impressive ternary and 1-bit variants of Qwen3.6 27B. The 1-bit version is only 3.9 GB, making it dramatically smaller than the original, which is roughly 55 GB.

• Bonsai 27B Models (Hugging Face)

These Bonsai 27B models behave quite differently from the original Qwen3.6 model, particularly in terms of accuracy and token efficiency.

Can you get Qwen3.6 27B-level accuracy from a 3.9 GB model?

With reasoning enabled, Bonsai 27B can nearly match the accuracy of Qwen3.6 27B running without reasoning. Moreover, that dramatic reduction in memory comes at a substantial computational cost: Bonsai needs to generate far more tokens to achieve the same result, up to 14 times more on some coding tasks.

When reasoning is enabled for both models, Qwen3.6 27B remains clearly ahead. It uses fewer tokens while achieving significantly higher accuracy. Still, what Prism ML has achieved is remarkable.

In this article, I take a deep dive into Bonsai 27B’s recipe, benchmark performance (my own numbers), and token efficiency. We will examine where the model struggles most, why its raw accuracy is lower, and why the results are nevertheless highly promising.

Read more

Hi everyone,

In this edition of The Weekly Kaitchup, I’m looking at two of the week’s most interesting releases for agentic workloads: Nanbeige4.2-3B and Laguna S 2.1.

Agentic workloads include multi-step reasoning, tool use, interaction with external environments, and tasks that may require many actions before completion.

• Nanbeige/Nanbeige4.2-3B

• poolside/Laguna-S-2.1

Despite this shared focus, they operate at very different scales.

Nanbeige4.2-3B is a compact dense model with approximately 4 billion total parameters and 3 billion non-embedding parameters. It is intended to make capable agentic behavior practical on consumer and workstation hardware.

Laguna S 2.1 is a 118-billion-parameter Mixture-of-Experts (MoE) model. It activates approximately 8 billion parameters for each token.

The two models represent different approaches to agentic AI. Nanbeige uses repeated computation over a relatively small set of weights, while Laguna uses sparse access to a much larger pool of learned parameters.

Nanbeige4.2-3B

While Nanbeige4.1 used a basic Llama architecture, Nanbeige4.2-3B uses a Looped Transformer.

Instead of containing a large number of unique transformer layers, the model has 22 physical decoder layers that are executed twice. The same weights are reused during the second pass, giving the model the computational depth of approximately 44 layer executions without storing 44 independent layers.

This design reduces weight memory, but it does not reduce inference computation to that of a normal 22-layer model. Each token must still pass through the transformer stack twice.

The model has 48 attention heads with a dimension of 128, and eight key/value heads. This is a lot. For instance, Qwen3.6 35B A3B has only two KV heads.

Looping introduces a further memory consideration. Unless the implementation explicitly shares KV-cache state between the two passes, each loop may require separate key/value entries. Nanbeige’s default configuration does not appear to enable loop-level KV sharing.

Moreover, I think models using the Looped Transformer should be better named. When you use a 3B model, you don’t expect it to be nearly as slow as a 6B model. small MoE models, like Qwen3.6 and Gemma 4, show the number of active parameters in their name, like 26B-A4B. For Looped Transformers, we should adopt a similar convention, like 3B-2P (for “2 passes”), for instance.

Nanbeige4.2-3B’s memory consumption

A 16GB GPU should be able to run the model, without quantization, at moderate context lengths with careful settings. A 24GB GPU provides more room for longer prompts, greater concurrency, and runtime overhead.

Based on the model’s 22 physical layers, two loop passes, eight key/value heads, 128-dimensional heads, and BF16 KV values, a conservative estimate places KV-cache consumption at approximately 176 KiB per cached token for each active sequence.

I show in the following article how to estimate the KV cache memory consumption:

At 16K tokens, this would require roughly 2.75 GiB of KV-cache memory. Using the complete 256K context could require roughly 44 GiB of KV-cache memory for a single active sequence.

This is very large. Almost twice as much as what Qwen3.6 27B would consume for the same context length.

Using Nanbeige4.2-3B with vLLM

At release, Nanbeige provided a dedicated vLLM branch for the model:

git clone -b nanbeige42 https://github.com/Nanbeige/vllm.git
cd vllm
pip install -e .

The model can then be served through vLLM’s OpenAI-compatible API:

vllm serve Nanbeige/Nanbeige4.2-3B \
  --host 0.0.0.0 \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser nanbeige \
  --reasoning-parser nanbeige

 #use --host 127.0.0.1 if you are running it locally

The reasoning and tool-call parsers are important for agent frameworks. They allow vLLM to separate reasoning content and structured tool calls from ordinary assistant text.

Nanbeige’s chat template exposes two controls.

The enable_thinking option determines whether the current response contains an explicit reasoning phase. The preserve_thinking option determines whether reasoning content from previous assistant turns remains in the conversation history.

For normal chat and question answering, preserved reasoning can usually be disabled. For multi-turn tool use, office workflows, and coding agents, the developers recommend retaining earlier reasoning content but I don’t understand how this can work well. Reasoning traces can be very long, like 50K+ tokens. Keeping them in the context, even if it’s just one, means that the model may reach its max context length at next turn.

Performance

According to the published evaluation, Nanbeige exceeded Qwen3.5-9B on most of the included agent, coding, and reasoning tasks. It also surpassed Gemma 4 12B on benchmarks including GDPval, SWE-Bench Verified, SWE-Bench Pro, and Terminal-Bench 2.0.

Agent results are influenced not only by the base model but also by the allowed reasoning budget, tool definitions, prompting strategy, conversation-state handling, retry policy, and maximum number of actions.

Reference: Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model

Laguna S 2.1

Laguna S 2.1 is a much larger coding-focused MoE model.

It contains 118 billion total parameters but activates approximately 8 billion parameters for each token. A routing mechanism selects a subset of experts during inference, allowing the model to access a large learned parameter pool without executing every parameter for every token.

Laguna contains 48 transformer layers. Twelve use global attention, while the remaining 36 use sliding-window attention with a window of 512 tokens. The global and local layers are interleaved in approximately a one-to-three ratio.

The global layers allow information to move across the complete input context. The sliding-window layers restrict attention to nearby tokens, reducing the cost of long-context inference and limiting KV-cache growth.

The model contains 256 routed experts and one shared expert. For each token, the router selects the top ten routed experts in addition to the shared computation.

Laguna also uses per-head softplus output gating.

Per-head softplus output gating gives each attention head its own learned volume control. For every token, the model can reduce, preserve, or amplify the contribution of individual heads before combining their outputs. The softplus function keeps these gates positive while allowing values above one, so useful attention heads can be strengthened rather than merely switched on or off.

It uses interleaved reasoning. The model can reason, issue a tool call, receive the tool result, and resume its reasoning before selecting another action.

Poolside also provides a DFlash draft model for speculative decoding. These models generate candidate tokens that the main model can verify in parallel, potentially increasing output speed.

Using Laguna S 2.1 with vLLM

Laguna support is documented for vLLM 0.25.0 or newer:

uv pip install -U "vllm>=0.25.0"

The model can run on a single B300 GPU:

vllm serve poolside/Laguna-S-2.1 \
  --enable-auto-tool-choice \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --default-chat-template-kwargs '{"enable_thinking": true}'

The NVFP4 and INT4 versions consume fewer than 80 GB but you would need a 96 GB GPU, like an RTX Pro 6000, to exploit the context length of the model.

vllm serve poolside/Laguna-S-2.1-INT4 \
  --enable-auto-tool-choice \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --default-chat-template-kwargs '{"enable_thinking": true}'

For coding-agent workloads, Poolside also recommends retaining prior reasoning_content in the conversation history.

As for the memory consumption of the KV cache, 256K tokens should consume around 24 GB.

Target tasks

Laguna S 2.1 is more specialized than Nanbeige.

Its primary target is long-horizon software engineering. This includes repository-level bug fixing, terminal interaction, shell-based workflows, multilingual code maintenance, codebase exploration, and answering questions that require understanding large software repositories.

Performance

Laguna’s sparse architecture and coding-focused training are particularly effective for terminal interaction, repository-level reasoning, and multilingual software engineering.

It looks like a very good model but so far, I have found community feedback to be mixed.

reference: Introducing Laguna S 2.1

I’ll spend some time with Nanbeige and Laguna. If everything goes well, I’ll probably write an article using them for agentic coding.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

Alibaba’s Qwen3.8 is a 2.4-trillion-parameter frontier model. The company has indicated that it plans to release the model’s weights, potentially making Qwen3.8 one of the largest openly downloadable AI models ever produced.

For comparison, Alibaba’s largest open-weight model to date, Qwen3-Coder-480B-A35B-Instruct, is roughly five times smaller. Qwen3.8 would still have about 400 billion fewer parameters than Kimi K3.

With both K3 and Qwen3.8 performing nearly at the frontier, was closing the accuracy gap with OpenAI and Anthropic largely a matter of scaling from hundreds of billions of parameters to more than two trillion?

An open-weight release for Qwen3.8 is excellent news. It could give researchers, companies, and the broader AI community unprecedented access to a model at this scale.

But what could most people realistically do with a 2.4-trillion-parameter model?

Very little locally. Even with aggressive quantization, compression, or distributed inference, a model of this size would be far too large to run on a typical personal computer. Its release could still be highly significant for research institutions, infrastructure providers, and well-resourced community projects. For casual users, however, direct local deployment would remain largely impractical.

Let’s check, with a few assumptions, what you would need to run NVFP4, Q4, and Q1 GGUF versions.

Qwen3.8: A Sparse MoE?

As I write this, Alibaba has not disclosed when it plans to release the model’s weights. That could change quickly. They may even be released by the time this article is published.

Alibaba has not yet disclosed how many parameters are activated for each token, how many experts the model contains, or whether every layer uses mixture-of-experts routing.

Nevertheless, a dense 2.4-trillion-parameter transformer would be extraordinarily expensive to serve. It is thus reasonable to assume that Qwen3.8 uses a highly sparse mixture-of-experts, or MoE, architecture.

Until Alibaba publishes the architecture, Moonshot AI’s Kimi K2 provides a useful proxy for estimating how a trillion-parameter MoE model might distribute its weights.

Kimi K2 contains:

• Total parameters: 1.04 trillion

• Activated parameters: 32.6 billion

• Routed experts: 384

• Experts selected per token: 8

• Routing sparsity: 8/384, or 1/48

• Shared experts: 1

Note: These figures come from the Kimi K2 technical report.

98.9% of Kimi K2’s parameters are located inside routed experts, while only about 1.1% are always-active or otherwise non-routed weights. The trend is the same for all the MoE with 500B+ parameters: Nearly 99% of the parameters are in the routed experts.

Applying the same ratio to Qwen3.8

In other words, Qwen3.8 could store 2.4 trillion parameters while using only a small fraction of them for each token. If its routing sparsity resembled Kimi K2’s, the computation required per token might be comparable to that of a roughly 75-billion-parameter dense model, before accounting for routing and distributed-communication overhead.

BF16 stores each parameter using 16 bits, or two bytes.

This does not include the KV cache, inference-engine workspace, CUDA graphs, temporary activations, multimodal components, tokenizer files or checkpoint metadata.

A practical deployment would consequently need more than 4.8 TB of combined accelerator memory. I won’t speculate on the KV cache size. There are too many variables that influence it: number of linear layers, number of KV heads, etc.

Experts in NVFP4, remainder in BF16

I hope Alibaba releases an official NVFP4 version. Producing a high-quality NVFP4 quantization of a model this large would be prohibitively expensive for most community projects. NVIDIA has also been relatively slow to publish its own NVFP4 conversions of open-weight models; for example, its version of Qwen3.6 appeared only recently.

NVFP4 stores each parameter as a 4-bit value, with a single FP8 scaling factor shared across all 16-value blocks. This results in an effective storage cost of approximately 4.5 bits per parameter, plus a negligible FP32 scaling value for each tensor.

Fortunately, routed experts tend to be relatively robust to quantization. In MoE models, the expert weights can often be stored at lower precision while the comparatively small set of shared, attention, embedding, and other always-active parameters remains at higher precision. This mixed-precision approach can substantially reduce storage requirements while limiting the effect on model quality.

So, with 99% of parameters quantized to NVFP4:

• 2.374 trillion routed-expert parameters at 4.5 bits each

• 25.8 billion remaining parameters at 16 bits each

The expert weights consume:

The BF16 remainder consumes:

Total:

That’s an average of 4.62 bits per parameter.

Compared with a 4.8 TB BF16 checkpoint, that saves approximately:

or 71.1% of the original weight memory.

What about a typical Q4 GGUF?

A common GGUF format such as Q4_K_M is approximately 4.8 bits per weight in representative models, while Q4_K_S is closer to 4.6 bits per weight. Actual ratios vary with architecture and which tensors remain at higher precision.

For a 2.4-trillion-parameter checkpoint:

• Q4_K_S-like: 4.58 bits; approximately 1.374 TB

• Q4_K_M-like: 4.84 bits; approximately 1.452 TB

• Higher-overhead Q4: 5.0 bits; approximately 1.50 TB

With runtime allocations and a useful KV cache, a deployment should budget at least 1.6–1.8 TB of usable memory, and more for long contexts or concurrent users.

What would TQ1 weigh?

I’m mentioning TQ1 since Unsloth has previously released very good TQ1 versions of Qwen3.5. We have no guarantee they can do/will do the same for Qwen3.8.

TQ1_0 uses a compact ternary representation at approximately 1.69 bits per weight.

Purely as storage arithmetic:

Could Qwen3.8 run “locally”?

It will not be a normal desktop model. Even the Q4 version would be roughly twenty times larger than a 70-billion-parameter Q4 model.

Full BF16

A BF16 deployment would need at least 4.8 TB for weights and roughly 5.5 TB after adding a modest 15% operational allowance.

That points toward configurations such as:

• Approximately 24 B300 GPUs, assuming 288 GB each.

• Approximately 32 B200 GPUs, assuming 180 GB each.

NVIDIA’s eight-GPU DGX B200 provides 1.44 TB of HBM and can be configured with 2–4 TB of system RAM. NVIDIA’s Blackwell Ultra B300 provides 288 GB of HBM per GPU, or approximately 2.3 TB in an eight-GPU system.

BF16 Qwen3.8 is firmly a multi-server deployment.

NVFP4, Q4, and TQ1

A 1.4–1.5 TB quantized model is more approachable, but “approachable” still means rack-scale hardware.

An eight-GPU B300 server with approximately 2.3 TB of HBM should have enough room for the quantized weights, runtime allocations, and a reasonable KV cache.

An eight-GPU B200 system has only 1.44 TB of HBM. It might barely load the 1.387 TB mixed-NVFP4 estimate, but it would leave almost no memory for the inference engine or KV cache. A two-node, 16-B200 configuration would be much more practical.

A 507 GB TQ1 checkpoint would still need roughly 600 GB or more after runtime overhead.

It could fit in:

• A 768 GB or 1 TB RAM server.

• Four B200 GPUs with sufficient aggregate memory.

• Three B300 GPUs in a custom configuration.

• An eight-GPU node with considerable unused capacity.

A current maximum-memory Mac Studio offers up to 512 GB of unified memory, which would be too tight once runtime overhead and the KV cache are included.

So, running even the most compressed version of Qwen3.8 won’t be cheap.

Hopefully, Alibaba will also release smaller versions of Qwen3.8.

Hi everyone,

In this edition of The Weekly Kaitchup, we will discuss:

• Inkling and the Return of Global Competition in Open-Weight AI

• Gemma 4: Small Updates with Significant Improvements

• Bonsai Binary and Ternary Qwen3.6: Only 3.9 GB and It Still Works!

Inkling and the Return of Global Competition in Open-Weight AI

GLM, Kimi, MiniMax, and many of the largest recent open-weight language models have come from China. Outside China, NVIDIA has been one of the few companies to release a comparably large model recently, with Nemotron 3 Ultra.

Against this backdrop, the release of Inkling by Thinking Machines is a welcome development. Strong open-weight models from a wider range of organizations and countries are important for maintaining healthy global competition in frontier AI.

• thinkingmachines/Inkling

Inkling is a 975-billion-parameter multimodal model with approximately 41 billion parameters active per token. Its most interesting technical contributions are not its mixture-of-experts architecture or hybrid attention, both of which are now relatively common, but its approach to long-context modeling, optimization, and reinforcement learning.

Long-context architecture

Inkling uses learned relative positional representations instead of rotary positional embeddings. Thinking Machines reports that this approach performed better in its long-context extrapolation experiments.

The model combines these representations with an attention pattern consisting of five sliding-window attention layers for every global-attention layer. Most computation remains local, while periodic global-attention layers allow information to move across the full context. This idea of global-local layers is somewhat close to what Gemma 4 does.

Short convolutions are also applied after the attention key and value projections, as well as to the outputs of the attention and MLP branches. This introduces explicit local sequence mixing at several points within each transformer block, rather than relying exclusively on attention.

Mixture-of-experts design

Each mixture-of-experts layer contains:

• 256 routed experts

• Two shared experts (so, one more than most open-weight models)

• Six routed experts selected for each token

The outputs of the shared and routed experts are normalized together. This allows the routing mechanism to control the relative contribution of general-purpose shared computation and more specialized expert computation.

Parameter-specific optimization

Inkling uses Muon for large matrix parameters and Adam for other parameter types. Its weight decay is also adjusted alongside the learning-rate schedule instead of remaining fixed throughout training.

This represents a notably large-scale example of assigning different optimizers according to parameter structure.

Learning to reason within a compute budget

Thinking Machines reports running more than 30 million asynchronous rollouts while varying both the requested level of reasoning effort and the cost associated with generating additional tokens.

The objective is to teach the model to adapt its reasoning strategy to a given compute budget. This differs from simply imposing a maximum output length after training. Instead, the model learns to estimate when additional reasoning is likely to improve its answer and when the expected benefit is not worth the additional token cost.

Thinking Machines also randomized the available tools and their schemas during training. The goal was to prevent the model from becoming overly dependent on fixed function names, argument structures, or a single tool-calling format.

This could help the model generalize more reliably across unfamiliar tools and changing software environments.

Competitive, but not yet the leader

Inkling performs competitively with other leading open-weight models. However, on the reported benchmarks, GLM-5.2 remains slightly stronger across most evaluations despite being smaller.

They also released an NVFP4-quantized version that requires just 592 GB of memory, compared with roughly 1.9 TB for the original model.

• thinkingmachines/Inkling-NVFP4

The smallest GGUF made by Unsloth requires 270 GB + KV cache. I expect it to work very well despite the heavy compression since at that scale most of the parameters are routed experts, which are usually very robust to quantization.

• unsloth/inkling-GGUF

Gemma 4: Small Updates with Significant Improvements

The recent Gemma 4 changes are mainly runtime and interface updates rather than a new checkpoint.

One update is broader FlashAttention 4 support on NVIDIA Hopper GPUs. Google reports higher prompt-processing throughput and lower time to first token. The main benefit is during prefill, when the model processes the complete prompt before generating its first output.

Gemma 4 also now exposes several visual-token budgets per image. Developers can use smaller allocations for classification, captioning, or video frames and larger allocations for OCR, documents, handwriting, or images containing small text.

This makes visual resolution a direct inference control. More visual tokens preserve more detail but also increase prompt length, prefill time, cache use, and the amount of context consumed by each image. The setting is useful for workloads in which some images require detailed analysis while others do not.

The other relevant change concerns chat and tool-use templates. These templates convert structured messages into the special-token format expected by the model.

Recent fixes address turn boundaries, tool-response handling, structured arguments, multimodal messages, and reasoning continuity across multiple tool calls. These are implementation changes, but they can directly affect agent reliability. An incorrect marker may cause the model to treat a tool result as a user message or to restart a reasoning sequence instead of continuing it.

Google reports accuracy improvements on benchmarks using tools thanks to these updates:

Important: The Gemma 4 repositories were updated in place. As a result, reproducing earlier evaluation scores requires loading the models with the corresponding revision explicitly specified.

Bonsai Binary and Ternary Qwen3.6: Only 3.9 GB and It Still Works!

PrismML’s Bonsai applies binary and ternary weights to most of the language network of Qwen3.6-27B.

• prism-ml/bonsai-27b

The ternary version uses negative, zero, and positive weight states and occupies about 5.9GB. The binary version uses only negative and positive states and occupies about 3.9GB. Both use one half-precision scale for groups of 128 weights.

The low-bit format covers embeddings, attention projections, MLP layers, and the final language-model head. Many quantized models keep sensitive layers at higher precision, especially embeddings or output heads. Bonsai does not use those higher-precision exceptions in the language network.

The vision tower remains in four-bit precision. This suggests that the visual encoder is more sensitive to binary or ternary quantization than the language network. Vision performance still declines, likely because errors accumulate across the visual encoder, multimodal projection, and low-bit decoder.

Bonsai also requires custom CUDA and MLX kernels. A compressed weight file does not automatically produce fast inference if the runtime first expands the weights into a higher-precision matrix.

The custom kernels operate directly on the packed representation. Binary weights can be handled mainly as sign changes, while ternary weights add a zero state that can represent an absent connection. Accumulation and activations still require higher precision.

The reported phone deployment should be interpreted mainly as a weight-memory result. A 3.9GB model can fit within the memory range of a high-end mobile application, but runtime memory must also include the KV cache, activations, workspaces, the vision tower, and operating-system overhead. The full advertised context length is unlikely to be practical on a phone.

Despite the heavy compression, the model remains very capable:

The degradation compared with the original model is significant, but the accuracy is far better than all the other sub-4 GB models I have evaluated.

I’ll publish full analysis, accuracy, and token efficiency next week!

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning it with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

During generation, attention layers store keys and values for every previous token. As context length and concurrency increase, this cache can consume tens of gigabytes and become a major memory-bandwidth bottleneck.

KV-cache quantization reduces that cost by storing keys and values at lower precision. Moving from 16-bit to approximately 4-bit storage can shrink the cache by nearly four times, enabling longer contexts and more concurrent requests.

In practice, however, a method that works well in a research implementation may perform poorly inside an optimized engine such as vLLM or llama.cpp. Quantization can add computational overhead, require specialized kernels, reduce hardware compatibility, and interfere with features such as speculative decoding.

In this article, I compare three 4-bit KV-cache formats in vLLM, turboquant_4bit_nc, int4_per_token_head, and nvfp4, using Qwen3.6-27B. I evaluate their memory usage, accuracy, token generation, and inference speed.

Acknowledgments

Verda provided the B200s and RTX Pro 6000s to run the experiments described in this article.

Verda is the full-stack frontier AI cloud, built for high-performance inference, training, and agentic workloads with data privacy and sustainability at its core.

You can check them out here. There is a $50 coupon that you can redeem in your Verda account, after provisionning it with $5, to try their GPUs.

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Qwen3.6-27B KV Cache Size

I’m going to compare KV-cache memory usage across different quantization methods. First, let’s establish the memory required for Qwen3.6-27B’s native 262,144-token context window.

Qwen3.6-27B has 64 language layers, but only 16 are Gated Attention layers that maintain a full KV cache. The remaining layers are Gated DeltaNet blocks, so they are not included when calculating the conventional attention KV cache.

Each Gated Attention layer has four KV heads, with a head dimension of 256.

In BF16, the KV cache for these full-attention layers consumes approximately 17.2 GB. This is substantially less than for models that use full attention in every layer, but it is still significant relative to the model weights themselves, which occupy roughly 54 GB.

On a device with 96 GB of memory, that leaves:

96 − 54 − 17.2 = 24.8 GB

This is not enough to run several full-context requests in parallel, for example, multiple sub-agents, especially once additional overhead is included. CUDA graphs, temporary buffers, runtime allocations, and other inference-engine components all consume extra memory.

Read more

Hi everyone,

In this edition of The Weekly Kaitchup, I’ll highlight some of the most interesting work I saw at ACL 2026, which I attended this week in San Diego.

I’ll also discuss OpenAI’s recent take on benchmarking with SWE-Bench Pro, one of the most widely used coding benchmark. Once again, a SWE-Bench is being declared “bad.” I can’t say I’m too upset about having never spent compute on it.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning it with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

ACL 2026: Why I Went, and Why I Skipped ICML 2026

This year, two of the major annual research conferences shaping the development of AI, ACL and ICML, took place at the same time, on opposite sides of the Pacific: ACL in San Diego and ICML in Seoul.

While there is significant, and increasing, overlap between the two communities, they tend to emphasize different areas. ICML is often the place for deeper discussions about machine learning methods, algorithms, and theoretical foundations. ACL, by contrast, is where I usually find the most relevant work on evaluation, multilinguality, benchmarks, datasets, and language-centered applications.

That distinction is far from absolute. Many of the key ideas behind today’s Transformers and LLMs, as well as many of the benchmarks and evaluation tasks we still rely on, originated in papers published within the ACL community.

I chose to attend ACL this year primarily because it aligns more closely with my own research background. It is also the community where I know more people, having published several papers there during my Ph.D.

ACL 2026 was capped at 3,500 onsite attendees. For reference, ACL 2025 in Vienna had more than 5,000 in-person attendees.

The smaller scale made ACL easier to navigate, but it was still impossible to see everything I had planned. As I did for NeurIPS 2025, I will mainly focus on the papers I actually saw and discussed at the conference rather than trying to summarize the full program.

I spent most of my time in the poster sessions, which were by far the best place to have useful conversations with authors. The talks were less crowded, and some large rooms were surprisingly almost empty, especially on the last day.

To keep this edition useful and focused, I will mostly discuss papers on LLM reasoning and efficiency, two of the major themes of ACL 2026 along with reinforcement learning, as it has been at most recent AI and machine learning conferences.

The program chairs even noted that papers on some of these topics seemed to have higher acceptance rates, which suggests that they were not only popular among authors, but also particularly appealing to reviewers and area chairs.

Note: For each paper discussed below, I also included a photo of the corresponding poster. The images are high-resolution, but I do not fully trust Substack to display them cleanly (depending on your device), so apologies if some details are difficult to read. ACL may also publish the posters directly on each paper’s page, or may have already done so by the time you read this. You can check by clicking the paper links.

Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models
Hoang Phan, Xianjun Yang, Yuanshun Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, Deren Lei
Meta; New York University; Johns Hopkins University

This paper looks at a well-known problem in reinforcement learning for reasoning models: training that improves reasoning can also make the model forget broader capabilities. I often see that when running RL for recent models like Qwen3.5/Gemma 4.

The authors show that RLVR-style post-training may improve mathematical or multimodal reasoning while degrading skills such as perception, OCR, robustness, and general instruction following.

They propose RECAP, a replay-based method that keeps general-capability data in the training loop and dynamically reweights objectives. The main contribution is a clearer diagnosis of “reasoning gain” as a trade-off problem: a model can become better at narrow reasoning benchmarks while becoming less generally useful. RECAP is presented as a way to preserve the original model’s breadth while still gaining reasoning ability.

Very simple and intuitive.

SSSD: Simply-Scalable Speculative Decoding
Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K Muller, Lukas Cavigelli
Huawei; ETH Zurich

This paper proposes SSSD, a training-free speculative decoding method for making LLM inference faster.

Instead of relying on an additional trained draft model, like EAGLE3 or DFlash, SSSD retrieves likely n-gram continuations from the prompt, previously generated tokens, and a datastore, then verifies them with the main model.

It avoids extra model training and is easier to deploy across changing workloads, domains, and languages.

The paper reports latency reductions of up to 2.9x over standard autoregressive decoding, making it relevant for production serving where simplicity and robustness matter as much as peak speed.

Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty
Salesforce AI Research; University of Wisconsin-Madison; The University of Hong Kong

Early layers of an LLM can already identify many of the tokens that will matter for answering a query. The authors turn this into GemFilter, a training-free method that uses early layers as filters to select a much smaller subset of the input before later computation.

The method is aimed at long-context inference, where latency and GPU memory grow quickly with input length. By compressing the input token set, GemFilter speeds up inference while keeping the important evidence visible to the model.

A nice aspect of the work is interpretability: because the method explicitly selects input tokens, humans can inspect what the model kept. The paper reports a 2.4x speedup and 30% GPU memory reduction compared with strong baselines, while performing well on Needle-in-a-Haystack and competitively on LongBench.

A Survey on Memory-Efficient Fine-Tuning for Large Language Models
Yeachan Kim, Mingyu Lee, SangKeun Lee
Hankuk University of Foreign Studies; Korea University

This survey organizes the landscape of memory-efficient fine-tuning for LLMs. I recommend reading it if you are lost in the recent progress made in this area.

One point I strongly agree with, although it is not very well developed in the paper, is that these methods are often poorly evaluated.

The paper mentions GLUE-style tasks. I did not double-check how widely GLUE is still used in this line of work, but even replacing it with old and relatively easy benchmarks such as QNLI, PIQA, or CommonsenseQA, like they did in this paper, would not solve the problem. People use these benchmarks because they are cheap to run.

As I often argue here, if your benchmarks do not require the model to generate millions of tokens, they are unlikely to tell you much about how a generative language model actually performs, unless of course you thouroughly study what the benchmarks has generated instead of just looking at the scores.

KV Compression

LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning
Haoyue Zhang, Hualei Zhang, Xiaosong Ma, Jie Zhang, Song Guo
The Hong Kong University of Science and Technology; The Hong Kong Polytechnic University

This paper studies KV-cache compression for long chain-of-thought reasoning. The key finding is “Token Importance Recurrence”: tokens that seem unimportant at one decoding step can become important again later. This makes simple greedy KV eviction risky, because it may remove tokens that the model will need in future reasoning steps.

LazyEviction addresses this by delaying eviction and observing attention patterns over a window before deciding what to discard. The method is designed for long reasoning sequences, where memory cost grows with every generated token, and it aims to reduce memory while avoiding sudden reasoning failures caused by prematurely evicting useful context.

This is another interesting piece of work on KV cache compression, one of many presented at the conference.

As usual, though, the real test will be implementation. Unless a method lands in frameworks such as vLLM or llama.cpp, it is hard to know whether it actually improves inference efficiency in practice, or whether it interferes with other optimizations already implemented in these systems.

Anchoring the Cache: Mitigating Contextual Hallucination in KV-Compressed Long-Context Summarization
Yu Fu, Chen Luo, Josef Valvoda, Xin Zhang, Xuejing Lei, Xiao Pan, Hui Liu, Yue Dong
University of California, Riverside; Amazon

This paper studies an important side effect of KV-cache compression in long-context summarization: it can make models hallucinate more.

The authors show that aggressive compression may preserve surface-level summarization metrics while weakening factual grounding, because retrieval heads drift away from the source context during generation.

Their method, HalluKV, anchors selected retrieval heads to the source by selectively removing generated KV pairs from those heads during decoding. The idea let the model keep the efficiency benefits of cache compression, but prevent the heads responsible for source retrieval from over-attending to the model’s own generated text.

The result is a decoding-time intervention that reduces hallucination while maintaining long-context efficiency. The paper reports that KV compression can increase hallucination scores by up to 3.36×, and that HalluKV reduces hallucination across multiple models and datasets while preserving competitive summary quality.

Quantize What Counts: More for Keys, Less for Values
Mohsen Hariri, Alan Luo, Weicong Chen, Tianyi Zhang, Qifan Wang, Xiaotian Han, Vipin Chaudhary
Case Western Reserve University; Rice University; Meta

Keys and values of the KV cache should not necessarily receive the same number of bits: key projections tend to have larger spectral and Frobenius norms than value projections, meaning keys carry higher-magnitude, more quantization-sensitive information.

The authors use this observation to justify a simple rule: allocate more precision to keys and compress values more aggressively.

The paper turns this into a geometry-driven design principle for mixed-precision KV-cache quantization. Instead of tuning key/value bit splits heuristically for each model or task, it argues that model weight geometry can predict which cache component needs more protection. Empirically, key-favored allocations such as 4-bit keys and 2-bit values preserve up to 98.3% of the accuracy of uniform 4-bit KV quantization while saving memory, making the result useful for long-context inference where KV cache memory dominates.

A very informative paper to better understand why keys are more sensitive to quantization.

More Comments on the ACL 2026

The social events at the ACL are always great. This year, the organizers reserved the USS Midway for an evening dinner. The USS Midway is an aircraft carrier turned museum, and the venue was genuinely impressive.

These social events are often among the most useful parts of a conference. They make it much easier to meet people outside your usual circle, and this one was no exception.

The next ACL will be in Kyoto, so I will probably end up playing local guide: I lived and worked there for six years.

Hopefully, the conference will not coincide with the worst of Kyoto summer: 35°C or more, 70%+ humidity, and the occasional typhoon. Otherwise, the most important survival tips may have nothing to do with AI.

SWE-Bench Is Not a Good Benchmark, Again

Unrelated to the ACL, but very interesting findings published by OpenAI this week.

OpenAI has used SWE-Bench Pro to benchmark all its models since GPT 5.2.

And yet, according to OpenAI’s recent audit, many SWE-bench Pro tasks fail for reasons unrelated to model capability.

Examples include:

• hidden requirements that never appear in the issue description

• contradictory specifications

• unit tests that reject perfectly reasonable implementations

• grading logic that expects one specific implementation instead of checking whether the issue is actually fixed

• mismatches between the GitHub issue, the merged pull request, and the evaluation tests

Because SWE-bench Pro is built automatically from real repositories, OpenAI argues that these inconsistencies are common enough to substantially affect reported scores.

OpenAI’s main conclusion is:

Approximately 30% of SWE-bench Pro tasks are broken.

Their recommendation is therefore to “carefully inspect benchmark results instead of treating leaderboard rankings as ground truth.” Yes, of course, but who actually does this? Most AI labs want to report the strongest possible numbers. If you publish a score of 80 and your competitor publishes 90, most people will not ask whether that 10-point gap reflects a real performance difference or a benchmarking artifact. They will simply see the lower score, and it will look bad.

Ironically, OpenAI itself helped create SWE-bench Verified in 2024 because it believed the original SWE-bench contained many impossible or ambiguous tasks. Now, after auditing SWE-bench Pro, it argues that even this newer benchmark still contains enough flawed tasks that raw leaderboard scores should not be over-interpreted.

My take is that these benchmarks are becoming increasingly complex: automatically generated, frequently refreshed, available in multiple versions, and often difficult to interpret consistently. As a result, I think people will develop more and more trust issues around published scores and may eventually start disregarding them altogether.

However, this is less of a problem when evaluating different versions of the same model. For example, when comparing a quantized model to the original, the goal is not necessarily to establish the model’s absolute performance. What matters is whether the quantized version performs as closely as possible to the non-quantized one.

In that setting, the original score matters less. The main objective is to minimize the accuracy delta between the two versions. Even a flawed benchmark like SWE-Bench can still be useful for this purpose, as long as it is used consistently across both model variants.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

LFM2.5 is a family of small language models. In this article, I will focus specifically on the tiny 230M and 350M versions. I ran both models on a large set of benchmarks with two goals:

1. Reproduce some of the results published by Liquid AI.

2. Better highlight what these models can and cannot do.

At first glance, the size difference between the 350M and 230M models may seem negligible: only 120M parameters. But that gap feels much larger when you realize that 120M parameters represent roughly 34% of the 350M model’s total size. On memory-constrained devices, especially when using longer context lengths, this difference can matter a lot. In 16-bit precision, it reduces weight memory consumption from about 0.71 GB for the 350M model to about 0.46 GB for the 230M model.

And given how accurate is quantization now, it could be possible to make a quantized version of the 350M that is both smaller and better than the 230M.

This raises another question that I will address in this article:

Should you use a low-bit quantized version of the 350M model, or a higher-precision version of the 230M model?

To answer that, we will compare accuracy versus model size for both models across different GGUF quantization formats released by Liquid AI and Unsloth.

LFM2.5 230M and 350M: What They Can and Cannot Do

Read more

Hi everyone,

In this edition of The Weekly Kaitchup:

• Which NVFP4 Version of Qwen3.6 27B Should You Use?

• DSpark: DeepSeek’s New Speculative Decoding Method

I’m at the ACL 2026 and stopped by the Qwen corner to ask the important question:

Should we expect Qwen3.7 27B soon?

Answer:
We haven’t made a final decision yet, but we’re committed to open-sourcing more models and have some exciting releases coming soon.

Not very informative… It sounds like a well-prepared answer. And they told me they already got this question a lot during just a single morning at the conference.

Which NVFP4 Version of Qwen3.6 27B Should You Use?

NVIDIA released an NVFP4 version of Qwen3.6 27B. It is a good occasion to revisit the other NVFP4 variants that have been released and to put NVIDIA’s checkpoint into context.

When I published my evaluations and analysis of quantized Qwen3.6 27B, only a few NVFP4 versions were available. There were so few that I had to make one myself with AutoRound.

kaitchup/Qwen3.6-27B-autoround-nvfp4-linearattn-mtp-BF16
This is my version made with AutoRound. It uses NVFP4 while keeping the linear-attention and MTP layers in 16-bit. The model size is 28.6 GB.

Peutlefaire/Qwen3.6-27B-NVFP4
This version goes further and quantizes more linear_attn layers too. The model size is 20.6 GB, with a 19.7 GB main model file and an 849 MB MTP file. Compared with my AutoRound MTP BF16 version, that is roughly 8.0 GB smaller.

In my analysis, I showed that quantizing the attention path to NVFP4 significantly degraded the model. It did not break the model, but it made it more prone to endless thinking, and accuracy went down on nearly all benchmarks. Check the full analysis for details.

unsloth/Qwen3.6-27B-NVFP4
Unsloth also released an NVFP4 checkpoint. Its config applies NVFP4 quantization to most Linear layers, while leaving some of the linear-attention projections (linear_attn.out_proj) in BF16. You can see it as a version between mine and a full NVFP4 quantization. The model size is 26.4 GB.

nvidia/Qwen3.6-27B-NVFP4
The NVIDIA version is interesting because it is mixed precision: attention and linear-attention projections are FP8 W8A8, while the MLP layers and LM head are marked as W4A16_NVFP4. The checkpoint is 21.9 GB, which makes it 6.7 GB smaller than my 28.6 GB AutoRound MTP BF16 version and 4.5 GB smaller than Unsloth’s 26.4 GB version.

This also means that the NVFP4 parts of NVIDIA’s model do not use NVFP4 activations. They are W4A16: 4-bit NVFP4 weights, but 16-bit activations. The attention path, meanwhile, is FP8 rather than NVFP4. With Blackwell GPUs, this likely leaves speed on the table compared with W4A4 NVFP4. In my experience, NVFP4 activations can be almost harmless with good calibration.

Another important detail: NVIDIA’s config.json includes an FP8 KV-cache scheme. That matters a lot for speed and memory use, especially on devices where memory bandwidth is the bottleneck. I already saw some third-party speed comparisons on X comparing this checkpoint against other NVFP4 models without matching the KV-cache format, which is not an apples-to-apples comparison. At minimum, KV-cache dtype should be controlled explicitly across all models; vLLM, for example, can be run with an explicit FP8 KV cache.

NVIDIA also reports its own evaluation numbers against an FP8 baseline. In their table, the NVFP4 checkpoint is very close to the FP8 checkpoint across MMLU-Pro, GPQA Diamond, HLE, τ²-Bench, MMMU Pro, SciCode, AIME 2025, AA-LCR, and IFBench. This is useful, but it is still not a community-wide comparison against the other NVFP4 checkpoints under identical settings.

It is also worth mentioning the PrismaQuant family.

rdtand/Qwen3.6-27B-PrismaQuant-5.5bit-vllm
This version uses a sensitivity-driven per-Linear allocation. PrismaQuant chooses between NVFP4, MXFP8, and BF16 under a 5.5-bit-per-parameter target. According to the model card, bulk dense MLP layers are usually NVFP4 W4/A4, higher-sensitivity dense linears can be MXFP8 W8/A8, and the most sensitive tensors, norms, biases, embeddings, and LM head stay BF16.

The model size is 22.7 GB, which slighly larger to the one released by NVIDIA, probably due to the LM head staying at higher precision.

There are newer PrismaQuant-family variants too. PrismaAURA 5.5-bit uses an AURA/KL-Fisher allocation over NVFP4, FP8, and BF16, while PrismaSCOUT 5.31-bit is described as a Blackwell-oriented successor to the earlier 5.5-bit PrismaQuant artifact. PrismaSCOUT uses NVFP4 plus selected BF16, includes MTP tensors, and its repository is 20.2 GB.

How to Choose?

We do not yet have a good comparison. Quantization is cheap; evaluation is expensive.

What we need is a benchmark suite run on all these checkpoints with exactly the same settings: same inference engine, same chat template, same thinking mode, same KV-cache dtype, same context length, same sampling settings, same MTP/speculative-decoding settings, and same hardware.

Until then, I would assume the following:

• There is no strong reason to keep all activations in 16-bit on Blackwell if NVFP4 activation quantization is well calibrated.

• Keeping the attention path out of NVFP4 is safer.

• Unsloth’s version and my AutoRound version are safer choices if your priority is a more conservative quantization strategy.

• PrismaSCOUT and the other PrismaQuant-family models are worth testing, but I would not rank them without independent, apples-to-apples accuracy numbers. This could be the best version. We simply don’t know.

So, for now, I would use either Unsloth’s version or my AutoRound MTP BF16 version if accuracy safety is the main concern. They are also better documented and easier to reason about than many one-off community checkpoints. If memory footprint matters more, NVIDIA’s version and PrismaSCOUT are especially interesting.

DSpark: DeepSeek’s New Speculative Decoding Method

DSpark is a speculative decoding method for making LLMs generate faster without changing the final model that verifies the text.

Instead of asking the full model to produce one token at a time, DSpark, like any other speculative decoding method, lets a smaller draft module propose several future tokens, then asks the full model to verify those tokens in a batch. When the draft is right, the server accepts multiple tokens from one target-model pass. When the draft is wrong, the server keeps the longest valid prefix and continues from there.

The target model we want to accelerate remains the source of truth, while the drafter tries to guess what the target model is likely to accept next.

What DSpark Changes

The weakness of many speculative decoders is suffix decay. Drafting the first token is easy but drafting the fifth, sixth, or seventh token is much harder because every position depends on earlier guesses. Fully parallel drafters are fast, but later draft positions often become noisy. Autoregressive drafters are more faithful, but they give back too much of the latency advantage because they draft sequentially.

DSpark tries to sit between those two extremes. It keeps a DFlash-style parallel draft backbone, then adds a lightweight Markov logit-bias head. Note: I explained how DFlash works here:

That head lets each draft position condition on previously sampled tokens inside the draft block, which improves the later positions without turning the whole drafter into a slow autoregressive model.

The other important piece is confidence. DSpark adds a confidence head that predicts the acceptance probability of each draft position. That gives the serving system a practical scheduling signal. When the drafter is confident, the runtime can verify more proposed tokens. When confidence drops, or when the server is under load, it can avoid wasting verification work on tail tokens that are likely to be rejected. In other words, DSpark is not just a better drafter; it is a more serving-aware drafter.

How it works in the decoding loop

At each generation step, DSpark first prepares a block of draft tokens. The target model then scores that block in one verification pass. The runtime accepts the longest prefix that agrees with the target model’s sampling path and discards the rest. If several drafted tokens survive, the user sees several tokens produced for roughly one target-model step. If only the first token survives, the system behaves closer to normal decoding for that step.

The Markov head just helps the drafter keep later draft positions coherent. The confidence head helps the runtime decide how much speculation is worth attempting. Together, they attack the two practical bottlenecks of speculative decoding: low acceptance at later positions and wasted verification work under real serving load.

Speed results

DSpark improves per-user generation speed by about 60–85% over the previous MTP-1 baseline on DeepSeek-V4-Flash, and by about 57–78% on DeepSeek-V4-Pro at matched throughput. Offline accepted-length results also reportedly improve over Eagle3 and DFlash. As usual with speculative decoding, the exact gain depends on the prompt mix, sampling settings, hardware, batch pressure, and how often the drafter’s later tokens are accepted. Once widely available for other open-weight models, like Qwen3.6 and the larger Gemma 4, it will be interesting to see how fast is it compared with high MTP, like MTP-4 or 5 which are much faster than MTP-1 for these two models.

You can find more results in the technical report:

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

What DeepSeek released

The DeepSpec repository currently includes three draft-model algorithms: DSpark, DFlash, and Eagle3.

DeepSeek also released trained draft checkpoints for four target models: Qwen3-4B, Qwen3-8B, Qwen3-14B, and Gemma-4-12B-it. For each target, the repo lists Eagle3, DFlash, and DSpark checkpoints, so the open release covers twelve reference draft checkpoints.

DSpark checkpoints are also available for DeepSeek V4 Flash and Pro.

Running DSpark in vLLM

DSpark support is landing in vLLM through the speculative decoding path. The relevant vLLM PR for DSpark speculators-format checkpoint support was merged on July 2, 2026, so use a nightly build or a release that includes that change.

uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly

vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
  --tensor-parallel-size 8 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}'

Training your own DSpark speculator

You can train a DSpark speculator on your own data for Qwen3 and GLM 5.2 (only these architectures are support for now) using the speculators project. This is very similar to training a DFlash a speculator.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!