The interesting part of today's hot r/LocalLLaMA thread is not that someone posted another speedup chart.

It is that llama.cpp is starting to absorb one of the tricks that people usually associate with heavier serving stacks. A new beta pull request adds MTP, short for multi-token prediction, to llama.cpp's speculative decoding path. If it lands cleanly, the local stack stops looking like the place where good ideas arrive last.

That is a bigger shift than it sounds.

For the past year, the usual story around local inference has been familiar: open weights get better, quantization gets better, kernels get better, but the nicest serving features tend to show up first in systems like vLLM. llama.cpp remains the practical workhorse because it runs almost everywhere, not because it reaches feature parity first. This PR matters because it attacks that division directly.

What is actually verified

The Reddit thread points to an open pull request in ggml-org/llama.cpp titled "llama + spec: MTP Support", created on May 4. The PR is not merged. It is large, around 1,500 additions across 40 changed files, including common/speculative.cpp, convert_hf_to_gguf.py, and multiple core llama source files.

In the PR description, contributor Aman says the implementation adds support for MTP heads, tested on Qwen3.6-27B and Qwen3.6-35BA3B. The design described there matters. Instead of requiring a completely separate draft model package, the MTP path loads a separate MTP model from the same GGUF and gives it its own context and KV cache. That keeps the feature closer to the model's native structure than the more familiar "small external draft model helps a larger model" setup.

The PR also includes benchmark tables. On the author's reported DGX Spark runs, a baseline Qwen3.6-27B setup sits around 7 tokens per second. With --spec-type mtp and --spec-draft-n-max 3, several prompt classes move into the mid-to-high teens, with one code case at 21.6 tok/s and aggregate wall time dropping from 201.07 seconds to 83.8 seconds across the posted nine-request batch. With --spec-draft-n-max 2, aggregate acceptance rises further, though absolute throughput shifts by workload.

Those numbers are worth reading, but they are still author-reported benchmark results inside an open PR. They are evidence of serious progress, not a settled new baseline for everyone running llama.cpp.

There is also outside corroboration that the broader feature category is real and maturing. vLLM's current documentation has a dedicated page for MTP (Multi-Token Prediction). It describes MTP as a speculative decoding method where the target model already contains native multi-token prediction capability, so no separate draft model is required. The docs also make an important limitation explicit: MTP only works for model families that support it in vLLM, and a small num_speculative_tokens value such as 1 is the recommended starting point.

That point is easy to miss in the Reddit excitement. MTP is not a universal free lunch you can switch on for any GGUF lying around. It depends on model support, conversion support, backend support, and enough systems work to make the extra path worthwhile.

Why this is more interesting than the benchmark headline

The headline version of this story is simple: llama.cpp might get faster token generation on MTP-capable models.

The stronger version is that local inference is getting less structurally behind.

There are two ways to speed up generation. One is to make the same old path faster through kernels, batching, quantization, and memory handling. The other is to change the shape of the work so the large model does less fully serial next-token labor. MTP belongs to the second camp. It is not just a cleaner CUDA graph or a better quant. It changes how much useful work the stack can extract from one expensive pass through the model.

That is why this thread landed well on Reddit. The better comments are not reacting like this is a novelty patch. They are reacting like a missing piece is finally arriving. One commenter on the PR said this should "massively bridge the TG gap with vLLM," and the Reddit thread itself quickly turned into a discussion of what MTP means for actual users, not just benchmark chasers.

That mood tells you something real. Local model users have spent enough time comparing llama.cpp against more feature-rich serving systems that they know exactly where the pain points are. They are not impressed by abstract architecture names. They care when one of those names turns into a flag they might plausibly run.

The caveats are the story too

The PR discussion is also a good reminder that feature arrival is not the same thing as feature maturity.

A reviewer asked whether earlier MTP attempts had leaned too much on host-to-device copies. Aman replied that tensor sharing between two llama contexts is part of the design discussion. Another commenter asked about extra memory use on constrained systems. The answer from the PR author was that the current implementation is opt-in via --spec-type mtp, and that memory overhead should stay under roughly 10 percent of overall memory use because the MTP component is much lighter than a full draft model.

That is encouraging, but not universal proof. Memory pressure is precisely where local inference claims go to die.

There are backend limits too. One user reported garbage output on Vulkan with very low draft acceptance. Aman responded that the PR currently relies on another change, #22400, which is not implemented for Vulkan yet, and that testing had so far focused on a small number of CUDA devices. That matters. A feature is not truly a llama.cpp feature in the social sense until it survives the messy zoo of hardware and backend combinations that people actually run.

So the cleanest reading is this: the idea is real, the implementation is substantial, the early numbers are promising, and the portability story is still in progress.

What people are missing

People keep talking about open models as if the contest is mostly about weights. It is not.

The practical contest is model plus runtime plus conversion plus backend behavior. Qwen3.6 helped expose that because it gave the local community a model family worth pushing hard. This MTP beta is the next step in the same arc. Once the model has the right structure, the question becomes whether the runtime stack can stop leaving performance on the table.

That is why this PR is more important than a normal "look, faster tokens" post. It suggests the local ecosystem is beginning to integrate model-native acceleration features instead of waiting for centralized inference stacks to normalize them first.

If that continues, the gap between "good open weights" and "good local experience" gets narrower for reasons that are architectural, not cosmetic.

What remains uncertain

Several important pieces are still unsettled.

First, the PR is open, not merged. The implementation can change, regress, or stall.

Second, the benchmark tables are from the contributor, not an independent multi-hardware bakeoff.

Third, backend support is incomplete. The public discussion already shows at least one rough edge on Vulkan.

Fourth, MTP support is model-specific. vLLM's own docs underline that point, and llama.cpp will face the same reality even if the core plumbing lands.

So no, this does not mean every local model suddenly got a 2x speedup today.

It does mean one of the most important local runtimes is moving closer to the place where these gains can become normal instead of exotic.

Practical takeaway

If you run llama.cpp, the thing to watch is not just whether this specific PR merges. Watch whether MTP becomes easy to convert, easy to enable, and boringly reliable across the hardware people actually own.

That is when the story changes.

The local AI stack does not need more theoretical reasons to be fast. It needs more features that survive contact with ordinary machines. This beta looks like one of the better signs that the gap is no longer mostly conceptual.

Sources