When I was Young…
There’s a good chance you are more tuned into the world of AI and LLMs than I. There’s also another good chance that you were not on holiday for most of June this year. I reckon though there’s probably an even chance that you were not avidly following ML/AI research papers back in November 2022.
Why November 2022 specifically? That predates the mass hysteria about AI (read LLMs) courtesy of OpenAI and ChatGPT. What is important about November 2022?
Speculative Decoding
Now, I won’t pretend I can speak in detail about Speculative Decoding. I am, however, a key admirer of those that exemplify the Dunning-Kruger Effect. Here’s my attempt to join the DKE club.
In 2022 it was clear that Next Token Prediction (NTP) was sub-optimal. The process of consistently bottleneck compute by loading weights in-and-out of memory was identified as an architectural challenge of the Transformer architecture. Those savvy folk at Google presented a solution, a smaller ‘drafter’ model that sat alongside the larger model and guessed 3-5 (k=3/4/5) tokens ahead. If the guess was right, happy days! Efficiencies gained! If incorrect, there was little negative consequence as NTP still occurred regardless.
Yes, I used AI to create the above image.
The speed up here is found in tuning the ‘k value’, the token lookahead value, to the appropriate sweet spot. Too small and you’re leaving performance on the table, too high and you’re using up too much compute while risking increased waste - if the Drafter went wrong at lookahead token #2 tokens #3-10 were a waste to compute.
Multi-Token Prediction (MTP)
Zoom ahead 18-24 months later and Meta released a paper looking at Multi-Token Prediction in earnest. The premise of the paper being that performance can be increased by taking that speculative decoding concept and baking it into the model. This approach requires a change in how models are trained. Models were trained with NTP, but achieves Speculative Decoding without the drafter.
The evolution required LLMs to consider n tokens ahead of the current token, t, with branches off t. The sweet spot was found at n=4 and allowing the LLM to branch off so you have:
t + 1 | t + 2 | t + 3 | t + 4
One can look at that concept and immediately see how this could speed up token generation. If the LLM generates both the immediate token and the next 4 tokens that’s massive compounding savings. A more interesting benefit was that this approach improved model performance on benchmarks via improved reasoning and planning. Predicting ahead of the immediate token forced models to build better long-term context.
This was a win-win, not only does the model improve in performance benchmarks, but inference speed is also improved.
Today?
MTP models are increasingly appearing as of June 2026. We are seeing models from DeepSeek, Alibaba, and Google release with MTP architectures at their core. The models are not the whole story. Runtimes and engines must be available to leverage the architecture shift. Fortunately, major runtimes and engines are rapidly delivering support for MTP. vLLM, Ollama, llama.cpp are all ready to serve you with MTP enabled models and the associated gains.
It’s worth noting that trying to run MTP with Mixture of Expert (MoE) models is not a great mix. This is primarily because each increment on t can lead to different experts being activated. When different experts are called for the same contextual thread the results are inconsistent, often falling back to NTP. Ditch the MoE models, go for Dense MTP models, and profit.
The Times They Are A-Changing
The next post will look at how things are changing in the local world beyond architectural advancements such as the above. I have never been more excited by local AI. We are starting to see Open Source models reach major lab levels of capability. Huge models with multiple trillion parameters available for download and running. Truly local models, seriously exciting stuff, that can run on an iPhone.