MoE on KnightLi Blog

LLM Architecture Evolution from 2023 to 2026: Tokenizers, Positional Encoding, Attention, MoE, Normalization, and Activation Functions

Sun, 17 May 2026 08:53:29 +0800

From 2023 to 2026, LLM architecture seemed to change in many directions: tokenizers got larger, positional encoding shifted toward RoPE, attention evolved from MHA to GQA, sliding windows, and MLA, MoE became mainstream again, and normalization and activation functions moved toward combinations like RMSNorm and SwiGLU.

But the main story is not that Transformer was overturned. The main story is that the Transformer core stayed in place, while almost every component around it was optimized for longer context, lower inference cost, higher training efficiency, and stronger multilingual capability.

Start with the Big Picture

An LLM can be roughly divided into several parts:

Tokenizer: turns text into tokens the model can understand.
Positional encoding: tells the model where each token is in the sequence.
Attention mechanism: decides which context each token should look at.
Feed-forward network: applies more complex nonlinear transformations at each position.
Normalization: keeps training more stable.
Activation function: gives the network nonlinear expressive power.
MoE: splits part of the feed-forward network into multiple experts and activates only a few at a time.

The 2023-2026 evolution is basically these components being optimized one by one.

Tokenizers: From “Can Split Text” to “Uses Fewer Tokens”

The tokenizer turns natural language into token sequences. The model does not see text directly; it sees token IDs.

Earlier tokenizers were often more efficient for English and less efficient for Chinese, code, and multilingual text. If the same sentence is split into too many small pieces, it consumes more context window and increases both training and inference cost.

One clear trend in recent years is larger vocabularies and better multilingual support. Llama 3 uses a 128K-token vocabulary, and Meta explicitly says this encodes language more efficiently and improves model performance. Qwen, DeepSeek, and other models also pay close attention to token efficiency for Chinese, code, and multilingual scenarios.

For beginners, think of it this way: the better the tokenizer, the less fragmented the same text becomes, and the more useful information the model can fit into the same context length.

Positional Encoding: RoPE Became Mainstream

Language has order. “Dog bites man” and “man bites dog” contain similar words, but the order changes the meaning. Positional encoding injects that order information into the model.

Early Transformers used absolute positional encodings, where position 1, position 2, and position 3 each had their own vector. Later LLMs more often used RoPE, or Rotary Positional Embedding. RoPE integrates position information into attention computation and is friendlier to long-context extension.

From the Llama family to many open models, RoPE has become one of the de facto standards. To support longer context, models may also adjust the RoPE base frequency, apply RoPE scaling, or combine it with sliding-window or chunked attention.

Simply put, RoPE does not make a model “suddenly smarter,” but it helps the model handle relative position relationships better in longer text.

Attention: From MHA to GQA, Sliding Windows, and MLA

Attention is the core of Transformer. It lets each token look at the most relevant tokens in the context for the current task.

The classic version is MHA, or Multi-Head Attention. It has multiple attention heads, each learning a different way to focus. The problem is that as models and contexts grow, KV cache becomes expensive and inference cost rises.

After 2023, the main direction of attention optimization was reducing inference cost.

GQA, or Grouped-Query Attention, is an important step. It lets multiple query heads share fewer key/value heads, reducing KV cache pressure. Meta explicitly adopted GQA in Llama 3 to improve inference efficiency.

Mistral 7B represents another direction: sliding-window attention. It does not require every token to attend to the entire history, but focuses mainly on a nearby window, reducing long-sequence computation pressure. For many tasks, local context already carries much of the useful information.

DeepSeek-V2/V3 pushed attention optimization further with MLA, or Multi-head Latent Attention. Its focus is compressing KV cache to reduce inference memory pressure. The DeepSeek-V3 technical report lists MLA and DeepSeekMoE as core architectural features.

You can understand these methods together:

MHA: the classic approach, strong but expensive.
GQA: greatly reduces KV cache cost with little loss in expressiveness.
Sliding-window attention: reduces full-attention cost in long context.
MLA: further compresses attention cache for efficient inference.

MoE: Many Parameters, but Only Some Are Used Each Time

MoE means Mixture of Experts.

A normal dense model activates all parameters for every token. MoE puts many experts inside the model, but routes each token to only a few experts. This allows the total parameter count to be large while the number of active parameters per inference step stays smaller.

Mixtral 8x7B, released at the end of 2023, was an important moment that brought MoE back into broad attention. Mistral’s paper explains that Mixtral 8x7B largely follows the Mistral 7B architecture, but replaces each feed-forward block with 8 experts and uses sparse routing to select part of them for computation.

DeepSeek-V3 later made MoE a core route. It has a very large total parameter count, but activates only a subset for each token, using DeepSeekMoE to reduce training and inference cost. Qwen3 and other model families also provide both dense and MoE variants, showing that MoE has moved from a research trick to a mainstream engineering option.

For beginners, a dense model is like a company where everyone attends every meeting. MoE is like dividing the company into expert teams and calling only the most relevant teams for each problem.

MoE also has clear difficulties:

The router must learn to send tokens to suitable experts.
Expert load must be balanced, so not all tokens crowd into a few experts.
Distributed training and inference become more complex.
Large total parameters do not automatically make deployment cheap.

Normalization: RMSNorm Became Common

Normalization stabilizes the distribution of intermediate values inside the neural network. When training large models, unstable values make convergence harder and training less reliable.

Early Transformers commonly used LayerNorm. Many Llama-style models later switched to RMSNorm. RMSNorm is simpler than LayerNorm: it does not compute the mean and focuses on root-mean-square scaling. It is lighter and stable enough in practice.

You do not need to memorize the formula. Just remember that RMSNorm is a lighter stabilizer. It does not determine model capability by itself, but it affects training stability, speed, and engineering implementation.

Activation Functions: From ReLU/GELU to SwiGLU

Activation functions add nonlinear capability to neural networks. Without them, a deep network would easily collapse into a linear transformation.

Earlier Transformers often used GELU. In modern LLMs such as Llama, Mistral, Qwen, and DeepSeek, SwiGLU or similar GLU variants are more common. SwiGLU usually appears inside the feed-forward network and controls information flow through a gating mechanism.

A rough analogy: a normal activation function is like a fixed switch, while SwiGLU is more like a learnable valve. It does not just decide whether information passes through; it can learn which information should be amplified.

SwiGLU makes the feed-forward layer slightly more complex, but in large-model practice it has become a common high-performance component.

The Overall Trend from 2023 to 2026

The timeline can be summarized like this:

2023: Llama, Mistral 7B, Mixtral, and other open models popularized combinations such as RoPE, RMSNorm, SwiGLU, GQA, sliding-window attention, and MoE.
2024: Llama 3, Qwen2.5, DeepSeek-V2/V3, and others expanded vocabularies, improved long context, strengthened inference efficiency, and made MoE and efficient attention central topics.
2025: DeepSeek-V3/R1 made more people pay attention to MLA, DeepSeekMoE, FP8, MTP, and the deep connection between architecture optimization and system engineering.
2026: The trend remains efficiency and engineering maturity: dense models continue to pursue stable general capability, MoE models expand capacity, and efficient attention reduces long-context cost.

The most important change was not one component replacing Transformer. It was the realization that adding parameters alone is not enough: architecture, data, training systems, and inference services must be optimized together.

How Beginners Should Learn This

If you are starting from zero, do not begin by forcing yourself through every paper. A better order is:

Understand the basic Transformer structure: tokens, embeddings, attention, and FFN.
Learn why RoPE, RMSNorm, and SwiGLU became common.
Study GQA and KV cache to understand why inference consumes so much memory.
Learn MoE, focusing on the difference between total parameters and active parameters.
Finally, read model reports such as DeepSeek-V3, Mixtral, and Llama 3 to place these components back into real models.

Do not treat these terms as isolated facts. Most of them answer the same question: how can models become stronger while remaining trainable, deployable, and fast enough to serve?

Summary

The 2023-2026 evolution of LLM architecture can be seen as the engineering maturation of Transformer. Tokenizers reduce token waste, RoPE represents position more effectively, GQA, sliding-window attention, and MLA reduce attention cost, MoE expands capacity while controlling active computation, and RMSNorm plus SwiGLU make training and representation more stable and efficient.

For beginners, the key is not memorizing terms. The key is understanding the main tradeoff: almost every modern LLM architecture change is about cost, efficiency, context length, and scalability.

References:

Running DeepSeek V4 Locally: VRAM Estimates for Pro, Flash, and Base Versions

Fri, 01 May 2026 11:55:25 +0800

DeepSeek V4 and Gemma 4 are not in the same class for local deployment. With Gemma 4, it still makes sense to discuss how to run 26B or 31B models on 24GB or 32GB GPUs. DeepSeek V4 is a huge MoE model, and full local deployment quickly moves into multi-GPU workstation or server territory.

The official DeepSeek V4 Preview release mainly includes two inference models:

DeepSeek-V4-Pro: 1.6T total / 49B active params
DeepSeek-V4-Flash: 284B total / 13B active params

The official Hugging Face collection also includes two Base models:

DeepSeek-V4-Pro-Base
DeepSeek-V4-Flash-Base

This article only discusses rough VRAM requirements when the full model weights are loaded. For MoE models, active params mainly affects per-token compute. It does not mean only those parameters need to be loaded. Without expert-on-demand loading, CPU/NVMe offload, distributed inference, or specialized runtime optimizations, VRAM should still be estimated from the full weight size.

Quick Summary

VRAM Scale	What Is Realistic	Do Not Expect
24GB	Cannot fully run DeepSeek V4; use smaller distilled models or API	Full V4-Flash / V4-Pro local loading
48GB	Still not suitable for full loading; good for small models or remote API clients	Stable V4-Flash Q4
80GB	Theoretically try V4-Flash Q2/Q3 or heavy offload	V4-Pro
128GB	V4-Flash Q4 becomes more realistic; Q5/Q6 still tight	V4-Pro Q4
192GB	V4-Flash FP8/Q6 is more comfortable; Pro Q2 enters experimental range	V4-Pro Q4
256GB	V4-Flash FP8 is fairly comfortable; Pro Q2/Q3 can be tested	V4-Pro Q5 and above
512GB	V4-Pro Q4 starts to become discussable	V4-Pro FP8
1TB+	V4-Pro FP8 and low-bit Pro-Base are more realistic	Low-cost single-machine deployment
2TB+	Pro-Base FP8 class	Ordinary workstation deployment

If your goal is to run a model on a personal computer, DeepSeek V4 is not the right target. More realistic options are:

Use the official DeepSeek API or compatible services.
Wait for stable community GGUF/EXL2/MLX quantizations and inference support.
Use smaller DeepSeek distilled models.
Use local models in the 7B to 70B range from Qwen, Gemma, Llama, and similar families.

Official Weight Sizes

The following figures come from model.safetensors.index.json in the official Hugging Face repositories. They reflect current public weight file sizes, not full runtime VRAM use under long context.

Model	Parameter Scale	Official Weight Size	Notes
`DeepSeek-V4-Flash`	284B total / 13B active	159.61GB	Inference model, smallest in this group
`DeepSeek-V4-Pro`	1.6T total / 49B active	864.70GB	Inference model, stronger but enormous
`DeepSeek-V4-Flash-Base`	284B total	294.67GB	Base model, closer to full FP8 weight size
`DeepSeek-V4-Pro-Base`	1.6T total	1606.03GB	Base model, about 1.6TB

Even the smallest V4-Flash is already close to 160GB of official weights. That is why it should not be treated like a 13B model just because it has 13B active params.

DeepSeek V4 Flash VRAM Estimate

V4-Flash is the most approachable DeepSeek V4 variant for local experiments. But that only means “more approachable than Pro”; it is still not a consumer single-GPU model.

The table below uses the official 159.61GB weight size as the baseline. Q4/Q3/Q2 rows are bit-width estimates and do not imply that stable official GGUF versions currently exist.

Version / Quantization	Estimated Weight Size	Minimum VRAM	Safer VRAM	Best For
`FP8 / official weights`	159.61GB	192GB	256GB	Multi-GPU servers, inference service
`Q6`	120GB	160GB	192GB	Quality-first quantization tests
`Q5`	100GB	128GB	160GB	Quality/size balance
`Q4`	80GB	96GB	128GB	More realistic starting point for Flash
`Q3`	60GB	80GB	96GB	Large-VRAM single GPU or multi-GPU tests
`Q2`	40GB	48GB	64GB	Extreme low-bit experiments with clear quality risk

If mature V4-Flash Q4 builds appear later, it still probably will not be a 24GB GPU model. A more realistic starting point is 96GB to 128GB total VRAM, or CPU/offload setups that trade speed for capacity.

DeepSeek V4 Pro VRAM Estimate

V4-Pro is the flagship inference model, with official weights around 864.70GB. Even at 4-bit quantization, the full weights remain in the hundreds of GB.

Version / Quantization	Estimated Weight Size	Minimum VRAM	Safer VRAM	Best For
`FP8 / official weights`	864.70GB	1TB	1.2TB+	Multi-node or multi-GPU inference service
`Q6`	648GB	768GB	1TB	High-quality quantized service
`Q5`	540GB	640GB	768GB	Quality/cost balance
`Q4`	432GB	512GB	640GB	Lowest practical quality line for Pro
`Q3`	324GB	384GB	512GB	Low-bit experiments
`Q2`	216GB	256GB	320GB	Extreme experiments with high quality and stability risk

For individual users, V4-Pro is better consumed through an API. If the goal is full local deployment, treat it as a multi-GPU server model, not a 4090, 5090, or RTX PRO single-GPU model.

DeepSeek V4 Flash-Base VRAM Estimate

Base models are usually for research, fine-tuning, or continued training, not ordinary chat deployment. V4-Flash-Base has official weights of about 294.67GB.

Version / Quantization	Estimated Weight Size	Minimum VRAM	Safer VRAM	Best For
`FP8 / official weights`	294.67GB	384GB	512GB	Research, preprocessing, evaluation
`Q6`	221GB	256GB	320GB	High-quality quantization research
`Q5`	184GB	224GB	256GB	Quality/size balance
`Q4`	147GB	192GB	224GB	Lower-cost Base experiments
`Q3`	111GB	128GB	160GB	Low-bit experiments
`Q2`	74GB	96GB	128GB	Extreme experiments

If you only want to use DeepSeek V4 capabilities, do not start with the Base model. Base models cost more to deploy and tune; most applications should use the inference model or API.

DeepSeek V4 Pro-Base VRAM Estimate

V4-Pro-Base is the heaviest variant, with official weights around 1606.03GB. That is already a 1.6TB-class model file.

Version / Quantization	Estimated Weight Size	Minimum VRAM	Safer VRAM	Best For
`FP8 / official weights`	1606.03GB	2TB	2.4TB+	Large-scale research clusters
`Q6`	1205GB	1.5TB	2TB	High-quality quantization research
`Q5`	1004GB	1.2TB	1.5TB	Research and evaluation
`Q4`	803GB	1TB	1.2TB	Low-bit research
`Q3`	602GB	768GB	1TB	Extreme low-bit research
`Q2`	402GB	512GB	640GB	Extreme experiments

This kind of model should not be discussed in the framework of “can a home GPU run it?” Even Q4 is already beyond the comfortable range of most single-machine workstations.

Why Active Params Are Not Enough

DeepSeek V4 is an MoE model. MoE means each token activates only part of the experts, so compute is much lower than the total parameter count. But this does not mean VRAM only needs to hold the active parameters.

Full local inference also depends on:

Whether all expert weights must stay resident on GPU.
Whether on-demand expert loading is supported.
CPU memory to GPU memory transfer costs.
NVMe offload latency.
KV cache growth under long context.
Extra runtime overhead under 1M context.
Multi-node and multi-GPU communication cost.

So V4-Pro with 49B active should not be deployed like a 49B model. V4-Flash with 13B active should not be treated like a 13B small model either.

How to Choose

If you are an ordinary individual user:

Do not try to fully self-host DeepSeek V4.
Use the official API when you need DeepSeek V4 capabilities.
For private local deployment, first check whether you have mature inference infrastructure or internal multi-GPU servers.
With only 24GB to 48GB VRAM, 7B, 14B, 32B, or 70B quantized models are more practical.

If you have 128GB to 256GB total VRAM:

Watch for stable community implementations of V4-Flash Q4/Q5.
Do not treat V4-Pro as your main local model.

If you have 512GB+ total VRAM:

V4-Pro Q4 starts to become an engineering validation target.
You still need to care about inference framework support, expert scheduling, KV cache, throughput, and concurrency.

The key question for DeepSeek V4 local deployment is not “which quantized file should I download?” It is “do I have the system-level inference capacity for this model?” It is closer to a server model than a desktop model.