Inside DeepSeek V4.1: 763 Billion Numbers, 8 Billion at a Time

The short version

DeepSeek V4.1 is one model: DeepSeek-V4.1-Flash. It came out in September 2026, and a weight update, DeepSeek-V4.1-Flash-1017, followed on October 17 with the same architecture. No V4.1-Pro has been released. The open weights are MIT licensed, and the API name deepseek-flash now runs the 1017 weights.

It holds 763 billion numbers on disk. When it reads your prompt, each token touches about 8 billion of them. When it writes, each new token touches about 16 billion. It reads images natively, and it keeps up to a million tokens of context.

The headline of the paper is not size or scores. It is memory. Every token you give it costs 890 bytes of fast GPU memory to remember. That is about a quarter of V4-Flash, and 437 times less than DeepSeek's first model in 2023. A full million-token prompt fits in less than 1 GB.

Four ideas make that work, and they are what this post is about:

• The 40 layers are split into a causal encoder and a decoder, and the prompt mostly runs through only the first half.
• Attention is sparse and shared: each new token looks at a fixed 640 places, not the whole million, and most layers borrow what an earlier layer already computed.
• Engram is a 196-billion-number lookup table of short word sequences, read by hashing, not by thinking.
• DSpark is a small drafter on top that guesses five tokens at once, and a scheduler that decides how many of them are worth checking.

"Engram" and "DSpark" are both real DeepSeek names, and both are in V4.1. DSpark is not sparse attention, though the name sounds like it. It is speculative decoding. The sparse attention has its own name: CSA2.

What V4.1 came from

DeepSeek V4 came out in April 2026 in two sizes. V4-Pro had 1.6 trillion parameters, 49 billion active per token. V4-Flash had 284 billion, 13 billion active. Both read a million tokens, using a mix of two attention types, CSA and HCA, that squeeze old tokens into fewer cache entries. Both had a new kind of residual connection called mHC. Both were text only.

V4.1-Flash keeps the family's habits: mixture of experts, a sliding window in every layer, compressed sparse attention, mHC, and the Muon optimizer. Then it changes almost everything else. It is bigger than V4-Flash on disk (552 billion backbone parameters against 284) but lighter per token (8 or 16 billion against 13). It drops HCA and uses one attention type everywhere. It drops the multi-token prediction module that V3 and V4 trained with. It adds images, Engram and DSpark.

Its base model scores about like V4-Pro's base on knowledge, reasoning and code, with a third of the parameters and a quarter of the active ones.

How big it really is

The 763 billion on the Hugging Face page is everything in the checkpoint. DeepSeek counts it differently, and both are right once you split it.

The backbone is everything a token can flow through in the 40 layers: experts, attention, embeddings. That is 552 billion. Engram's tables are another 196 billion, and they are a different kind of parameter: they are never multiplied, only looked up. The drafter and the vision encoder are small next to them.

One token's trip

Here is the whole path, before we zoom in.

Text becomes token ids, from a vocabulary of 129,280. An image becomes a grid of vision tokens (more below). Each token becomes a vector of 5,120 numbers. Then that vector is copied four times. V4.1 does not carry one residual stream through the layers, it carries four, and learns how to mix them at every layer. That is mHC.

Then come 40 layers. Each has attention and a mixture-of-experts feed-forward block. Two of them, layers 1 and 14 (counting from 0), first get a write from Engram. The first 20 layers are the encoder. The last 20 are the decoder. After layer 39, the four streams collapse back into one, and the output head turns it into a score for every one of the 129,280 tokens.

While that happens, the drafter on top reads the inputs of layers 37, 38 and 39 and guesses the next five tokens, so the big model can check them all in one go.

The encoder and the decoder

Agents send long prompts again and again: every tool call adds a little text to a huge conversation. So most of the work is reading, not writing. DeepSeek calls reading the prompt "prefill". V4.1 is built to make prefill cheap.

The idea comes from a 2024 paper called YOCO. Split the layers in two. The lower half, the encoder, works like any causal transformer. The upper half, the decoder, does not build its own long-range memory from its own hidden states. It builds it from the encoder's last output. DeepSeek calls this a Causal Encoder-Decoder, CED. "Encoder" can mislead here: both halves are causal, and every token still only sees the past.

Why this saves work: to read a prompt, you need the memory (the KV cache) of every layer, for every prompt token. In a normal model, layer 39's memory needs layer 38's output, which needs layer 37's, all the way down. So every prompt token must go through all 40 layers. In V4.1, the decoder's long-range memory is a projection of layer 19's output. So a prompt token goes through 20 layers, gets projected once, and is done. That is why a prompt token costs about 8 billion parameters and an output token about 16.

There is one catch. Every layer also has a sliding window: it looks at the last 128 tokens in full detail, and that part does come from each layer's own hidden state. So the decoder's window memory can't be skipped. V4.1's answer: run the decoder only on the last 128 tokens of the prompt. That is enough to start writing. DeepSeek calls this Decoder SWA Bounded Replay. The result is not exactly what a full run would give, and they say the quality cost is negligible, and they also train with the same shortcut so the model gets used to it.

Attention that looks at 640 places

A token in a million-token context can't look at a million earlier tokens. It would be too slow, and the memory would not fit. V4.1's attention, CSA2 (Compressed Sparse Attention 2), gives each new token a fixed budget.

Every layer has two sources it can read:

• The window. The last 128 tokens, in full detail, stored in FP8. Every layer has its own window.
• The long-range memory. All older tokens, compressed. In the encoder, every 2 tokens are pooled into one 512-number entry with a learned softmax weight. In the decoder, entries are not pooled: one per token.

Then a small helper, the indexer, decides which 512 long-range entries this token should read. The indexer is a cheap attention of its own: 32 heads of 128 numbers in FP4, one key per entry. It scores every entry and keeps the top 512. The main attention then reads those 512 plus the 128 in the window.

So the main attention always reads 640 places, whether the context is 4,000 tokens or a million. The indexer still scans everything, but it is much cheaper per entry than the real attention.

The main attention itself is the DeepSeek style: 64 query heads, each 512 wide, but only one key-value vector per place, shared by all heads. Queries come through a narrow 1,280-wide bottleneck. Outputs go back through 8 groups of a 1,024-wide bottleneck. Each head also has a learned "sink", a place to put attention when nothing is worth reading.

Most layers borrow

The window and the 640-place budget fix compute. They don't fix memory: if every layer kept its own long-range cache, a million tokens times 40 layers would still be huge. So V4.1 makes most layers borrow.

Each of the 38 CSA2 layers has one of three fixed modes:

• Full. The layer builds its own long-range cache, builds the indexer's keys from it, and runs its own indexer to pick its 512.
• Reindex. The layer borrows the cache and the indexer keys from the last Full layer, but runs its own indexer query. So it reads the same memory, but can pick different entries.
• Reuse. The layer borrows the cache and also the last picks. It runs no indexer at all.

Every layer still makes its own query and its own window. Only the long-range part is shared.

The layout is regular. Layers 0 and 1 only have the window. Layers 2 to 19 form three groups of six: one Full layer, then five Reuse layers. Layers 20 to 39 form five groups of four. The first group starts with a Full layer, the other four start with a Reindex layer, and the rest are Reuse.

So in the whole model, only four layers keep a long-range cache: 2, 8 and 14 in the encoder, and 20 in the decoder. Layer 20 is the one that builds its cache from the encoder's output, which is the CED projection from the last section. Eight layers run an indexer. The other 30 just read.

The decoder has one more trick, the Hierarchical Sparse Indexer. Layer 20, the first Full layer of the decoder, scores every entry anyway. While it does, it also groups the entries into blocks of 8, takes the best score in each block, and keeps the top 2,048 blocks. That is a pool of 16,384 candidates. The Reindex layers above it score only those candidates, not the whole context. So their cost stops growing with context length. The block holding the newest tokens is always kept in the pool, so a full old block can't push out the newest text.

890 bytes

Now the payoff. Count the bytes one token leaves behind in the long-range cache, which has to sit in GPU memory the whole time.

A long-range entry is 512 numbers. V4.1 stores it in FP4: 4 bits a number, so 256 bytes, plus one 8-bit scale for every 16 numbers, 32 bytes. That is 288 bytes an entry. An indexer key is 128 numbers in FP4 with one scale per 32: 64 + 4 = 68 bytes.

The encoder has three caches, each holding one entry per 2 tokens: 3 x 288 / 2 = 432 bytes per token. The decoder has one, with one entry per token: 288. Indexer keys add 3 x 68 / 2 + 68 = 170. Total: 432 + 288 + 170 = 890 bytes. It matches DeepSeek's number to the byte.

The FP4 cache is new in V4.1. The model is trained for it (quantization-aware training in post-training), and DeepSeek shows why no global scale is needed: after normalization and RoPE, no number in a 512-wide entry can go above about 22.6, and in practice they stay under 10. The window cache stays FP8, because it is more sensitive.

When the cache leaves the GPU

There is a second memory problem. DeepSeek keeps the caches of recent conversations on SSD for at least 72 hours, so your next message in the same chat does not have to reread everything. In V4, about half of that stored cache was window cache, which is only useful for a few minutes, until the next turn starts.

V4.1 stops storing the window cache on SSD. It keeps it only in a small pool in host memory, 10% of each machine's RAM, for minutes. If a request comes back after that, the long-range cache is still there, but the window cache is gone. Rebuilding it exactly would mean rerunning 40 layers x 128 tokens = 5,120 tokens, because each layer's window depends on the window below it. V4.1 replays only the last 128 tokens and accepts a slightly different state. DeepSeek calls this Encoder SWA Bounded Replay.

Together with the 890-byte cache, this cuts the stored cache to about 1/8 of V4-Flash.

Mixture of experts

Each of the 40 layers has 385 small feed-forward networks, called experts. One is shared: every token goes through it. The other 384 are routed: a small router picks 6 of them for each token. Each expert is a SwiGLU block 2,304 wide, about 35 million parameters, stored in FP4.

The router is a single matrix: 384 scores from the token's 5,120 numbers. V4.1 turns each score into sqrt(softplus(score)), which keeps them positive without the "winner takes all" of a softmax. Then it adds a bias to each expert's score and takes the top 6. The bias exists only to keep the load even: an expert that gets too many tokens has its bias lowered after each training step, one that gets too few has it raised. This is DeepSeek's auxiliary-loss-free balancing. The bias picks the experts, but it does not weigh them: the weights come from the raw scores, normalized to sum to 1, then times 1.5.

One new detail in V4.1: images and text have separate biases. Image tokens and text tokens like different experts, and balancing them together would hide an image-only overload. So each modality balances on its own.

The activation is clamped at 10 inside each expert, a trick to keep FP8 and FP4 values in range.

Four streams instead of one

A transformer normally has one residual stream: each layer reads it and adds its output back. mHC, from DeepSeek's V4, keeps four. Before each block, a small learned mix A says how much of each stream goes into the block. After the block, a mix C says how much of the output goes into each stream. And a 4 x 4 matrix B shuffles the streams among themselves. All three are computed fresh for each token from the streams themselves. B is pushed to be doubly stochastic (every row and column sums to 1) with 20 Sinkhorn steps, so signals neither blow up nor die across 40 layers.

V4.1's change, Single-Pass mHC, is about speed. In V4, a block had to finish computing its mix A before it could start reading its input, which forced a second pass over memory. V4.1 uses the A computed by the previous block. That removes the wait, so one GPU kernel can read the streams once and write them once. DeepSeek says this halves the memory traffic of this step and costs almost no quality.

Engram: a memory you look up

Some of what a language model knows is not reasoning at all. "Alexander the" is followed by "Great". "import numpy as" is followed by "np". A normal model learns these with its early layers, by computing. Engram, from a DeepSeek paper in January 2026, gives the model a giant lookup table for them instead.

It works like this for every token:

1. Normalize. Map the token id through a fixed table that merges ids that mean the same text after lowercasing, Unicode normalization and accent removal. "The", " the" and "THE" become one id. V4.1's 129,280 ids collapse to 99,092, 23% fewer.
2. Make n-grams. Take the last 2, last 3 and last 4 normalized ids ending at this token. N-grams stop at the start of the text and never cross an image.
3. Hash. For each of the 3 n-gram sizes, 8 different hash heads each turn the n-gram into a row number: multiply each id by a fixed odd number, XOR them together, and take the remainder by a prime near 16 million. Every (size, head) pair has its own prime, so its own slice of the table. That is 24 row numbers.
4. Fetch. Read those 24 rows. Each row is 256 numbers in FP8. Together they make 6,144 numbers.
5. Gate. Project the 6,144 numbers to one key per mHC stream and one shared value. For each stream, compare the key with the stream's current state (a normalized dot product, then a signed square root, then a sigmoid). If the memory fits the context, the gate opens. If it doesn't, for example because of a hash collision or a word with two meanings, the gate stays near zero.
6. Write. Add gate x value into each stream, then let the layer run as usual.

V4.1 has two Engram modules, at layers 1 and 14, each with a table of about 384 million rows. That is the 196.6 billion. DeepSeek placed them there partly to balance memory across training machines. Compared with the original paper, V4.1 drops a small convolution after the gate, which didn't pay for its complexity.

The best part is the systems side. The row numbers depend only on the token ids, not on anything the model computes. So the server knows which rows it needs as soon as the text arrives, and can fetch them from host memory over the network while the first layer is still running. The 183 GB table does not need to sit on the GPU. The vLLM recipe has a switch for exactly that.

The Engram paper also explains how big the table should be. Under a fixed budget of parameters, moving about 20 to 25% of the sparse parameters from experts to Engram gave the best loss, and the curve is U-shaped on both sides. V4.1's split is close: 196.6 billion of Engram against 545 billion of experts is 26.5%.

Training it needed its own optimizer trick. Adam would need two extra numbers for each of the 196 billion. Instead DeepSeek uses momentum followed by Sinkhorn balancing of the update's rows and columns, which needs only one extra number and, they say, works better than Adam. The same is used for the token embedding and output head. Engram's learning rate is 5 times the rest.

How images get in

V4.1 reads images from the start of pre-training, not as a later add-on. The path is short:

1. Resize. Keep the aspect ratio. Make the image at least 544 x 544 pixels in area, and small enough to fit in 1,024 tokens. Pad the leftover space with grey.
2. Patch. Cut it into 14 x 14 pixel patches. A linear layer turns each patch (588 numbers) into 1,024.
3. Encode. DeepSeek-ViT, 32 layers wide 1,024 with 16 heads, trained from scratch. Every patch sees every other patch of the same image. Positions use 2D RoPE, so any size works. It uses RMSNorm and SwiGLU like the language model, and a linear patch layer instead of a convolution so the Muon optimizer can train it.
4. Merge. Each 3 x 3 block of patches is folded into one vector of 9 x 1,024 = 9,216 numbers. So one language token covers 42 x 42 pixels, and an image costs 9 times fewer tokens.
5. Project. A two-layer MLP maps 9,216 numbers to 5,120, the width of the language model.
6. Insert. The image becomes a span in the token sequence: a start marker, then each row of image tokens followed by a newline marker, then an end marker. The markers are learned vectors. Then the span flows through the backbone like text, except that Engram skips it and the router uses the image bias.

The newline markers count toward the 1,024 limit, so a square image tops out at 31 x 31 tokens, about 1,300 pixels on a side. DeepSeek's "about 1,344 x 1,344" is the same thing, counted loosely.

The ViT was trained in two steps before it met the big model. First contrastively, the SigLIP way, on about 47 billion image and caption pairs at only 224 x 224 pixels. Then attached to a small 4-billion-parameter MoE language model, on 236 billion tokens of captions, charts and OCR, at 544 to 1,344 pixels. Then that small model was thrown away. During V4.1's pre-training the ViT stayed frozen until the learning rate started to decay, and then trained along with a smaller learning rate. Text-only data and image data were mixed 7 to 1.

DSpark: guessing five tokens ahead

Writing is slow because each new token needs a full pass through the model, and one pass on one token leaves most of a GPU idle. Speculative decoding fixes that: a small drafter guesses several tokens, and the big model checks all of them in one pass. If the first three guesses are right, you get three tokens for the price of one pass.

DSpark is DeepSeek's drafter, from a July 2026 paper. In V4.1 it is three extra blocks on top of the model, built like the backbone's blocks but with only the sliding window and 128 experts (3 per token). It reads the inputs of the top three layers, 37, 38 and 39, averaged over the four streams and projected into one vector.

It has three parts:

• Parallel draft. Feed the last real token plus four "noise" placeholder tokens, and get scores for all five next positions in one pass. That is fast, but each position guesses without knowing what the earlier positions picked. "Of" then "problem" might each be likely, but "of problem" is wrong.
• Markov head. A small fix for that: after picking position k, add a bias to the scores of position k+1 based on the token just picked. It is a 129,280 x 129,280 table of "what follows what", stored as two thin 256-wide matrices. Once position 1 picks "of", position 2 leans to "course".
• Confidence head. For each position, a tiny linear layer predicts the chance that this guess survives, given the ones before it survived. Multiply them and you get the chance that the first j guesses all survive.

Then a scheduler uses those chances. Checking a guess is cheap when the server is quiet and expensive when it is full, because every checked guess takes a slot another user could use. So for the whole batch, the scheduler sorts all guesses by survival chance and adds them in that order while the expected tokens per second keeps going up, using a speed curve measured once when the server starts. A user writing code might get all five guesses checked. A user chatting on a busy server might get one.

DSpark is trained after pre-training, with the backbone frozen, and then kept in step during post-training without sending its gradients into the backbone. On V4's servers, DeepSeek measured 60 to 85% faster generation per user for V4-Flash than the old one-token MTP, at the same total throughput.

How it was trained

Pre-training ran on 45 trillion tokens, text and images mixed, with no loss spikes reported. The batch stayed at 100.6 million tokens the whole way, which is about 447,000 steps. The learning rate warmed up for 2,000 steps, stayed at 2.6e-4 until 28 trillion tokens, fell on a cosine to 2.6e-5 by 40 trillion, and stayed there to the end. Sequences were 64K tokens long from the very first step, with sparse attention and no dense warm-up, and grew to 1 million at 34 trillion tokens.

Three optimizers share the work. Muon trains the matrices, with a twist: the query and key matrices are split per head first, so each head gets its own preconditioner. AdamW trains norms and small vectors. The Sinkhorn-balanced momentum update trains the embeddings, the output head and Engram.

DeepSeek does not say how many GPUs or GPU hours it took. A rough count of my own, using 6 x active parameters x tokens with every layer running in training, gives about 4 x 10^24 operations. Treat that as an order of magnitude.

The post-training recipe is the standard one: supervised fine-tuning, then reinforcement learning, then on-policy distillation. DeepSeek says plainly that it added no new algorithm here. The work went into data: automated generation of agent tasks and environments, at growing scale, and a sandbox system to run huge numbers of agents.

One feature came from that stage: a reasoning effort dial from 1 to 100. During RL, the system prompt carries "Reasoning Effort: N", and longer reasoning is penalized more at low effort, with a penalty that falls exponentially as effort grows. The API's low, high and max are 50, 75 and 100.

How good it is

V4.1-Flash is a "Flash" model, and DeepSeek compares it with the top closed models anyway. On agent coding it is close or ahead: DeepSWE v1.1 74.2 for the launch weights and 77.0 for 1017. On the hardest science agent tasks it is behind: Terminal-Bench 4.0 at 31.2, against 51.8 for the best closed model they list. DeepSeek says this itself.

What I take from it

V4.1 is a model designed backward from a bill. Agents send long prompts, over and over, and the cost is in reading and remembering, not in writing. Every big idea in the paper attacks that: read with half the layers, remember with four caches instead of forty, in 4 bits, keep only what lasts on disk, look up instead of compute what is fixed, and guess ahead when writing.

DeepSeek is also honest about the risk. The shared picks can pick wrong, and the bounded replays are approximations. They say they tested a lot and saw no systematic loss, and also that no test suite covers every edge.

If you want to run it yourself, the weights are about 511 GB on disk. The vLLM recipe says that fits one 8-GPU H200 node, with room left for the cache.

Sources

• DeepSeek-AI, "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression", tech report, September 2026: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf
• Model card and config: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash and https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/config.json
• Reference code (model.py, engram.py, vision.py, image_processor.py): https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/tree/main/inference
• The 1017 update: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
• DeepSeek-AI, "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence", arXiv 2606.19348: https://arxiv.org/abs/2606.19348
• Cheng et al., "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models" (Engram): https://github.com/deepseek-ai/Engram
• Cheng et al., "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation", arXiv 2607.05147: https://arxiv.org/abs/2607.05147 and code at https://github.com/deepseek-ai/DeepSpec
• DeepSeek API docs, model names: https://api-docs.deepseek.com/
• vLLM recipe for V4.1-Flash, for the memory breakdown and serving notes: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash