DeepSeek V4.1 Flash inference explorer · memory tiers

A deliberately small 5-layer causal encoder + 5-layer decoder schematic. It illustrates dataflow, not exact layer placement or dimensions.

Global KV 0

SWA KV 0

Engram rows 0

CSA2 policy FP4 global KV

Token flow and state placementREADY

INPUT / OUTPUT TOKENS

CAUSAL ENCODER — runs in prefill

Encoder final hidden stream H[t]

DECODER — skipped in prompt prefill

Decoder reads projected global KV

MEMORY HIERARCHY — HARDWARE OVERLAY

width ≈ capacity · stripe ≈ latency · five orders of magnitude top to bottom

GPU SRAM~1–5 ns~50 MB · 20–100 TB/s on-chip

Kernel working set — activations, indexer scores, Top-K sortlives only inside a kernel's lifetime — invisible in the time budget

GPU HBM3e~0.4–0.7 µs141–192 GB/GPU · 5–8 TB/s

MoE weights — 552B total · 16B active~32 GB/token streamed → ~0.4–0.5 ms floor — ≥95% of every decode step

Global KV — active sessions · FP4 · token-addressable

FP4: compact retained scalar values ~1 µs/step — Top-512 slice ≈ 450 KB at 8 TB/s

CSA2 sparse index — selected global positions

sub-µs — HSI candidate pool caps scoring, independent of context length

SWA KV — active window<1 µs — last n_win tokens only; capacity cost ≈ zero

Host DRAM (PCIe 5 / CXL)~1–2 µsTBs/host · ~64 GB/s hop

SWA KV — recent-session pool (10% of host DRAM)

hit: 1.5–3 µs, hidden by prefetch · miss: bounded replay ≈ 10–100 µs of compute — never SSD

Engram — 196B learned weight tables (not activations)

Model weights (not activations) · hash current n-grams → sparse rows → learned gate deterministic addresses → RDMA prefetch: ~0 on critical path · ~2 µs if stalled

NVMe SSD (Gen5)~50–100 µstens of TB · ~12–14 GB/s per drive

Global KV — persistent prefix cache (≥ 72 h)direct hit ~60–100 µs — so hits are promoted SSD → DRAM → HBM before the turn starts

Start here. Prefill streams the prompt through the causal encoder. Click an animation, then inspect the highlighted state.

What this visual separates

Compute pathShared KV / Engram weightsSWA local stateFP4 stored values

Causal Encoder Decoder (CED) answers where global KV is authored: per-token from the encoder final stream. CSA2 answers which layers own vs share that state. FP4 answers its numeric storage cost. SWA gets a separate short-life tier.

Compressed Sparse Attention (CSA2) layer modes

Animation ledger

Ready: choose a path above.

Engram: weights, not activations

Engram's tables are part of the model's weights — 196B parameters trained offline, just like an MoE expert's matrices. Nothing is written, updated, or invalidated as tokens stream by. The KV caches above are the opposite: activations, request-specific state created by this very request.

The conditioning works like MoE routing, but with a hash instead of a learned router. An MoE layer computes router logits and fires a few of many experts; Engram deterministically hashes the current 2-, 3-, and 4-token patterns (n-grams) into addresses in multi-head, prime-sized tables and reads only those rows. The condition is the local token pattern itself.

A learned gate then meters how much of the retrieved vectors enters the residual stream — the analog of MoE's expert weighting. So Engram is best read as a hash-routed, n-gram-conditioned memory: sparse and conditional like MoE, but addressed by content pattern rather than a router network, and read-only at inference.