Step through the diagram using the Next/Prev buttons at the bottom
Picture an LLM as one thick cable of d_model wires running through the full length of the model, one such cable per token. At each attention head the cable is unbundled into individual subcables (colored in different colors below). The number of subcables is the same as the number of heads (num_heads). Each of these heads or subcables carries d_head wires inside. Each of these wires carries a single dimension or coordinate of the embedding vector. So the embedding vector has d_model dimensions and occupies the full cable.
One token = One vector = One embedding = One fat cable = d_head subcables = d_model wires or dimensions/token.
Each of the dimensions (wires or channels) is typically an FP16 = 16 bits = 2 bytes of information. GPT-2 small had an embedding size of d_model=768 dimensions, and 12 heads for 64 dimensions/head.
The diagram illustrates transformer processing steps using a cable diagram to build intuition.
Cycle through the ten steps by pressing next below. Pressing next after step 10 loops back to step 1 so you can rewatch - in a real model there are multiple such layers and the input tokens change in each round. New tokens appear only between forward passes: after the final layer, an unembedding matrix and the model's last softmax turn the final token's cable into next-token probabilities, one token is sampled, appended as a new cable, and the whole stack runs again (with KV caching so old cables' K/V aren't recomputed).
What a wire carries. One scalar per token — wire i is coordinate i of the token's d_model-dimensional vector. Its numeric format is an engineering choice, not architecture: fp32 (classic training), bf16/fp16 (standard today), fp8 or int8/int4 (quantized inference).
Scale. 8 wires are drawn so you can count them. GPT-2 small has 768 per cable, GPT-3 12,288, with d_ff at 4× that; the geometry is identical, just a denser cable. Nothing about the picture changes with size — which is also why d_model can't be changed without retraining everything: every box here has a side of length d_model.
A head is only a subcable after projection. The thin bundles are genuinely d_head wires each, and concat really is laying them side by side (h × d_head = d_model). But each head's W_Q, W_K, W_V read all 8 input wires and project them down to d_head new ones — shown by the fan-in to each small box. The residual stream is not pre-partitioned among heads.
Other simplifications. Attention weights here are illustrative numbers, not computed from data; only one reading token's arrows are drawn at a time (all tokens read simultaneously in reality); biases, dropout, positional information, and GQA/MQA-style K/V sharing are omitted; modern models often use RMSNorm and GELU/SwiGLU in place of LayerNorm and ReLU — same positions in the wiring.
Add the bypass again, and 8 wires per token continue to the next layer, which has its own copies of every box. No softmax between layers — softmax lives only inside heads and at the final unembedding. Next wraps to step 1 for review; see the note below on why a real model is a stack, not a loop.
x ← x + Δ_ffn ; layer out ∈ ℝ^{3×8} = layer in’s shape