The tokenizer is character-level with a fixed vocabulary of 64 symbols (a–z, digits, punctuation). Every prediction and every generated token is one character; "words" exist only insofar as the model learns them from data.
Units. The training loss is the negative log-likelihood of the next character, −log p(target), averaged over positions. Taken with the natural log it is measured in nats (base e); divided by ln 2 the same quantity becomes bits (base 2). So 1 nat ≈ 1.44 bits — the loss chart shows nats, the bits/char chart shows the identical loss converted, which makes it directly comparable to Shannon entropies and the Markov baselines.
An untrained model guesses uniformly: log₂ 64 = 6 bits/char. Markov-1 (bigram) reaches ~2–2.5 bits on English-like text, markov-2 lower. The interesting moment is when the model line crosses below markov-2: from then on the transformer extracts structure no fixed 2-character context can — attention is paying for itself. On the small repeated default corpus a trained model goes far below 1 bit/char: the text is nearly deterministic given enough context (that is memorization — paste a few KB of varied text for an honest gap).
Early in training the maps are a uniform lower-triangle smear (causal mask, no learned structure). Trained, heads differentiate: typically one goes near-diagonal (local/previous-character copying) and another develops off-diagonal stripes jumping back to earlier occurrences of the same word — the long-context advantage the Markov panel quantifies. Residual-stream norms growing across stages is normal for pre-norm transformers; the final LayerNorm absorbs the scale.
Blue = the model's uncertainty before seeing the next character; red = how surprised it actually was. Inside a memorized word both drop toward 0 (after "th", predicting "e" is easy). Spikes sit at genuine branch points — after a space or period, where several continuations are legal. Red far above blue = confidently wrong. Mean entropy vs the 6-bit maximum gives redundancy; character-level English is highly redundant, so 60–80%+ after training is expected.
Bars per position: wherever the model bar is shorter than markov-2 — typically at the start of a repeated phrase — the model is using context deeper than 2 characters. Ties on short common pairs are expected; a bigram is already optimal there.
H_z = diag(p) − p pᵀ is the curvature of the loss in prediction space (the same per-token object the Python repo's Hessian analysis factors through). Trace = 1 − Σp²: a confident prediction has curvature ≈ 0, an uncertain one has high trace, and r_PR / erank count how many characters are genuinely still in play — a three-way branch point shows r_PR ≈ 2–3, not 64. It tracks the entropy curve in shape but measures something different: not how unsure the model is, but how many directions of the output distribution the data at that position can still bend. Watch it collapse as training pins predictions down — the per-token seed of effective-dimension analysis.