Interpretability research moves quickly, but the questions underneath it are stable: What kind of mathematical object is a network’s internal state? Can we name the parts and trace the wiring? If concepts are directions, can we steer? And when an explanation is offered, how do we know it is true of the machine rather than merely plausible to us? This book builds the foundations for those questions from first principles, then takes them to the frontier of current research.
The treatment is deliberately concrete. Every chapter closes with
takeaways, open research directions scoped so a reader could start on
them, and runnable programs — the full suite runs on a laptop with
Python, torch, and GPT-2-class models. Short
“neuroscience parallel” passages trace which methods were
inherited from systems neuroscience, and which failure modes came with
them.
Part I — From Random Machines to Trained Machines
Chapters 1–4 · the foundations
- Statistical Foundations — Markov, Shannon, and the modern objective
- Geometric Foundations — KL divergence, Fisher information, the Fisher–Rao metric
- Neural Networks as Distribution Machines — the transformer decoder, flows, mixtures of experts
- Training Dynamics — natural gradient, flat minima, training-time levers
Part II — Geometry, Circuits, Control, and Evidence
Chapters 5–8 · the interpretability frontier
- Representation Geometry — the latent space: manifolds, superposition, probes and lenses
- Features and Circuits — sparse autoencoders, induction heads, activation patching
- Control and Steering — steering vectors, conceptors, fine-tuning compared
- Evaluation and Frontiers — faithfulness, grokking, open problems
Hands-on programs
The programs/ github directory contains 22 runnable experiments
keyed to the chapters — from reproducing Markov’s 1913 letter
statistics to finding induction heads, steering refusal directions, and
watching a grokking phase transition form its internal circuit before
the behavioral jump. See programs/README_programs.txt for
what each program does and what to expect, and run them with
programs/run_tests.sh.
Interactive demos
A browser companion to Chapter 1: uniform and Markov sentence generators with adjustable dials, scored by cross-entropy against Shakespeare.
Markov based sentence generators entropy and cross-entropy demo
PicoGPT based entropy and cross-entropy demo
Backpropagation - an exploration of activation gradients, weight gradients, vanishing gradients problem for deep layers and its fix with a residual stream
Transformer as a fat cable, including splicing, mixing and processing
Representation hypotheses atlas