Backpropagation and vanishing gradients

What this page covers. Backpropagation is the algorithm that computes, for every weight in a neural network, how much that weight should change to reduce the loss. This page derives the algorithm step by step from two calculus facts, shows why training stalls in deep networks without skip connections (vanishing gradients), and explains a fix for it that stabilizes the gradients - called residual stream.
Notation in the diagram. W = weight matrices (learned parameters, what training adjusts). h = activations (hidden states, output of each layer during the forward pass; stored but never directly updated except indirectly via weight updates). δ = activation gradients = ∂Loss/∂h, how much the loss would change if that layer's output (activation h) were nudged slightly. These travel backward and are used to compute weight gradients ∂Loss/∂W — the actual update signal for each weight. See below the diagram for how δ is calculated.
Forward pass (data flows left → right) x = h₀ h₁ h₂ Loss W₁ W₂ W₃ Backward pass (δ flows right ← left) δ₁ δ₂ δ₃ seeded = 1 ∂L/∂W₁ = h₀ᵀ · δ₁ activation × activation gradient ∂L/∂W₂ = h₁ᵀ · δ₂ activation × activation gradient
How δ is calculated — and why it is called a gradient.
δ_l is literally the partial derivative of the loss with respect to the activation h_l at layer l: δ_l = ∂Loss/∂h_l. It is a vector — one entry per activation dimension — telling you how much the loss would change if you nudged that dimension up by a tiny amount.

Seeding. At the loss node, δ_L is seeded to 1 (the loss with respect to itself is 1). This is the starting point.
Propagating backward. At each layer, δ is multiplied by that layer's Jacobian (the matrix of all partial derivatives of the layer's output with respect to its input). This gives the δ for the layer below. The chain rule guarantees this is the correct gradient. In PicoGPT's autograd.js, each operation stores a backward() closure that performs exactly this multiplication and accumulates into the input's gradient.

Why you care. The weight gradient ∂Loss/∂W at any layer is the outer product of the activation that entered that layer and the δ that arrives from above (equation 6 below). If δ shrinks to near zero as it travels backward through many layers — the vanishing gradient problem — every weight gradient at those early layers also becomes near zero. The weights stop updating. The layer stops learning. The activations at that layer can be arbitrarily large and it makes no difference.
The central dependency. Every weight gradient ∂Loss/∂W at a layer = (activation stored during the forward pass) × (activation gradient δ arriving from the loss). If δ vanishes before reaching a layer, every weight in that layer receives a near-zero update and stops learning, regardless of activation magnitude. Everything depends on δ arriving with useful magnitude.

1 · What backpropagation computes — parameter gradients and activation gradients

Training updates weights: for every weight W we need ∂L/∂W, then W ← W − η·∂L/∂W. The input x is fixed; we never update it. The derivation below shows, step by step, why activation gradients (the δ values) control every weight gradient. Two calculus facts are all that is needed: the chain rule and "derivative of a sum = sum of derivatives".

Setup. x is one training sequence — a batch of T token positions processed together. Each position's embedding is a vector in ℝ^d_model, so x has shape T × d_model. The loss is averaged over all T positions. h_l is the activation (hidden state) coming out of layer l; in a transformer, this is the residual stream at that depth. W_l are the weight matrices of layer l. f_l is the computation layer l performs (attention + FFN in a transformer block):

h₀ = x,   h_l = f_l(h_{l−1}, W_l) for l = 1…L,   Loss = loss(h_L)(1)

Two calculus facts. In (2) and (3) below, u and v stand for any two intermediate quantities in the computation — for example, u could be a weight entry W[i][j] and v could be an activation vector h_l. The chain rule says: to find how a change in u affects the loss, multiply how u affects v by how v affects the loss.

One-variable chain rule, and its several-variable form — if u affects the loss through multiple intermediate quantities v[j], each route contributes one term and the routes add:

y = g(u), u = k(v) ⟹ dy/dv = (dy/du)·(du/dv)(2)
∂Loss/∂u = Σ_j (∂Loss/∂v[j]) · (∂v[j]/∂u) — one term per route u→v[j]→Loss(3)
from (2): apply it along each route, then add routes (derivative of a sum).

The messenger δ. Define, for every layer boundary, the row vector of partials of the loss with respect to that activation:

δ_l := ∂Loss/∂h_l,   i.e. δ_l[j] = ∂Loss/∂h_l[j](4)

δ_l answers: "how much does the loss change if I nudge h_l[j] slightly?" It is not used to update h_l — activations are never updated. It is the information that travels backward so that weight gradients can be computed at each layer. The term "activation gradient" means: gradient of the loss with respect to an activation, not gradient of the activation itself.

What one matmul contributes to one weight. Take a linear step h_out = h_in·W, written per entry so the partial is a one-liner:

h_out[j] = Σ_i h_in[i]·W[i][j] ⟹ ∂h_out[j]/∂W[i][j] = h_in[i],  ∂h_out[j′]/∂W[i][j] = 0 for j′≠j(5)
differentiate the sum term by term; W[i][j] appears in exactly one output entry.

The weight gradient formula. W[i][j] reaches the loss only through h_out[j] (one route), so (3) has a single term:

∂Loss/∂W[i][j] = δ_out[j] · h_in[i]  — in matrix form ∂Loss/∂W = h_inᵀ·δ_out(6)
from (3) with one route, (4) naming ∂Loss/∂h_out[j], and (5) for the local partial.

Every weight gradient is a product of two factors: the activation h_in stored during the forward pass, and the activation gradient δ_out arriving from the loss. The first factor is local and always available. The second factor is what depth puts at risk.

Propagating the messenger. Collect all partials of one step into its Jacobian:

J_l[j][i] := ∂h_l[j]/∂h_{l−1}[i](7)

h_{l−1}[i] reaches the loss through all entries of h_l — (3) with routes h_{l−1}[i] → h_l[j] → Loss:

δ_{l−1}[i] = Σ_j δ_l[j]·J_l[j][i]  — in matrix form δ_{l−1} = δ_l·J_l(8)
from (3) with routes through h_l, (4) for δ_l, (7) for the partials.
apply (8) repeatedly:  δ_l = δ_L · J_L · J_{L−1} ··· J_{l+1}(9)

Combining (6) and (9) — the key result:

∂Loss/∂W_l = h_{l−1}ᵀ · ( δ_L · J_L ··· J_{l+1} )(10)
from (6) with δ_out given by (9).

We never use ∂Loss/∂x to update anything — but by (10) the Jacobian product is a shared factor in every weight gradient at layer l. If each Jacobian shrinks vectors by at most a factor s (its largest singular value; this is |slope| in the 1-dim case):

‖δ_l‖ ≤ ‖δ_L‖ · s^{L−l}  ⟹ by (10), ‖∂Loss/∂W_l‖ ≤ ‖h_{l−1}‖·‖δ_L‖·s^{L−l}(11)
from (9), bounding each factor; then (10). s<1 ⟹ geometric decay with depth — Bengio 1994's vanishing-gradient result.

With the residual stream. A pre-norm block writes into the stream instead of replacing it:

h_l = h_{l−1} + F_l(h_{l−1})  ⟹  J_l = I + F_l′(12)
differentiate (12) term by term using (7): ∂h_{l−1}[j]/∂h_{l−1}[i] gives the identity matrix I, plus the branch Jacobian F′.
δ_l = δ_L·(I+F_L′)···(I+F_{l+1}′) = δ_L·( I + Σ_k F_k′ + cross terms )  — the term δ_L·I contains no weights(13)
substitute (12) into (9); multiply out — one term per subset of blocks (§3). This is He et al. 2016, eq. (5).
∂Loss/∂W_l = h_{l−1}ᵀ · δ_L·( I + … )  — weight gradients inherit the unkillable identity term(14)
from (10) with (13). Contrast with (11): no s^{L−l} factor multiplies the leading term, so deep-layer weights keep learning.

2 · Vanishing gradient: what happens without the residual connection

This demo shows what happens when gradient signal travels backward through a network with no residual connections, compared to one that has them. From equation (9), the gradient at each layer is the gradient from the layer above multiplied by that layer's local slope. Without a residual connection, that multiplication happens at every layer. If the slope is less than 1, the gradient shrinks at every step — a geometric decay that leaves early layers with almost no signal. At slope 0.5 and 12 layers, the first layer receives 0.5¹² ≈ 1/4096 of the loss gradient, and every weight update in that layer is scaled by that same factor.

Why normalizing the gradient does not fix this. A natural response is to keep only the gradient's direction and discard its magnitude, similar to what LayerNorm does for activations in the forward pass. This does not work for two reasons. First, the gradient's magnitude carries information: a near-zero gradient means "this weight barely affects the loss right now — leave it alone." Normalizing it to unit length causes a large, unwarranted update. Second, normalizing at one layer does not help layers further down. The attenuated signal still passes through every remaining Jacobian before reaching those layers, shrinking at each one. You would have to normalize after every single layer, and each normalization discards the information that earlier layers need about how much to change.

The residual connection does something different: it adds 1 to every Jacobian (equation 12). The backward pass then includes one path through which the loss signal travels unchanged, skipping every layer's weights (the identity path). This path cannot vanish no matter how small the weights become — there are no weights in it.

Drag the slope slider to see gradient magnitude change across layers (top diagram) and how gradient mass shifts across path lengths (bottom diagram). Both diagrams respond to the same slider. Gradient mass is the share of the total gradient signal flowing through paths of a given length — where length counts how many layer branches the signal passes through rather than skipping.

layers L 12   local slope a 0.50

Red row: plain network. Each box multiplies the gradient by slope a. Green row: residual network. Each box multiplies by 1+a. The colored bar below each box is the gradient magnitude at that point, on a log scale — taller bars mean more signal. The dashed line marks gradient = 1. By equation (10), every weight gradient at a given depth carries the same factor as the messenger bar shown there.

The bar at k=0 (leftmost) is the pure identity path — the gradient arrives unchanged, bypassing all layer weights. With slope less than 1, most gradient mass concentrates at short path lengths. Veit et al. 2016 measured this in a 110-layer ResNet: effective paths are only 10–34 layers deep, even in a 110-layer network.

3 · The path expansion — why +1 before the product works

Multiply out the residual chain's backward factor (algebra, no calculus):

(1+a₁)(1+a₂)···(1+aₙ) = 1 + Σaᵢ + Σaᵢaⱼ + ··· + a₁a₂···aₙ — one term per subset of blocks = one term per path (2ⁿ paths)

The leading 1 is the all-skip path: it delivers δ_L to the bottom unchanged. This is equation (5) of He et al. 2016 [1603.05027], quoted verbatim in §4: the gradient splits into ∂E/∂x_L · (1 + ∂/∂x_l ΣF) — our equation (13) — and therefore "the gradient of a layer does not vanish even when the weights are arbitrarily small." In a plain chain, making weights small kills every path; in the residual chain the identity path contains no weights, so no weight can kill it.

4 · Sources

Hochreiter 1991; Bengio, Simard & Frasconi 1994 — Learning long-term dependencies with gradient descent is difficult (IEEE Trans. Neural Networks). First precise statement of the problem: backpropagated error is a product of Jacobians, and with contractive factors it decays exponentially in depth/time.
He, Zhang, Ren, Sun 2015 — Deep Residual Learning for Image Recognition [1512.03385]. Introduced the residual block to fix the degradation problem: deeper plain nets had higher training error — an optimization failure, not overfitting.
He, Zhang, Ren, Sun 2016 — Identity Mappings in Deep Residual Networks [1603.05027]. The math this page animates. Eq. (4): x_L = x_l + Σ_{i=l}^{L−1} F(x_i, W_i) — any deep feature is a shallow feature plus a sum of residual writes (our "bus" panel). Eq. (5): ∂E/∂x_l = ∂E/∂x_L · (1 + ∂/∂x_l Σ_{i=l}^{L−1} F(x_i, W_i)) — the additive gradient decomposition, with the conclusion: "the gradient of a layer does not vanish even when the weights are arbitrarily small."
Veit, Wilber, Belongie 2016 — Residual Networks Behave Like Ensembles of Relatively Shallow Networks [1605.06431]. Unrolls the 2ⁿ-path expansion (§3) and measures where gradient flows: in a 110-layer ResNet, most gradient comes from paths only 10–34 layers deep; deleting single blocks barely hurts (ensemble behavior).
Balduzzi et al. 2017 — The Shattered Gradients Problem: If resnets are the answer, then what is the question? [1702.08591]. In plain deep nets, gradients wrt the input decorrelate like white noise (correlations decay ~2^−L); with skips the decay is sublinear — gradients stay structured and carry usable learning signal.
Elhage et al. 2021 — A Mathematical Framework for Transformer Circuits [ transformer-circuits.pub]. The residual-stream-as-communication-bus framing used across these pages: blocks read from and write to a shared linear stream; the stream itself is untouched. Pre-norm transformers like PicoGPT are He-2016-style identity mappings.