Backpropagation, Step by Step

A neural network learns by trial and error: it makes a prediction, measures how wrong that prediction was, and then adjusts every one of its parameters — the weights and biases — slightly in the direction that would have made the prediction better. Backpropagation is the systematic procedure that determines, for every parameter at once, which direction is an improvement and by how much.

This note develops that idea step by step, in plain language, with a figure for every stage. The only genuine mathematics involved is the chain rule from calculus, and even that is introduced as a simple, intuitive statement before any symbols appear.

Summary. The forward pass sends an input through the network to produce a prediction. Backpropagation then sends the resulting error backward through the same connections, and as the error passes each layer it records how much that layer contributed to it. Those recorded quantities are the gradients used to improve the network.

The following diagram shows the loop a network repeats many times as it trains; every section below examines one stage of Figure 1 in detail.

Throughout we use the same names as the forward/backward reference notes: a layer is $[l]$, a training example is $(i)$, activations are $\mathbf{a}^{[l]}$, pre-activations (the value before the nonlinearity) are $\mathbf{z}^{[l]}$, weights are $\mathbf{W}^{[l]}$, and biases are $\mathbf{b}^{[l]}$. A leading "$d$" always denotes "the gradient of the loss with respect to this quantity" — so $d\mathbf{W}^{[l]}$ is shorthand for "how much the loss changes when $\mathbf{W}^{[l]}$ changes."

Setup: what the forward pass leaves behind

Backpropagation never runs on its own. It always follows a forward pass, and it relies on several quantities that the forward pass sets aside for it.

As the forward pass moves through the network, each layer not only computes its output but also stores the values it used — a record known as the cache. Backpropagation later traverses the same layers in reverse and reuses those stored values, so it never has to repeat the forward computation.

For a small network — two inputs, one hidden layer, one output — the forward pass is shown in Figure 1.1.

The values a layer stores are just three:

Why these three? Because, as we will see, the weight gradient requires the layer's input $\mathbf{a}^{[l-1]}$, the activation step requires the pre-activation $\mathbf{z}^{[l]}$ (to evaluate the slope of the nonlinearity), and passing the error further back requires the weights $\mathbf{W}^{[l]}$. Nothing is stored that is not later used.

The goal: what we are trying to compute

It is worth stating the objective precisely. A network improves by gradient descent, which needs only one piece of information about each parameter: if this parameter increases slightly, does the loss increase or decrease, and how quickly? That number is a gradient.

So for every layer $l$, backpropagation must produce two gradients — one for the weights and one for the biases:

Everything else backpropagation computes along the way is intermediate scaffolding that makes these two inexpensive to obtain. Keep Equation 2.1 in mind as the objective; the steps below are the route to it.

Big picture: the error flows backward

The network only learns that it was wrong at the very end, where the prediction is compared with the true answer. That is therefore where we begin. Backpropagation takes that error and passes it from the output back toward the input, one layer at a time (Figure 3.1).

A useful way to picture it: the forward pass propagates information from left to right, and the backward pass propagates the error signal from right to left. When the error reaches a layer, that layer determines how much each of its neurons contributed, and then passes the remaining error signal to the layer before it.

In Figure 3.1 each hidden neuron's returning gradient is drawn in its own colour, so it can be followed individually: the blue signal is neuron $a_1^{[1]}$'s contribution, the purple one is $a_2^{[1]}$'s, and the teal one is $a_3^{[1]}$'s. Note that each input neuron on the right is reached by all three colours at once — its total gradient is the sum of the contributions arriving along every path (discussed in §4.1).

Before assembling the general rules, it helps to see every operation at ground level. Figure 3.2 zooms into a single hidden neuron — $a_3^{[1]}$ — and lays out everything backpropagation does there: the gradient $d\mathbf{a}_3^{[1]}$ arrives from the output, passes back through the activation to become the shared signal $dz_3^{[1]}$, and from that one quantity the neuron produces its two weight gradients, its bias gradient, and the share of the gradient it hands back to each input. This is the backward counterpart of a neuron's forward computation, and a concrete preview of the five general steps derived in §5.

That is the overall structure of the algorithm. The remainder of this note works out exactly what each layer does to the error signal as it passes through.

The one idea: the chain rule

Why does propagating the error backward yield the correct gradient? Because a parameter deep in the network affects the final loss only through a chain of intermediate quantities. A weight sets a pre-activation, which determines an activation, which feeds the next layer, and so on, until it finally influences the loss (Figure 4.1).

The chain rule states that, to find the total effect of the first link on the last, one multiplies the effects of every link in between. The analogy is a train of gears: if gear A turns gear B twice as fast, and B turns C three times as fast, then A turns C six ($2\times3$) times as fast.

The reason backpropagation is efficient is that it never re-multiplies the entire chain for each weight separately. It computes the shared error signal accumulated so far once at each layer and reuses it everywhere. That shared quantity is $d\mathbf{z}^{[l]}$, introduced formally in Step 2.

Why the previous layer's gradient is a sum

There is one place where the chain rule requires a summation, and it is worth examining on its own because it explains the colour-mixing in Figure 3.1.

A single neuron in one layer does not feed only one neuron in the next layer — it feeds all of them (Figure 4.2). So to determine a neuron's total contribution to the error, we must add together the error signal returning along every path it took forward. The chain rule accounts for this by summing over the paths.

In words: a neuron's gradient is the total error it caused, summed over every neuron it fed. That single sum over paths is the only place where the four backward rules use addition rather than a simple multiplication.

One layer, four gradients

Now consider a single layer $l$ in detail. It receives one quantity from its right — the gradient on its own output, $d\mathbf{a}^{[l]}$ — and from that it must produce four quantities (Figure 5.1):

1. $d\mathbf{z}^{[l]}$ — the error signal at its pre-activation (the shared signal),
2. $d\mathbf{W}^{[l]}$ — its weight gradient (retained for the update),
3. $d\mathbf{b}^{[l]}$ — its bias gradient (also retained),
4. $d\mathbf{a}^{[l-1]}$ — the error signal passed to the layer before it.

The next five steps derive these four gradients in order. To keep the notation uncluttered we drop the example index $(i)$ — every formula is written for a single training example.

Step 1 — the incoming gradient

Each layer's computation begins when the layer to its right passes it a gradient on its own output — how strongly the loss depends on this layer's activations:

For the final layer, this first gradient comes directly from the loss function (see §6). For every other layer, it is exactly what the layer above passed down in its own Step 5. This is what makes the process a chain: one layer's output is the next layer's input.

Step 2 — back through the activation

The nonlinearity treats each neuron independently — neuron $j$'s output depends only on neuron $j$'s pre-activation. So passing the error signal back through it simply means scaling each neuron's signal by that neuron's local slope, $g^{[l]\prime}$. Because every neuron is scaled by its own value, this is an element-wise product (written $\odot$), not a matrix product:

Intuition: if a neuron was in a flat region of its activation (slope near $0$), changing it barely affects the output, so it receives almost no error signal. If it was in a steep region (large slope), it receives the full share. This $d\mathbf{z}^{[l]}$ is the shared error signal — the next three gradients are all built from it, which is why the entire layer is inexpensive to compute.

Step 3 — the weight gradient

A weight $w_{j,k}^{[l]}$ acts by multiplying the input $a_k^{[l-1]}$. So that weight's gradient is simply its neuron's error signal times the input it multiplied. Computing this for every weight at once gives an outer product:

In words: each weight's gradient = (error signal at the neuron it feeds) × (the activation it multiplied). A weight that carried a large input into a highly erroneous neuron receives a large gradient — precisely the parameter we most want to adjust.

Step 4 — the bias gradient

The bias is the simplest case. It is added directly to the pre-activation, so increasing the bias by one increases $z$ by one — no scaling and no input involved. Its gradient is therefore just the neuron's error signal, unchanged:

Step 5 — hand off to the previous layer

Finally, the layer passes the remaining error signal backward. Because each input neuron fed every neuron in this layer (the sum over paths from §4.1), collecting the error signal for the previous layer means multiplying by the transpose of the weight matrix:

The transpose is not a trick — it is the same connections used in the opposite direction. In the forward pass, $\mathbf{W}^{[l]}$ carries activations from layer $l-1$ up to layer $l$; in the backward pass, $(\mathbf{W}^{[l]})^{T}$ carries the error signal from layer $l$ back down to $l-1$. This output is exactly the input (Step 1) of the previous layer, so the five steps simply repeat, one layer down.

Where the very first gradient comes from

Steps 1–5 all assume the layer has already been given its $d\mathbf{a}^{[l]}$. That holds for every layer except the last, which has no layer to its right. Its first gradient must come from the loss function itself, by asking how the loss changes as the prediction changes.

For the common case of a binary (yes/no) prediction with a sigmoid output and the standard cross-entropy loss, the result is:

This expression looks complicated, but a notable simplification occurs when it is passed through Step 2. The sigmoid's slope is $g^{[L]\prime}=a(1-a)$, and upon multiplication the fractions cancel completely, leaving:

Prediction minus the true answer. That is the entire starting signal: if the prediction was $0.9$ and the truth was $1$, the error signal is $-0.1$; if the prediction was $0.9$ and the truth was $0$, it is a large $0.9$. This form is simple, inexpensive, and numerically well-behaved — which is why the sigmoid and cross-entropy are almost always used together.

The whole network: the backward sweep

Now reassemble the layers. Begin at the output with the seed from Equation 6.2, then proceed backward, applying Steps 2–5 at each layer and feeding each layer's Step 5 output into the previous layer's Step 1 input (Figure 7.1). A single pass from the last layer down to the first produces $d\mathbf{W}^{[l]}$ and $d\mathbf{b}^{[l]}$ for every layer.

Using the gradients: the update step

Backpropagation ends here — it has produced a gradient for every parameter. What we do with those gradients is the final stage of the training loop (Figure 1), a single operation called gradient descent.

A gradient points in the direction that makes the loss increase. Since we want the loss to decrease, we step in the opposite direction — subtracting a small multiple of the gradient (Figure 8.1):

The scalar $\alpha$ is the learning rate — the size of the step. If it is too small, learning is slow; if it is too large, the updates overshoot and never settle. After taking one such step, return to the top of Figure 1 and repeat the process on the next batch of data. After many iterations, the network's predictions become accurate.

Vectorized summary and pseudocode

Everything above was written for a single example. To train on a batch of $m$ examples at once, we stack the examples side by side into matrices (capital letters). The four steps are identical in form; the only change is that the two parameter gradients gain a factor $\tfrac{1}{m}$ — because the loss for a batch is the average over its examples, so each parameter gradient is an average too. (The activation gradients carry no $\tfrac{1}{m}$: each example retains its own error signal.)

In code, the entire backward sweep is a short loop over the cached layers, from last to first — five lines, one per step:

# caches[l] = (A_prev, W, Z) from the forward pass; g_prime[l] = the activation's slope
grads = {}
dA = A_final - Y                     # seed: sigmoid + cross-entropy  (eq-init-simpl)
for l in reversed(range(1, L + 1)):
    A_prev, W, Z = caches[l]
    dZ = dA * g_prime[l](Z)                       # Step 2  (element-wise)
    grads[f"dW{l}"] = (1 / m) * dZ @ A_prev.T      # Step 3
    grads[f"db{l}"] = (1 / m) * dZ.sum(axis=1, keepdims=True)   # Step 4
    dA = W.T @ dZ                                  # Step 5  → becomes next loop's input

And the update (Equation 8.1) is one more line per parameter:

for l in range(1, L + 1):
    W[l] -= alpha * grads[f"dW{l}"]     # gradient descent
    b[l] -= alpha * grads[f"db{l}"]

That is the complete procedure: a forward pass to predict and cache the required values, a five-line backward sweep to assign the error to every parameter, and a one-line update to improve them — repeated until the network performs well.