Forward and Backward Propagation for Single and Multiple Training Examples

This document develops the mechanics of a feed-forward neural network in full: how an input is transformed into a prediction (forward propagation), and how the resulting error is used to update the parameters (backward propagation). Each result is presented twice. We first derive it for a single training example, where the reasoning is most transparent, and then state it in vectorized form for a batch of $m$ examples, which corresponds to a practical implementation. A single, fixed set of symbols is used throughout and is summarized in Table 1.1.

Overview. In the forward direction, every layer performs two operations: a linear transformation followed by a nonlinearity. Backward propagation reverses this chain, reusing the quantities cached during the forward pass. Tracking the shape of each matrix is often sufficient to reconstruct the formulas, since in most cases only one product is dimensionally consistent.

Notation Conventions

Before introducing the equations, we fix a consistent naming scheme. A neural-network expression must identify four things at once: the layer, the neuron, the training example, and whether the quantity is a scalar, a vector, or a matrix. The following conventions encode all four unambiguously:

• Scalars (single numbers): $m, n_l, y^{(i)}, z_j^{[l]}$.
• Vectors (column vectors): bold lowercase, e.g. $\mathbf{a}^{[l]} \in \mathbb{R}^{n_l}$, $\mathbf{x}$.
• Matrices: bold uppercase, e.g. $\mathbf{W}^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}$, $\mathbf{A}^{[l]} \in \mathbb{R}^{n_l \times m}$.
• Superscript $[l]$: layer index.
• Superscript $(i)$: training example index.
• The symbol $\odot$ denotes the element-wise (Hadamard) product.

The two superscripts are the most important to distinguish: $[l]$ denotes the layer and $(i)$ denotes the training example, while a subscript ($j$ or $k$) selects an individual neuron. For example, $a_j^{[l]{}(i)}$ is a single scalar — the activation of neuron $j$ in layer $l$ for example $i$.

Throughout, it is helpful to move between two equivalent views of the same object. In the single-example view, a quantity is a column vector, which is convenient for applying the chain rule. In the vectorized view, these columns are placed side by side to form a matrix, one column per example. Every batched equation below is simply its single-example counterpart with the examples arranged as columns.

A complete summary of the notation is given in Table 1.1.

The table distinguishes the cost $J$ from the per-example losses $L^{(i)}$: the cost is the average of the losses over the batch. Training minimizes $J$, but each example contributes its own loss and its own gradient, so the batch gradient is the average of the per-example gradients. This averaging is the origin of the factor $\tfrac{1}{m}$ that appears in the vectorized parameter gradients derived below.

Forward Propagation: Single Training Example

Forward propagation computes the network's prediction. Beginning with the input $\mathbf{x}^{(i)}$ (equivalently, $\mathbf{a}^{[0]{}(i)}$), each layer transforms the activations of the previous layer into its own, until the final layer produces the output $\hat{y}^{(i)}$. Every layer applies the same two operations: a linear transformation followed by a nonlinear activation function. We first establish the exact dimensions of each object, since these dimensions determine the form of nearly every equation that follows.

A concrete instance of such a network is drawn in Figure 2.1, using exactly the symbols of Table 1.1 and shown for a single training example $i$, so every activation carries the example superscript $(i)$. The input is layer $0$ (features $x_k^{(i)} = a_k^{[0]{}(i)}$, here $n_0 = 3$); the two hidden layers are $l = 1, 2$ with activations $a_j^{[l]{}(i)}$ (here $n_1 = n_2 = 4$); and the output is layer $L = 3$ with $\hat y_j^{(i)} = a_j^{[L]{}(i)}$ ($n_3 = 2$). The weights $w_{j,k}^{[l]}$ and biases $b_j^{[l]}$ carry no $(i)$, since the same parameters are shared across all examples. Every neuron in layer $l-1$ connects to every neuron in layer $l$, and each connection carries a weight $w_{j,k}^{[l]}$ with row $j$ = destination neuron in layer $l$ and column $k$ = source neuron in layer $l-1$ — the same convention laid out for $\mathbf{W}^{[l]}$ in Equation 2.2. To keep the picture readable, only the weights feeding the top neuron of each layer are labelled (the first row of each weight matrix $\mathbf{W}^{[l]}$); every other edge carries its own $w_{j,k}^{[l]}$ in the same way. Each layer also has a dashed bias unit ($1$) whose edges broadcast a bias $b_j^{[l]}$ to every neuron of that layer, one component of the bias vector $\mathbf{b}^{[l]}$ of Equation 2.3; the bias edges of every layer are labelled $b_j^{[l]}$ to show the full bias vector $\mathbf{b}^{[l]}$. The number of weight connections is the product of adjacent layer sizes, $4\cdot 3 + 4\cdot 4 + 2\cdot 4 = 36$.

Inside a Single Neuron

The network figure draws each neuron as a single circle, which hides the two operations every neuron actually performs: a linear step — scale each incoming activation by its weight and add the bias — followed by a nonlinearity $g$. Figure 2.2 zooms in on one neuron, neuron $3$ of hidden layer $1$ ($a_3^{[1]{}(i)}$), and unfolds it. Reading left to right, each input $a_k^{[0]{}(i)}$ is multiplied ($\times$) by its weight $w_{3,k}^{[1]}$; the three products are added ($+$) together with the bias $b_3^{[1]}$ to form the pre-activation

and finally the activation function is applied to obtain the neuron's output $a_3^{[1]{}(i)} = g(z_3^{[1]{}(i)})$. Every other neuron in the network — in every layer — works in exactly this way; the full network is just this same multiply–sum–activate unit repeated and wired together.

Detailed Matrix and Vector Structures

Before stating the forward equations, we give the explicit structure of each matrix and vector used in layer $l$.

Weight Matrix $\mathbf{W}^{[l]}$

The weight matrix $\mathbf{W}^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}$ holds the connection strengths from layer $l-1$ (with $n_{l-1}$ neurons) to layer $l$ (with $n_l$ neurons):

Read the matrix by its indices:

• Rows ($j = 1, \ldots, n_l$): correspond to each neuron in the current layer $l$.
• Columns ($k = 1, \ldots, n_{l-1}$): correspond to each neuron in the previous layer $l-1$.
• Entry $w_{j,k}^{[l]}$: the weight connecting neuron $k$ in layer $l-1$ to neuron $j$ in layer $l$.

The index order follows the convention row = destination, column = source. Row $j$ of $\mathbf{W}^{[l]}$ therefore contains all incoming weights for neuron $j$, which is the weight vector $\mathbf{w}_j^{[l]}$ defined in Table 1.1. The dimensions follow directly: mapping an $n_{l-1}$-dimensional input to an $n_l$-dimensional output requires $n_l$ rows and $n_{l-1}$ columns.

Bias Vector $\mathbf{b}^{[l]}$

The bias vector $\mathbf{b}^{[l]} \in \mathbb{R}^{n_l}$ provides one adjustable offset per neuron in layer $l$:

Here Entry $b_j^{[l]}$ is the bias added to the pre-activation of neuron $j$ in layer $l$. The bias shifts the neuron's threshold, allowing it to activate when the weighted input is zero or to remain inactive until the input is sufficiently large; without it, the neuron's response would be constrained to pass through the origin.

Input Matrix $\mathbf{X}$

The input matrix collects the raw feature vectors $\mathbf{x}^{(i)}$ of all $m$ examples as columns. It is the base case $\mathbf{X} = \mathbf{A}^{[0]} \in \mathbb{R}^{n_0 \times m}$ from which forward propagation starts, so the first layer receives $\mathbf{X}$ in place of a previous layer's activations:

Its layout is the same as every activation matrix:

• Rows ($k = 1, \ldots, n_0$): correspond to the $n_0$ input features.
• Columns ($i = 1, \ldots, m$): correspond to the $m$ training examples.
• Entry $x_k^{(i)}$: the $k$-th feature of training example $i$, equivalently the layer-0 activation $a_k^{[0]{}(i)}$.

Pre-activation Matrix $\mathbf{Z}^{[l]}$ (Vectorized)

The pre-activation of a neuron is its weighted sum of inputs plus bias, computed before the nonlinearity is applied. For $m$ training examples, these values are collected in the matrix $\mathbf{Z}^{[l]} \in \mathbb{R}^{n_l \times m}$:

Notice the layout convention that will hold for every activation-like matrix:

• Rows ($j = 1, \ldots, n_l$): correspond to each neuron in layer $l$.
• Columns ($i = 1, \ldots, m$): correspond to each training example.
• Entry $z_j^{[l]{}(i)}$: the pre-activation (weighted sum plus bias) of neuron $j$ in layer $l$ for training example $i$, computed as:

Equation Equation 2.6 is a dot product followed by a shift: for each neuron $k$ in the previous layer, its activation $a_k^{[l-1]{}(i)}$ is multiplied by the connection strength $w_{j,k}^{[l]}$, the products are summed, and the bias $b_j^{[l]}$ is added. Equivalently, it is row $j$ of $\mathbf{W}^{[l]}$ applied to the input column, which is why the matrix form below is compact.

Forward Equations

With the dimensions established, the forward pass for a single layer reduces to two equations. Consider a layer $l$, and let

• $\mathbf{a}^{[l-1]{}(i)} \in \mathbb{R}^{n_{l-1}}$ be the input activations from the previous layer for example $i$,
• $\mathbf{W}^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}$ be the weight matrix for layer $l$,
• $\mathbf{b}^{[l]} \in \mathbb{R}^{n_l}$ be the bias vector for layer $l$,
• $g^{[l]}(\cdot)$ be the activation function used in layer $l$ (e.g. ReLU, sigmoid).

The layer applies its two operations in sequence.

Linear step

Combine the inputs with the weights and add the bias; this is Equation 2.6 written for all neurons simultaneously:

The dimensions are consistent: $(n_l \times n_{l-1})\times(n_{l-1}\times 1)$ yields an $n_l \times 1$ column, and adding $\mathbf{b}^{[l]}\in\mathbb{R}^{n_l}$ preserves that size.

Activation step

Pass each score through the nonlinearity:

where $g^{[l]}$ is applied element-wise to the components of $\mathbf{z}^{[l]{}(i)}$. The nonlinearity is essential to the model's expressiveness: a stack of purely linear layers is equivalent to a single linear map, so without $g$ the network could represent only linear functions.

Cache

During forward propagation, we store the intermediate values required to compute gradients during the backward pass:

Caching is essential rather than a convenience. Backpropagation reuses precisely these quantities — $\mathbf{a}^{[l-1]{}(i)}$ for the weight gradient, $\mathbf{z}^{[l]{}(i)}$ for the slope of the nonlinearity, and $\mathbf{W}^{[l]}$ to pass the signal to the previous layer — so they are stored during the forward pass rather than recomputed.

Forward Propagation: $m$ Training Examples (Vectorized)

Processing one example at a time is conceptually clear but inefficient. In practice the entire batch is propagated through the network simultaneously, replacing $m$ small matrix–vector products with a single matrix–matrix product. The underlying mathematics is unchanged: the per-example columns are simply stacked into matrices.

The input matrix is formed by placing each example's activation column side by side:

Understanding the structure of $\mathbf{A}^{[l-1]}$

Written out entry by entry:

The layout is the same as always — rows are features/neurons, columns are examples:

• Rows ($j = 1, \ldots, n_{l-1}$): correspond to each of the $n_{l-1}$ neurons in layer $l-1$.
• Columns ($i = 1, \ldots, m$): correspond to each of the $m$ training examples.
• Entry $a_j^{[l-1]{}(i)}$: the activation of the $j$-th neuron in layer $l-1$ for training example $i$.

Bias Broadcasting

When all $m$ examples are processed together, the product $\mathbf{W}^{[l]}\mathbf{A}^{[l-1]}$ already has the correct shape ($n_l \times m$), but the same bias vector $\mathbf{b}^{[l]}$ must be added to every column, since every example uses the same biases. This is written as the outer product $\mathbf{b}^{[l]}\mathbf{1}_m^T$, which copies $\mathbf{b}^{[l]}$ across all $m$ columns:

The factor $\mathbf{1}_m^T$ (a row of $m$ ones) formally expresses the repetition of the bias across all $m$ columns. In NumPy and similar libraries it is not written explicitly; the addition is handled automatically by broadcasting:

Z = W @ A_prev + b   # b is broadcast across columns

Vectorized forward equations

The two operations now apply to the entire batch simultaneously.

Linear step

The single matrix product $\mathbf{W}^{[l]}\mathbf{A}^{[l-1]}$ computes the scores for all examples simultaneously; we then add the broadcast bias from Equation 3.3:

where $\mathbf{1}_m \in \mathbb{R}^m$ is a vector of ones and $\mathbf{Z}^{[l]} \in \mathbb{R}^{n_l \times m}$. Column $i$ of this equation is exactly the single-example Equation 2.7 — the batch version is just all those columns computed together. In code the broadcast is implicit:

Z = W @ A_prev + b

Activation step

Apply the nonlinearity element-wise to every entry:

where the activation function is again applied element-wise.

Cache

As before, we store what backprop will need — now in matrix form:

Backward Propagation: Single Training Example

Forward propagation determines the network's prediction; backward propagation determines how to adjust the network to reduce the error. Its purpose is to compute the gradient of the loss with respect to every weight and bias, which gradient descent then uses to update the parameters.

Backpropagation is based on the chain rule. Because the loss depends on a weight only through a chain of intermediate quantities ($w \to z \to a \to \cdots \to L$), its derivative is the product of the local derivatives along that chain. Backpropagation organizes this computation so that factors shared between derivatives are reused rather than recomputed.

We assume that the gradient of the loss $L$ with respect to this layer's activations for example $i$ is already known; it is the signal supplied by the subsequent layer:

From this single incoming signal, three quantities must be produced: the gradients of this layer's parameters, and the gradient passed to the preceding layer:

The first two ($d\mathbf{W}^{[l]}, d\mathbf{b}^{[l]}$) update the current layer; the third ($d\mathbf{a}^{[l-1]{}(i)}$) becomes the incoming signal for the preceding layer, which is what propagates the error backward. Using the cached quantities $\mathbf{a}^{[l-1]{}(i)}, \mathbf{W}^{[l]}, \mathbf{b}^{[l]}, \mathbf{z}^{[l]{}(i)}$, we first record the shape of each gradient, then derive the four from the chain rule and state them in vector form.

Gradient shapes

Before deriving the gradients, we fix the shape of each one. A gradient always has the same shape as the thing it differentiates, so the three objects produced by a layer's backward pass — $d\mathbf{W}^{[l]}$, $d\mathbf{Z}^{[l]}$, $d\mathbf{b}^{[l]}$ — mirror the forward $\mathbf{W}^{[l]}$, $\mathbf{Z}^{[l]}$, $\mathbf{b}^{[l]}$ exactly.

Weight Gradient Matrix $d\mathbf{W}^{[l]}$

The gradient of the loss with respect to the weights, $d\mathbf{W}^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}$, has the same shape as $\mathbf{W}^{[l]}$ — one partial derivative per weight:

Each Entry $\frac{\partial L}{\partial w_{j,k}^{[l]}}$ measures how the loss responds to a small change in that single weight; this is the quantity gradient descent uses to update the weight. Over the full batch, the per-example contributions are averaged:

Pre-activation Gradient Matrix $d\mathbf{Z}^{[l]}$

The gradient with respect to the pre-activations, $d\mathbf{Z}^{[l]} \in \mathbb{R}^{n_l \times m}$, is the central quantity in backpropagation: the error signal at each neuron's pre-activation.

Its Entry $dz_j^{[l]{}(i)} = \frac{\partial L}{\partial z_j^{[l]{}(i)}}$ is the gradient of the loss with respect to the pre-activation of neuron $j$ in layer $l$ for example $i$, obtained by passing the activation gradient back through the nonlinearity:

where $g^{[l]\prime}$ is the derivative of the activation function. This relation is derived from the chain rule in Section 4.2 below; it is the product of the upstream gradient and the local slope of the nonlinearity.

Bias Gradient Vector $d\mathbf{b}^{[l]}$

The gradient with respect to the biases, $d\mathbf{b}^{[l]} \in \mathbb{R}^{n_l}$, has one entry per neuron:

Each Entry $\frac{\partial L}{\partial b_j^{[l]}}$ measures how much the loss changes with respect to the bias of neuron $j$. Because the bias is added directly to the score, its gradient is simply the score's gradient, summed (averaged) over the examples:

In NumPy: db = (1/m) * np.sum(dZ, axis=1, keepdims=True), which averages $d\mathbf{Z}$ over the example axis.

The chain rule, component by component

Every equation in this section follows from one principle: a parameter influences the loss $L$ only through the pre-activation $z_j^{[l]}$, which determines the activation $a_j^{[l]}$ and, through it, the rest of the network. The chain rule multiplies the local derivatives along this path. Each result is derived below for a single scalar component, with the example index $(i)$ suppressed for clarity; the vector and matrix forms then follow by collecting components.

The pre-activation gradient comes from the activation $a_j^{[l]} = g^{[l]}(z_j^{[l]})$, which depends on $z_j^{[l]}$ alone, so the chain rule contributes a single factor:

$$ dz_j^{[l]} = \frac{\partial L}{\partial z_j^{[l]}} = \frac{\partial L}{\partial a_j^{[l]}}\,\frac{\partial a_j^{[l]}}{\partial z_j^{[l]}} = da_j^{[l]}\, g^{[l]\prime}(z_j^{[l]}). $$

Collecting this over all neurons $j$ yields the Hadamard product in Equation 4.9.

The weight gradient uses the linear step $z_j^{[l]} = \sum_{k} w_{j,k}^{[l]} a_k^{[l-1]} + b_j^{[l]}$: the weight $w_{j,k}^{[l]}$ enters only $z_j^{[l]}$, with $\partial z_j^{[l]}/\partial w_{j,k}^{[l]} = a_k^{[l-1]}$. Therefore

$$ \frac{\partial L}{\partial w_{j,k}^{[l]}} = \frac{\partial L}{\partial z_j^{[l]}}\,\frac{\partial z_j^{[l]}}{\partial w_{j,k}^{[l]}} = dz_j^{[l]}\, a_k^{[l-1]}, $$

and stacking over all $j$ and $k$ gives the outer product in Equation 4.10.

The bias gradient is the simplest case: the bias enters $z_j^{[l]}$ additively, so $\partial z_j^{[l]}/\partial b_j^{[l]} = 1$ and the factor collapses,

$$ \frac{\partial L}{\partial b_j^{[l]}} = \frac{\partial L}{\partial z_j^{[l]}}\,\frac{\partial z_j^{[l]}}{\partial b_j^{[l]}} = dz_j^{[l]}, $$

which is Equation 4.11.

The previous-activation gradient is the one case that requires a sum. The activation $a_k^{[l-1]}$ feeds into every pre-activation $z_j^{[l]}$ of the current layer, so the loss depends on it through all $n_l$ neurons, and the multivariable chain rule adds the contribution of each path:

$$ da_k^{[l-1]} = \frac{\partial L}{\partial a_k^{[l-1]}} = \sum_{j=1}^{n_l} \frac{\partial L}{\partial z_j^{[l]}}\,\frac{\partial z_j^{[l]}}{\partial a_k^{[l-1]}} = \sum_{j=1}^{n_l} dz_j^{[l]}\, w_{j,k}^{[l]}. $$

The sum over $j$ is exactly the $k$-th entry of $(\mathbf{W}^{[l]})^{T} d\mathbf{z}^{[l]}$, which is Equation 4.12. This is the general pattern to remember: when a quantity affects the loss through several paths, the chain rule adds each path's contribution.

Backward equations (single example)

The same four results, now written directly in vector and matrix form.

Through the activation

The signal is first propagated back through the nonlinearity. Since $\mathbf{a}^{[l]{}(i)} = g^{[l]}(\mathbf{z}^{[l]{}(i)})$ element-wise, the chain rule introduces a factor of the slope $g^{[l]\prime}$ evaluated at each neuron's pre-activation:

where $g^{[l]\prime}$ denotes the derivative of the activation function, applied element-wise. The Hadamard product $\odot$ appears, rather than a matrix product, because the nonlinearity acts on each neuron independently: neuron $j$'s pre-activation affects only neuron $j$'s activation. This is Equation 4.6 in vector form.

Gradient with respect to weights

Since $\mathbf{z}^{[l]{}(i)} = \mathbf{W}^{[l]}\mathbf{a}^{[l-1]{}(i)} + \mathbf{b}^{[l]}$, each weight $w_{j,k}$ is multiplied by the input $a_k^{[l-1]{}(i)}$, so its gradient is the score's gradient scaled by that input. Collecting all $j,k$ gives an outer product:

which has shape $n_l \times n_{l-1}$, matching $\mathbf{W}^{[l]}$ as any weight gradient must. (Shape check: $(n_l\times1)\times(1\times n_{l-1})=n_l\times n_{l-1}$.) The outer product expresses each weight's gradient as the product of the error at its output neuron and the activation of its input neuron.

Gradient with respect to biases

Because the bias is added directly to the pre-activation, a unit change in $b_j$ changes $z_j$ by one unit, so the local derivative is $1$. The bias gradient is therefore equal to the pre-activation gradient:

since each component of $\mathbf{b}^{[l]}$ is added directly to the corresponding component of $\mathbf{z}^{[l]{}(i)}$.

Gradient with respect to previous activations

Finally, the signal is passed to the preceding layer. Each input activation $a_k^{[l-1]{}(i)}$ influences every pre-activation $z_j^{[l]{}(i)}$ through the weight $w_{j,k}$; these contributions are collected by the transpose of $\mathbf{W}^{[l]}$:

The transpose reflects the same weights used in the opposite direction: in the forward pass, $\mathbf{W}^{[l]}$ maps layer $l-1$ to layer $l$; in the backward pass, $(\mathbf{W}^{[l]})^T$ maps the error in layer $l$ back to layer $l-1$. (Shape check: $(n_{l-1}\times n_l)\times(n_l\times1)=n_{l-1}\times1$.)

These four equations are sufficient to implement backpropagation for a single layer and a single training example. Applying them from the final layer to the first — feeding each layer's Equation 4.12 into the preceding layer's Equation 4.9 — propagates the error through the entire network.

Backward Propagation: $m$ Training Examples (Vectorized)

As in the forward pass, the batch is processed at once by stacking the per-example columns. The four steps are unchanged except for a factor of $\tfrac{1}{m}$, which arises because the cost is the average loss over the batch (see Table 1.1).

We stack the incoming activation gradients into a matrix:

Understanding the structure of $d\mathbf{A}^{[l]}$

• Layer index $[l]$: this indicates we are at layer $l$ of the network (e.g., $l = 1, 2, \ldots, L$).
• Example index $(i)$: each column corresponds to training example $i$, where $i = 1, 2, \ldots, m$.
• Number of neurons $n_l$: layer $l$ has $n_l$ neurons, so each column vector $d\mathbf{a}^{[l]{}(i)}$ has $n_l$ components.

The full matrix structure is:

where:

• Rows ($j = 1, \ldots, n_l$): correspond to each of the $n_l$ neurons in layer $l$.
• Columns ($i = 1, \ldots, m$): correspond to each of the $m$ training examples.
• Entry $da_j^{[l]{}(i)}$: the gradient of the loss with respect to the $j$-th neuron's activation in layer $l$ for training example $i$.

Using the cached matrices $\mathbf{Z}^{[l]}, \mathbf{A}^{[l-1]}, \mathbf{W}^{[l]}$, we compute the batched versions of the four steps.

Vectorized backward equations

Through the activation

Applied element-wise, exactly as in the single-example case, but over the entire matrix:

Gradient with respect to weights

The single-example outer product in Equation 4.10 becomes a matrix product that sums over the examples automatically: the product $d\mathbf{Z}^{[l]}(\mathbf{A}^{[l-1]})^T$ accumulates each example's contribution, and the factor $\tfrac{1}{m}$ converts the sum into an average:

This is Equation 4.4 in matrix form. (Shape check: $(n_l\times m)\times(m\times n_{l-1}) = n_l\times n_{l-1}$ — the $m$ contracts away, which is the summation over examples.)

Gradient with respect to biases

Averaging Equation 4.11 over the batch means summing $d\mathbf{Z}^{[l]}$ across its columns, which the product with $\mathbf{1}_m$ performs:

which corresponds, in NumPy, to summing over the example axis:

db = (1/m) * np.sum(dZ, axis=1, keepdims=True)

Gradient with respect to previous activations

The signal passed to the preceding layer, identical to Equation 4.12 but applied to all columns at once. No factor of $\tfrac{1}{m}$ appears here, because this is a gradient of activations rather than an averaged parameter update; each example retains its own signal:

As before, these four equations are sufficient to implement backpropagation for a single layer in the fully vectorized setting. Applying them from layer $L$ back to layer $1$ yields the gradients for the entire network.

Initial Gradient at the Final Layer (Binary Classification)

One element remains to be specified. Every backward step above assumes that $d\mathbf{a}^{[l]{}(i)}$, the gradient of the loss with respect to the layer's activations, is already available. The initial such gradient — the gradient of the loss with respect to the network's output — is obtained directly from the loss function.

Consider binary classification with a sigmoid output layer $L$ having $n_L = 1$ neuron (a single output). For a single training example $i$, the single output is the predicted probability:

where:

• $a_1^{[L]{}(i)}$: activation of neuron 1 (the only neuron) in the final layer $L$ for training example $i$.
• $\hat{y}^{(i)}$: the predicted probability for example $i$.
• $y^{(i)}$: the true binary label for example $i$.

The appropriate loss for a predicted probability is the logistic (binary cross-entropy) loss, which is large when the prediction is both confident and incorrect:

Only one of the two terms is active per example: if $y^{(i)}=1$ the loss is $-\log \hat{y}^{(i)}$ (penalizing a low predicted probability of the true class), and if $y^{(i)}=0$ it is $-\log(1-\hat{y}^{(i)})$. Differentiating Equation 6.2 with respect to the output activation gives:

This quantity is the required initial gradient. We denote it:

This serves as the starting point for backpropagation at the final layer: substituting it into Equation 4.9 (with $g^{[L]}$ the sigmoid) initiates the backward pass.

Simplification for the sigmoid. When the output nonlinearity is the sigmoid, $g^{[L]\prime} = a(1-a)$. Substituting Equation 6.4 into Equation 4.9, the denominators cancel and the expression reduces to $dz_1^{[L]{}(i)} = a_1^{[L]{}(i)} - y^{(i)}$ — the prediction minus the target. This is a principal reason the sigmoid and the cross-entropy loss are commonly paired.

General case: multiple output neurons ($n_L > 1$)

For multi-class classification or multiple outputs, the final layer $L$ has $n_L$ neurons, each with its own initial gradient. The activation and gradient for neuron $j$ in layer $L$ for example $i$ are:

Vectorized form for all $m$ examples

Stacking predictions and labels in the usual layout (neurons in rows, examples in columns), let $\mathbf{A}^{[L]}, \mathbf{Y} \in \mathbb{R}^{n_L \times m}$:

where:

• Rows ($j = 1, \ldots, n_L$): correspond to each output neuron in layer $L$.
• Columns ($i = 1, \ldots, m$): correspond to each training example.
• Entry $a_j^{[L]{}(i)}$: predicted output of neuron $j$ for example $i$.
• Entry $y_j^{(i)}$: true label for output $j$ of example $i$.

For binary classification with $n_L = 1$, both matrices reduce to row vectors in $\mathbb{R}^{1 \times m}$ — one row of predictions and one row of labels.

The initial gradient for the whole batch is then given by a single element-wise expression:

where the divisions and operations are applied element-wise, and each entry is just Equation 6.4 written per neuron and per example:

This $d\mathbf{A}^{[L]}$ is the input to the backward step of the final layer. The layer-by-layer backward equations of Section 5 are then applied recursively from layer $L$ down to layer $1$. This completes the forward–backward cycle: the network computes a prediction, the error is measured, the gradient is initialized at the output, and it is propagated backward to update every parameter.