# 2. Neural Networks :::{image} pics/mlp.png :width: 60% :align: center ::: A linear classifier can represent only a limited family of relationships between inputs and outputs. When the structure of a problem is more complicated, training for longer does not necessarily help. The optimizer may be working correctly, the loss may be decreasing, and the implementation may contain no errors. The model may simply be unable to represent the relationship required by the task. Neural networks address this limitation by combining several learned transformations with nonlinear activation functions. This allows the model to construct intermediate representations that make more complex predictions possible. The central question of this lesson is whether the model can represent a sufficiently good solution, not whether the optimizer can find one.
Loading interactive ReLU visualizations…
## 1. Limits of a linear classifier Consider a dataset with two input features. Each example can be represented as a point on a two-dimensional plane. Suppose one class forms a small group near the centre while the second class surrounds it. A possible arrangement might look like this; the exact coordinates are not important. :::{image} pics/concentric.png :width: 400px :align: center ::: A linear classifier separates classes using a straight decision boundary. No matter where we place or rotate a straight line, it cannot separate the inner class from the surrounding class. Improving the weights may move the line to a better position, but some examples will always remain on the wrong side. A linear classifier therefore has a representational limitation. It cannot describe every possible relationship between the features and the target. ### Linear decision boundary For binary classification, a linear classifier calculates a score such as $$ z = xW+b. $$ The prediction changes when the score crosses a threshold. In two dimensions, the inputs producing exactly the threshold form a straight line. In three dimensions, they form a plane. In higher-dimensional spaces, the corresponding boundary is called a hyperplane. :::{note} :class: dropdown Strictly speaking, $xW+b$ is an **affine transformation** because it includes a bias term. PyTorch and most deep-learning texts nevertheless use the conventional term **linear layer**, and we will follow that convention. ::: Training can only search among the functions that the model is capable of representing. If the model is incapable of representing the boundary required to distinguish the classes, no amount of training will help. This gives us an important distinction: - **Optimization failure:** The model could represent a good solution, but training has not found suitable parameters. - **Capacity failure:** The model cannot represent a sufficiently good solution, regardless of how its parameters are trained. A model with too little capacity is said to **underfit** the problem. Its predictions remain poor because its family of possible functions is too restricted. ### Stacking linear layers A natural response is to add more linear transformations. Suppose the first transformation produces an intermediate vector: $$ h=xW_1+b_1, $$ and a second transformation converts it into logits: $$ z=hW_2+b_2. $$ This appears to create a deeper model. However, substituting the first equation into the second and expanding the expression produces a single linear transformation: $$ z=x \underbrace{W_1W_2}_{W'} + \underbrace{b_1W_2 + b_2}_{b'} = xW'+b'. $$ The above implementation may contain more tensors, but it has not gained the ability to represent a different relationship between input and output. A sequence of linear transformations is still linear. ## 2. Activation function To prevent several layers from collapsing into one linear transformation, we place a **nonlinear activation function** between them. A small neural network can be written as follows. $$ \begin{aligned} h &= \operatorname{ReLU}(xW_1+b_1)\\ z &= hW_2+b_2. \end{aligned} $$ The first layer transforms the original input into an intermediate vector. ReLU changes that vector nonlinearly. The second layer converts the resulting representation into class logits. In code, the computation is like this. ```python hidden_scores = features @ weights_1 + bias_1 hidden_activations = torch.relu(hidden_scores) logits = hidden_activations @ weights_2 + bias_2 ``` ### ReLU The Rectified Linear Unit, usually called **ReLU**, is defined as $$ \operatorname{ReLU}(a)=\max(0,a). $$ It keeps positive values and replaces negative values with zero. $$ \operatorname{ReLU} \left( \begin{bmatrix} 2.4 & -1.1 & 0.7 & -0.3 \end{bmatrix} \right) = \begin{bmatrix} 2.4 & 0 & 0.7 & 0 \end{bmatrix} $$ The operation is simple, but its position between learned layers is crucial. Once ReLU has changed the intermediate values, the surrounding matrix multiplications can no longer be merged into one equivalent linear transformation. The model can now treat different regions of the input space differently. Some hidden units may be active for one group of inputs and inactive for another. The output layer combines these activation patterns to form more complex decision boundaries. ### Piecewise-linear boundary ReLU does not make the network an arbitrary curved function. Within a region where the same hidden units remain active, the network behaves like a linear model. When an input crosses a point at which a hidden unit switches between active and inactive, the effective linear relationship changes. The complete function is therefore **piecewise linear**. A useful geometric picture is to imagine the input space divided into polyhedral regions. Each region has its own linear behaviour, while the boundaries between regions are determined by the activation thresholds of the hidden units. By combining many such regions, the network can approximate relationships that a single linear model cannot represent. *(See the widget at the top for an interactive demonstration.)* This explains why nonlinear activation functions are essential. Stacking linear layers without nonlinear activations would cause the entire network to collapse into a single affine transformation, regardless of its depth. ReLU prevents this collapse by changing the effective transformation from one region of the input space to another. Depth then allows the network to compose these transformations and create increasingly complex arrangements of regions and decision boundaries. ## 3. Forward pass Consider a flower represented by four standardized measurements: $$ x= \begin{bmatrix} 0.5 & -1.0 & 2.0 & 1.5 \end{bmatrix}. $$ We will pass this input through a two-layer network with three hidden units and three output classes. ### Hidden representation Using the row-vector notation, the first weight matrix has shape $(4,3)$ and the first bias has shape $(3,)$. Let these parameters be the following. $$ W_1= \begin{bmatrix} 1.0 & 0.0 & -1.0 \\ 0.5 & 1.0 & 0.0 \\ 0.0 & 0.5 & 1.0 \\ 1.0 & -0.5 & 0.5 \end{bmatrix} \qquad b_1= \begin{bmatrix} 0.1 & 0.0 & -0.2 \end{bmatrix}. $$ The first layer calculates $a=xW_1+b_1$ and yields the following vector of **pre-activations**. $$ a= \begin{bmatrix} 1.6 & -0.75 & 2.05 \end{bmatrix}. $$ Applying ReLU produces the final **activations**, also called the **hidden representation** of the input. $$ h= \operatorname{ReLU}(a) = \begin{bmatrix} 1.6 & 0 & 2.05 \end{bmatrix}. $$ The negative second pre-activation has become zero. For this particular input, the second unit is inactive. ### Class logits The output layer receives the three hidden activations and produces three class logits. Let the second weight matrix and bias be the following. $$ W_2= \begin{bmatrix} 1.0 & 0.0 & -0.5 \\ 0.5 & 1.0 & 0.0 \\ -1.0 & 0.5 & 1.0 \end{bmatrix} \qquad b_2= \begin{bmatrix} 0.2 & -0.1 & 0.0 \end{bmatrix}. $$ The output is $z=hW_2+b_2$, which gives the following logits. $$ z= \begin{bmatrix} -0.25 & 0.925 & 1.25 \end{bmatrix}. $$ The third class receives the largest logit, so the model predicts class index $2$. This example uses fixed numbers so that the forward computation is visible. During training, the weight matrices and biases are not fixed. Their values are adjusted by gradient-based optimization. ### Batch dimensions For one flower, the shape progression is like this. ```text input features: (4,) hidden pre-activations: (3,) hidden activations: (3,) class logits: (3,) ``` For a batch of 120 flowers, the first dimension represents the number of examples in the batch and remains present throughout the forward pass. ```text input batch: (120, 4) hidden pre-activations: (120, 3) hidden activations: (120, 3) class logits: (120, 3) ``` The model applies the same parameters to every example in the batch. :::{admonition} Quiz :class: dropdown A network receives a batch with shape `(64, 10)`. Its hidden layer contains 20 units, and the task has four classes. What shapes should the two weight matrices have when using the row-vector notation? What should the final logit shape be?
Answer The first weight matrix has shape (10, 20), and the second has shape (20, 4). The final logits have shape (64, 4).
::: ### Learned representations The hidden layer converts the input features into a representation that the output layer can use. For the flower dataset, one hidden unit may respond strongly to flowers with a particular combination of petal length and width. Another may react to unusual proportions between sepals and petals. The model is therefore creating new features from the original ones. The hidden representation is learned because the weights that produce it are adjusted to reduce the classification loss. The representation learned by the hidden layer is usually difficult for humans to understand. A unit is not guaranteed to become a simple “large petal detector” or “setosa detector.” Its role is determined jointly with the rest of the network, and useful information may be distributed across several units. It is often safer to think of a hidden layer as a new coordinate system constructed for the task. ## 4. Backward pass The forward pass transforms the input features into hidden activations, logits, and finally a loss. The backward pass operates in the reverse direction to calculate the gradient of the loss with respect to every model parameter. These gradients describe how a small change in each parameter would affect the loss. They will be used by an optimizer to update the parameters in a way that is expected to reduce the loss. *NOTE: The backward pass is also called **backpropagation** because it propagates gradients backwards through the network using the chain rule for differentiation of composite functions.* ### Through the loss For softmax followed by cross-entropy, the gradient with respect to the logits has a particularly simple form. $$ \frac{\partial L}{\partial z} = p - y_{\text{one-hot}} $$ The forward pass produced the logits $z=[-0.25, 0.925, 1.25]$. Applying softmax to the logits gives the class probabilities $p= [0.115, 0.371, 0.514]$. Suppose the correct target is class index $1$, with a one-hot encoding of $[0, 1, 0]$. The gradient of the loss with respect to the logits is therefore $[0.115, -0.629, 0.514]$. The gradient for the correct class is negative. This tells us that increasing the correct-class logit would reduce the loss. The backward pass is already expressing the correction the model needs to make. :::{note} The logits themselves are not trainable parameters. Backpropagation must transfer this information through the output layer and into its weights, biases, and hidden inputs. ::: ### Through the output layer The output layer is calculated as $z = h W_2+b_2$ for one hidden representation $h$. The gradient with respect to the layer parameters is obtained by applying the chain rule. $$ \frac{\partial L}{\partial W_2} = h^\top\frac{\partial L}{\partial z} \qquad \frac{\partial L}{\partial b_2} = \frac{\partial L}{\partial z} $$ These derivatives combine the hidden activation that passed through the connection with the gradient arriving from the corresponding output logit. To continue backwards, we also need the gradient with respect to the hidden representation, using the chain rule again. $$ \frac{\partial L}{\partial h} = \frac{\partial L}{\partial z}W_2^\mathsf{T} $$ The output layer received the hidden representation $h = [1.6, 0, 2.05]$. Substituting the values of $h$ and $W_2$ in the above equation gives approximately $[-0.142, -0.571, 0.085]$. These values describe how changes to the three hidden activations would affect the loss. ### Through ReLU The hidden representation is calculated as $h=\operatorname{ReLU}(a)$. The derivative of ReLU is simply a step function. $$ \operatorname{ReLU}'(a)= \begin{cases} 1 & \text{if } a>0,\\ 0 & \text{if } a\leq 0. \end{cases} $$ The gradient with respect to the pre-activations is obtained by multiplying the incoming gradient with it. $$ \frac{\partial L}{\partial a} = \frac{\partial L}{\partial h} \odot \operatorname{ReLU}'(a) $$ The pre-activations were $a=[1.6, -0.75, 2.05]$, so the ReLU derivative is $[1, 0, 1]$. Plugging the known values in the above equation yields $[-0.142, 0, 0.085]$. The second derivative has become zero because the second unit was inactive during the forward pass. ReLU therefore affects both directions of computation: it sets negative activations to zero in the forward pass and blocks their gradients in the backward pass. ### Through the hidden layer The hidden layer is calculated as $a=xW_1+b_1$. The gradient is obtained by applying the chain rule again. $$ \frac{\partial L}{\partial W_1} = x^\mathsf{T}\frac{\partial L}{\partial a} \qquad \frac{\partial L}{\partial b_1} = \frac{\partial L}{\partial a} $$ The backward pass has now produced gradients for all four parameter tensors $W_1, b_1, W_2, b_2$. ### Chain rule A parameter in the hidden layer does not affect the loss directly. It changes a hidden pre-activation, which changes a hidden activation, which changes the logits, which changes the loss. Backpropagation uses the **chain rule** to connect these effects. Schematically, the dependency can be represented as $$ \frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial z} \frac{\partial z}{\partial h} \frac{\partial h}{\partial a} \frac{\partial a}{\partial W_1}. $$ Because these quantities include vectors and matrices, the formal calculation involves Jacobians and tensor operations. Backpropagation performs the required operations efficiently without constructing every complete Jacobian explicitly. The important idea is that each operation receives a gradient from the operation after it, combines that gradient with its own local derivative, and passes the result backwards. :::{note} An important pattern is that every gradient has the same shape as the corresponding parameter. | Tensor | Shape | Gradient shape |--------|----------|---------------- | $W_1$ | `(4, 3)` | `(4, 3)` | $b_1$ | `(3,)` | `(3,)` | $a$ | `(3,)` | `(3,)` | $h$ | `(3,)` | `(3,)` | $W_2$ | `(3, 3)` | `(3, 3)` | $b_2$ | `(3,)` | `(3,)` | $z$ | `(3,)` | `(3,)` ::: ### Automatic differentiation Calculating every gradient manually is useful for understanding the process, but deep-learning libraries perform these calculations automatically. During the forward pass, PyTorch records the tensor operations used to calculate the loss. Together, these operations form a computational graph. When we call `loss.backward()`, PyTorch traverses the graph in reverse and calculates the gradient of the loss with respect to every registered parameter involved in the computation. The gradients are stored in each parameter’s `.grad` attribute. The parameters themselves are not changed. :::{admonition} Quiz :class: dropdown A hidden unit has a negative pre-activation for one example. What happens to the gradient passing through that unit during the backward pass?
Answer ReLU blocks the gradient because its derivative is zero for a negative pre-activation.
::: ## 5. Multilayer perceptron A neural network made from linear layers and activation functions is commonly called a **multilayer perceptron (MLP)**. A simple MLP for the flower problem might use the following architecture: - 4 input features, - 8 hidden units, - ReLU activation, - 3 class logits. The first layer is called a *hidden layer* because its output is internal to the model. The final layer is called the *output layer* because it produces the logits used by the loss and prediction rule. ### PyTorch model In principle, the weights and biases of a neural network can be created directly, and the forward computation can refer to these tensors explicitly. As models become larger, manually managing every parameter tensor becomes cumbersome. PyTorch provides `nn.Module` as the basic abstraction for models and layers. ```python from torch import nn class FlowerClassifier(nn.Module): def __init__(self, n_features: int, n_hidden: int, n_classes: int): super().__init__() self.hidden = nn.Linear(n_features, n_hidden) self.output = nn.Linear(n_hidden, n_classes) def forward(self, features): hidden_scores = self.hidden(features) hidden_activations = torch.relu(hidden_scores) logits = self.output(hidden_activations) return logits ``` The constructor creates the layers and stores them as attributes of the model. Each `nn.Linear` layer owns a weight matrix and a bias vector. The `forward` method defines how the data moves through those layers. We can then create the model as a Python object. ```python model = FlowerClassifier(n_features=4, n_hidden=8, n_classes=3) ``` Calling the model runs the forward computation. ```python logits = model(features) ``` PyTorch recommends calling `model(features)` rather than invoking `model.forward(features)` directly. Calling the module allows the framework to perform additional work around the forward pass. :::{admonition} No output activation :class: danger The output layer returns raw logits. ReLU is not applied to them. Class logits must be free to take positive or negative values, and restricting them to non-negative values provides no benefit for cross-entropy classification. We also do not apply softmax before the loss. PyTorch’s cross-entropy function performs the required normalization internally using a numerically stable computation. ::: ### Registered parameters Because the linear layers are stored inside the module, PyTorch automatically recognizes their weights and biases as trainable parameters. This code prints the names and shapes of every registered parameter. ```python for name, parameter in model.named_parameters(): print(name, parameter.shape) ``` ```text hidden.weight torch.Size([8, 4]) hidden.bias torch.Size([8]) output.weight torch.Size([3, 8]) output.bias torch.Size([3]) ``` PyTorch stores the weight matrix of `nn.Linear` using the shape `(output features, input features)`. This is the transpose of the row-vector notation used in our equations. Internally, `nn.Linear` performs the equivalent affine transformation. The `model.parameters()` function returns the model parameters without their names. We can use this method to pass the parameters to an optimizer without having to list them explicitly. It is an abstraction that reduces bookkeeping, but the underlying objects remain weight and bias tensors with stored gradients. :::{admonition} Quiz :class: dropdown Suppose a tensor is stored inside a module as an ordinary tensor rather than as part of a registered layer. Will it automatically appear in `model.parameters()`?
Answer No. Ordinary tensors are not automatically registered as trainable parameters.
::: :::{dropdown} Custom parameters In PyTorch, `nn.Parameter` is a special kind of tensor that represents a model’s learnable weight. When you assign it to a module, it gets automatically added to that module’s `parameters()` list. ::: ### Sequential models When the model follows one direct chain of operations, it can be expressed more compactly with `nn.Sequential`. The following code creates a model with the same architecture as `FlowerClassifier`. ```python model = nn.Sequential( nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 3), ) ``` The general computation is the same: the first layer produces a hidden representation, ReLU changes it nonlinearly, and the second layer produces class logits. Once constructed, a sequential model is called in the same way as a custom module. ```python logits = model(features) ``` The `nn.Sequential` class is convenient for simple chains. A custom `nn.Module` is more useful when the forward computation has multiple branches, several inputs, skip connections, or intermediate outputs. Neither form is inherently more powerful. They are two ways of organizing the same layers and parameters. ### Training with an optimizer It is entirely possible to update the model parameters manually. ```python with torch.no_grad(): weights -= learning_rate * weights.grad bias -= learning_rate * bias.grad ``` But this bookkeeping is tedious and error-prone. PyTorch provides optimizers that perform the update automatically for every registered model parameter. ```python optimizer = torch.optim.SGD( model.parameters(), lr=0.1, ) ``` A training step becomes: ```python optimizer.zero_grad() logits = model(features) loss = F.cross_entropy(logits, targets) loss.backward() optimizer.step() ``` Each line has a specific role. - `optimizer.zero_grad()` clears gradients left from the previous update. - `model(features)` performs the forward pass. - `F.cross_entropy` calculates the objective. - `loss.backward()` computes gradients for every registered parameter involved in the loss. - `optimizer.step()` updates those parameters according to the optimizer’s rule. The optimizer defines how the parameters change when their gradients are available. It does not decide what the model should predict or which loss is appropriate. Those choices come from the task formulation. :::{note} PyTorch API shortens the implementation, but the underlying learning cycle remains the same. ::: ## 6. Architecture choices The architecture of an MLP includes choices such as the number of hidden layers, the number of units in each layer, and the activation functions. Some dimensions are determined by the task. A dataset with four input features requires the first layer to accept four values. A three-class classification task requires the final layer to produce three logits. Hidden dimensions are hyperparameters chosen by the user. A model with more hidden units or layers has greater potential capacity than a smaller model. It does not automatically have better test performance. A larger model can still fail because the learning rate is unsuitable, the loss is incorrect, the data contains errors, or the training procedure is unstable. It may also fit the training examples very closely while generalizing poorly to unseen data, a phenomenon called **overfitting**. Architecture choices are hypotheses that must be tested by controlled experiments. A useful diagnostic question is therefore the following. > Is the model unable to represent the relationship, or has training failed to find suitable parameters? Capacity and optimization are different problems, even though they interact. :::{seealso} [Neural Network Playground](https://playground.tensorflow.org) -- OR -- [Playground With More Options](https://playground.scienxlab.org/) ::: ### Depth efficiency Deep neural networks (more layers with less units) are more expressive than shallow networks (fewer layers with more units) because they can divide the input space into many more linear regions without requiring a proportional increase in parameters. As network depth increases, the number of regions can grow very rapidly, and this effect becomes even stronger for higher-dimensional inputs. This means that, for the same parameter budget, a deep network can represent much more complex functions than a shallow one. However, these extra regions are not completely independent: they are linked by structural dependencies and symmetries created by the repeated transformations of each layer. Therefore, having more regions is most useful when the target function has a similar hierarchical or compositional structure. This leads to the idea of **depth efficiency**: although both shallow and deep networks can approximate arbitrary functions, certain functions can be represented efficiently by a deep network but would require exponentially more hidden units in a shallow network. The main open question is whether the real-world functions we want to model actually possess this kind of structure. ## 7. Conclusion A neural network processes data through a sequence of transformations. In its simplest form, it consists of linear layers separated by nonlinear activation functions. The hidden layers progressively transform the original features into a representation that the output layer can use to make predictions. The activation functions prevent the full sequence of layers from collapsing into a single linear transformation, allowing the network to behave as a nonlinear function with a more flexible decision boundary. --- ## Final quiz :::{admonition} Quiz — Model capacity Why might a correctly implemented and correctly trained linear classifier still perform poorly?
Answer The relationship between inputs and targets may require a function that a linear classifier cannot represent.
::: :::{admonition} Quiz — Nonlinearity Why is ReLU placed between linear layers?
Answer Without a nonlinear operation, the layers collapse into one equivalent linear transformation. ReLU allows different linear behaviours in different regions of the input space.
::: :::{admonition} Quiz — Hidden representations What is the purpose of a hidden layer?
Answer It transforms the original input features into an intermediate learned representation that the output layer can use to produce logits.
::: :::{admonition} Quiz — Output dimensions A dataset has seven input features and five classes. A model uses a hidden layer with 12 units. Which architecture has the correct dimensions?
Answer ```python nn.Sequential( nn.Linear(7, 12), nn.ReLU(), nn.Linear(12, 5), ) ```
::: :::{admonition} Quiz — Optimizer role Does the optimizer decide which loss function should be used?
Answer No. The loss is selected according to the task and model output. The optimizer uses gradients of that loss to update the parameters.
:::