# 6. Deeper Architectures :::{image} pics/resnet-line.png :width: 100% :align: center ::: A convolutional neural network builds an image representation by applying learned filters to local regions, combining the resulting feature maps through nonlinear transformations, and progressively reducing spatial resolution. Early layers operate on relatively simple local patterns, while deeper layers combine those responses into broader and more abstract features. The final representation is then converted into a prediction by a few additional layers. Deep convolutional networks can build rich representations by stacking many layers of convolution. However, increasing depth also makes optimization more difficult: information and gradients must travel through a longer sequence of operations, and additional capacity does not automatically translate into a model that trains effectively. Convolutional architectures use several mechanisms to make deeper networks easier to optimize and organize. This lesson introduces four important ones: batch normalization, residual connections, projection shortcuts, and global average pooling. ## 1. Batch Normalization Every layer in a neural network receives the representation produced by the layers before it. During training, those earlier layers are continuously changing because their parameters are being updated. The numerical distribution of an intermediate representation can therefore also change during optimization. Large differences in activation scale can make optimization more difficult. A layer may receive values concentrated in a narrow interval during one stage of training and values with a much larger magnitude later. The optimizer must then adjust several interacting layers while the numerical conditions under which they operate are themselves changing. **Batch normalization** introduces an explicit normalization operation into the network so that intermediate feature channels are presented at a controlled scale. ### Normalizing activations Consider a batch of intermediate activations with shape $(N,C,H,W)$, where $N$ is the batch size, $C$ the number of channels, and $H$ and $W$ the spatial dimensions. Batch normalization treats every channel independently. For channel $c$, it calculates a mean using all examples and all spatial positions in the batch: $$ \mu_c = \frac{1}{NHW} \sum_{n=1}^{N} \sum_{i=1}^{H} \sum_{j=1}^{W} x_{n,c,i,j}. $$ It also calculates a variance on the same values: $$ \sigma_c^2 = \frac{1}{NHW} \sum_{n=1}^{N} \sum_{i=1}^{H} \sum_{j=1}^{W} \left(x_{n,c,i,j}-\mu_c\right)^2. $$ The activations in that channel are then normalized: $$ \hat{x}_{n,c,i,j} = \frac{x_{n,c,i,j}-\mu_c} {\sqrt{\sigma_c^2+\epsilon}}, $$ where $\epsilon$ is a small constant that prevents numerical problems when the variance is close to zero. After this operation, the normalized values in a channel are centered near zero and have a controlled scale. ### Learned scale and shift Normalization alone would force every channel toward the same center and scale. The network may instead benefit from activations with a different magnitude or offset. Batch normalization therefore follows normalization with a learned affine transformation: $$ y_{n,c,i,j} = \gamma_c \, \hat{x}_{n,c,i,j} + \beta_c. $$ Each channel has its own learned parameters $\gamma_c$ and $\beta_c$ to adjust the scale and offset of its normalized activations. For example, a batch-normalization layer with 64 channels contains 64 learned scale parameters and 64 learned shift parameters. These parameters are updated through backpropagation along with the other parameters in the network. ### Effect of normalization Batch normalization does not add new spatial information and does not increase the receptive field. Instead, it controls the numerical distribution of intermediate activations. The normalization step ensures that each channel has zero mean and unit variance, while the learned scale and shift allow the network to adjust that distribution to a more useful form. The layer therefore has two effects: it keeps intermediate activations on a more controlled numerical scale while parameters throughout the network are changing, and it allows the network to learn an appropriate scale and offset for each channel. In many architectures, batch normalization makes optimization more stable and permits learning rates or depths that would otherwise be difficult to use. It is not guaranteed to improve every model, and its effect depends on factors such as batch size, architecture, and optimizer. Like other model-development choices, its usefulness must ultimately be evaluated experimentally. ### Training and evaluation During training, batch normalization uses the mean and variance calculated from the current mini-batch. This is natural because training already proceeds batch by batch, and the statistics can be computed directly from the activations available at that step. The consequence is that the normalization applied to one example depends partly on the other examples in the same batch. This is acceptable during training, where batch-to-batch variation is part of the optimization process. The situation is different during evaluation. A prediction should not depend on which unrelated examples happen to be evaluated alongside it. Consider the same image evaluated once in a batch of mostly dark images and once in a batch of mostly bright images. If normalization always used the current batch statistics, the image could be normalized differently in the two cases and therefore produce different predictions. Batch normalization avoids this problem by maintaining **running estimates** of each channel's mean and variance during training. These estimates accumulate information from many training batches and provide fixed normalization statistics for later evaluation. The layer therefore has two operating modes. | Mode | Statistics used | Running statistics | | ---------- | --------------------------------------------- | ------------------ | | Training | Mean and variance of the current mini-batch | Updated | | Evaluation | Running estimates accumulated during training | Kept fixed | This distinction explains why a model containing batch normalization must be explicitly switched between training and evaluation modes. During training, the layer uses the current batch and updates its stored statistics. During validation or testing, it uses the stored statistics instead, so that each prediction is independent of the other examples in the evaluation batch. ## 2. Residual Learning A conventional sequence of layers transforms its input $x$ into a new representation $y = F(x)$. Every piece of information passed to the next part of the network must therefore pass through the learned transformation. A **residual block** introduces a second path that carries the input directly to the output: $$ y=x+F(x). $$ The learned path that calculates $F(x)$ is called the **residual branch**, while the direct path carrying $x$ is called a **shortcut** or **skip connection**. The addition operation is element-wise, so the two branches must produce compatible tensors. If the residual branch changes the number of channels or spatial dimensions, the shortcut must also be transformed to match. ### Learning a residual A skip connection changes what the learned branch needs to learn. Suppose the representation $x$ is already useful and the best action for the block would be to preserve it. A conventional block must learn an approximation of the identity transformation. A residual block only needs its residual branch to produce something close to zero. The output then becomes $y=x+F(x)\approx x$. The identity transformation is therefore directly available through the shortcut. The residual branch can be interpreted as learning a correction to the input representation. If the best output is a small change to the input, the residual branch can learn that change directly. But the residual branch is not required to produce only small corrections. It may learn a substantial transformation when that is useful. The important structural property is that preserving an existing representation does not require the new layers to reconstruct it. ### Multiple paths A residual block contains two routes from its input to its output. The shortcut preserves the existing representation, while the residual branch processes the same representation through additional transformations. Their outputs are combined through addition. Consequently, adding more layers no longer means that information must pass exclusively through every new transformation. A representation can survive through the shortcut while the residual branch learns additional features. This changes the structure of a deep network considerably. A long sequence of residual blocks contains many paths of different lengths rather than one strictly sequential path through every convolution. Skip connections also affect backpropagation. For the residual block $y=x+F(x)$, the derivative of the block output with respect to its input is $$ \frac{\partial y}{\partial x} = I + \frac{\partial F(x)}{\partial x}. $$ The identity term $I$ means that the gradient from subsequent layers can propagate directly through the shortcut to previous layers. This makes it easier for the optimizer to adjust earlier layers in a deep network. Residual connections do not eliminate every optimization problem, and they do not guarantee that arbitrarily deep networks will train successfully. They provide shorter routes through which both activations and gradients can propagate, which makes much deeper architectures substantially more practical. ## 3. Projection Shortcuts A network cannot preserve the same tensor dimensions indefinitely. As convolutional representations become deeper, spatial resolution is usually reduced and the number of channels may increase. Suppose a residual block needs to change both the channel count and the spatial resolution of its input feature map. The output of the residual branch can no longer be added directly to the input. To preserve the residual structure, the shortcut must also be transformed. ### Learned projection To make the shortcut compatible with the residual output, we can learn a transformation that converts the input to the required shape. Instead of $y=x+F(x)$, the block now computes $$ y = P(x) + F(x), $$ where $P$ is a learned **projection** that converts the input to the required shape. A particularly convenient operation for this purpose is a $1\times1$ convolution. ### The 1-by-1 convolution A $1\times1$ convolution processes each spatial position independently. At a given position, it takes a vector containing one value from each input channel. The convolution combines these values using learned weights and produces a new vector of output channel values at the same position. Such a transformation is applied at every spatial location. A $1\times 1$ convolution therefore mixes information **across channels** without combining information from neighbouring positions. The operation is distinct from a larger convolution. A $k\times k$ kernel combines information across both channels and a local spatial neighbourhood, whereas a $1\times 1$ kernel only transforms the channel representation at each position. This makes $1\times 1$ convolutions particularly useful when a network needs to change the number of channels while preserving the spatial organization of the feature maps. They are also comparatively inexpensive. Transforming $d_{\text{in}}$ input channels into $d_{\text{out}}$ output channels with a $1\times 1$ convolution requires $d_{\text{in}} d_{\text{out}}$ weights, excluding bias terms. A $k\times k$ convolution with the same input and output channels requires $k^2$ times as many weights. A $1\times 1$ convolution can therefore change the channel dimension with substantially fewer parameters when no additional spatial processing is required. ### Changing spatial resolution A projection shortcut may need to change not only the number of channels, but also the spatial dimensions of the representation. This happens when the residual branch performs downsampling. Since the two branches must have the same shape before they can be added, the shortcut must reproduce the same change in spatial resolution. A $1\times 1$ convolution can perform this downsampling by using a stride greater than one. With stride $s$, the convolution is evaluated only at every $s$-th spatial position, so the height and width of its output are reduced accordingly. The projection can therefore change the channel dimension through its learned $1\times 1$ weights while changing the spatial resolution through its stride. ## 4. Global Average Pooling After a convolutional feature extractor, the network must convert its spatial feature maps into the representation used for classification. One approach is to flatten the remaining feature maps into a single vector. A linear classifier then produces class logits from that vector. This approach has two disadvantages. First, it ties the classifier to a particular spatial resolution. If the final feature maps are $7\times 7$, the classifier expects $7\times 7$ activations. If they are instead $8\times 8$, the classifier cannot be used without modification. Second, flattening produces a large number of values, which can make the classifier expensive in terms of parameters and computation. A different approach is to summarize each feature map. ### One value per channel **Global average pooling** replaces every feature map by its mean activation. For channel $c$, the pooled value is $$ z_c = \frac{1}{HW} \sum_{i=1}^{H} \sum_{j=1}^{W} x_{c,i,j}. $$ The pooling operation itself contains no learned parameters. Consider a representation containing 128 feature maps of size $7\times 7$. Global average pooling produces one value per channel, so the output contains 128 values, rather than the $128\times 7\times 7=6,272$ values that would result from flattening. This represents a substantial reduction in the number of parameters required for classification, since the next linear layer now receives only 128 inputs instead of 6,272. Reducing the number of classifier parameters places more of the model's representational capacity in the convolutional feature extractor, which is usually more effective at producing a rich representation of the input. ### Spatial independence Flattening ties the classifier to a particular spatial resolution. A classifier expecting $128\times7\times7$ activations cannot directly receive $128\times8\times8$ activations because the vector length changes. Global average pooling always produces one value per channel, no matter the spatial dimensions. This makes the classifier independent of the final spatial resolution. The classification head can therefore be used with different spatial dimensions, as long as the feature extractor produces the same number of channels. ### What is discarded? Global average pooling removes explicit spatial position from the final representation. If a feature responds strongly near the top of one image and near the bottom of another, the pooled value may be similar as long as the average activation is similar. This is appropriate for many classification tasks, where the final decision depends primarily on whether useful features are present rather than on their exact coordinates. It would be inappropriate as the final representation for tasks that require spatial outputs, such as object localization or segmentation. Global average pooling is therefore not merely a parameter-saving device. It also expresses an architectural assumption about which information the final classifier should retain. ## 5. Building a Residual CNN The mechanisms introduced in this lesson can now be combined into one architecture. Consider a network for $28\times28$ grayscale images. Its feature extractor begins with 32 channels and then passes through three residual blocks. The first preserves the original spatial resolution. The next two reduce the spatial dimensions while increasing the number of feature channels. The progression is as follows. | Stage | Output shape | Shortcut | | ---------------------- | -------------------- | ---------- | | Input | $1\times28\times28$ | — | | Initial convolution | $32\times28\times28$ | — | | Residual block | $32\times28\times28$ | Identity | | Residual block | $64\times14\times14$ | Projection | | Residual block | $128\times7\times7$ | Projection | | Global average pooling | $128$ | — | | Linear classifier | $10$ | — | The first residual block leaves the tensor shape unchanged, so its shortcut can carry the input directly. The second block increases the number of channels and reduces the spatial resolution, so its shortcut must be projected to match. The third block repeats this pattern. After the final block, global average pooling summarizes the feature channels into a vector of length 128. A small linear layer then produces the ten class logits. ### Batch normalization For convolutional feature maps, PyTorch provides `nn.BatchNorm2d`. Its argument is the number of channels being normalized. A typical block can be written as: ```python nn.Conv2d(32, 64, kernel_size=3, padding=1, bias=False), nn.BatchNorm2d(64), nn.ReLU(), ``` The convolution produces 64 output channels, so the batch-normalization layer also contains 64 channel-wise sets of parameters and running statistics. The convolutional bias is usually disabled because a constant shift is removed by centering, while batch normalization already provides a learned offset. During training, a call to `model.train()` causes batch-normalization layers to use the current batch and update their running statistics. During evaluation, a call to `model.eval()` causes the layers to use the stored running statistics instead. Remember that `model.eval()` does not disable gradient recording, and a call to `torch.inference_mode()` does not change the behaviour of batch normalization. The two operations are therefore often used together. ### Reusable residual block A residual block can be expressed through one module whose behaviour depends on the input and output dimensions. The residual branch contains two convolutional layers. The first is followed by batch normalization and ReLU, while the second is followed by batch normalization. The shortcut is then added to the residual output, and ReLU is applied to the combined result. The shortcut is either an identity or a learned projection, depending on whether the input and output shapes match. The learned projection is implemented by a `nn.Conv2d` layer with a kernel size of 1 and a stride matching the downsampling performed by the residual branch. A batch-normalization layer is usually included after the convolution to control the scale of the projected activations. ```python class ResidualBlock(nn.Module): def __init__(self, in_channels, out_channels, stride=1): super().__init__() self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1, bias=False) self.bn1 = nn.BatchNorm2d(out_channels) self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1, bias=False) self.bn2 = nn.BatchNorm2d(out_channels) if stride == 1 and in_channels == out_channels: self.shortcut = nn.Identity() else: self.shortcut = nn.Sequential( nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, bias=False), nn.BatchNorm2d(out_channels), ) def forward(self, x): residual = self.conv1(x) residual = self.bn1(residual) residual = torch.relu(residual) residual = self.conv2(residual) residual = self.bn2(residual) shortcut = self.shortcut(x) return torch.relu(residual + shortcut) ``` ### Global pooling PyTorch can reduce arbitrary spatial dimensions to one value per channel with adaptive average pooling. For an input with shape `(batch_size, 128, height, width)`, the output has shape `(batch_size, 128, 1, 1)` and can be flattened to `(batch_size, 128)` before classification. A complete classification head can therefore be written as follows. ```python nn.AdaptiveAvgPool2d((1, 1)), nn.Flatten(), nn.Linear(128, 10), ``` ### Complete model The complete network can be expressed as a single module that combines the initial convolution, three residual blocks, global average pooling, and a linear classifier. The main new requirement is that batch normalization makes the distinction between training and evaluation modes part of the model's computation. The training workflow already provides that distinction by using `model.train()` during parameter updates and `model.eval()` during validation and testing. ```python class ResidualCNN(nn.Module): def __init__(self): super().__init__() self.stem = nn.Sequential( nn.Conv2d(1, 32, kernel_size=3, padding=1, bias=False), nn.BatchNorm2d(32), nn.ReLU(), ) self.features = nn.Sequential( ResidualBlock(32, 32), ResidualBlock(32, 64, stride=2), ResidualBlock(64, 128, stride=2), ) self.classifier = nn.Sequential( nn.AdaptiveAvgPool2d((1, 1)), nn.Flatten(), nn.Linear(128, 10), ) def forward(self, images): features = self.stem(images) features = self.features(features) logits = self.classifier(features) return logits ``` ## 6. Conclusion A deeper convolutional network requires more than additional convolutional layers. Its architecture must also control intermediate representations and provide effective paths through which information and gradients can propagate. Batch normalization standardizes feature channels during training and learns a scale and offset for each one. Running statistics allow the same layer to operate independently of evaluation-batch composition at inference time. Residual connections give every block a direct shortcut in addition to its learned residual transformation. The identity path makes it possible to preserve useful representations and contributes a direct term to gradient propagation. When the dimensions of a representation change, a $1\times1$ projection transforms the shortcut so that residual addition remains possible. The same operation provides an economical mechanism for changing channel width without mixing neighbouring spatial locations. Finally, global average pooling reduces each final feature map to one value. This produces a compact classifier, avoids tying the output layer to one fixed spatial resolution, and deliberately discards exact spatial position at the end of an image-classification network. These mechanisms form the basis of residual convolutional architectures. Their importance lies not in adding isolated features to a CNN, but in making deeper representations practical to construct and optimize. --- ## Final Quiz :::{admonition} Quiz 1 A batch-normalization layer receives a tensor with shape `(64, 128, 20, 20)`. Over which dimensions is the mean for one channel calculated?
Answer For one channel, the mean is calculated across the 64 examples and all 20×20 spatial positions. Different channels have separate statistics.
::: :::{admonition} Quiz 2 Why does batch normalization apply learned parameters after normalization?
Answer They allow each channel to learn an appropriate scale and offset instead of forcing the layer output to remain centered at zero with unit variance.
::: :::{admonition} Quiz 3 Why does batch normalization behave differently during training and evaluation?
Answer Training uses statistics from the current mini-batch and updates running estimates. Evaluation uses the stored running estimates so that a prediction does not depend on which other examples happen to share its batch.
::: :::{admonition} Quiz 4 Why is preserving the identity transformation particularly simple in a residual block?
Answer The shortcut already carries the input directly to the output. If the residual branch learns to approximate zero, then their sum is approximately the input.
::: :::{admonition} Quiz 5 Why is a projection shortcut required when a residual block changes from $32\times28\times28$ to $64\times14\times14$?
Answer Residual addition requires compatible tensor shapes. The original input differs from the residual output in both channel count and spatial dimensions, so the shortcut must be transformed before the two branches can be added.
::: :::{admonition} Quiz 6 What is the main distinction between a $1\times1$ convolution and a $3\times3$ convolution?
Answer A 1×1 convolution mixes information across channels at each spatial position without combining neighbouring positions. A 3×3 convolution combines both channel information and information from a local spatial neighbourhood.
::: :::{admonition} Quiz 7 A network finishes with 128 feature maps of size $7\times7$. Compare the number of inputs received by the classifier after flattening and after global average pooling.
Answer Flattening produces 128×7×7 = 6272 inputs. Global average pooling produces one value per channel, so the classifier receives only 128 inputs.
:::