# 5. Convolutional Networks :::{image} pics/cnn.png :width: 90% :align: center ::: Images are represented as grids of values arranged across spatial dimensions. A color image, for example, can be stored as a tensor of shape `(3, height, width)` with separate red, green, and blue channels. A multilayer perceptron (MLP) may process this image by flattening it into a vector of `3 × height × width` values and passing the vector through fully-connected layers. This approach can learn useful functions, but it does not directly use the structure that distinguishes an image from an arbitrary sequence of numbers. Nearby pixels often form meaningful patterns, such as edges, textures, and object parts. After flattening, an MLP no longer represents these spatial relationships explicitly: each hidden unit learns a separate weight for every pixel position. A **convolutional neural network** (ConvNet or CNN) uses a different architecture. Instead of connecting every hidden unit independently to every pixel, it applies small filters to local regions and reuses the same filter weights across spatial positions. This design preserves the spatial organization of intermediate representations, detects recurring patterns without learning a separate detector for each location, and combines simple local features into progressively more complex ones. The learning process remains the same: the network produces outputs, a loss measures prediction error, backpropagation calculates gradients, and an optimizer updates the parameters. What changes is the structure of the learned function, which is better suited to image data. ## 1. Spatial Structure An image can be represented as a 3D tensor with dimensions for channels, height, and width. A grayscale image typically has one channel, while a color image typically has three channels for red, green, and blue intensity values. Using the channels-first convention, a single image has the shape $$ \textsf{Image Shape} = (\text{channels}, \text{height}, \text{width}). $$ When multiple images are processed together, an additional batch dimension records how many images the tensor contains. A batch of images therefore has the shape $$ \textsf{Batch Shape} = (\text{batch size}, \text{channels}, \text{height}, \text{width}). $$ Nearby pixels often belong to the same edge, texture, or object part. For example, adjacent vertical regions with contrasting intensity values may indicate an edge, while multiple spatially related edges may define the contour of an object part. The two-dimensional organization of an image therefore contains information not only in its pixel values, but also in how those pixels are arranged. ### Flattening An MLP that processes images typically begins by flattening the input. This converts the spatially organized pixel values into a one-dimensional vector of length $\text{channels} \times \text{height} \times \text{width}$. After flattening, neighboring pixels become ordinary vector entries, and the architecture has no built-in mechanism for treating adjacent values as a local group. A linear layer learns a separate connection weight from every input position to every hidden unit. Each hidden unit can combine information from any part of the image, but its connections remain tied to specific pixel positions. An MLP can still learn useful image patterns; flattening does not make image processing impossible. The limitation is that the architecture does not encode the useful prior assumptions that images contain local structure and that similar patterns can appear in different locations. ### Locality Many useful visual patterns are local. An edge can often be recognized by comparing nearby pixels; nearby edges can form a corner; repeated local changes can define a texture. An object part also occupies a limited region of the image. These observations suggest a different form of connectivity. > Instead of connecting one hidden unit to the entire image, examine small local regions. Local connectivity is only part of the idea. The same edge or texture may appear in many positions. If the model had to learn a separate detector for the top-left, center, and bottom-right of an image, much of its capacity would be spent relearning the same operation. This motivates a second assumption. > Instead of learning a separate detector at each position, reuse the same operation at multiple positions. Together, local connectivity and weight sharing form the basis of convolutional layers. ## 2. Convolution A convolution combines an input with a small array of weights called a **filter** or **kernel**. The operation produces an output that records how strongly the filter responds at each spatial position. Two properties make convolution especially useful for image processing. 1. **Local connectivity:** Each output value depends on a small local region of the input. 2. **Weight sharing:** The same filter weights are applied at every spatial position. The following matrix example shows how the operation works. ### Kernel as a local detector Consider a small artificial image containing a bright vertical stripe surrounded by dark pixels: $$ X = \begin{bmatrix} 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \end{bmatrix}. $$ The values represent grayscale intensities from $0$ (dark) to $1$ (bright). Now consider the $3 \times 3$ matrix: $$ K = \begin{bmatrix} -1 & 0 & 1 \\ -1 & 0 & 1 \\ -1 & 0 & 1 \end{bmatrix}. $$ This matrix will serve as a simple local detector. We will call it a **kernel** or **filter**. ### Calculating one response Let us begin with the $3 \times 3$ patch in the upper-left corner of the image: $$ P_{0:3,0:3} = \begin{bmatrix} 0 & 0 & 1 \\ 0 & 0 & 1 \\ 0 & 0 & 1 \end{bmatrix}. $$ The slicing notation $i:j$ selects rows or columns from $i$ to $j-1$ using 0-based indexing. To calculate the detector response, multiply corresponding entries in the patch and kernel, then sum the products: $$ \begin{aligned} Y_{0,0} &= (0)(-1)+(0)(0)+(1)(1) \\ &+(0)(-1)+(0)(0)+(1)(1) \\ &+(0)(-1)+(0)(0)+(1)(1) = 3. \end{aligned} $$ The response is positive because the patch contains dark values on the left and bright values on the right, matching the kernel’s sign pattern. Now consider the patch at the opposite side of the stripe: $$ P_{0:3,2:5} = \begin{bmatrix} 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \end{bmatrix}. $$ Using the same calculation, its response is $Y_{0,2} = -3$. The negative response indicates the opposite orientation: bright values on the left and dark values on the right. The kernel response is therefore not a simple measure of brightness; it depends on how values are arranged within the local patch. ### General expression For a single-channel image and one kernel, the response at spatial position $(i,j)$ can be written as $$ Y_{i,j} = b + \sum_{u=0}^{k_h-1} \sum_{v=0}^{k_w-1} K_{u,v} \, X_{i+u,j+v} $$ where $K$ is the kernel, $X$ is the image, $b$ is an optional bias, and $k_h$ and $k_w$ are the kernel height and width. The equation describes the calculation we performed manually. It selects one local image patch, multiplies it by the kernel, sums the products, and adds the bias. :::{admonition} Cross-correlation :class: dropdown Technically, this formula defines **cross-correlation** because the kernel is not reversed before it is applied. Deep-learning literature conventionally calls the operation **convolution**. Because the filters are learned, the distinction has little practical consequence in neural networks. ::: :::{admonition} Quiz :class: dropdown A $3 \times 3$ kernel produces one scalar response at one image position. What information does that scalar contain?
Answer It measures how the values in one $3 \times 3$ image patch interact with the kernel values. A large positive or negative response indicates that the patch strongly matches one of the spatial patterns to which the kernel responds.
::: ### Feature map A single convolutional response indicates whether a pattern appears in one local region. To search for that pattern across the entire image, the kernel is applied at every position where its window fits within the input. Each position produces one scalar response. For the vertical-stripe image, the resulting responses are $$ Y = \begin{bmatrix} 3 & 0 & -3 \\ 3 & 0 & -3 \\ 3 & 0 & -3 \end{bmatrix}. $$ This output is called a **feature map**. A strong response at a given feature-map position indicates that the pattern appeared near the corresponding location in the input. The feature map therefore preserves spatial organization by recording both the presence of the pattern and where it occurred. :::{admonition} Weight sharing :class: dropdown A convolution reuses the same kernel at every spatial position, a property called **weight sharing**. This allows a filter to detect a local pattern regardless of its position without increasing the number of parameters. By contrast, a dense layer assigns different weights to different input positions, so it must learn a separate detector for each location. Weight sharing therefore reduces the number of learned parameters and can lead to better generalization. ::: :::{admonition} Translational equivariance :class: dropdown A convolution is **translation equivariant**: if an input pattern moves, the corresponding response moves with it. A convolution is however not *translation invariant*, which would require the output to be identical regardless of where the input pattern appears. Nevertheless, limited invariance can emerge in deeper convolutional networks through downsampling and aggregation, making a final classifier less sensitive to an object's exact position. ::: :::{admonition} Quiz :class: dropdown A $3 \times 3$ grayscale filter is applied to a $28 \times 28$ image and then to a $256 \times 256$ image. Does the filter require more learned weights for the larger image?
Answer No. The same nine weights and one bias are reused at more spatial positions. The output becomes larger, but the filter parameter count is unchanged.
::: ## 3. Convolutional Layer A convolutional layer turns the operation described earlier into a trainable component of a neural network. The layer contains several kernels whose values are learned during training. Each kernel is applied across the input and produces one feature map. Together, these feature maps form the output channels of the layer. ```text Kernel 1 → Feature map 1 Kernel 2 → Feature map 2 ... Kernel M → Feature map M ``` A layer with 16 kernels produces 16 output channels. One kernel might become sensitive to a vertical transition, another to a horizontal transition, and another to a texture-like arrangement. These examples are illustrative. During training, kernels develop whichever responses help the network reduce the loss function, so the learned patterns are task-specific and may not be easy to interpret. ### Input channels A convolutional layer can process inputs with multiple channels. For each output channel, the layer uses one kernel containing a 2D weight matrix for every input channel. At each spatial location, the kernel combines values across all input channels, sums their contributions, and adds a bias to produce one output value. Thus, one complete kernel connects every input channel to one output channel. For example, a detector applied to an RGB image combines values from the red, green, and blue channels. ```text Red channel ┐ Green channel ├→ One complete kernel → One feature map Blue channel ┘ ``` :::{admonition} Quiz :class: dropdown A convolutional layer contains 32 filters. How many output channels does it produce?
Answer It produces 32 output channels, one feature map for each filter.
::: ### Parameters A complete kernel is a 3D tensor with dimensions for input channels, kernel height, and kernel width. Because a convolutional layer has one kernel for each output channel, its full weight tensor has the shape $$ \textsf{Weight Shape} = (\text{out channels}, \text{in channels}, \text{kernel height}, \text{kernel width}). $$ The total number of trainable parameters in the layer, including one bias per output channel, is therefore $$ \text{Total Parameters} = \text{out channels} \times (\text{in channels} \times \text{kernel height} \times \text{kernel width} + 1). $$ For example, a convolutional layer with 4 input channels, 16 output channels, and a $3 \times 3$ kernel contains $16 \times (4 \times 3 \times 3 + 1) = 592$ trainable parameters. :::{admonition} Quiz :class: dropdown A convolutional layer receives RGB images and uses $5 \times 5$ kernels. How many weights belong to one kernel, excluding the bias?
Answer The kernel spans three input channels, so it contains $3 \times 5 \times 5=75$ weights.
::: ### General expression Consider a convolutional layer with $C_{\text{in}}$ input channels, $C_{\text{out}}$ output channels, and kernel size $k_h \times k_w$. Mathematically, the response at position $(i,j)$ in output channel $c$ is $$ Y_{c,i,j} = b_c + \sum_{d=0}^{C_{\text{in}}-1} \sum_{u=0}^{k_h-1} \sum_{v=0}^{k_w-1} K_{c,d,u,v} \, X_{d,i+u,j+v} $$ where $X$ is the input tensor, $b_c$ is the bias for output channel $c$, and $K_{c,d,u,v}$ is the kernel weight connecting input channel $d$ to output channel $c$ at kernel position $(u,v)$. Although the notation is more involved, the operation is unchanged: the layer extracts a local patch from each input channel, combines those values with one learned 3D kernel, and produces one output response. ### Nonlinear activation A convolutional layer performs an affine transformation: each output value is a weighted sum of input values plus a bias. Weight sharing changes the connectivity pattern, but the operation remains linear in the input. A nonlinear activation is therefore needed to prevent a stack of convolutional layers from collapsing into a single affine transformation. The typical sequence of operations looks like this: ```text Convolution → ReLU → Convolution → ReLU → ... ``` ReLU is a common choice, although other nonlinear activation functions can be used. :::{admonition} Negative response :class: dropdown A convolutional filter followed by ReLU may respond to one edge orientation, but not the reverse orientation, since negative responses are suppressed. A convolutional layer can preserve both orientations by learning separate filters. ::: ## 4. Controlling Spatial Dimensions Convolutional layers are usually stacked so that each layer can process the feature maps produced by the previous layer. The first layer operates on image pixels, while later layers combine nearby activations from earlier feature maps. This allows the network to build its representation gradually across multiple layers. Stacking convolutional layers requires careful control of the height and width of their feature maps. Normally, a convolution reduces the spatial dimensions because the kernel can be evaluated only where it fits entirely inside the input. This shrinking is not always desirable. A network may need to preserve spatial dimensions for several layers, or it may want to downsample the feature maps more quickly. The following sections describe three mechanisms for controlling spatial dimensions: padding, stride, and pooling. ### Valid convolution A valid convolution applies the kernel only where it fits entirely inside the input, so the output shrinks a little. In one spatial dimension, a kernel of width 3 can occupy the following positions within an input of width 5. ```text input: 1 2 3 4 5 pos. 1: 1 2 3 pos. 2: 2 3 4 pos. 3: 3 4 5 ``` More generally, the output size of a convolution with stride one and no padding is $$ \text{Output Size} = n - k + 1, $$ where $n$ is the input size and $k$ is the kernel size. This calculation applies separately to height and width. :::{admonition} Example :class: tip Consider a $28 \times 28$ image processed by a $3 \times 3$ kernel. The output size is $28 - 3 + 1 = 26$. ::: ### Padded convolution Padding adds zeros around the input boundary before the convolution is applied. The added values allow the kernel to produce outputs near the original border. In one spatial dimension, a kernel of width 3 can occupy the following positions within an input of width 5 that has been padded with one zero on each side. ```text input: 1 2 3 4 5 padded: 0 1 2 3 4 5 0 pos. 1: 0 1 2 pos. 2: 1 2 3 pos. 3: 2 3 4 pos. 4: 3 4 5 pos. 5: 4 5 0 ``` For one spatial dimension, the output size of a padded convolution is $$ \text{Output Size} = n + 2p - k + 1, $$ where $p$ is the amount of padding on each side. For an odd kernel size $k$, the following choice of padding preserves the spatial dimensions: $$ p=\frac{k-1}{2} \quad\Rightarrow\quad \text{Output Size} = n. $$ Padding defines how convolution behaves at the boundary and can prevent feature maps from shrinking. Preserving spatial size is often convenient, but it is an architectural choice rather than a requirement. :::{admonition} Example :class: tip Consider a $28 \times 28$ image processed by a $3 \times 3$ kernel and padding of one. The output size is $28 + 2(1) - 3 + 1 = 28$. ::: ### Strided convolution Stride controls how far the kernel moves between evaluated positions. A stride of one moves the kernel one position at a time; a stride of two skips every other position and produces a smaller feature map; a stride of three skips two positions, and so on. In one spatial dimension, a kernel of width 3 can occupy the following positions within an input of width 5, using a padding of one and stride two. ```text input: 1 2 3 4 5 padded: 0 1 2 3 4 5 0 stride: 2 pos. 1: 0 1 2 pos. 2: 2 3 4 pos. 3: 4 5 0 ``` For one spatial dimension, the output size of a strided and padded convolution is $$ \text{Output Size} = \left\lfloor\frac{n + 2p - k}{s} + 1\right\rfloor, $$ where $s$ is the stride. For an odd kernel size $k$, the following choice of padding makes the output size equal to the input size divided by the stride, rounded up. $$ p = \frac{k-1}{2} \quad\Rightarrow\quad \text{Output Size} = \left\lceil \frac{n}{s} \right\rceil $$ ::: A larger stride evaluates fewer positions and therefore reduces spatial resolution. Like padding, stride is an architectural choice, not merely a technical setting. :::{admonition} Example :class: tip Consider a $28 \times 28$ image processed by a $5 \times 5$ kernel with padding 2 and stride 2. The output size is $\left\lfloor \frac{28 + 2(2) - 5}{2} + 1 \right\rfloor = 14$. ::: ### Pooling layers Pooling downsamples a feature map while preserving its number of channels. It has **no learned parameters**; a sliding window moves across the feature map and selects or aggregates the values in each local region. Max pooling is a common choice, where the maximum value in each local region is retained. ```text --- input --- 1 4 2 0 3 2 1 5 0 2 3 1 1 6 2 4 --- 2x2 max pooling --- 4 5 6 4 ``` For one spatial dimension, the output size is determined by the window size $k$, padding $p$, and stride $s$. $$ \text{Output Size} = \left\lfloor\frac{n + 2p - k}{s} + 1\right\rfloor $$ When the stride equals the window size, the pooling regions do not overlap. In this case, assuming no padding, the output size is equal to the input size divided by the window size, rounded down. $$ \text{Output Size} = \left\lfloor\frac{n + 2(0) - k}{k} + 1\right\rfloor = \left\lfloor\frac{n}{k}\right\rfloor $$ Pooling may follow a convolutional layer to summarize local information and provide limited robustness to small translations. But it is not required in every CNN. Strided convolutions can also reduce spatial resolution. ## 5. Building a CNN We can now combine convolution, nonlinear activation, and pooling into a complete neural network. The model is designed to process $28 \times 28$ grayscale images and produce ten class logits. The architecture is divided into two main parts: a feature extractor that preserves the spatial structure of the input image, and a classifier that converts the final feature maps into class logits. ### Architecture The feature extractor contains two blocks of convolution, nonlinear activation, and pooling. The first convolution receives grayscale images, so it has one input channel. The second convolution receives the 16 feature maps produced by the first block and produces 32 new feature maps. ```python features = nn.Sequential( nn.Conv2d(1, 16, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2), ) ``` :::{important} The input channels of a convolutional layer must match the output channels of the previous layer. ::: The classifier flattens the final feature maps into a vector of length $32 \times 7 \times 7 = 1568$, and passes it through a linear layer to produce ten class logits. The number $1568$ is determined by the output of the feature extractor, so that the two modules can be connected together. ```python classifier = nn.Sequential( nn.Flatten(), nn.Linear(32 * 7 * 7, 10), ) ``` Finally, the feature extractor and classifier modules can be combined into a single model. ```python class SmallCNN(nn.Module): def __init__(self): super().__init__() self.features = nn.Sequential(...) self.classifier = nn.Sequential(...) def forward(self, images): features = self.features(images) logits = self.classifier(features) return logits ``` :::{admonition} Quiz :class: dropdown Why does the second convolution use 16 input channels?
Answer The first convolution produces 16 output channels. These feature maps become the input channels of the second convolution.
::: ### Tracing the Shapes The following table traces an image through the network, showing the tensor shape, parameter count, and receptive field at each stage. The spatial dimensions start at $28 \times 28$. The first convolution produces $28 \times 28$ feature maps. The first pooling layer reduces them to $14 \times 14$. The second convolution produces $14 \times 14$ feature maps, and the second pooling layer reduces them to $7 \times 7$. The final feature maps are flattened into a vector of length $32 \times 7 \times 7 = 1568$ before being passed to the linear layer. | Layer | Output shape | Parameter Count | Receptive Field | | ------- | ------------ | --------------- | ---------------- | | Input | $1 \times 28 \times 28$ | - | 1 | | Conv2d | $16 \times 28 \times 28$ | $16(1 \times 3 \times 3+1)=160$ | 3 | | Pooling | $16 \times 14 \times 14$ | - | 4 | | Conv2d | $32 \times 14 \times 14$ | $32(16 \times 3 \times 3+1)=4,640$ | 8 | | Pooling | $32 \times 7 \times 7$ | - | 10 | | Flatten | $1568$ | - | 10 | | Linear | $10$ | $1568 \times 10+10=15,690$ | Entire image | ### Receptive field The **receptive field** of an activation is the region of the original input that can influence it. In the first convolutional layer, this relationship is easy to see: an activation produced by a $3\times3$ kernel depends directly on a $3\times 3$ region of the input image. In deeper layers, however, each activation depends on earlier activations that already summarize local regions of the input. Receptive fields therefore grow as convolutional and pooling operations are composed. To calculate this growth, we must keep track of a second quantity, the **effective stride**, which measures the spacing between neighbouring activations relative to the original input. Let $r_\ell$ denote the receptive-field size after layer $\ell$, and let $j_\ell$ denote the effective stride after that layer. At the input, $r_0=j_0=1$ because one input activation corresponds to one pixel and neighbouring input activations are one pixel apart. For a layer with kernel size $k_\ell$ and stride $s_\ell$, the receptive field and effective stride are updated as $$ \begin{aligned} r_\ell &= r_{\ell-1} + (k_\ell-1)j_{\ell-1},\\ j_\ell &= j_{\ell-1}s_\ell. \end{aligned} $$ The first equation shows that a kernel adds $k_\ell-1$ steps to the previous receptive field, but those steps must be measured at the spacing already created by earlier layers. The second equation shows how stride increases the spacing in proportion to the previous effective stride. The following table illustrates the calculations for the feature extractor described above. | Stage | Kernel | Stride | Receptive field | Effective stride | Spatial Dimensions | | ----- | -----: | -----: | --------------: | ---------------: | -----------------: | | Input | — | — | 1 | 1 | $H \times W$ | | Conv | 3 | 1 | 3 | 1 | $H \times W$ | | Pool | 2 | 2 | 4 | 2 | $H/2 \times W/2$ | | Conv | 3 | 1 | 8 | 2 | $H/2 \times W/2$ | | Pool | 2 | 2 | 10 | 4 | $H/4 \times W/4$ | ### Hierarchical representations The growth of receptive fields explains why convolutional layers are stacked. The first convolution operates directly on pixel values and can respond to local contrasts, edges, or simple textures. The next convolution combines the feature maps produced by the first layer and can respond to more complex patterns. Subsequent layers can combine those patterns into increasingly large-scale structures. The network therefore develops a **hierarchy of representations**, with each layer building on the previous one. This description is useful, but it should not be interpreted too literally. A specific channel is not guaranteed to represent one clean human concept. Feature meanings may be distributed across several channels, and a single channel may respond to several related patterns. The hierarchy refers primarily to the increasing scale and complexity of the information that later layers can combine. ## 6. Conclusion An image is more than a collection of pixel values. Its spatial organization captures relationships among nearby pixels, and similar local patterns can appear in different positions. An MLP can learn from flattened images, but its architecture does not explicitly encode those properties. A convolutional network does. It applies small learned filters to local regions, shares those filters across positions, and preserves the spatial arrangement of their responses in feature maps. Several convolutional layers combine local patterns into increasingly broad representations. Pooling layers reduce spatial resolution, and receptive fields grow as operations are composed. The final feature maps are eventually converted into class logits. A CNN is not universally superior. Its assumptions are useful when the input has meaningful local and spatial structure. For ordinary tabular data, the order of columns may not define a neighbourhood, so applying image-style convolution may be inappropriate. | Property | MLP | CNN | | ----------------- | ---------------------------------------- | ----------------------------------------- | | Input handling | Flattens the input image | Preserves channel, height, and width | | Connectivity | Hidden units receive every pixel | Early units receive local regions | | Weight use | Separate weights for each input position | Filters are shared across positions | | Spatial structure | Lost after flattening | Preserved in feature maps | | Assumption | No explicit spatial prior | Local patterns can recur across positions | --- ## Final Quiz :::{admonition} Quiz 1 What does each dimension of a batch with shape `(64, 3, 32, 32)` represent?
Answer The batch contains 64 images. Each image has three channels, a height of 32 pixels, and a width of 32 pixels.
::: :::{admonition} Quiz 2 What is the main difference between the connectivity of an MLP’s first hidden layer and a convolutional layer?
Answer An MLP hidden unit normally receives every flattened input value. A convolutional activation initially depends only on a local input region, although later layers can acquire larger receptive fields.
::: :::{admonition} Quiz 3 Why can one convolutional filter detect a similar pattern in several positions?
Answer The same filter weights are reused at every evaluated spatial position.
::: :::{admonition} Quiz 4 What is the weight shape of `nn.Conv2d(8, 20, kernel_size=3)`?
Answer The layer has 20 output filters. Each filter spans eight input channels and has spatial size $3 \times 3$. The weight shape is `(20, 8, 3, 3)`.
::: :::{admonition} Quiz 5 A tensor with shape `(32, 8, 20, 20)` is processed by `nn.Conv2d(8, 16, kernel_size=3, padding=1, stride=1)`. What is the output shape?
Answer Padding one preserves the spatial size for a $3 \times 3$ kernel with stride one. The output has 16 channels, so its shape is `(32, 16, 20, 20)`.
::: :::{admonition} Quiz 6 What happens to the shape `(64, 32, 14, 14)` after `nn.MaxPool2d(kernel_size=2)` with its default stride?
Answer The channel count remains 32, while height and width are halved. The output shape is `(64, 32, 7, 7)`.
::: :::{admonition} Quiz 7 Why is ReLU still needed in a convolutional neural network?
Answer Convolution is an affine operation. Nonlinear activations prevent the complete sequence of layers from remaining an affine transformation and allow the network to construct more flexible functions.
::: :::{admonition} Quiz 8 Does convolution make a classifier completely invariant to the position of an object?
Answer No. Convolution is approximately translation equivariant: moving a pattern tends to move the feature response. Pooling and later aggregation can reduce sensitivity to small shifts, but complete invariance is not automatic.
::: :::{admonition} Quiz 9 A CNN achieves higher validation accuracy than an MLP, but the two models were trained on different data splits. Can the difference be attributed to architecture?
Answer Not reliably. The data split and architecture changed together, so their effects are confounded. A useful comparison should use the same partitions and evaluation procedure.
::: :::{admonition} Quiz 10 Does a convolutional filter require a special learning algorithm that differs from backpropagation?
Answer No. Convolutional parameters participate in the same differentiable computation as other model parameters. Backpropagation calculates their gradients, and an optimizer updates them.
:::