Build a Residual CNN

Build a Residual CNN#

(Click the button in the top right corner to download the lab.)

Deep convolutional networks are more expressive than shallow networks, but they are also more difficult to optimize. This lab explores several architectural mechanisms that help make deeper convolutional networks easier to train. You will first examine batch normalization, then implement residual blocks and combine them into a complete residual CNN with global average pooling. Finally, you will compare this model with a plain CNN that uses neither batch normalization nor residual shortcuts.

1. Experimental Setup#

This lab uses FashionMNIST to study architectural changes in a small image-classification problem. A fixed subset of the official training data is used throughout the lab so that the convolutional models can be trained repeatedly without making the experiments unnecessarily expensive.

Data partitions#

The official FashionMNIST training set contains the examples available for model development. From it, 12,000 examples are used for training and 6,000 for validation; the remaining training examples are left unused. The split is generated reproducibly. No fitted preprocessing is required. The official test set is loaded but no test loader is created yet, so it remains untouched until the final evaluation.

The class counts printed below provide a quick check that the reduced training subset has not introduced a severe class imbalance.

The data loaders are constructed with a default batch size of 128. For a shuffled loader, the random seed is fixed so that the batch order can be reproduced within the lab.

BATCH_SIZE = 128

def make_loader(dataset, shuffle: bool, seed = SEED, batch_size = BATCH_SIZE):
    generator = torch.Generator().manual_seed(seed) if shuffle else None
    return DataLoader(dataset, batch_size, shuffle, generator=generator)

Training workflow#

The functions below provide the training and evaluation workflow used throughout the lab.

train_epoch performs one parameter-updating pass through the training loader and accumulates epoch-level loss and accuracy.
evaluate measures the model at a fixed parameter state using evaluation mode and inference mode.
fit_model combines these operations across epochs. After every training epoch, it evaluates the model on the validation set, records the history, and saves the parameters that achieve the lowest validation loss. At the end of training, that best validation checkpoint is restored.

2. Batch Normalization#

Batch normalization is a layer that rescales activations using statistics computed for each channel separately.

For an activation tensor with shape \((N,C,H,W)\), BatchNorm2d calculates one mean and one variance for each channel using all examples in the batch and all spatial positions. An activation \(x\) is first normalized as

\[ \hat{x}=\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}, \]

where \(\mu\) and \(\sigma^2\) are the statistics for its channel. The normalized value is then transformed using two learned parameters,

\[ y=\gamma\hat{x}+\beta, \]

where \(\gamma\) controls the scale and \(\beta\) controls the offset.

Calculate channel statistics#

We start with a small three-channel tensor whose channels deliberately have different offsets and scales. This makes the effect of channel-wise normalization easy to observe.

set_seed(SEED)

activations = torch.randn(4, 3, 5, 5)
activations[:, 0] = activations[:, 0] * 0.5 + 3.0
activations[:, 1] = activations[:, 1] * 2.0 - 1.0
activations[:, 2] = activations[:, 2] * 5.0 + 10.0

Exercise 1#

Calculate the mean and variance of each channel across the batch and spatial dimensions. When computing the variance, use unbiased=False so that the calculation matches the batch-normalization formula.

# TODO: reduce over batch, height, and width while preserving channels
channel_mean = None # YOUR CODE HERE
channel_variance = None # YOUR CODE HERE

print("Means:", channel_mean)
print("Variances:", channel_variance)

Checkpoint#

Verify that your calculations are correct.

# TODO: assert that channel_mean and channel_variance each contain 3 values
# TODO: compare channel_mean[0] with a direct mean of activations[:, 0, :, :]
# TODO: compare channel_variance[0] with a direct variance of activations[:, 0, :, :]

Check with PyTorch#

Now compare the manual calculation with PyTorch’s BatchNorm2d. Configure the layer without learned scale and shift parameters, and use the statistics from the current batch. This isolates the normalization step that you calculated above.

Exercise 2#

Using the channel means and variances calculated previously, normalize the activation tensor manually. Then compare your result with the output produced by BatchNorm2d.

Hint: You can use broadcasting to subtract the channel means and divide by the channel standard deviations. Use eps to avoid division by zero.

bn_without_affine = nn.BatchNorm2d(3, affine=False, track_running_stats=False)
bn_without_affine.train()

# TODO: calculate the manual normalization using channel_mean and channel_variance
eps = bn_without_affine.eps
manual_normalized = None # YOUR CODE HERE

pytorch_normalized = bn_without_affine(activations)

print("Maximum absolute difference:", (manual_normalized - pytorch_normalized).abs().max().item())

Before the learned scale and shift are applied, normalization should produce values with mean close to zero and variance close to one in each channel. An ordinary BatchNorm2d layer can then rescale and recenter these normalized activations through its learned weight and bias parameters.

normalized_means = pytorch_normalized.mean(dim=(0, 2, 3))
normalized_variances = pytorch_normalized.var(dim=(0, 2, 3), unbiased=False)

print("Normalized means:", torch.round(normalized_means, decimals=2))
print("Normalized variances:", torch.round(normalized_variances, decimals=2))

Learned parameters and running statistics#

An ordinary BatchNorm2d layer contains two different kinds of state. The learned weight and bias are trainable parameters, one pair for each channel. The layer also stores running estimates of the channel mean and variance. These running statistics are not optimized by gradient descent; they are updated from training batches and later used during evaluation.

Inspect the initial state of a three-channel batch-normalization layer.

bn = nn.BatchNorm2d(3)

print("Learned scale shape:", tuple(bn.weight.shape))
print("Learned shift shape:", tuple(bn.bias.shape))
print("Running mean:", bn.running_mean)
print("Running variance:", bn.running_var)

Training and evaluation behaviour#

In training mode, BatchNorm2d normalizes activations using the statistics of the current mini-batch and updates its running estimates. In evaluation mode, it instead normalizes activations using the stored running estimates and leaves them unchanged.

Run the following code and compare the running mean and variance before training, after several training batches, and after an evaluation batch.

bn.train()

running_mean_before = bn.running_mean.clone()
running_var_before = bn.running_var.clone()

for _ in range(5):
    training_batch = torch.randn(8, 3, 5, 5) * 2 + 4
    _ = bn(training_batch)

running_mean_after_training = bn.running_mean.clone()
running_var_after_training = bn.running_var.clone()

bn.eval()

_ = bn(torch.randn(8, 3, 5, 5) * 10 - 20)

running_mean_after_evaluation = bn.running_mean.clone()
running_var_after_evaluation = bn.running_var.clone()

print("Running mean before training:    ", running_mean_before)
print("Running mean after training:     ", running_mean_after_training)
print("Running mean after evaluation:   ", running_mean_after_evaluation)
print()
print("Running variance before training: ", running_var_before)
print("Running variance after training:  ", running_var_after_training)
print("Running variance after evaluation:", running_var_after_evaluation)

Checkpoint#

Verify that the running mean and variance changed during training but remained unchanged during evaluation. Also verify that bn.weight and bn.bias are trainable parameters.

# TODO: assert that running statistics changed during training
# TODO: assert that they remained unchanged during evaluation
# TODO: assert that bn.weight and bn.bias require gradients

The distinction between training and evaluation modes has an important consequence. During training, the normalized representation of an example can depend on the other examples in the same mini-batch because the normalization statistics come from that batch. During evaluation, the stored running statistics provide a fixed reference instead.

3. Residual Block#

A residual block combines a learned transformation with a shortcut path. If the residual branch computes a transformation \(F(x)\) and the input already has the correct shape, the block can use an identity shortcut:

\[ y = x + F(x). \]

If the residual branch changes the number of channels or the spatial resolution, the shortcut must transform the input as well:

\[ y = P(x) + F(x), \]

where \(P(x)\) is a projection that makes the shortcut compatible with the residual output.

Reusable implementation#

The residual branch below follows the sequence Conv → BatchNorm → ReLU → Conv → BatchNorm. The shortcut uses nn.Identity when the shapes already match. Otherwise, a 1×1 convolution with the required stride changes the channel count and spatial dimensions, followed by BatchNorm2d. The two branches are then added and passed through a final ReLU.

Exercise 3#

Complete the ResidualBlock class. Use an identity shortcut when the input and residual output shapes match. Otherwise, construct the projection shortcut with a 1×1 convolution using the block stride, followed by BatchNorm2d.

class ResidualBlock(nn.Module):

    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)

        # TODO: use nn.Identity when shapes already match.
        #       Otherwise use a 1x1 projection with the required stride,
        #       followed by BatchNorm2d.
        if in_channels == out_channels and stride == 1:
            self.shortcut = None # YOUR CODE HERE
        else:
            self.shortcut = nn.Sequential(
                # YOUR CODE HERE
            )


    def forward(self, x):
        residual = self._residual_branch(x)
        shortcut = self.shortcut(x)

        # TODO: combine the two paths and apply ReLU
        output = None # YOUR CODE HERE
        return output

    def _residual_branch(self, x):
        residual = self.conv1(x)
        residual = self.bn1(residual)
        residual = torch.relu(residual)
        residual = self.conv2(residual)
        residual = self.bn2(residual)
        return residual

Exercise 4#

Indicate the shapes of the residual output and shortcut output for both blocks in the table below.

Block	Input	Residual output	Shortcut output
`ResidualBlock(32, 32)`	`(8, 32, 28, 28)`
`ResidualBlock(32, 64, stride=2)`	`(8, 32, 28, 28)`

Then create the two blocks and inspect the output of the residual and shortcut branches directly.

identity_block = ResidualBlock(32, 32)
projection_block = ResidualBlock(32, 64, stride=2)

x = torch.randn(8, 32, 28, 28)

# TODO: compute the outputs of both branches for each block
identity_residual = None # YOUR CODE HERE
identity_shortcut = None # YOUR CODE HERE
projection_residual = None # YOUR CODE HERE
projection_shortcut = None # YOUR CODE HERE

print("Identity block residual: ", tuple(identity_residual.shape))
print("Identity block shortcut: ", tuple(identity_shortcut.shape))
print("Projection block residual:", tuple(projection_residual.shape))
print("Projection block shortcut:", tuple(projection_shortcut.shape))

Checkpoint#

Verify that the residual and shortcut branches produce compatible shapes in both cases, and confirm that the projection shortcut has the expected 1×1 convolution.

# TODO: assert that residual and shortcut shapes match for both blocks
# TODO: assert that identity_block.shortcut is nn.Identity
# TODO: verify that the projection convolution has weight shape (64, 32, 1, 1)

4. Residual CNN#

The residual blocks can now be combined into a complete convolutional network. Starting from a FashionMNIST image with shape (1, 28, 28), the model follows this progression:

input    →  stem      →  block 1   →  block 2   →  block 3  →  GAP  →  classifier
-------     --------     --------     --------     -------     ---     ----------
1×28×28  →  32×28×28  →  32×28×28  →  64×14×14  →  128×7×7  →  128  →  10

The stem first produces 32 feature channels. An identity residual block keeps the same shape, then two projection blocks reduce the spatial resolution while increasing the channel count. Global average pooling converts the final feature maps into one value per channel before the linear classifier produces ten logits.

Global average pooling#

Global average pooling converts each feature map into a single value by averaging over all of its spatial positions. For an activation tensor with shape \((N,C,H,W)\), it produces a tensor with shape \((N,C,1,1)\).

Each output value summarizes how strongly one learned feature is present across the entire feature map, without preserving its exact spatial location. The resulting vector contains one value per channel and can be passed directly to a classifier.

In this network, global average pooling reduces a representation of shape (128, 7, 7) to 128 values. Flattening the same representation would produce \(128 \times 7 \times 7 = 6272\) values, requiring a much larger final linear layer. Global average pooling therefore provides a compact transition from convolutional features to class logits while deliberately discarding exact spatial position.

Exercise 5#

Calculate the number of parameters required by a ten-class linear classifier in two cases: after flattening the (128, 7, 7) feature maps, and after global average pooling. Include both weights and biases, but do not instantiate the classifier layers.

flattened_features = 128 * 7 * 7
global_features = 128
n_classes = 10

# TODO: count parameters include both weights and biases
flatten_classifier_parameters = None # YOUR CODE HERE
global_classifier_parameters = None # YOUR CODE HERE

print("Flatten classifier parameters:", flatten_classifier_parameters)
print("Global-pooling classifier parameters:", global_classifier_parameters)
print("Reduction factor:", flatten_classifier_parameters / global_classifier_parameters)

Exercise 6#

Complete the feature extractor in ResidualCNN by assembling the three residual blocks shown in the architecture above.

class ResidualCNN(nn.Module):

    def __init__(self):
        super().__init__()

        self.stem = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, padding=1, bias=False),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )

        # TODO: assemble the three residual blocks
        self.features = nn.Sequential(
            # YOUR CODE HERE
        )

        self.pool = nn.AdaptiveAvgPool2d((1, 1))
        self.classifier = nn.Linear(128, 10)

    def forward(self, images):
        x = self.stem(images)
        x = self.features(x)
        x = self.pool(x)
        x = torch.flatten(x, start_dim=1)
        return self.classifier(x)

Checkpoint#

Complete the expected shape at each stage in the table below.

Stage	Expected shape for batch size 16
Input	`(16, 1, 28, 28)`
Stem
Residual block 1
Residual block 2
Residual block 3
Global average pooling
Flatten
Logits

Then, run the following cell and compare your trace with the actual tensors

set_seed(SEED)
residual_cnn = ResidualCNN().to(device)
example_images = torch.randn(16, 1, 28, 28, device=device)

residual_cnn.eval()
with torch.inference_mode():
    stem_output = residual_cnn.stem(example_images)
    block1_output = residual_cnn.features[0](stem_output)
    block2_output = residual_cnn.features[1](block1_output)
    block3_output = residual_cnn.features[2](block2_output)
    pooled_output = residual_cnn.pool(block3_output)
    flat_output = torch.flatten(pooled_output, start_dim=1)
    logits = residual_cnn.classifier(flat_output)

for name, tensor in [
    ("input", example_images),
    ("stem", stem_output),
    ("block 1", block1_output),
    ("block 2", block2_output),
    ("block 3", block3_output),
    ("pooled", pooled_output),
    ("flattened", flat_output),
    ("logits", logits),
]:
    print(f"{name:10s}: {tuple(tensor.shape)}")

Train the residual CNN#

The model now has a valid end-to-end architecture. Train a fresh ResidualCNN for 6 epochs with a learning rate of 0.001. This run establishes the residual model that will later be compared with a plain CNN under the same training budget.

set_seed(SEED)
EPOCHS = 6
LEARNING_RATE = 1e-3

# TODO: create and train a fresh ResidualCNN
model = ResidualCNN()
residual_result = None # YOUR CODE HERE

Analysis#

Inspect the training and validation loss curves. Does the training loss decrease consistently? At which epoch is the validation loss lowest? Do the curves show an obvious optimization failure, or does the model appear to learn the task over this training run?

5. Plain CNN#

The residual CNN combines batch normalization and residual connections. To provide a simpler reference, the model below mirrors the sequence of 3×3 convolutions in the residual branches, together with the same channel progression, downsampling locations, global average pooling, and final classifier. It omits both batch normalization and the shortcut paths, including their 1×1 projection convolutions. Each 3×3 convolution is followed directly by ReLU.

This is a broad architectural comparison between a plain CNN and the complete residual CNN. Because batch normalization and residual shortcuts change together, the comparison cannot isolate the individual contribution of either mechanism.

class PlainCNN(nn.Module):

    def __init__(self):
        super().__init__()

        self.stem = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, padding=1),
            nn.ReLU(),
        )

        self.features = nn.Sequential(
            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.ReLU(),

            nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.ReLU(),

            nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.ReLU(),
        )

        self.pool = nn.AdaptiveAvgPool2d((1, 1))
        self.classifier = nn.Linear(128, 10)

    def forward(self, images):
        x = self.stem(images)
        x = self.features(x)
        x = self.pool(x)
        x = torch.flatten(x, start_dim=1)
        return self.classifier(x)

Train the plain CNN#

Train the plain CNN for the same 6 epochs and with the same learning rate used for the residual CNN. This makes the resulting learning curves directly comparable under these conditions.

set_seed(SEED)
EPOCHS = 6
LEARNING_RATE = 1e-3

plain_model = PlainCNN()

# TODO: train the plain CNN
plain_result = None # YOUR CODE HERE

comparison_results = {
    "Plain CNN": plain_result,
    "Residual CNN": residual_result,
}

plot_metric(comparison_results, "train_loss", "Plain CNN versus residual CNN")
plot_metric(comparison_results, "valid_loss", "Plain CNN versus residual CNN")

Analysis#

Compare the training-loss curves of the two models. Which model reduces training loss more quickly, and which reaches the lower training loss by the end of the run? Then compare their best validation losses and the relationship between training and validation loss.

Use these observations to describe how the two architectures behaved under the tested training conditions. What can this comparison establish, and what can it not establish because batch normalization and residual shortcuts changed together?

6. Test Evaluation#

Validation loss has been used throughout model development to compare the two candidates and choose among their checkpoints. The test set serves a different purpose: it estimates the generalization performance of the model selected without test feedback.

Select the candidate with the lowest validation loss first. Only after that choice has been made should a test loader be created and the selected model evaluated once on the test set.

final_candidates = {
    "Plain CNN": plain_result,
    "Residual CNN": residual_result,
}

candidate_table = pd.DataFrame(
    [
        {
            "model": label,
            "best epoch": result["best_epoch"],
            "validation loss": result["valid_loss"],
            "validation accuracy": result["valid_accuracy"],
        }
        for label, result in final_candidates.items()
    ]
).sort_values("validation loss")

candidate_table

Exercise 7#

Select the lowest-validation-loss candidate programmatically, then create the test loader and evaluate the selected checkpoint once.

selected_label, selected_result = min(
    final_candidates.items(),
    key=lambda item: item[1]["valid_loss"],
)

print("Selected model:", selected_label)
print("Validation loss:", selected_result["valid_loss"])
print("Validation accuracy:", selected_result["valid_accuracy"])

# TODO: create a non-shuffled test loader only now
# TODO: evaluate selected_result["model"] once on the test set
test_loader = None # YOUR CODE HERE
final_test_metrics = None # YOUR CODE HERE

print("Test loss:", final_test_metrics["loss"])
print("Test accuracy:", final_test_metrics["accuracy"])

Analysis#

Compare the final test loss and accuracy with the selected model’s validation performance. Are they broadly consistent? Use the test result to describe how well the selected model generalizes to examples that were not used for training or model selection.

The completed model combines several ideas that support deeper convolutional architectures. Batch normalization controls activation scale during training, residual shortcuts preserve a direct path for existing representations, projection shortcuts reconcile incompatible shapes, and global average pooling converts the final feature maps into a compact representation for classification. The comparison with the plain CNN shows how the complete residual architecture behaves under the same training budget, without attributing any difference to a single mechanism.

Final Reflection#

How do the learned parameters and running statistics of BatchNorm2d play different roles during training and evaluation?
When can a residual block use an identity shortcut, and when is a projection shortcut required?
What does global average pooling preserve, and what spatial information does it discard?
What evidence from the learning curves distinguishes the residual CNN from the plain CNN in this experiment, and what causal conclusion can you not draw from that comparison?
How does the selected model’s test performance compare with its validation performance, and what does that suggest about its generalization?

Build a Residual CNN

Contents

Build a Residual CNN#

1. Experimental Setup#

Data partitions#

Training workflow#

2. Batch Normalization#

Calculate channel statistics#

Exercise 1#

Checkpoint#

Check with PyTorch#

Exercise 2#

Learned parameters and running statistics#

Training and evaluation behaviour#

Checkpoint#

3. Residual Block#

Reusable implementation#

Exercise 3#

Exercise 4#

Checkpoint#

4. Residual CNN#

Global average pooling#

Exercise 5#

Exercise 6#

Checkpoint#

Train the residual CNN#

Analysis#

5. Plain CNN#

Train the plain CNN#

Analysis#

6. Test Evaluation#

Exercise 7#

Analysis#

Final Reflection#