Residual Networks

Use this page to quickly reference batch normalization, residual connections, projection shortcuts, global average pooling, and the residual-block patterns commonly used in deeper convolutional networks.

Batch Normalization

BatchNorm2d

Use BatchNorm2d to normalize convolutional feature channels.

batch_norm = torch.nn.BatchNorm2d(num_features=C)

For an input tensor with shape (N, C, H, W), batch normalization treats each of the C channels separately and preserves the complete tensor shape. The num_features argument is the number of input channels and must match the channel dimension of the input tensor.

Parameters and Buffers

With its default settings, batch normalization learns one scale parameter and one shift parameter for each channel. It also maintains running estimates of the mean and variance for each channel. A layer with C channels therefore has \(2C\) learnable parameters, together with \(2C\) running statistics.

Scale and shift are registered as parameters. They are exposed as model parameters and are normally updated during training. By contrast, the running statistics are registered as buffers. They are persistent model state but are not model parameters. Both parameters and buffers are part of the layer state.

batch_norm.weight        # scale (γ) - shape (C,)
batch_norm.bias          # shift (β) - shape (C,)
batch_norm.running_mean  # mean      - shape (C,)
batch_norm.running_var   # variance  - shape (C,)

Training and Evaluation Behaviour

Batch normalization behaves differently during training and evaluation.

Mode	Statistics used	Running statistics
Training	Current mini-batch	Updated
Evaluation	Stored running estimates	Fixed

Switch modes with model.train() and model.eval().

Warning

torch.inference_mode() disables gradient recording but does not switch batch normalization into evaluation behaviour. During evaluation, use it together with model.eval().

Convolution Bias

A convolution immediately followed by batch normalization is commonly created with bias=False.

from torch import nn

block = nn.Sequential(
    nn.Conv2d(32, 64, kernel_size=3, padding=1, bias=False),
    nn.BatchNorm2d(64),
    nn.ReLU(),
)

Batch normalization centers each channel and already provides a learned shift parameter, so the preceding convolutional bias is normally redundant. ReLU is applied after batch normalization.

Residual Connection

A residual connection combines a learned transformation with a shortcut carrying the input:

\[ y = x + F(x). \]

Since the addition is element-wise, the two branches must produce tensors of the same shape. This occurs naturally when the residual branch \(F(x)\) preserves both the number of channels and the spatial dimensions.

Note

ReLU is normally applied after the residual addition.

Projection Shortcut

When the residual branch changes the channel count or spatial resolution, the original input can no longer be added directly to the branch output. The shortcut must first transform the input to the same shape as the residual branch. A learned transformation used for this purpose is called a projection shortcut.

A common projection uses a \(1\times 1\) convolution followed by batch normalization:

from torch import nn

shortcut = nn.Sequential(
    nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, bias=False),
    nn.BatchNorm2d(out_channels),
)

A \(1\times 1\) convolution can change the number of channels without combining neighbouring spatial positions. Using stride > 1 also reduces the spatial dimensions. The projection can therefore make both the channel count and spatial resolution match the residual branch output.

Residual Block

A residual block can be implemented as a reusable module whose behaviour depends on the input channels, output channels, and stride. The first convolution determines the block’s output channel count and performs any required downsampling. The second convolution preserves that shape. The shortcut uses either the identity or a projection so that both branches have the same shape before they are added.

from torch import nn

class ResidualBlock(nn.Module):

    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        ks1 = dict(kernel_size=1, bias=False)
        ks3 = dict(kernel_size=3, bias=False, padding=1)

        self.conv1 = nn.Conv2d(in_channels, out_channels, stride=stride, **ks3)
        self.bn1 = nn.BatchNorm2d(out_channels)

        self.conv2 = nn.Conv2d( out_channels, out_channels, **ks3)
        self.bn2 = nn.BatchNorm2d(out_channels)

        if stride == 1 and in_channels == out_channels:
            self.shortcut = nn.Identity()
        else:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, stride=stride, **ks1),
                nn.BatchNorm2d(out_channels),
            )

    def forward(self, x):
        residual = self.conv1(x)
        residual = self.bn1(residual)
        residual = torch.relu(residual)
        residual = self.conv2(residual)
        residual = self.bn2(residual)
        shortcut = self.shortcut(x)
        return torch.relu(residual + shortcut)

This implementation is one common residual-block pattern rather than a fixed template. Residual blocks can vary in the number and order of convolutions, normalization layers, and activation functions. They may also use different strategies for changing channel count or spatial resolution. The essential requirement is that the residual branch and shortcut produce compatible tensors before they are added.

Global Average Pooling

Global average pooling reduces each feature map to one value by averaging across its complete spatial extent. For every channel \(c\), the output is

\[ z_{n,c}=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}x_{n,c,i,j}. \]

For an input tensor with shape (N, C, H, W), global average pooling produces an output tensor with shape (N, C, 1, 1). Flattening the two singleton spatial dimensions gives (N, C).

AdaptiveAvgPool2d

Use adaptive pooling to produce one spatial value per channel regardless of the input height and width.

pool = torch.nn.AdaptiveAvgPool2d((1, 1))

Global average pooling makes the input size of the next linear layer independent of the final spatial dimensions, provided that the number of channels remains fixed. For example, a convolutional feature extractor that produces 128 channels can be connected to a linear classifier with 128 input features, regardless of the spatial dimensions of the feature maps.

from torch import nn

classifier = nn.Sequential(
    nn.AdaptiveAvgPool2d((1, 1)),
    nn.Flatten(),
    nn.Linear(128, 10),
)