Siamese Network#
A Siamese network is an architecture for comparing two inputs. The same neural network processes each input separately and converts it into an embedding. The two embeddings are then compared using a distance or similarity measure. During training, the model receives pairs labeled as similar or dissimilar, and it is encouraged to produce embeddings that reflect these relationships. The objective is not merely to compare the original inputs directly, but to learn which characteristics are relevant for the comparison.
In this tutorial, we use a Siamese network to compare handwritten digits from the MNIST dataset. The model is trained with the contrastive loss, which encourages matching digits to have similar embeddings while enforcing separation between nonmatching digits.
Note
A Siamese network is often drawn as two parallel processing paths. These paths do not represent two independently trained networks. They are two forward passes of the same network. The inputs also need not be supplied as explicit pairs. The network can embed an entire batch at once, and then positive and negative pairs can be constructed afterward.
Creating the network#
The first step is to define a neural network that processes the input images and produces embeddings. Generally, this network is composed of three parts.
Backbone: A series of convolutional and pooling layers to extract features from the input image.
Neck: A small number of layers to further process the extracted features into a compact representation. This typically involves a flattening layer followed by one or more fully-connected layers, nonlinear activations, and possibly batch normalization.
Head: A final set of layers to prepare the embeddings for comparison with the chosen loss function. This usually involves a normalization layer, and depending on the loss function, a fully-connected layer. As the head is only used during training, it may be included in the loss function, rather than the model itself.
Backbone#
The backbone of the network is responsible for extracting features from the input images. It is best practice to use a pre-trained model as the backbone, since this usually improves performance and reduces the amount of training data required. For the present tutorial, however, we will build a simple backbone from scratch, made up of convolutional and pooling layers with ReLU activations.
class Backbone(nn.Sequential):
def __init__(self):
super().__init__(
nn.Conv2d(1, 64, 5, padding=2),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(64, 128, 5, padding=2),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(128, 256, 3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2),
)
Neck#
The neck of the network takes the features extracted by the backbone and processes them into a more compact representation. In this tutorial, the neck is implemented using a few fully-connected layers with ReLU activations. Before passing the data to these layers, a flatten operation is applied to convert the 3D tensor output from the backbone into a 1D tensor suitable for fully-connected processing.
class Neck(nn.Sequential):
def __init__(self, out_dim):
super().__init__(
nn.Flatten(),
nn.Linear(256 * 3 * 3, 1024),
nn.ReLU(),
nn.Linear(1024, 256),
nn.ReLU(),
nn.Linear(256, out_dim),
)
Embedding model#
The embedding model combines the backbone and neck in a single model that takes an input image and produces an embedding. An optional L2 normalization scales each embedding to unit length. This removes variations in vector magnitude, so similarity depends primarily on direction. It also bounds Euclidean distances between 0 and 2, giving distance-based margins a fixed and interpretable scale. Without normalization, the model could alter distances simply by increasing or decreasing embedding norms rather than by learning a more meaningful arrangement of the data.
class EmbeddingNet(nn.Module):
def __init__(self, out_dim, normalize=True):
super().__init__()
self.backbone = Backbone()
self.neck = Neck(out_dim)
self.normalize = normalize
def forward(self, x):
x = self.backbone(x)
x = self.neck(x)
if self.normalize:
x = F.normalize(x, p=2, dim=1, eps=1e-12)
return x
Let’s summarize the architecture of the embedding model using the torchinfo package. We will set (1, 28, 28) as the input shape, as this is the size of the MNIST images. We will also set 50 as the output dimension of the embeddings, a somewhat arbitrary choice for this tutorial.
model = EmbeddingNet(50)
==========================================================================================
Layer (type:depth-idx) Input Shape Output Shape
==========================================================================================
EmbeddingNet [1, 1, 28, 28] [1, 50]
├─Backbone: 1-1 [1, 1, 28, 28] [1, 256, 3, 3]
│ └─Conv2d: 2-1 [1, 1, 28, 28] [1, 64, 28, 28]
│ └─ReLU: 2-2 [1, 64, 28, 28] [1, 64, 28, 28]
│ └─MaxPool2d: 2-3 [1, 64, 28, 28] [1, 64, 14, 14]
│ └─Conv2d: 2-4 [1, 64, 14, 14] [1, 128, 14, 14]
│ └─ReLU: 2-5 [1, 128, 14, 14] [1, 128, 14, 14]
│ └─MaxPool2d: 2-6 [1, 128, 14, 14] [1, 128, 7, 7]
│ └─Conv2d: 2-7 [1, 128, 7, 7] [1, 256, 7, 7]
│ └─ReLU: 2-8 [1, 256, 7, 7] [1, 256, 7, 7]
│ └─MaxPool2d: 2-9 [1, 256, 7, 7] [1, 256, 3, 3]
├─Neck: 1-2 [1, 256, 3, 3] [1, 50]
│ └─Flatten: 2-10 [1, 256, 3, 3] [1, 2304]
│ └─Linear: 2-11 [1, 2304] [1, 1024]
│ └─ReLU: 2-12 [1, 1024] [1, 1024]
│ └─Linear: 2-13 [1, 1024] [1, 256]
│ └─ReLU: 2-14 [1, 256] [1, 256]
│ └─Linear: 2-15 [1, 256] [1, 50]
==========================================================================================
Total params: 3,137,330
Trainable params: 3,137,330
Non-trainable params: 0
Total mult-adds (Units.MEGABYTES): 58.57
==========================================================================================
Input size (MB): 0.00
Forward/backward pass size (MB): 0.71
Params size (MB): 12.55
Estimated Total Size (MB): 13.27
==========================================================================================
Contrastive loss#
The contrastive loss trains the model from labeled pairs. Each pair is marked as either similar or dissimilar, and the loss treats the two cases differently. For similar pairs, the loss penalizes large distances between their embeddings. For dissimilar pairs, the loss penalizes distances that fall below a specified margin. This leads to the following mathematical formulation.
Here, \(z_1\) and \(z_2\) are the embeddings of two inputs, \(y\) is a label denoting whether a pair is similar (\(y=1\)) or dissimilar (\(y=0\)), and \(m\) is the margin imposed on dissimilar pairs.
Intuition#
The contrastive loss encourages similar examples to cluster together while keeping dissimilar examples sufficiently separated. This behavior can be attributed to the two terms in the loss function, each of which is active for only one type of pair.
Positive term. For a similar pair, the loss is the squared Euclidean distance between the embeddings. A large distance produces a large penalty, so training brings the embeddings closer together.
Negative term. For a dissimilar pair, the loss is the squared hinge distance between their embeddings. This term is positive only when the embeddings are less than \(m\) units apart. Training pushes the embeddings away until their distance reaches the margin. Once the distance exceeds the margin, the loss is zero, and no further effort is made to separate the pair.
The margin therefore defines sufficient separation rather than requiring every dissimilar pair to be as far apart as possible. This concentrates learning on negative pairs that are still too close, while ignoring those that are already adequately separated.
Note
The negative term applies a squared hinge to the ordinary Euclidean distance. Some references use squared distance inside the hinge instead; their margin has a different scale and should not be confused with the margin used here.
Implementation#
We implement the contrastive loss in PyTorch using class labels rather than preconstructed pairs. Given a batch of embeddings, the loss computes the distance between every unordered pair. Pairs from the same class are treated as positive, while pairs from different classes are treated as negative.
Most batches contain many more negative pairs than positive pairs, especially when the dataset has many classes. A simple average over all pairs would therefore give disproportionate weight to negative pairs. To avoid this imbalance, we average the positive and negative losses separately.
Note
Each batch must contain at least one class with multiple examples, so that positive pairs exist, and at least two distinct classes, so that negative pairs exist. For datasets with rare or highly imbalanced classes, a class-balanced sampler is more reliable than random batching.
Helper function#
We begin with a helper function that identifies the positive and negative pairs in a batch. It returns Boolean masks indicating which pairs share the same class label and which pairs belong to different classes. These masks are then used to select the appropriate distances when computing the loss. Duplicate pairs and self-comparisons are removed, so each unordered pair is considered only once.
def get_similarity_masks(labels: torch.IntTensor) -> tuple[torch.BoolTensor, torch.BoolTensor]:
"""
Args
- labels: 1D tensor containing labels
Returns
- pos_mask: 2D tensor where pos_mask[i, j] = True if labels[i] == labels[j] and i < j
- neg_mask: 2D tensor where neg_mask[i, j] = True if labels[i] != labels[j] and i < j
"""
# Matrix of pairwise comparisons
labels_equal = labels.unsqueeze(0) == labels.unsqueeze(1)
# Negative pairs
neg_mask = torch.triu(~labels_equal, diagonal=1) # Remove duplicates
# Positive pairs
pos_mask = torch.triu(labels_equal, diagonal=1) # Remove duplicates and self-comparisons
return pos_mask, neg_mask
Let’s test this function with a batch of class labels to get an idea of how it works.
labels = torch.tensor([0, 0, 1, 2, 1, 2, 2, 0, 1, 2])
pos_mask, neg_mask = get_similarity_masks(labels)
Positive pairs:
[(0, 1), (0, 7), (1, 7)]
[(2, 4), (2, 8), (4, 8)]
[(3, 5), (3, 6), (3, 9), (5, 6), (5, 9), (6, 9)]
Negative pairs:
[(0, 2), (0, 3), (0, 4), (0, 5), (0, 6), (0, 8), (0, 9), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (1, 8), (1, 9), (7, 8), (7, 9)]
[(2, 3), (2, 5), (2, 6), (2, 7), (2, 9), (4, 5), (4, 6), (4, 7), (4, 9), (8, 9)]
[(3, 4), (3, 7), (3, 8), (5, 7), (5, 8), (6, 7), (6, 8)]
Balanced loss module#
The loss module receives a batch of embeddings and their class labels. It identifies positive and negative pairs, computes the Euclidean distance between every pair of embeddings, and evaluates the corresponding loss terms. Positive and negative losses are averaged separately before being added. This prevents the more numerous negative pairs from dominating the objective. A clear error is raised when the batch cannot form at least one positive and one negative pair, which indicates a problem with the data or sampling strategy. The margin is restricted to \((0,2]\) because the embeddings are assumed to be L2-normalized.
class ContrastiveLoss(nn.Module):
def __init__(self, margin: float):
super().__init__()
self.margin = margin
if not 0 < margin <= 2:
raise ValueError("For normalized embeddings, margin must be in (0, 2]")
def forward(self, embeddings, labels):
# Identify positive and negative pairs
pos_mask, neg_mask = get_similarity_masks(labels)
# Ensure at least one positive and one negative pair
if not pos_mask.any() or not neg_mask.any():
raise ValueError("Each batch needs at least one positive and one negative pair.")
# Compute all pairwise distances
distances = torch.cdist(embeddings, embeddings, p=2)
# Compute the loss terms separately
positive_loss = distances[pos_mask].pow(2).mean()
negative_loss = F.relu(self.margin - distances[neg_mask]).pow(2).mean()
return positive_loss + negative_loss
Training the network#
Our implementation of the contrastive loss has the same signature as a loss function for classification. This allows us to use the Trainer class defined in training.py file without any modification.
from training import Trainer
First, we load MNIST data and reserve 10% of the official training split for validation.
preprocess = v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)])
full_ds = MNIST('.data', train=True, download=True, transform=preprocess)
test_ds = MNIST('.data', train=False, download=True, transform=preprocess)
train_idx, valid_idx = train_test_split(
range(len(full_ds)),
test_size=0.1,
stratify=full_ds.targets,
random_state=42,
)
train_ds = Subset(full_ds, train_idx)
valid_ds = Subset(full_ds, valid_idx)
100%|██████████| 9.91M/9.91M [00:00<00:00, 13.5MB/s]
100%|██████████| 28.9k/28.9k [00:00<00:00, 352kB/s]
100%|██████████| 1.65M/1.65M [00:00<00:00, 3.69MB/s]
100%|██████████| 4.54k/4.54k [00:00<00:00, 6.07MB/s]
Then, we define the model, loss function, optimizer, and data loaders.
torch.manual_seed(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = EmbeddingNet(50, normalize=True).to(device)
optimizer = optim.Adam(model.parameters(), lr=0.001)
loss_fn = ContrastiveLoss(margin=1.0)
train_loader = DataLoader(train_ds, batch_size=128, shuffle=True)
valid_loader = DataLoader(valid_ds, batch_size=256, shuffle=False)
Note
The embedding size affects the quality of the embeddings produced by the Siamese network. We set the embedding size to 50 because MNIST is a relatively simple dataset. A larger embedding size may be necessary for more complex datasets and tasks.
Finally, we train the model for a few epochs, while monitoring the validation loss.
epochs = 5
trainer = Trainer()
history = trainer.fit(model, train_loader, loss_fn, optimizer, epochs, valid_loader)
===== Training on cpu device =====
Epoch 1/5: 100%|██████████| 422/422 [01:44<00:00, 4.04it/s, train_loss=0.0809, valid_loss=0.0273]
Epoch 2/5: 100%|██████████| 422/422 [01:45<00:00, 4.02it/s, train_loss=0.0216, valid_loss=0.0332]
Epoch 3/5: 100%|██████████| 422/422 [01:56<00:00, 3.62it/s, train_loss=0.0145, valid_loss=0.0378]
Epoch 4/5: 100%|██████████| 422/422 [02:00<00:00, 3.51it/s, train_loss=0.0103, valid_loss=0.0190]
Epoch 5/5: 100%|██████████| 422/422 [02:07<00:00, 3.32it/s, train_loss=0.0089, valid_loss=0.0196]
After training, we can plot the loss curves. Their absolute scales are comparable because both use the same balanced pair objective.
Evaluation#
A Siamese network produces embeddings rather than class predictions, so its evaluation should reflect how those embeddings will be used. We consider three complementary protocols.
K-nearest neighbors classification provides an intuitive measure of class structure. Each query is assigned the most common label among its nearest embeddings in a reference gallery. High accuracy indicates that examples from the same class tend to form local neighborhoods.
Retrieval evaluation measures the quality of the complete ranking. For each query, gallery examples are ordered by distance, and metrics such as Recall@K or mean average precision assess whether relevant examples appear near the top.
Verification evaluation determines whether two examples form a match by comparing their distance with a threshold. Because the threshold affects performance, it must be selected using validation data and then kept fixed when evaluating the test set.
In these protocols, embeddings from the training set serve as the gallery, while validation or test examples serve as queries. The sets are disjoint, preventing a query from retrieving itself and ensuring that the evaluation measures generalization to unseen examples. Low-dimensional projections such as t-SNE may also be used to inspect the embedding structure visually. They can reveal clusters or overlaps, but they are qualitative diagnostic tools and should not replace quantitative evaluation.
Embedding generation#
Evaluation begins by computing embeddings for the reference and query datasets. Each dataset is passed through the trained model, and the resulting embeddings and labels are collected into tensors. Since this step does not involve training, the model is placed in evaluation mode and gradient computation is disabled. The stored embeddings can then be reused for different evaluation protocols.
gallery_embeddings, gallery_labels = get_embeddings(model, train_ds)
valid_embeddings, valid_labels = get_embeddings(model, valid_ds)
test_embeddings, test_labels = get_embeddings(model, test_ds)
k-NN classification#
k-nearest neighbors evaluates whether nearby embeddings tend to belong to the same class. For each test embedding, we find the \(k\) closest embeddings in the training gallery and predict the label by majority vote among their class labels. The proportion of correct predictions provides a simple measure of how well the embedding space organizes examples by class. High accuracy indicates that examples from the same class form consistent clusters, even though the embedding model was not trained with a classification head.
knn = KNeighborsClassifier(n_neighbors=5, metric="euclidean")
knn.fit(gallery_embeddings.numpy(), gallery_labels.numpy())
predictions = knn.predict(test_embeddings.numpy())
score = accuracy_score(test_labels.numpy(), predictions)
print(f"Test k-NN accuracy: {score:.2%}")
Test k-NN accuracy: 99.44%
Retrieval#
Retrieval evaluates the ordering of gallery examples rather than predicting a single class label. For each query embedding, the gallery is ranked from nearest to farthest. In MNIST, a gallery image is considered relevant when it has the same digit label as the query. A good embedding model places relevant images near the top of this ranking. Precision and recall metrics quantify how consistently this occurs across queries.
test_retrieval = retrieval_metrics(test_embeddings, test_labels, gallery_embeddings, gallery_labels)
for name, value in test_retrieval.items():
print(f"{name}: {value:.2%}")
recall@1: 99.41%
recall@5: 99.61%
mAP: 99.49%
Verification#
Verification determines whether two examples should be considered a match based on the distance between their embeddings. For evaluation, we sample equal numbers of positive and negative pairs from separate dataset splits. This keeps the computation manageable and prevents the result from being dominated by one pair type. A decision threshold is chosen using validation pairs: distances below the threshold are classified as matches, while larger distances are classified as nonmatches. Once selected, the threshold is kept fixed and applied to the test pairs. Performance is reported using balanced accuracy, which gives equal importance to correctly identifying matches and nonmatches.
valid_distances, valid_targets = sample_pair_distances(valid_embeddings, valid_labels, gallery_embeddings, gallery_labels)
test_distances, test_targets = sample_pair_distances(test_embeddings, test_labels, gallery_embeddings, gallery_labels)
thresholds = torch.linspace(valid_distances.min(), valid_distances.max(), steps=401)
scores = [
balanced_accuracy_score(valid_targets.numpy(), (valid_distances <= threshold).numpy())
for threshold in thresholds
]
best_threshold = thresholds[int(torch.tensor(scores).argmax())]
test_matches = test_distances <= best_threshold
test_balanced_accuracy = balanced_accuracy_score(test_targets.numpy(), test_matches.numpy())
print(f"Validation-selected threshold: {best_threshold:.3f}")
print(f"Test verification balanced accuracy: {test_balanced_accuracy:.2%}")
Validation-selected threshold: 0.823
Test verification balanced accuracy: 99.69%
Qualitative visualization#
t-SNE projects high-dimensional embeddings into two dimensions while attempting to preserve local neighborhoods. It can therefore help reveal whether examples from the same class tend to appear near one another. The resulting plot must be interpreted cautiously. The axes have no intrinsic meaning, and apparent cluster sizes, distances, and separations can change with the random seed and t-SNE hyperparameters. The visualization is therefore a qualitative diagnostic, not a measure of embedding quality.
To obtain a coherent view, we fit t-SNE once on a bounded sample from a held-out dataset. Training and test embeddings should not be projected separately, because independently fitted t-SNE plots use unrelated coordinate systems and cannot be compared directly.
Summary#
In this tutorial, we trained a Siamese network on MNIST using a balanced contrastive loss. The model learned normalized embeddings in which images of the same digit were encouraged to remain close, while images of different digits were separated by a margin. Averaging positive and negative losses separately prevented the more numerous negative pairs from dominating training.
We then evaluated the learned representation through k-NN classification, retrieval, and verification on held-out data. These protocols examine different properties of the embedding space: local class structure, ranking quality, and pairwise matching. The t-SNE projection provided a qualitative view of the learned neighborhoods, but it should not be treated as quantitative evidence. Conclusions about embedding quality should instead rely primarily on downstream metrics.
More generally, the effectiveness of contrastive learning depends on how similarity is defined, how batches are constructed, and how the margin relates to the embedding scale. Informative negative pairs are essential: poor sampling may yield weak representations even when the training loss appears satisfactory. Embedding variance, pairwise-distance distributions, and retrieval results should therefore be monitored throughout development.