7. Representation Learning

7. Representation Learning#

../_images/repr.png

A neural network is often described as a function that maps an input to a prediction. An image enters the model, several layers process it, and the final layer produces the class logits. This description is correct, but it leaves out an important part of the computation. Before the model can make a useful prediction, it repeatedly transforms the input into new forms. These intermediate forms are called representations.

Earlier lessons focused on how those transformations contribute to solving a specific task. This lesson asks a broader question: what makes a learned representation useful on its own? A representation may simplify the original prediction problem, transfer to another dataset or task, provide a meaningful way to compare examples, or support unsupervised learning tasks. These uses depend on what information the representation preserves and how that information is organized.

1. Useful Representations#

Raw inputs rarely organize information in a form that is convenient for a task. Two images of the same object can differ greatly at the pixel level because of position, lighting, viewpoint, or background. Conversely, two images with similar average intensities may depict entirely different objects. A representation becomes useful when it reorganizes the input so that relevant distinctions and relationships become easier to express.

Encoder and head#

A representation is an encoding of an input made available for subsequent computation. To distinguish the learned representation from the task-specific computation that uses it, we can describe a neural network as two separate but connected components:

\[ \mathbf{h}=f_{\theta}(\mathbf{x}), \qquad \hat{\mathbf{y}}=g_{\phi}(\mathbf{h}). \]

The encoder \(f_{\theta}\) transforms the input \(\mathbf{x}\) into a representation \(\mathbf{h}\). The head \(g_{\phi}\) uses the representation for a particular task. For an image classifier, the encoder may contain convolutional layers and the head may be the final linear layers. The boundary between the two is a modelling choice. A deep network contains many intermediate representations, and different applications may choose different layers as the encoder output.

Important

The encoder produces a representation. The head interprets a representation for a particular task.

The representation supplied to the head may be fixed in advance or learned jointly with the head. A model trained on a fixed representation can only learn a decision rule in that representation space. When the encoder and head are trained together, the encoder can also change the representation on which the decision rule operates. The architecture determines which transformations the encoder can express, while the training objective rewards those that help solve the task.

Representation space#

Consider a classification problem in which two classes cannot be separated by a straight line in the original input space. A linear classifier acts directly on the original coordinates, so no choice of its parameters can produce the required boundary. A neural network inserts a learned transformation before the classifier:

\[ \mathbf{h}=f_{\theta}(\mathbf{x}), \qquad \mathbf{z}=W\mathbf{h}+\mathbf{b}. \]

The classifier is still linear, but it no longer operates on the original coordinates. The encoder reorganizes the coordinates presented to the classifier, so that a straight boundary in representation space can correspond to a nonlinear boundary in input space. This is one of the central ideas of representation learning:

A useful representation makes a difficult relationship easier to express by a simple model.

(See the widget at the top for an interactive demonstration.)

Desirable properties#

There is no universally best representation. The same input can be represented in many ways, but a representation that is useful for one task may be useless for another. Consequently, the quality of a representation depends on the task for which it needs to be used. Three properties are especially important.

  • Transferability describes whether the representation remains useful when the dataset, prediction head, or task changes. A representation learned from a broad collection of images may transfer because it captures shapes, textures, object parts, and other patterns that recur across several visual tasks.

  • Meaningful similarity means that distance or angle in representation space reflects a relationship relevant to the application. Similarity is not an intrinsic property waiting to be discovered. It depends on what the system is intended to treat as equivalent.

  • Appropriate invariance means that the representation changes little under variations that should not matter. A classifier may benefit from being relatively insensitive to small translations. A model estimating position cannot ignore translations because position is precisely what it must predict.

These properties can conflict. A representation that suppresses small visual differences may be excellent for category classification yet poor for distinguishing examples within a class. The intended use of the representation determines which properties are desirable.

2. Learning Representations for Classification#

The models developed earlier in the course already perform representation learning. When an encoder and classification head are trained jointly, the classification loss changes not only the final layer but also the transformations that produce its inputs. Classification is therefore a representation learning objective, even though the loss is applied to the logits rather than directly to the representation.

Classification head#

For a labelled example \((\mathbf{x},y)\), a classification objective can be written as

\[ \mathbf{h}=f_{\theta}(\mathbf{x}), \qquad \mathbf{z}=g_{\phi}(\mathbf{h}), \qquad \mathcal{L}=\operatorname{CE}(\mathbf{z},y). \]

Cross-entropy measures whether the logits support the correct class. The encoder still receives a learning signal because its output affects the logits. Across many examples, the encoder is rewarded for retaining distinctions that help the classification head. If shape is consistently useful for distinguishing classes, it will be preserved in the representation. If a background pattern predicts a class in the training data, the encoder may also preserve that pattern, even when it is not part of the intended concept.

The encoder is not explicitly told which properties of the input are relevant. Cross-entropy shapes the representation through the requirements of correct prediction, while leaving many properties of its geometry unspecified. The dataset determines which variations and correlations are observed, and the labels specify which ones must be retained to solve the training problem. Any information unrelated to this goal is not guaranteed to survive in the representation.

Transfer learning#

A model trained to recognize many categories of objects may learn a representation useful for a different classification problem, even when the new classes were not present in the original dataset. This is the basis of transfer learning: a classifier trained on a source task can be reused for a target task. The source task may be a large-scale classification problem, or it may be a smaller task that is related to the target task.

Transfer learning supports several forms of reuse. A pretrained encoder can provide a better starting point than random initialization, particularly when the target dataset is small. It can also produce features that are already useful for the new task, even without further adaptation. In either case, transfer is most effective when the source task encourages the encoder to retain information that is also relevant to the target task.

Once a model has been pretrained, the source classification head is no longer essential. A new system can attach another head to the representation and choose how much of the encoder to update.

  • With frozen feature extraction, the encoder parameters remain fixed. Target examples are passed through the encoder, and only the new head is trained. This is computationally efficient and reduces the number of parameters estimated from limited target data.

  • With fine-tuning, some or all encoder parameters are updated on the target task. This is more flexible than feature extraction, but it can also overfit when the target dataset is small or destroy useful features.

The practical details of freezing layers and configuring optimizers are covered by transfer-learning workflows. For representation learning, the important point is that the encoder can be separated from its source head, used as a feature extractor, and adapted when the target requires different information.

Limits of class supervision#

Classification can produce excellent representations, but they emerge indirectly as a byproduct of the source task, rather than through direct supervision of the representation itself. This leaves the encoder free to organize the representation space in ways that are not always useful for other tasks.

  1. Classification imposes only a partial structure on the representation space.

    • The encoder has no incentive to arrange examples within a class in any particular way, nor to separate examples from different classes beyond what is necessary for the head.

  2. Classification can be too coarse (or too fine).

    • A category label may group together examples that are too different for a target task. On the other hand, very fine labels can encourage specific distinctions that do not transfer to broader categories.

  3. Shortcut learning.

    • If an accidental feature predicts the label in the source data, the representation is encouraged to preserve that feature, even if it is not relevant outside the source distribution.

  4. Classification encourages invariance only to variations observed in the data.

    • If all training images of a class share the same background, the model has no reason to ignore it. Data augmentation can add variation, but its design determines which invariances are learned.

These limitations motivate objectives that supervise relationships between representations more directly.

3. Learning Representations for Similarity#

Some applications require a representation that supports more than classification. A retrieval system must rank examples by similarity. A verification system must decide whether two observations refer to the same object. A nearest-neighbour classifier may need to handle classes that were not present during training. In such cases, the arrangement of examples in representation space is part of the learned solution.

Embeddings#

When a vector representation is intended for comparison or grouping, it is commonly called an embedding. An encoder maps an input \(\mathbf{x}\) to the corresponding embedding \(\mathbf{h}\) in a vector space of dimension \(d\):

\[ \mathbf{h}=f_{\theta}(\mathbf{x}) \in \mathbb{R}^{d}. \]

Once examples are embedded, they can be compared using Euclidean distance or cosine similarity.

Choosing a similarity measure does not define what it means for two examples to be similar. A representation space can be organized in many ways, and the same distance function can express different relationships depending on how the encoder arranges the vectors. The training objective and supervision determine which relationships the encoder is encouraged to preserve.

Learning from pairs#

The simplest way to supervise similarity is through pairs of examples. A pair consists of two inputs \((\mathbf{x}_i,\mathbf{x}_j)\) and a relationship label \(s_{ij} \in \{0,1\}\) indicating whether the examples should be treated as similar (\(s_{ij}=1\)) or dissimilar (\(s_{ij}=0\)). Both examples pass through the same encoder:

\[ \mathbf{h}_i=f_{\theta}(\mathbf{x}_i), \qquad \mathbf{h}_j=f_{\theta}(\mathbf{x}_j). \]

The shared encoder is important to ensure that the resulting embeddings are comparable. A pairwise objective can encourage positive examples to become closer and negative examples to remain separated. One common formulation is a margin-based contrastive loss:

\[ \mathcal{L}_{ij} = s_{ij}\,d(\mathbf{h}_i,\mathbf{h}_j)^2 + (1-s_{ij}) \max\big\{0,m-d(\mathbf{h}_i,\mathbf{h}_j)\big\}^2. \]

For a positive pair, only the first term is active, so large distance is penalized. For a negative pair, the second term is active and penalizes the pair only when the embeddings are closer than the margin \(m\). The margin ensures that dissimilar examples are not pushed apart indefinitely.

Learning from triplets#

Another way to supervise similarity is through the relative comparison of three examples. A triplet consists of an anchor \(\mathbf{x}_a\), a positive \(\mathbf{x}_p\) that should be similar to the anchor, and a negative \(\mathbf{x}_n\) that should be dissimilar. The goal is to ensure that the anchor is closer to the positive than to the negative in the embedding space, after passing through the same encoder:

\[ \mathbf{h}_a=f_{\theta}(\mathbf{x}_a), \qquad \mathbf{h}_p=f_{\theta}(\mathbf{x}_p), \qquad \mathbf{h}_n=f_{\theta}(\mathbf{x}_n). \]

The desired relation is \(d(\mathbf{h}_a,\mathbf{h}_p) < d(\mathbf{h}_a,\mathbf{h}_n)\), which can be enforced by a margin-based triplet loss:

\[ \mathcal{L}_{a,p,n} =\max\big\{0, d(\mathbf{h}_a,\mathbf{h}_p) - d(\mathbf{h}_a,\mathbf{h}_n) + m \big\}. \]

The loss is zero only when the positive is closer to the anchor than the negative by at least the margin \(m\). This formulation does not prescribe one absolute distance for all positive pairs. It asks for a ranking: the positive should be closer than the negative. The construction of triplets is part of the learning problem. Several techniques exist to select informative triplets and collect them efficiently.

Important

Pair and triplet objectives reveal a broader principle: supervision consists not only of labels, but also of decisions about which relationships should structure the representation.

Learning from many comparisons#

Pair and triplet objectives reason about only a small number of examples at a time. A batch, however, may contain many useful relationships. Suppose a batch contains examples from classes A, B, and C. For an anchor from class A, the other A examples can act as positives while examples from B and C act as negatives.

batch: A1  A2  B1  C1  B2  A3  C2

anchor: A1
positives: A2, A3
negatives: B1, B2, C1, C2

Supervised contrastive learning uses this idea to compare one anchor with many positive and negative examples during the same update. A common formulation is as follows. For an anchor \(i\), let \(P(i)\) be the set of indices of other positive examples in the batch. The supervised contrastive loss for that anchor is

\[ \mathcal{L}_i = - \frac{1}{|P(i)|} \sum_{p\in P(i)} \log \frac{\exp(\operatorname{sim}(\mathbf{q}_i,\mathbf{q}_p)/\tau)} {\sum_{a\neq i}\exp(\operatorname{sim}(\mathbf{q}_i,\mathbf{q}_a)/\tau)}. \]

The function \(\operatorname{sim}\) is cosine similarity, and the temperature \(\tau\) controls how strongly similarity differences influence the comparison. For each positive, the numerator measures the anchor’s similarity to that positive. The denominator compares this value with the anchor’s similarities to every other eligible example in the batch. The loss becomes smaller when positive similarities are high relative to the similarities of negatives.

Important

Pair, triplet, and contrastive objectives inherit the meaning of similarity from the labels in the training data. They do not discover a universal notion of similarity.

Projection head#

Contrastive learning often introduces a projection head to separate the representation intended for later use from the vectors on which the loss is applied.

\[ \mathbf{h}=f_{\theta}(\mathbf{x}), \qquad \mathbf{q}=p_{\psi}(\mathbf{h}). \]

The encoder \(f_{\theta}\) produces the representation \(\mathbf{h}\), and the projection head \(p_{\psi}\) maps it to a vector \(\mathbf{q}\), often followed by normalization. The contrastive objective shapes the projected vectors, while the encoder representation can be retained for downstream tasks. After training, the projection head is discarded.

Important

The representation on which a training loss is applied does not need to be the representation eventually exposed to another task.

4. Comparison of training objectives#

Classification, metric learning, and contrastive learning use different objectives, but they all train an encoder by imposing requirements on its output. The main difference lies in what is supervised directly.

Method

Direct requirement

Classification

The head must predict the correct class

Metric learning (pairs)

Selected pairs should be close or separated

Metric learning (triplets)

Selected triplets should maintain a relative distance order

Supervised contrastive learning

Positives should be more similar than many negatives

None of these objectives produces a universally best representation. Each emphasizes the structure expressed by its supervision.

When to use each approach#

Classification is often a strong choice when abundant labelled data is available and the main goal is to learn a broadly useful encoder that supports prediction on related tasks. It is simple to optimize, scales well to large datasets, and can produce representations that transfer effectively. Its main limitation is that the representation space is shaped only indirectly: the encoder must provide enough information for the classification head to predict correctly, but distances among representations are not themselves part of the objective.

Metric learning is more appropriate when comparison is central to the application. The representation may be used directly through distances or similarities rather than through a learned classification head. The effectiveness of pair and triplet objectives depends strongly on how positive and negative examples are defined. If those relationships do not reflect the intended notion of similarity, the resulting geometry may be poorly suited to the application.

Supervised contrastive learning serves a similar purpose. It encourages the representation space to be organized around the supplied notion of similarity, while retaining a strong class-oriented geometry. This can be useful when the downstream task benefits from both class-oriented and similarity-oriented representations. As with metric learning, however, the learned geometry inherits the assumptions encoded by the labels and positive-example construction.

This list should not be interpreted as a ranking from weaker to stronger method. The choice of which approach to use depends on the available data, the intended use of the representation, and the desired properties of the resulting geometry.

Evaluating a representation#

A representation should ultimately be evaluated through what it enables. One common approach is to freeze the encoder and train a simple linear classifier on top of its output. If the representation is useful, a linear model should achieve good performance on the target task. This evaluation can be applied to any representation, regardless of how it was learned.

Another approach is to measure the quality of the representation for retrieval or verification. A nearest-neighbour classifier can be applied directly in the representation space. Given a query example, the evaluator retrieves the closest examples in the representation space and measures whether the returned examples satisfy the intended similarity relation.

Visualizing the representation can provide intuition, especially after projecting a high-dimensional embedding into two dimensions. Such plots should be treated cautiously. Dimensionality reduction necessarily distorts some relationships, so attractive clusters are not by themselves evidence that a representation is useful.

5. Conclusion#

Representation learning provides a broader way to understand neural networks. An encoder does not merely pass information toward a prediction. It constructs a representation in which selected information, distinctions, and relationships are preserved, as long as they support the training objective. The usefulness of that representation depends on the task: it may need to support accurate classification, transfer to another dataset, preserve a meaningful notion of similarity, or remain stable under variations that should not affect the result.

Supervised classification already performs representation learning. Cross-entropy rewards correct predictions, and the encoder develops whichever internal structure that allows the classification head to separate the required classes. However, this objective constrains the representation only indirectly. It does not require same-class examples to occupy nearby regions of the space, nor does it guarantee that Euclidean distance or cosine similarity will express a useful relationship for another application.

Metric learning and contrastive learning introduce more direct relational supervision, asking the encoder to place selected examples close together or maintain particular distance orderings. Pair-based objectives specify which examples should be close or separated. Triplet objectives express relative similarity. Supervised contrastive learning uses many such relationships simultaneously. In each case, the supervision defines what the representation should preserve, what it should ignore, and which examples should be considered related.

The broader lesson is that deep networks do not simply learn to make predictions. They learn hidden representations that organize information in a way that makes predictions easier. The encoder transforms the input into a form that preserves relevant distinctions and relationships while discarding irrelevant details.

Final Quiz#

Quiz 1

What is the difference between an encoder and a task-specific head?

Answer The encoder transforms the input into a representation. The head consumes that representation to perform a particular task, such as classification.

Quiz 2

A classifier achieves perfect accuracy. Does this guarantee that examples from the same class are close together under Euclidean distance in its hidden representation?

Answer No. Classification requires the head to produce the correct decision, but it does not uniquely determine distances among hidden representations.

Quiz 3

Why can a pretrained encoder remain useful after its original classification head is removed?

Answer The encoder may have learned representations that preserve information useful for other datasets or tasks. A new head can operate on those representations.

Quiz 4

In a triplet containing an anchor, positive, and negative, what relationship does the loss encourage?

Answer The positive should be closer to the anchor than the negative, usually by at least a chosen margin.

Quiz 5

Two photographs show different shoes from the same category. Should they be treated as a positive pair?

Answer It depends on the task. They may be positives for category retrieval but negatives when the goal is to retrieve the exact same product.

Quiz 6

What does strong performance from a linear probe suggest?

Answer It suggests that the frozen representation already organizes the information in a way that makes the target task accessible through a simple linear decision rule.

Quiz 7

Why is an attractive two-dimensional visualization not sufficient evidence that an embedding is useful?

Answer The projection from a high-dimensional representation into two dimensions distorts some relationships. Representation quality should ultimately be evaluated through the downstream task or similarity relation that matters.