# Evaluation

Use this page to quickly reference how to evaluate a fixed model, aggregate metrics over a dataset, select a model using validation data, and reserve the test set for final evaluation.

## Data Roles

### Training / Validation / Test

Data is typically split into three partitions, each with a different role.
- The training set is used to update model parameters and fit any preprocessing that learns statistics or parameters from data. 
- The validation set is used to guide development decisions without updating model parameters.
- The test set is reserved for final evaluation after development choices are complete.

| Partition  | May influence |
| ---------- | ------------- |
| Training   | Model parameters and fitted preprocessing |
| Validation | Model selection and development decisions |
| Test       | Final reported evaluation only |


:::{warning}
Do not use test performance to choose hyperparameters or take development decisions.
:::


### Data Leakage

Data leakage occurs when information that should be unavailable during training or model selection enters the development process. Common examples include:

- fitting preprocessing on validation or test data;
- duplicated examples across data partitions;
- splitting related examples from the same group across partitions;
- selecting models or hyperparameters using test performance.

For splitting and fitted preprocessing, see [Data Management](quickref-data.md).


## Evaluating a Model

Use both `model.eval()` and `torch.inference_mode()` during evaluation.

```python
model.eval()

with torch.inference_mode():
    outputs = model(inputs)
```

These functions have different responsibilities.
- `model.eval()` changes the behaviour of layers such as dropout and batch normalization.
- `torch.inference_mode()` disables gradient recording.

If evaluation occurs inside a training workflow, restore training mode before the next parameter update:

```python
model.train()
```

### Evaluation Loop

A canonical evaluation loop measures the loss of a fixed model over the complete dataset.

```python
def evaluate(model, loader, loss_fn, device):
    model.eval()

    total_loss = 0.0
    total_weight = 0

    with torch.inference_mode():
        for inputs, targets in loader:
            inputs = inputs.to(device)
            targets = targets.to(device)

            outputs = model(inputs)
            loss = loss_fn(outputs, targets)

            batch_weight = targets.numel()
            total_loss  += loss.item() * batch_weight
            total_weight += batch_weight

    return total_loss / total_weight
```

This pattern applies to multiclass classification, binary classification, and regression when the loss uses `reduction="mean"` and averages over the target values.

### Aggregating Loss

A mean batch loss must be weighted by the number of values represented by that mean before losses from several batches are combined. If batch $b$ contains $M_b$ target values and reports mean loss $L_b$, the complete evaluation loss is

$$
L_{\text{eval}} = \frac{\sum_b M_b L_b}{\sum_b M_b}.
$$

The corresponding pattern is:

```python
batch_weight = targets.numel()

total_loss += loss.item() * batch_weight
total_weight += batch_weight

mean_loss = total_loss / total_weight
```

For classification targets with shape `(N,)`, `targets.numel()` is the batch size. For regression with multiple targets per example, `targets.numel()` also counts the additional target values to weight the loss correctly.


### Predictions and Metrics

Loss can be evaluated with the same general workflow across tasks, but predictions and metrics depend on what the model is predicting.

| Task                      | Model output                   | Prediction or error    | Common metric |
| ------------------------- | ------------------------------ | ---------------------- | ------------- |
| Multiclass classification | logits `(N, K)`                | largest logit          | accuracy      |
| Binary classification     | logits `(N, 1)` or `(N,)`      | logit threshold at `0` | accuracy      |
| Regression                | predictions `(N, T)` or `(N,)` | prediction error       | MAE or MSE    |

When several metrics are required, update them during the same pass through the evaluation loader rather than creating a separate evaluation workflow for each one.

:::{admonition} Multiclass classification
:class: dropdown, tip

A multiclass classifier produces one logit per class for each example. The predicted class is the one with the largest logit. Accuracy then measures the fraction of examples for which this predicted class matches the target class.

Before the batch loop, initialize the accumulators.

```python
total_correct = 0
total_examples = 0
```

Inside the batch loop, accumulate the number of correct predictions and total examples.

```python
predictions = outputs.argmax(dim=1)
total_correct += (predictions == targets).sum().item()
total_examples += targets.shape[0]
```

After the batch loop, compute the overall accuracy.

```python
accuracy = total_correct / total_examples
```
:::

:::{admonition} Binary classification
:class: dropdown, tip

A binary classifier produces one logit per example. A positive logit corresponds to a probability greater than 0.5, so predictions can be obtained by comparing the logits with 0. Accuracy measures the fraction of examples for which this predicted class matches the target class.

Before the batch loop, initialize the accumulators.

```python
total_correct = 0
total_values = 0
```

Inside the batch loop, accumulate the number of correct predictions and total values.

```python
predictions = outputs >= 0
binary_targets = targets >= 0.5
assert predictions.shape == binary_targets.shape

total_correct += (predictions == binary_targets).sum().item()
total_values += targets.numel()
```

After the batch loop, compute the overall accuracy.

```python
accuracy = total_correct / total_values
```

The model outputs and targets should follow the same shape convention, such as `(N,)` for both.
:::

:::{admonition} Regression
:class: dropdown, tip

A regression model predicts continuous values. Regression metrics, such as mean absolute error (MAE) and mean squared error (MSE), measure how far the predictions are from their target values. 

Before the batch loop, initialize the accumulators.

```python
total_absolute_error = 0.0
total_squared_error = 0.0
total_values = 0
```

Inside the batch loop, accumulate the absolute and squared errors and the total number of values.

```python
total_absolute_error += (outputs - targets).abs().sum().item()
total_squared_error += ((outputs - targets) ** 2).sum().item()
total_values += targets.numel()
```

After the batch loop, compute the overall MAE and MSE.

```python
mae = total_absolute_error / total_values
mse = total_squared_error / total_values
```

These metrics describe different aspects of prediction error: MAE is more robust to outliers, while MSE penalizes large errors more heavily. Choose the metric that aligns with the goals of your regression task.
:::

### Stateful Metrics

The above accumulators implement a simple stateful metric: initialize the required state before evaluation, update it for each batch, then compute the final value after all batches have been processed. Metric libraries such as TorchEval and TorchMetrics package this pattern into metric objects with operations such as `update()`, `compute()`, and `reset()`. Some libraries also provide stateless functional interfaces when the complete inputs needed for a metric are available at once.


## Model Selection

### Validation-Based Checkpointing

Choose a checkpoint using a predefined validation criterion.

Initialize tracking before training:

```python
best_valid_loss = float("inf")
best_state = None
best_epoch = None
```

After each validation evaluation:

```python
if valid_loss < best_valid_loss:
    best_valid_loss = valid_loss
    best_epoch = epoch
    best_state = copy.deepcopy(model.state_dict())
```

The selected checkpoint may come from an epoch before the end of training.

:::{warning}
Do not use `best_model = model` to preserve the best model. That creates another reference to the same object, so later updates continue changing it. Use an independent copy of the state dictionary.
:::


### Restoring the Best State

Restore the selected validation checkpoint after training:

```python
model.load_state_dict(best_state)
```

The model now contains the parameters and persistent buffers from the selected state.


### Early Stopping

Early stopping and checkpointing solve different problems.

| Mechanism | Purpose |
| --------- | ------- |
| Checkpointing | Preserve the best observed model state |
| Early stopping | Decide when to stop training |

A simple stopping condition is:

```python
if epochs_without_improvement >= patience:
    break
```

A training run may use checkpointing without early stopping.


### Model State vs Training Checkpoint

A model state preserves the model itself:

```python
torch.save(best_state, "best_model.pt")
```

A resumable training checkpoint may also preserve optimizer state and training metadata:

```python
checkpoint = {
    "epoch": epoch,
    "model_state": model.state_dict(),
    "optimizer_state": optimizer.state_dict(),
    "valid_loss": valid_loss,
}

torch.save(checkpoint, "checkpoint.pt")
```

The model and optimizer states in a resumable checkpoint should come from the same training point.

For model-state saving and loading, see [Models](quickref-models.md).


## Final Evaluation

Restore the selected checkpoint before the final test evaluation:

```python
model.load_state_dict(best_state)

test_loss = evaluate(model, test_loader, loss_fn, device)
```

:::{warning}
If a test result causes you to modify the model or training procedure, the test set is no longer a valid measure of generalization. It has then become part of development. A new untouched test set is required to obtain another independent final evaluation.
:::

For interpreting learning curves and designing controlled comparisons, see [Diagnostics](quickref-diagnosis.md).