Evaluation#
Use this page to quickly reference how to evaluate a fixed model, aggregate metrics over a dataset, select a model using validation data, and reserve the test set for final evaluation.
Data Roles#
Training / Validation / Test#
Data is typically split into three partitions, each with a different role.
The training set is used to update model parameters and fit any preprocessing that learns statistics or parameters from data.
The validation set is used to guide development decisions without updating model parameters.
The test set is reserved for final evaluation after development choices are complete.
Partition |
May influence |
|---|---|
Training |
Model parameters and fitted preprocessing |
Validation |
Model selection and development decisions |
Test |
Final reported evaluation only |
Warning
Do not use test performance to choose hyperparameters or take development decisions.
Data Leakage#
Data leakage occurs when information that should be unavailable during training or model selection enters the development process. Common examples include:
fitting preprocessing on validation or test data;
duplicated examples across data partitions;
splitting related examples from the same group across partitions;
selecting models or hyperparameters using test performance.
For splitting and fitted preprocessing, see Data Management.
Evaluating a Model#
Use both model.eval() and torch.inference_mode() during evaluation.
model.eval()
with torch.inference_mode():
outputs = model(inputs)
These functions have different responsibilities.
model.eval()changes the behaviour of layers such as dropout and batch normalization.torch.inference_mode()disables gradient recording.
If evaluation occurs inside a training workflow, restore training mode before the next parameter update:
model.train()
Evaluation Loop#
A canonical evaluation loop measures the loss of a fixed model over the complete dataset.
def evaluate(model, loader, loss_fn, device):
model.eval()
total_loss = 0.0
total_weight = 0
with torch.inference_mode():
for inputs, targets in loader:
inputs = inputs.to(device)
targets = targets.to(device)
outputs = model(inputs)
loss = loss_fn(outputs, targets)
batch_weight = targets.numel()
total_loss += loss.item() * batch_weight
total_weight += batch_weight
return total_loss / total_weight
This pattern applies to multiclass classification, binary classification, and regression when the loss uses reduction="mean" and averages over the target values.
Aggregating Loss#
A mean batch loss must be weighted by the number of values represented by that mean before losses from several batches are combined. If batch \(b\) contains \(M_b\) target values and reports mean loss \(L_b\), the complete evaluation loss is
The corresponding pattern is:
batch_weight = targets.numel()
total_loss += loss.item() * batch_weight
total_weight += batch_weight
mean_loss = total_loss / total_weight
For classification targets with shape (N,), targets.numel() is the batch size. For regression with multiple targets per example, targets.numel() also counts the additional target values to weight the loss correctly.
Predictions and Metrics#
Loss can be evaluated with the same general workflow across tasks, but predictions and metrics depend on what the model is predicting.
Task |
Model output |
Prediction or error |
Common metric |
|---|---|---|---|
Multiclass classification |
logits |
largest logit |
accuracy |
Binary classification |
logits |
logit threshold at |
accuracy |
Regression |
predictions |
prediction error |
MAE or MSE |
When several metrics are required, update them during the same pass through the evaluation loader rather than creating a separate evaluation workflow for each one.
Multiclass classification
A multiclass classifier produces one logit per class for each example. The predicted class is the one with the largest logit. Accuracy then measures the fraction of examples for which this predicted class matches the target class.
Before the batch loop, initialize the accumulators.
total_correct = 0
total_examples = 0
Inside the batch loop, accumulate the number of correct predictions and total examples.
predictions = outputs.argmax(dim=1)
total_correct += (predictions == targets).sum().item()
total_examples += targets.shape[0]
After the batch loop, compute the overall accuracy.
accuracy = total_correct / total_examples
Binary classification
A binary classifier produces one logit per example. A positive logit corresponds to a probability greater than 0.5, so predictions can be obtained by comparing the logits with 0. Accuracy measures the fraction of examples for which this predicted class matches the target class.
Before the batch loop, initialize the accumulators.
total_correct = 0
total_values = 0
Inside the batch loop, accumulate the number of correct predictions and total values.
predictions = outputs >= 0
binary_targets = targets >= 0.5
assert predictions.shape == binary_targets.shape
total_correct += (predictions == binary_targets).sum().item()
total_values += targets.numel()
After the batch loop, compute the overall accuracy.
accuracy = total_correct / total_values
The model outputs and targets should follow the same shape convention, such as (N,) for both.
Regression
A regression model predicts continuous values. Regression metrics, such as mean absolute error (MAE) and mean squared error (MSE), measure how far the predictions are from their target values.
Before the batch loop, initialize the accumulators.
total_absolute_error = 0.0
total_squared_error = 0.0
total_values = 0
Inside the batch loop, accumulate the absolute and squared errors and the total number of values.
total_absolute_error += (outputs - targets).abs().sum().item()
total_squared_error += ((outputs - targets) ** 2).sum().item()
total_values += targets.numel()
After the batch loop, compute the overall MAE and MSE.
mae = total_absolute_error / total_values
mse = total_squared_error / total_values
These metrics describe different aspects of prediction error: MAE is more robust to outliers, while MSE penalizes large errors more heavily. Choose the metric that aligns with the goals of your regression task.
Stateful Metrics#
The above accumulators implement a simple stateful metric: initialize the required state before evaluation, update it for each batch, then compute the final value after all batches have been processed. Metric libraries such as TorchEval and TorchMetrics package this pattern into metric objects with operations such as update(), compute(), and reset(). Some libraries also provide stateless functional interfaces when the complete inputs needed for a metric are available at once.
Model Selection#
Validation-Based Checkpointing#
Choose a checkpoint using a predefined validation criterion.
Initialize tracking before training:
best_valid_loss = float("inf")
best_state = None
best_epoch = None
After each validation evaluation:
if valid_loss < best_valid_loss:
best_valid_loss = valid_loss
best_epoch = epoch
best_state = copy.deepcopy(model.state_dict())
The selected checkpoint may come from an epoch before the end of training.
Warning
Do not use best_model = model to preserve the best model. That creates another reference to the same object, so later updates continue changing it. Use an independent copy of the state dictionary.
Restoring the Best State#
Restore the selected validation checkpoint after training:
model.load_state_dict(best_state)
The model now contains the parameters and persistent buffers from the selected state.
Early Stopping#
Early stopping and checkpointing solve different problems.
Mechanism |
Purpose |
|---|---|
Checkpointing |
Preserve the best observed model state |
Early stopping |
Decide when to stop training |
A simple stopping condition is:
if epochs_without_improvement >= patience:
break
A training run may use checkpointing without early stopping.
Model State vs Training Checkpoint#
A model state preserves the model itself:
torch.save(best_state, "best_model.pt")
A resumable training checkpoint may also preserve optimizer state and training metadata:
checkpoint = {
"epoch": epoch,
"model_state": model.state_dict(),
"optimizer_state": optimizer.state_dict(),
"valid_loss": valid_loss,
}
torch.save(checkpoint, "checkpoint.pt")
The model and optimizer states in a resumable checkpoint should come from the same training point.
For model-state saving and loading, see Models.
Final Evaluation#
Restore the selected checkpoint before the final test evaluation:
model.load_state_dict(best_state)
test_loss = evaluate(model, test_loader, loss_fn, device)
Warning
If a test result causes you to modify the model or training procedure, the test set is no longer a valid measure of generalization. It has then become part of development. A new untouched test set is required to obtain another independent final evaluation.
For interpreting learning curves and designing controlled comparisons, see Diagnostics.