Diagnostics

Diagnostics#

Use this page to quickly reference how to diagnose poor learning, verify that the training pipeline works, distinguish common failure modes, and design controlled experiments.

        ---
config:
  flowchart:
    rankSpacing: 20

---
flowchart TD

    A[Model performs poorly]
    C{Can it fit a tiny batch?}
    D[Fix pipeline or optimization]
    E[Train on full training set]
    F[Adjust optimizer, learning rate, or number of epochs]
    G[Proceed to next stage]

    A --> C
    C -- No --> D
    C -- Yes --> E
    E -- Not converged --> F
    E -- Converged --> G

    H[Once training has stabilized]
    I{Training strong?}
    J[Underfitting]
    K{Validation strong?}
    L["<div style='text-align:center'>Overfitting or<br/> distribution mismatch</div>"]
    M[No obvious problem]

    H --> I
    I -- No --> J
    I -- Yes --> K
    K -- No --> L
    K -- Yes --> M

Training Behaviour#

Compare training and validation measurements across epochs before deciding what to change.

Training behaviour

Validation behaviour

Initial interpretation

Little or no improvement

Little or no improvement

Pipeline or optimization problem

Stabilizes at a weak level

Also weak

Underfitting or inadequate setup

Becomes very strong

Stagnates or deteriorates

Overfitting or distribution mismatch

Strong and stable

Strong and stable

Promising generalization

Unexpectedly weak

Stronger than expected

Check for data leakage or implementation error

These patterns suggest where to investigate first. They do not prove a diagnosis.

Optimization Failure#

When the model does not fit the training data, first establish that the pipeline and optimization procedure are working. Common signs of a pipeline or optimization problem include:

  • training loss is still improving substantially when the run ends;

  • training loss barely changes;

  • training loss oscillates strongly or becomes non-finite;

  • expected gradients are missing or non-finite;

  • model parameters do not update as expected;

  • a tiny-batch test fails.

Possible causes include an incorrect data or loss contract, optimizer misconfiguration, unsuitable learning rate, or insufficient training budget. Do not diagnose underfitting while these explanations remain plausible.

Underfitting#

A model underfits when training performance remains inadequate after the pipeline and optimization appear to work and training has stabilized. At that point, plausible causes include:

  • insufficient model capacity;

  • an inadequate input representation;

  • excessive regularization.

Increasing capacity is a reasonable experiment only after simpler optimization checks succeed.

Important

Poor training fit is not by itself evidence of underfitting. A model that has not trained successfully or has not yet converged should first be treated as an optimization problem.

Overfitting#

A typical sign of overfitting is that training performance continues to improve while validation performance stops improving or begins to deteriorate. The diagnosis should be based on a sustained divergence between the two, rather than on a single worse validation epoch.

Distribution Mismatch#

Weak validation performance may reflect a difference between the training and validation distributions rather than conventional overfitting. When a mismatch is plausible, inspect examples from both partitions, compare their sources and acquisition conditions, check the splitting procedure, and verify that the training data reflects the intended deployment conditions.

Training Checks#

If the model does not fit the training data, first verify that the pipeline and optimization procedure are working. Start with inexpensive checks and progress to more expensive ones until the problem is isolated.

Inexpensive Tests#

  • Inspect a Batch. Verify that input shapes match the model, targets match the loss function, inputs and targets contain the same number of examples, and labels correspond to the inspected inputs.

  • Check Finite Values. Verify that the loss is finite. Non-finite values should be investigated before interpreting learning curves. Check input tensors when numerical problems are suspected.

  • Inspect Gradients. Verify that expected gradients exist and are finite after loss.backward() and before clearing them. See Training for how to inspect gradients.

  • Confirm Parameter Updates. Store an independent copy of the parameters before one update. After loss.backward() and optimizer.step(), inspect which trainable parameters changed.

Parameters expected to participate in learning should change over repeated updates. If gradients exist but the parameters remain unchanged, check the optimizer configuration, learning rate, and update sequence.

Tiny Batch#

Train repeatedly on a very small fixed collection of examples. Success indicates that the model, loss, gradients, optimizer, and data pipeline can cooperate to fit at least a small set of examples. Failure indicates a pipeline, optimization, or capacity problem.

Warning

Fitting a tiny batch does not demonstrate generalization. It is a pipeline and optimization check.

Learning Rate#

Compare several learning rates while keeping the remaining conditions fixed. Typical values to test include:

learning_rates = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1]

The following table summarizes the expected behaviour of the training loss for each case.

Behaviour

Possible interpretation

Loss decreases extremely slowly

Learning rate may be too small

Loss decreases steadily

Learning rate may be suitable

Loss oscillates, increases, or becomes non-finite

Learning rate may be too large

Keep the data split, preprocessing, model architecture, initialization policy, optimizer type, training budget, and evaluation procedure fixed.

Training Budget#

An insufficient training budget is an optimization issue, not evidence of underfitting. If training loss is still improving when the run ends, the model has not yet demonstrated that it cannot fit the training data. Before changing the architecture, check whether additional update steps allow training performance to improve under otherwise unchanged conditions. Only diagnose underfitting after training has stabilized and the pipeline and optimization procedure are credible. Training duration is itself an experimental variable.

Designing Experiments#

Controlled Experiment#

A controlled experiment changes one planned factor while keeping the remaining conditions as stable as reasonably possible. The following table summarizes the components of a controlled experiment.

Component

Question

Hypothesis

What do you expect to happen, and why?

Independent variable

What factor are you deliberately changing?

Outcome

What predefined measurement will compare the conditions?

Controlled conditions

What relevant factors must remain fixed?

For example, here is an experiment to test whether weight decay improves generalization.

  • Hypothesis: Moderate weight decay will slightly reduce training fit but improve validation loss.

  • Independent variable: Weight-decay value.

  • Outcome: Best validation loss.

  • Controlled conditions: Data, preprocessing, model, optimizer, learning rate, batch size, training budget.

An experiment does not need to improve performance to be informative.

Baseline#

A baseline provides a reference point against which a proposed change can be judged. It may be a majority-class predictor, a linear classifier, a smaller neural network, the current best validated system, or the same architecture without the proposed change. The baseline and candidate should be evaluated using the same data split and evaluation procedure so that the comparison is meaningful.

Fixing Seeds#

Fix the sources of randomness used by the experiment when reproducibility matters.

import random
import numpy as np
import torch

seed = 42

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)

if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

Use the same seed policy when comparing experimental conditions.

Repeated Runs#

When differences are small enough that random initialization or batch ordering could matter, repeat the comparison across several seeds. Record the outcome for each run. Then, aggregate the results by reporting the mean and standard deviation of the outcome metric.

mean validation accuracy: 0.842
standard deviation:       0.006

The standard deviation across repeated runs provides a rough estimate of the expected variation due to random factors. If two conditions differ by only a small amount compared with this run-to-run variation, more evidence may be needed before concluding that one condition is better.