Build a Neural Network

(Click the button in the top right corner to download the lab.)

In this guided lab, you will implement a small neural network in PyTorch and train it with a PyTorch optimizer. You will work with a synthetic dataset made of two concentric classes. Despite the geometry being deliberately simple, it is a useful test case to illustrate the difference between a linear model and a nonlinear neural network. The goal is not merely to obtain a higher accuracy, but to understand how the model’s structure affects its ability to represent the data.

By the end of the lab, you should be able to explain why the linear model reaches a performance ceiling, why ReLU changes what the network can represent, and how PyTorch organizes the same learning cycle you implemented manually in the previous lab.

Technical references

Consult the tutorials when you need a specific operation or want to review a technical concept.

Tutorial	Content
PyTorch: Tensors	Tensor shapes, dtypes, operations, indexing.
PyTorch: Automatic Differentiation	Computational graphs, backward(), gradients.
MLP: Building a Neural Network	nn.Module, nn.Linear, nn.ReLU, nn.Sequential.

1. Data Preparation

The first stage of any modelling workflow is to understand the data contract. Before defining a network, we need to know what one example contains, what the target represents, and how the data will be divided for training and evaluation.

Run the setup cell below. The random seeds make the data split and model initialization reproducible, which is important when comparing experiments.

import copy
import random

import matplotlib.pyplot as plt
import numpy as np
import torch
import torch.nn.functional as F
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from torch import nn

SEED = 7

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)

Dataset

The dataset is generated using the make_circles function from the sklearn.datasets module. It creates two concentric circles in a 2D space, which are then used as the two classes for the classification task. The factor argument controls the radius of the inner ring relative to the outer ring, while noise prevents the examples from lying on perfectly smooth curves.

X, y = make_circles(
    n_samples=900,
    factor=0.4,
    noise=0.08,
    random_state=SEED,
)

X = X.astype(np.float32)
y = y.astype(np.int64)

print("Feature shape:", X.shape)
print("Target shape:", y.shape)
print("Observed classes:", np.unique(y))

The feature array has shape (900, 2). Each row is one example, and the two columns are its horizontal and vertical coordinates. The target array has shape (900,) because each example has one class index. Visualize the full dataset before applying any preprocessing.

plt.figure(figsize=(6, 6))
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.xlabel("Feature 1")
plt.ylabel("Feature 2")
plt.title("Concentric classes")
plt.axis("equal")
plt.show()

We selected this dataset for its non-linear separability. A straight line can divide the plane into two half-spaces, but it cannot isolate the inner ring from the outer ring. The linear classifier will therefore face a representational limitation.

Quiz

Would training a linear classifier for a very long time eventually produce a circular decision boundary?

Answer

No. Training changes the parameters of the linear model, which moves or rotates its straight boundary. It does not change the family of boundaries the model can represent.

Data splits

We need three distinct datasets because they answer different questions. The training set determines the model parameters. The validation set helps us compare architectures and select hyperparameters. The test set provides a final estimate after those choices have been made.

We first reserve 30% of the examples, then divide that temporary set equally between validation and test data. Stratification preserves the class proportions in each split.

X_train, X_temp, y_train, y_temp = train_test_split(
    X,
    y,
    test_size=0.30,
    random_state=SEED,
    stratify=y,
)

X_valid, X_test, y_valid, y_test = train_test_split(
    X_temp,
    y_temp,
    test_size=0.50,
    random_state=SEED,
    stratify=y_temp,
)

print("Training set:", X_train.shape, y_train.shape)
print("Validation set:", X_valid.shape, y_valid.shape)
print("Test set:", X_test.shape, y_test.shape)

Checkpoint

Verify the following conditions:

• every feature array is two-dimensional;

• every target array is one-dimensional;

• every feature array has two columns;

• each feature array contains the same number of examples as its target array;

• every split contains both classes.

# TODO: add assertions for the training, validation, and test sets

Inspect the class counts as an additional sanity check.

all_targets = {
    "training": y_train,
    "validation": y_valid,
    "test": y_test,
}

for name, targets in all_targets.items():
    counts = np.bincount(targets)
    print(f"{name:10s}: {counts}")

Because the complete dataset is balanced and the split is stratified, each subset should contain similar numbers of examples from both classes. In a real project, this check could reveal whether a split accidentally removed a rare class or created a severe imbalance.

Feature standardization

Both coordinates already have similar scales, so standardization is not essential for this particular dataset. We will still apply it because the preprocessing workflow itself matters. Later datasets may contain features with very different ranges, and using a consistent procedure avoids introducing a new source of variation when models are compared.

Standardization transforms each feature using statistics calculated from the training set:

\[ X_{\text{standardized}} = \frac{X-\mu}{\sigma}. \]

The validation and test sets must reuse the training mean and standard deviation. They are evaluation data, not additional sources of fitted preprocessing information.

Exercise 1

Complete the standardization code. Use axis=0 because each column is one feature and the statistics should be calculated across examples.

# TODO: calculate one mean per feature from X_train
feature_mean = ...

# TODO: calculate one standard deviation per feature from X_train
feature_std = ...

safe_feature_std = np.where(feature_std < 1e-8, 1.0, feature_std)

# TODO: standardize all three splits using the training statistics
X_train_standardized = ...
X_validation_standardized = ...
X_test_standardized = ...

print("Training means:", X_train_standardized.mean(axis=0))
print("Training standard deviations:", X_train_standardized.std(axis=0))

A correctly standardized training set should have means close to zero and standard deviations close to one. Small deviations are expected because the arrays use finite-precision floating-point numbers.

Checkpoint

Verify the transformation numerically.

# TODO: verify that the standardized training means are close to zero
# TODO: verify that the standardized training standard deviations are close to one

Quiz

Why should the validation and test sets not calculate their own means and standard deviations?

Answer

Preprocessing is part of the fitted pipeline. Using statistics from validation or test data would allow information from those sets to influence model development and would invalidate the evaluation.

Conversion to PyTorch

The models and optimizer operate on PyTorch tensors. The features should use floating-point values, while cross-entropy expects integer class indices for the targets.

X_train_t = torch.from_numpy(X_train_standardized)
X_validation_t = torch.from_numpy(X_validation_standardized)
X_test_t = torch.from_numpy(X_test_standardized)

y_train_t = torch.from_numpy(y_train)
y_validation_t = torch.from_numpy(y_valid)
y_test_t = torch.from_numpy(y_test)

print(X_train_t.shape, X_train_t.dtype)
print(y_train_t.shape, y_train_t.dtype)

Checkpoint

Verify that the feature tensors use torch.float32, the target tensors use torch.int64, and the training feature tensor has shape (n_training_examples, 2).

# TODO: add dtype and shape assertions

These checks may seem small, but dtype and shape mismatches are among the most common reasons a training pipeline fails before the model has a chance to learn.

Baseline

A trained model should be interpreted relative to a simpler strategy. For this balanced two-class problem, always predicting the most common training class should achieve roughly 50% accuracy on validation data. The baseline does not use the input features at all. Its purpose is to define a minimum level of performance that a useful classifier should clearly exceed.

Exercise 2

Find the most frequent class in the training targets, create one prediction for every validation example, and calculate the resulting validation accuracy.

# TODO: find the most common class in y_train_t
majority_class = ...

# TODO: create one baseline prediction per validation example
baseline_predictions = ...

# TODO: calculate the validation accuracy
baseline_accuracy = ...

print(f"Baseline accuracy: {baseline_accuracy:.3f}")

When the linear and nonlinear models are evaluated, the baseline will help distinguish genuine learning from a result that could be obtained without examining the features.

2. Model Definition

We will now define two models that solve the same classification task but have different capacities. Both receive tensors with shape (batch_size, 2) and return logits with shape (batch_size, 2). The loss and prediction rule therefore remain identical. The only conceptual difference is the path from inputs to logits. The linear model applies one affine transformation. The neural network first constructs a hidden representation, applies ReLU, and then produces the logits.

Linear classifier

The linear model acts as a capacity baseline. It is only able to represent straight decision boundaries.

Exercise 3

Complete the linear model implementation. The nn.Linear module applies an affine transformation to its input. The first argument is the number of input features, and the second is the number of output features.

class LinearClassifier(nn.Module):
    
    def __init__(self):
        super().__init__()

        # TODO: define one linear layer from 2 inputs to 2 logits
        self.output = ...

    def forward(self, features):
        # TODO: return the logits
        ...

Create the model and pass a small batch through it.

torch.manual_seed(SEED)

linear_model = LinearClassifier()

sample_logits = linear_model(X_train_t[:5])

print(linear_model)
print("Sample logit shape:", sample_logits.shape)
print("First row of logits:", sample_logits[0])

The model returns two unrestricted scores for every example. The scores do not need to sum to one because F.cross_entropy will consume the raw logits directly.

Checkpoint

Verify the output contract.

# TODO: assert that the output shape is (5, 2)
# TODO: assert that the logits use a floating-point dtype

Neural network

The neural network is a nonlinear model. It has a higher capacity than the linear model because it can represent more complex decision boundaries.

Exercise 4

Complete the nonlinear model implementation. The first nn.Linear module transforms the two input features into a hidden representation with a configurable width. The second nn.Linear module transforms the hidden representation into two class logits. ReLU is applied to the hidden representation before the second linear transformation. No activation function is applied to the output because cross-entropy expects unrestricted logits.

class NonlinearClassifier(nn.Module):
    def __init__(self, hidden_width: int):
        super().__init__()

        # TODO: define the hidden linear layer
        self.hidden = ...

        # TODO: define the output linear layer
        self.output = ...

    def forward(self, features):
        # TODO: calculate the hidden pre-activations
        hidden_scores = ...

        # TODO: apply ReLU
        hidden_activations = ...

        # TODO: calculate the class logits
        logits = ...

        return logits

Create a model with eight hidden units.

torch.manual_seed(SEED)
nonlinear_model = NonlinearClassifier(hidden_width=8)

sample_logits = nonlinear_model(X_train_t[:5])

print(nonlinear_model)
print("Sample logit shape:", sample_logits.shape)
print("First row of logits:", sample_logits[0])

The batch dimension should remain unchanged. Only the feature dimension changes as the data moves from two inputs to eight hidden activations and finally to two logits.

Checkpoint

Verify the output of the neural network.

# TODO: assert that the output shape is (5, 2)
# TODO: assert that the model contains four registered parameter tensors

Quiz 3

Why should the neural network not apply ReLU or softmax after its output layer?

Answer

The output layer should provide unrestricted logits. ReLU would unnecessarily restrict them to non-negative values, while `F.cross_entropy` performs the required normalization internally and should receive raw logits.

Registered parameters

Each nn.Linear layer owns its parameters, and the parent module registers them automatically. This means that model.parameters() can provide every trainable tensor to an optimizer. The following helper reports each parameter name, shape, and number of scalar values.

def summarize_parameters(model: nn.Module) -> int:
    total = 0

    for name, parameter in model.named_parameters():
        count = parameter.numel()
        total += count
        print(f"{name:20s} {str(tuple(parameter.shape)):12s} {count:4d}")

    print(f"Total parameters: {total}")
    return total

Run the helper for both architectures.

print("Linear model")
linear_parameter_count = summarize_parameters(linear_model)

print("\nNonlinear model")
nonlinear_parameter_count = summarize_parameters(nonlinear_model)

Analysis

Calculate the expected number of parameters from the layer dimensions. Your explanation should account for both weights and biases. Then answer a more important question: why does the neural network contain more parameters, and why does parameter count alone not explain the difference in representational capacity?

Answer

The key change is not only that the model is larger; it is that ReLU prevents the complete computation from collapsing into one affine transformation.

3. Training

The models are now defined, but their parameters are still close to random. Training is the process of adjusting those parameters to reduce the loss on the training data.

Optimizer step

Before writing a complete loop, we will inspect one update in isolation. This makes it possible to verify that gradients are created and that the optimizer actually changes the parameters. Use a fresh neural network so the investigation does not alter the model reserved for later comparisons.

torch.manual_seed(SEED)

step_model = NonlinearClassifier(hidden_width=8)

step_optimizer = torch.optim.SGD(step_model.parameters(), lr=0.1)

weights_before = step_model.hidden.weight.detach().clone()

Exercise 5

Complete one training step.

step_model.train()

# TODO: clear gradients left from any previous update

# TODO: calculate logits for the complete training set
step_logits = ...

# TODO: calculate the cross-entropy loss
step_loss = ...

# TODO: calculate parameter gradients

# TODO: update the parameters

Inspect the result.

weights_after = step_model.hidden.weight.detach().clone()

print("Loss:", step_loss.item())
print("Weights changed:", not torch.allclose(weights_before, weights_after))
print("Gradient norm:", step_model.hidden.weight.grad.norm().item())

Checkpoint

Verify that the hidden weights changed, that their gradient exists, and that every gradient value is finite.

# TODO: add assertions for the update and gradient

Quiz

Why must loss.backward() occur before optimizer.step()?

Answer

The optimizer reads the gradients stored in the parameters. Those gradients are created by backpropagation, so the update cannot use the current loss until `backward()` has run.

Smoke test

A useful debugging technique is to ask whether a model can fit a very small collection of examples. This is not an evaluation of generalization. It is a connectivity test for the model, loss, gradients, and optimizer. A sufficiently flexible model should usually be able to drive the loss on a tiny batch very low. If it cannot, it is better to investigate the implementation before running a long experiment on the complete training set.

Use the first 16 training examples.

tiny_X = X_train_t[:16]
tiny_y = y_train_t[:16]

print("Tiny-batch shape:", tiny_X.shape, tiny_y.shape)

Create a fresh model and optimizer.

torch.manual_seed(SEED)

tiny_model = NonlinearClassifier(hidden_width=8)
tiny_optimizer = torch.optim.SGD(tiny_model.parameters(), lr=0.1)

Exercise 6

Train for 1,000 updates and record the loss every 20 steps.

tiny_losses = []

for step in range(1000):
    # TODO: set the model to training mode
    # TODO: clear gradients
    # TODO: calculate logits and loss
    # TODO: backpropagate
    # TODO: update the parameters

    if step % 20 == 0:
        tiny_losses.append(...)

print("Initial recorded loss:", tiny_losses[0])
print("Final recorded loss:", tiny_losses[-1])

Plot the result.

plt.figure(figsize=(7, 4))
plt.plot(np.arange(len(tiny_losses)) * 20, tiny_losses)
plt.xlabel("Training step")
plt.ylabel("Cross-entropy loss")
plt.title("Tiny-batch fitting check")
plt.show()

Analysis

Describe how the loss changed. A substantial decrease indicates that the forward pass, loss, gradients, and optimizer are connected. If the loss remains near its initial value, inspect the training sequence before proceeding. Explain why success on this tiny batch does not prove that the model will generalize to validation or test data.

Training function

The one-step update and tiny-batch experiment used the same operations repeatedly. We will now organize those operations into a reusable function that trains a model, records learning curves, monitors validation performance, and restores the parameters that achieved the lowest recorded validation loss. The function will switch between two modes. During training, gradients are enabled and parameters are updated. During validation, the model is evaluated inside torch.no_grad() because no optimization is performed.

Note

The dataset is small, so every epoch will use the complete training set as one batch. This keeps the focus on model structure and experiment design. Mini-batches and DataLoader will be introduced later when we work with larger datasets.

Exercise 7

Complete the training function. A suitable accuracy calculation is:

(logits.argmax(dim=1) == targets).float().mean()

def train_model(
    model: nn.Module,
    X_train: torch.Tensor,
    y_train: torch.Tensor,
    X_validation: torch.Tensor,
    y_validation: torch.Tensor,
    learning_rate: float = 0.1,
    n_epochs: int = 2000,
    record_every: int = 20,
):
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

    history = {
        "epoch": [],
        "training_loss": [],
        "training_accuracy": [],
        "validation_loss": [],
        "validation_accuracy": [],
    }

    best_validation_loss = float("inf")
    best_state = None

    for epoch in range(n_epochs):
        # ----- parameter update -----
        # TODO: set training mode
        # TODO: clear gradients
        # TODO: calculate training logits and loss
        # TODO: backpropagate
        # TODO: update the parameters

        if epoch % record_every == 0 or epoch == n_epochs - 1:
            # ----- measurement -----
            # TODO: set evaluation mode

            with torch.no_grad():
                # TODO: calculate training and validation logits
                # TODO: calculate both losses
                # TODO: calculate both accuracies
                pass

            # TODO: append ordinary Python numbers to history

            if validation_loss.item() < best_validation_loss:
                best_validation_loss = validation_loss.item()
                best_state = copy.deepcopy(model.state_dict())

    model.load_state_dict(best_state)
    return history

The function records metrics only every record_every epochs. This reduces the amount of stored data while preserving the overall shape of the learning curves. The best model state is selected by validation loss rather than validation accuracy because the loss retains information about the strength of the logits even when predicted classes remain unchanged.

Checkpoint

After completing the function, train a small model for a short run and verify that all history lists have the same length and contain finite values.

# TODO: perform a short smoke test and add assertions for the history

Training runs

We can now perform the central experiment. Both models will receive the same data and training budget. Resetting the seed before each construction makes initialization deterministic, although the parameter values cannot be identical because the architectures have different shapes.

Exercise 8

Train both architectures.

def create_comparison_models(hidden_width: int = 8):
    models = {}

    torch.manual_seed(SEED)
    models["linear"] = ...

    torch.manual_seed(SEED)
    models["nonlinear"] = ...

    return models


comparison_models = create_comparison_models(hidden_width=8)
comparison_histories = {}

for name, model in comparison_models.items():
    print(f"Training {name} model")

    comparison_histories[name] = train_model(
        model,
        X_train_t,
        y_train_t,
        X_validation_t,
        y_validation_t,
        learning_rate=0.1,
        n_epochs=2000,
        record_every=20,
    )

Using the same protocol makes the comparison easy to interpret. It does not prove that every possible optimizer or learning rate would behave identically, but it is sufficient to test the main hypothesis: the nonlinear architecture can represent a solution that the linear architecture cannot.

4. Evaluation

Training curves show how a model changed, while final metrics summarize where it finished. We will use both. A useful evaluation should answer not only which model performed better, but also why the difference is consistent with the architecture.

Performance metrics

The helper below evaluates a model without creating gradients. It returns both cross-entropy loss and accuracy because they capture different aspects of the predictions.

def evaluate_model(model: nn.Module, features: torch.Tensor, targets: torch.Tensor):
    model.eval()

    with torch.no_grad():
        logits = model(features)
        loss = F.cross_entropy(logits, targets)
        accuracy = (logits.argmax(dim=1) == targets).float().mean()

    return {
        "loss": loss.item(),
        "accuracy": accuracy.item(),
    }

Exercise 9

Build the comparison table.

comparison_results = {}

for name, model in comparison_models.items():
    comparison_results[name] = {
        "training": evaluate_model(model, X_train_t, y_train_t),
        "validation": evaluate_model(model, X_validation_t, y_validation_t),
    }

print(
    f"{'model':12s} "
    f"{'train loss':>12s} "
    f"{'train acc.':>12s} "
    f"{'val. loss':>12s} "
    f"{'val. acc.':>12s}"
)

for name, result in comparison_results.items():
    # TODO: print one formatted row per model
    pass

Compare the validation accuracies with the majority-class baseline. If the linear model remains near the baseline while the nonlinear model performs much better, the difference is evidence that the hidden representation and ReLU provided useful capacity.

Learning curves

Plot the training and validation losses on the same axes.

plt.figure(figsize=(8, 5))

for name, history in comparison_histories.items():
    plt.plot(history["epoch"], history["validation_loss"], label=name)

plt.xlabel("Epoch")
plt.ylabel("Validation loss")
plt.title("Validation loss by architecture")
plt.legend()
plt.show()

Analysis

Describe the behaviour of both curves. Does the linear model improve initially and then plateau? Does the nonlinear model continue to reduce validation loss?

Use the evidence to distinguish two explanations:

• Optimization failure: the model could represent a good solution, but training did not find it.

• Capacity limitation: the model trained, but its architecture cannot represent the required boundary.

Answer

The expected linear-model plateau is more consistent with a capacity limitation because the loss stabilizes while the decision boundary remains restricted to a line.

Decision boundaries

Metrics summarize performance, but this two-dimensional dataset allows us to inspect the models geometrically. The helper below evaluates a model on a dense grid and colours each location according to its predicted class. The plotting implementation is provided because the purpose of this section is model interpretation rather than coding.

def plot_decision_boundary(model: nn.Module, features: torch.Tensor, targets: torch.Tensor, title: str):
    
    x_min = float(features[:, 0].min()) - 0.5
    x_max = float(features[:, 0].max()) + 0.5
    y_min = float(features[:, 1].min()) - 0.5
    y_max = float(features[:, 1].max()) + 0.5

    grid_x, grid_y = np.meshgrid(np.linspace(x_min, x_max, 300), np.linspace(y_min, y_max, 300))

    grid = torch.tensor(np.column_stack([grid_x.ravel(), grid_y.ravel()]), dtype=torch.float32)

    model.eval()
    with torch.no_grad():
        predictions = model(grid).argmax(dim=1).numpy()

    predictions = predictions.reshape(grid_x.shape)

    plt.figure(figsize=(6, 6))
    plt.contourf(grid_x, grid_y, predictions, alpha=0.3)
    plt.scatter(features[:, 0], features[:, 1], c=targets, edgecolors="black", linewidths=0.3)
    plt.xlabel("Standardized feature 1")
    plt.ylabel("Standardized feature 2")
    plt.title(title)
    plt.axis("equal")
    plt.show()

Plot both models on the validation examples. It could also be useful to visualize the training examples, but the validation set is a better indicator of generalization.

for name, model in comparison_models.items():
    plot_decision_boundary(model, X_validation_t, y_validation_t, title=f"{name.title()} decision boundary")

Analysis

Describe the boundary produced by each model. Explain how the plots complement the loss and accuracy values.

Answer

The linear model should divide the plane into two regions with one straight transition. The MLP should create a piecewise-linear boundary that approximates the circular separation. A high validation accuracy tells us that the MLP predicts correctly on many examples; the boundary plot shows the form of the function that made those predictions possible.

5. Experimentation

The first comparison used eight hidden units. That value was a reasonable starting point, but it was still a design choice. We will now treat hidden width as an experimental variable. Changing the width changes the number of intermediate activations and the number of trainable parameters. Very narrow networks may lack enough capacity to approximate the circular boundary. Once the model is sufficiently wide, additional units may offer little practical improvement on this simple problem. The validation set will guide this choice. The test set remains untouched.

Width and capacity

Before training the models, try to predict the outcome of the experiment. Which widths are most likely to underfit? At what point do you expect additional width to stop producing a meaningful validation improvement? Do you expect accuracy to increase monotonically with parameter count?

A guess is useful even when it is wrong. It makes the comparison an investigation rather than a sequence of unexplained training runs.

Exercise 10

Launch the width experiment. Every run will use the same split, preprocessing, optimizer, learning rate, epoch budget, and random seed. Holding these conditions constant makes hidden width the main planned difference between experiments.

hidden_widths = [1, 2, 4, 8, 16, 32]
width_results = {}

for width in hidden_widths:
    torch.manual_seed(SEED)
    model = NonlinearClassifier(hidden_width=width)

    history = train_model(
        model,
        X_train_t,
        y_train_t,
        X_validation_t,
        y_validation_t,
        learning_rate=0.1,
        n_epochs=2000,
        record_every=20,
    )

    width_results[width] = {
        "model": model,
        "history": history,
        "parameters": sum(parameter.numel() for parameter in model.parameters()),
        "validation": evaluate_model(model, X_validation_t, y_validation_t),
    }

Print the results.

print(
    f"{'width':>8s} "
    f"{'parameters':>12s} "
    f"{'val. loss':>12s} "
    f"{'val. acc.':>12s}"
)

for width, result in width_results.items():
    # TODO: print one formatted row per width
    pass

Plot validation accuracy against hidden width.

width_values = list(width_results.keys())
validation_accuracies = [width_results[width]["validation"]["accuracy"] for width in width_values]

plt.figure(figsize=(7, 4))
plt.plot(width_values, validation_accuracies, marker="o")
plt.xlabel("Hidden width")
plt.ylabel("Validation accuracy")
plt.title("Effect of hidden width")
plt.show()

Analysis

Compare the results with your guesses made before the experiment. Identify which widths appear to underfit and where the validation performance reaches a plateau. Explain why a wider model can have more parameters without producing a better validation result.

Answer

Once the network has enough capacity for this task, additional units may be redundant. They enlarge the family of representable functions, but the problem may not require that extra flexibility.

Model selection

A model-selection rule should be defined before looking at the test result. We will choose the smallest hidden width that achieves the best observed validation accuracy. This prefers the simpler model when several widths perform equally well.

best_validation_accuracy = max(result["validation"]["accuracy"] for result in width_results.values())

candidate_widths = [
    width
    for width, result in width_results.items()
    if np.isclose(result["validation"]["accuracy"], best_validation_accuracy)
]

selected_width = min(candidate_widths)
selected_model = width_results[selected_width]["model"]

print("Selected hidden width:", selected_width)
print("Selected validation accuracy:", best_validation_accuracy)

Reflection

Why prefer the smaller model?

Answer

When two architectures achieve the same validation accuracy, selecting the smaller one can reduce unnecessary complexity and make the result easier to reproduce and interpret. This is not an absolute law, but it is a reasonable default when the evidence does not show a benefit from the larger model.

6. Final Evaluation

All architecture and width decisions have now been made using training and validation data. We can finally evaluate the selected model on the test set. The test result should be reported as evidence from one held-out sample, not as a universal guarantee. Its main value is that it was not used to choose the model.

Exercise 11

Evaluate both the selected model and the majority-class strategy on the test set. The majority class is still the one identified from the training targets; only its final accuracy is being measured on the held-out examples.

final_test_result = evaluate_model(selected_model, X_test_t, y_test_t)

baseline_test_predictions = torch.full_like(y_test_t, majority_class)
baseline_test_accuracy = (baseline_test_predictions == y_test_t).float().mean().item()

print(f"Test loss:              {final_test_result['loss']:.4f}")
print(f"Test accuracy:          {final_test_result['accuracy']:.3f}")
print(f"Test baseline accuracy: {baseline_test_accuracy:.3f}")

A strong result should clearly exceed the baseline and be accompanied by a boundary that matches the geometry of the problem.

plot_decision_boundary(selected_model, X_test_t, y_test_t, title=f"Selected model: hidden width {selected_width}")

Analysis

Discuss whether the test performance is consistent with the validation result. If the two differ, suggest plausible explanations without immediately concluding that the code is wrong.

Answer

A finite held-out set contains sampling variation, and some examples may lie near the noisy class boundary.

7. Conclusion

You should now be able to define a small neural network in PyTorch, inspect its registered parameters, train it with a PyTorch optimizer, and compare it fairly with a linear baseline. More importantly, you should be able to connect the geometry of a task with the capacity of a model and explain why a nonlinear activation changes what a network can represent. The next lesson will preserve the same model and optimization principles while introducing a more complete training workflow built around mini-batches, training and evaluation modes, and reusable experiment structure.