Build a Linear Classifier

Build a Linear Classifier#

(Click the button in the top right corner to download the lab.)

In this guided lab, you will build a linear classifier from scratch using the Iris dataset. Each example represents a flower with four measurements (sepal length, sepal width, petal length, and petal width) and a label identifying one of three Iris species. The aim is not to produce the most accurate flower classifier. Instead, the model is intentionally simple, making it possible to inspect every tensor, gradient, and parameter.

You will use PyTorch tensors and automatic differentiation, but you will not use nn.Module, torch.optim, or a reusable training utility. Those abstractions will be introduced after you have constructed the underlying process once by hand. By the end of the lab, the short expression optimizer.step() should no longer feel like a mysterious command: you will know which update it performs and which information it depends on.

The lab is organized into five stages. You will first prepare and verify the data, then define the model and its outputs, train the parameters manually, evaluate the resulting classifier, and finally investigate how the learning rate and gradient handling affect the training process.

Technical references

Consult the tutorials when you need a specific operation or want to review a technical concept.

Tutorial	Content
NumPy: Arrays	shapes, dimensions, reshaping
NumPy: Operations	reductions, axes, broadcasting
NumPy: Indexing	slicing, masks, advanced indexing
NumPy: Linear algebra	vector and matrix products
PyTorch: Tensors	tensor construction, reshaping, indexing, NumPy conversion
PyTorch: Automatic Differentiation	computational graphs, gradients, optimization

1. Data Preparation#

A learning system can only be interpreted if the data entering it has a clear and verified structure. In this section, you will load the flower measurements, document the meaning of their dimensions, reserve examples for final evaluation, standardize the features without leaking information from the test set, and convert the resulting arrays to PyTorch tensors.

Run the following code to import the necessary libraries into the notebook.

import random
import matplotlib.pyplot as plt
import numpy as np
import torch
import torch.nn.functional as F
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

SEED = 7

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)

1.1 Dataset inspection#

The Iris dataset contains measurements from 150 flowers. Each flower belongs to one of three species: setosa, versicolor, or virginica. The four input features are sepal length, sepal width, petal length, and petal width, all measured in centimetres.

Run the next cell to load the data as NumPy arrays, then use the reported shapes to connect the code with the problem description.

dataset = load_iris()

X = dataset.data.astype(np.float32)
y = dataset.target.astype(np.int64)

feature_names = dataset.feature_names
class_names = dataset.target_names

print("X shape:", X.shape)
print("y shape:", y.shape)
print("Features:", feature_names)
print("Classes:", class_names)

Quiz

Before continuing, explain the meaning of X.shape and y.shape. State what one row of X represents, what one column represents, and why the number of rows in X must equal the number of entries in y.

Now inspect one example. The feature vector and target should be interpreted together: the four measurements describe one flower, while the target identifies its species.

example_index = 0

print("Features:", X[example_index])
print("Target index:", y[example_index])
print("Target name:", class_names[y[example_index]])

The class index is an identifier rather than a measured quantity. Class 2 is not numerically twice class 1, and the distance between the class numbers has no meaning. The model will treat the target as the position of the correct output score, which is why the targets must use integer class indices.

Exercise 1#

Complete check_dataset so that it verifies the minimum conditions required by the model and loss used in this lab. The checks should fail early when the arrays have incompatible shapes, invalid values, or an unsuitable target dtype.

def check_dataset(features: np.ndarray, targets: np.ndarray) -> None:
    """Check the basic contract of a multiclass tabular dataset."""
    # TODO: verify that features is two-dimensional
    # TODO: verify that targets is one-dimensional
    # TODO: verify that both arrays contain the same number of examples
    # TODO: verify that all feature values are finite
    # TODO: verify that the targets use an integer dtype
    pass


check_dataset(X, y)

Continue Exercise 1 by counting the examples in each class. The result is useful both as a data check and as preparation for the majority-class baseline that you will construct later.

# TODO: compute the number of examples in each class
class_counts = ...

for class_index, count in enumerate(class_counts):
    print(f"{class_names[class_index]:10s}: {count}")

The three classes happen to be balanced in the complete dataset. This makes accuracy relatively easy to interpret, but it is not a property you should assume in future datasets. When one class dominates, a model can obtain a deceptively high accuracy by predicting that class repeatedly.

1.2 Train-test split#

A machine learning project usually starts with a single dataset. But we cannot use the same examples to both learn the parameters and evaluate the resulting model. The evaluation must be performed on examples that were not used to update the parameters, otherwise we would be measuring how well the model memorized the training data rather than how well it generalizes to unseen examples. Therefore, we will split the dataset into two sets: a training set and a test set. The training set is used to learn the parameters of the model, while the test set is used to evaluate the trained model.

Note

In this lab, we will not use a validation set. In practice, you would usually split the dataset into three sets: training, validation, and test. The validation set is used to tune hyperparameters and make experimental choices without leaking information from the test set.

The train_test_split function from sklearn.model_selection randomly shuffles the examples and splits them into two sets. The stratify argument asks the splitting function to preserve approximately the same class proportions in both sets, while random_state makes the split reproducible.

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=SEED,
    stratify=y,
)

print("Training features:", X_train.shape)
print("Training targets:", y_train.shape)
print("Test features:", X_test.shape)
print("Test targets:", y_test.shape)

The two sets contain the same number of features but different numbers of examples. From this point onward, the test set should remain untouched until the final evaluation. Repeatedly checking test performance while changing the model would gradually turn the test set into part of the development process, so it would no longer provide an honest estimate of performance on unseen data.

Checkpoint

Print the class counts in the training and test sets. Compare proportions, not raw counts: the training set contains more examples, but both sets should preserve approximately the same class distribution as the complete dataset.

# TODO: print the class counts for y_train and y_test

1.3 Feature standardization#

The four measurements in the dataset use the same physical unit, but they have different means and ranges. A feature with a larger numerical scale can dominate the linear combination even when it is not more informative. We will therefore standardize each feature using statistics calculated from the training set:

\[ X_{\text{standardized}} = \frac{X-\mu}{\sigma}. \]

Here, \(\mu\) and \(\sigma\) contain one value per feature. Their expected shape is (4,) or (1, 4), which allows NumPy to broadcast the same training statistics across all rows.

Exercise 2#

Complete the following code. Compute one mean and standard deviation per feature from X_train, protect the calculation from a feature whose standard deviation is zero or extremely small, and apply the resulting training statistics to both the training and test sets. Do not calculate statistics from the test data.

# TODO: compute one mean per feature from X_train
feature_mean = ...

# TODO: compute one standard deviation per feature from X_train
feature_std = ...

# Replace unsafe standard deviations with 1.0.
safe_feature_std = np.where(feature_std < 1e-8, 1.0, feature_std)

# TODO: standardize the training and test features using the training statistics
X_train_standardized = ...
X_test_standardized = ...

print("Mean shape:", feature_mean.shape)
print("Standard deviation shape:", safe_feature_std.shape)

Important

The test set must use the training mean and standard deviation. Calculating new statistics from the test set is a form of data leakage and would invalidate the evaluation.

Verify the transformation numerically. The standardized training features should have means close to zero and standard deviations close to one. The test features will not satisfy those properties exactly because their values did not determine the preprocessing statistics.

print("Training feature means:")
print(X_train_standardized.mean(axis=0))

print("\nTraining feature standard deviations:")
print(X_train_standardized.std(axis=0))

Floating-point arithmetic has limited precision, so the standardized values will not be mathematically exact. They should nevertheless be close to the expected values. Add explicit assertions with np.allclose so that this important property is checked automatically rather than judged only by visual inspection.

# TODO: assert that the standardized training means are close to zero
# TODO: assert that the standardized training standard deviations are close to one

Quiz

Explain why the mean and standard deviation were calculated with axis=0 rather than axis=1. Refer to the meaning of rows and columns in your explanation. What would an axis=1 calculation standardize instead?

1.4 Conversion to PyTorch#

The model and training loop will use PyTorch. Convert the standardized feature arrays and class labels to tensors with suitable dtypes. This conversion marks the boundary between data preparation in NumPy and differentiable model computation in PyTorch.

X_train_t = torch.from_numpy(X_train_standardized)
X_test_t = torch.from_numpy(X_test_standardized)

y_train_t = torch.from_numpy(y_train)
y_test_t = torch.from_numpy(y_test)

print(X_train_t.shape, X_train_t.dtype)
print(y_train_t.shape, y_train_t.dtype)

Checkpoint

Cross-entropy expects floating-point logits and integer class indices. Add assertions showing that the feature tensors use torch.float32 and the targets use torch.int64. A shape can be correct while a dtype is still incompatible with the next operation.

# TODO: add assertions for the four tensor dtypes

The conversion with torch.from_numpy may share memory with the original NumPy array. Shared storage is useful because it avoids an unnecessary copy, but it also means that modifying one object may affect the other. In this lab, the NumPy arrays will remain unchanged after conversion, so the shared storage does not create a problem.

1.5 Baseline#

Before defining the trainable model, it is useful to establish a simple result that the model should improve upon. The majority-class baseline always predicts the class that appears most often in the training targets. It ignores the flower measurements entirely, so it provides a minimal reference rather than a competitive model.

Note

The baseline class must be chosen from the training data. At this stage, calculate its accuracy on the training set only. The test-set baseline will be calculated later, alongside the final model, so that the test labels remain reserved for the final evaluation.

Exercise 3#

Find the most frequent class in the training targets, create one prediction for every training example, and calculate the resulting training accuracy. Record the value because the trained classifier should eventually provide a meaningful improvement over this reference.

# TODO: find the most frequent class in y_train_t
majority_class = ...

# TODO: create one baseline prediction for every training example
baseline_predictions = ...

# TODO: calculate the training baseline accuracy
baseline_accuracy = ...

print(f"Training majority-class baseline: {baseline_accuracy:.3f}")

Because the dataset is balanced, the training baseline should be close to one third. The exact value depends on the split. A later dataset may have a much stronger majority baseline, which is one reason raw accuracy should never be interpreted without context.

2. Model Definition#

The data is now ready to enter a model. In this section, you will define the trainable parameters, use them to produce one class score per flower, convert those scores into predictions, and connect the outputs to both a loss function and an accuracy metric.

2.1 Model parameters#

The linear classifier uses the computation

\[ Z = XW + b. \]

The number of input features determines the first dimension of the weight matrix, while the number of classes determines the number of output scores and therefore the second weight dimension.

Quiz

Complete the shape table before running the next code cell. Use symbolic reasoning first: let n_train be the number of training examples, n_features the number of measurements, and n_classes the number of possible species.

Tensor	Meaning	Shape
`X_train_t`	Training examples	`(n_train, n_features)`
`weights`	Feature-to-class weights	?
`bias`	One offset per class	?
`logits`	One score per example and class	?

Derive the dimensions from the tensors rather than writing them as fixed constants. This makes the implementation reusable when the dataset contains a different number of features, examples, or classes.

n_features = X_train_t.shape[1]
n_classes = int(y_train_t.max().item()) + 1

print("Number of features:", n_features)
print("Number of classes:", n_classes)

Initialize the weights with small random values and the bias with zeros. The call to requires_grad_() or the argument requires_grad=True tells PyTorch that these tensors are trainable parameters whose gradients must be calculated. The seed ensures that repeated runs begin from the same initial values, which will become important when comparing learning rates.

generator = torch.Generator().manual_seed(SEED)

weights = 0.01 * torch.randn(n_features, n_classes, generator=generator)
weights.requires_grad_()

bias = torch.zeros(n_classes, requires_grad=True)

print("Weight shape:", weights.shape)
print("Bias shape:", bias.shape)

Checkpoint

Add assertions for the weight and bias shapes. Explain how each dimension follows from the model equation rather than from the particular values printed by the notebook.

# TODO: assert the expected shape of weights
# TODO: assert the expected shape of bias

2.2 Forward pass#

The forward pass applies the current parameters to a batch of examples. Its output contains one unrestricted class score, or logit, for every example and every class. Training will use the logits directly to calculate cross-entropy; predictions are produced separately by selecting the largest score in each row.

Exercise 4#

Implement linear_forward using the model equation XW + b. The function must process the complete batch with tensor operations and must not loop over individual examples.

def linear_forward(features: torch.Tensor, weights: torch.Tensor, bias: torch.Tensor) -> torch.Tensor:
    """Return one class logit per example and class."""
    # TODO: implement XW + b
    pass

Apply the function to the complete training set. At this point the parameters are still random, so the values are not expected to be useful. The immediate goal is to verify that the computation produces the correct structure.

train_logits = linear_forward(X_train_t, weights, bias)

print("Logit shape:", train_logits.shape)
print("First row of logits:", train_logits[0])

Checkpoint

Explain why the output shape is (n_train, n_classes), then verify it with an assertion. Your explanation should connect one row with one flower and one column with one possible species.

# TODO: assert the expected logit shape

2.3 Class prediction#

A prediction is the index of the largest logit in one row. This decision rule is used for evaluation, but it is deliberately kept separate from the forward pass because the loss needs the complete set of logits.

Exercise 5#

Implement predict without looping over examples. The function should return one integer class index for each row of logits.

def predict(logits: torch.Tensor) -> torch.Tensor:
    """Convert class logits into integer class predictions."""
    # TODO: return the index of the largest logit in each row
    pass


initial_predictions = predict(train_logits)
print(initial_predictions[:10])

Checkpoint

Check that the prediction tensor contains one integer per training example. Verify both its shape and dtype.

# TODO: assert the expected prediction shape
# TODO: assert the expected prediction dtype

Checkpoint

Select one training example and calculate its logits separately. The result should match the corresponding row of the batched forward pass. This check confirms that batching changes efficiency, not the model applied to each example.

index = 5

single_example_logits = linear_forward(X_train_t[index], weights, bias)

print("Single-example logits:", single_example_logits)
print("Batched logits row:    ", train_logits[index])

# TODO: assert that the two results are close

Matching the single-example and batched results is stronger evidence than checking the output shape alone. Incorrect matrix operations can sometimes produce an array of the expected dimensions while mixing information across examples.

2.4 Training objective#

Use PyTorch’s built-in cross-entropy implementation to compare the logits with the correct class indices. The function accepts raw logits, so do not apply softmax first. Cross-entropy combines the necessary normalization and logarithm in a numerically stable computation.

initial_loss = F.cross_entropy(train_logits, y_train_t)

print("Initial loss:", initial_loss.item())

2.5 Classification accuracy#

Loss and accuracy answer different questions. Cross-entropy measures how strongly the model supports the correct classes, while accuracy records only whether the largest logit corresponds to the correct class.

Exercise 6#

Convert the logits to predictions, compare them with the targets, and return the proportion of correct predictions.

def accuracy(logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:
    """Return the proportion of correct predictions."""
    # TODO: convert logits to predictions
    # TODO: compare predictions with targets
    # TODO: convert the Boolean results to floating point and average them
    pass


initial_accuracy = accuracy(train_logits, y_train_t)
print("Initial training accuracy:", initial_accuracy.item())

A randomly initialized three-class model will often begin near chance accuracy, but its exact result depends on the initial parameters. More importantly, the loss can improve before accuracy changes. The model may increase the correct-class score without yet making it the largest score in the row.

Exercise 7#

Create a copy of the first logit row and increase the score corresponding to the correct class. Calculate the loss before and after the change. The second loss should be lower, which demonstrates that cross-entropy reacts to the complete score pattern rather than only to the final predicted class.

example_logits = train_logits[0].detach().clone()
example_target = y_train_t[0].reshape(1)

loss_before = F.cross_entropy(example_logits.reshape(1, -1), example_target)

# TODO: increase the correct-class logit by 2.0
modified_logits = ...

loss_after = F.cross_entropy(modified_logits.reshape(1, -1), example_target)

print("Loss before:", loss_before.item())
print("Loss after: ", loss_after.item())

# TODO: assert that loss_after is lower than loss_before

3. Training#

The model structure is complete, but its parameters still contain their initial values. Training connects the scalar loss back to those parameters, calculates how they should change, and repeats the update until the model produces more useful outputs.

3.1 Parameter gradients#

The loss is connected to the weights and bias through the forward computation. Calling backward() follows that computation in reverse and stores a gradient in each trainable leaf tensor. The gradients describe the local effect of changing each parameter; they do not update the parameters by themselves.

initial_loss.backward()

print("Weight gradient shape:", weights.grad.shape)
print("Bias gradient shape:", bias.grad.shape)
print("Weight gradient norm:", weights.grad.norm().item())
print("Bias gradient norm:", bias.grad.norm().item())

The gradients have now been calculated, but the parameters have not yet changed. Store copies of the initial weights and bias so that the update can be verified directly.

weights_before_update = weights.detach().clone()
bias_before_update = bias.detach().clone()

Checkpoint

Verify that each gradient has the same shape as its parameter and that every value is finite. A gradient tensor with an unexpected shape, missing values, or infinities indicates that the learning pipeline is not ready for an update.

# TODO: assert that weights.grad has the same shape as weights
# TODO: assert that bias.grad has the same shape as bias
# TODO: assert that both gradients contain only finite values

3.2 One parameter update#

The gradient points in the local direction of increasing loss. Gradient descent therefore changes each parameter in the opposite direction. The learning rate determines the size of that change: a value that is too small makes little progress, while a value that is too large can overshoot useful parameter values.

Exercise 8#

Choose the supplied learning rate and update the weights and bias in the direction opposite their gradients. Place the update inside torch.no_grad() so that PyTorch does not record the training procedure itself as part of the differentiable model computation.

learning_rate = 0.1

with torch.no_grad():
    # TODO: update weights
    # TODO: update bias
    pass

Checkpoint

Compare the new parameters with the copies saved before the update. Both tensors should have changed. If one did not, inspect whether it received a gradient and whether the update used the correct tensor.

print("Weights changed:", not torch.allclose(weights, weights_before_update))
print("Bias changed:", not torch.allclose(bias, bias_before_update))

Complete Exercise 8 by clearing the stored gradients. PyTorch accumulates gradients by default, so leaving them in place would cause the next call to backward() to add new values to the previous ones.

# TODO: set the stored weight gradients to zero
# TODO: set the stored bias gradients to zero

print("Weight gradient norm after clearing:", weights.grad.norm().item())
print("Bias gradient norm after clearing:", bias.grad.norm().item())

Recalculate the loss using the updated parameters. A single decrease does not prove that the full training process will succeed, but it is a useful local check that the gradient direction and update sign are consistent.

updated_logits = linear_forward(X_train_t, weights, bias)
updated_loss = F.cross_entropy(updated_logits, y_train_t)

print("Loss before update:", initial_loss.item())
print("Loss after update: ", updated_loss.item())

For a suitable learning rate, the loss should decrease after one update. If it increases, inspect the update sign and try a smaller learning rate before concluding that the gradient computation is wrong.

Quiz

In one connected explanation, state why the update occurs after backward(), why it is placed inside torch.no_grad(), and why the stored gradients are cleared afterward.

3.3 Training loop#

You now have every component required for full-batch gradient descent. The next function should initialize a fresh model, repeat the learning cycle, and record the loss and accuracy at every step. Keeping initialization inside the function ensures that later experiments can begin from identical parameters.

Exercise 9#

Complete train_linear_classifier without using nn.Module or torch.optim. Each iteration must compute logits, loss, and accuracy; backpropagate the loss; update the parameters without tracking the update; clear the gradients; and record ordinary Python numbers in the history.

def train_linear_classifier(
    features: torch.Tensor,
    targets: torch.Tensor,
    learning_rate: float,
    epochs: int,
    seed: int = SEED,
):
    """Train a multiclass linear classifier with manual gradient descent."""
    n_features = features.shape[1]
    n_classes = int(targets.max().item()) + 1

    generator = torch.Generator().manual_seed(seed)

    weights = 0.01 * torch.randn(n_features, n_classes, generator=generator)
    weights.requires_grad_()
    bias = torch.zeros(n_classes, requires_grad=True)

    history = {
        "loss": [],
        "accuracy": [],
    }

    for step in range(epochs):
        # TODO: calculate logits
        # TODO: calculate loss
        # TODO: calculate accuracy
        # TODO: backpropagate

        with torch.no_grad():
            # TODO: update the parameters
            pass

        # TODO: clear both parameter gradients

        # Store ordinary Python numbers rather than graph-connected tensors.
        history["loss"].append(...)
        history["accuracy"].append(...)

    return weights.detach(), bias.detach(), history

Be consistent about when values are recorded. The simplest convention is to store the loss and accuracy computed before each parameter update. Whatever convention you use, state it clearly so that the first and last points of the curves can be interpreted correctly.

After completing the function, train a fresh model for 500 full-batch steps.

trained_weights, trained_bias, history = train_linear_classifier(
    X_train_t,
    y_train_t,
    learning_rate=0.1,
    epochs=500,
)

print("Initial loss:", history["loss"][0])
print("Final loss:  ", history["loss"][-1])
print("Initial accuracy:", history["accuracy"][0])
print("Final accuracy:  ", history["accuracy"][-1])

Analysis

Plot the loss and accuracy histories. Describe where the loss changes rapidly, where improvement slows, and whether accuracy changes smoothly or in discrete jumps. Relate the difference between the two curves to the distinction between a continuous loss and a discrete prediction rule.

steps = np.arange(len(history["loss"]))

plt.figure(figsize=(8, 4))
plt.plot(steps, history["loss"])
plt.xlabel("Training step")
plt.ylabel("Cross-entropy loss")
plt.title("Training loss")
plt.show()

plt.figure(figsize=(8, 4))
plt.plot(steps, history["accuracy"])
plt.xlabel("Training step")
plt.ylabel("Training accuracy")
plt.title("Training accuracy")
plt.show()

A useful curve description goes beyond saying that training “worked.” Identify the initial behaviour, the region of fastest improvement, and the point at which additional steps provide diminishing returns. If either curve behaves unexpectedly, use that observation to formulate a specific diagnostic question.

4. Evaluation#

Training performance shows how well the current parameters fit the examples that produced their gradients. Evaluation on the reserved test set asks a different question: whether the learned relationship transfers to examples that did not influence preprocessing or parameter updates. Gradients are unnecessary during this stage, so the computations are placed inside torch.no_grad().

4.1 Test Performance#

The final evaluation uses the same functions as training, but it does not update the parameters. The test loss and accuracy are calculated from the final model after training is complete.

with torch.no_grad():
    train_logits = linear_forward(X_train_t, trained_weights, trained_bias)
    test_logits = linear_forward(X_test_t, trained_weights, trained_bias)

    final_train_loss = F.cross_entropy(train_logits, y_train_t)
    final_test_loss = F.cross_entropy(test_logits, y_test_t)

    final_train_accuracy = accuracy(train_logits, y_train_t)
    final_test_accuracy = accuracy(test_logits, y_test_t)

    test_baseline_predictions = torch.full_like(y_test_t, majority_class)
    baseline_test_accuracy = (test_baseline_predictions == y_test_t).to(torch.float32).mean()

print(f"Training loss:       {final_train_loss.item():.3f}")
print(f"Test loss:           {final_test_loss.item():.3f}")
print(f"Training accuracy:   {final_train_accuracy.item():.3f}")
print(f"Test accuracy:       {final_test_accuracy.item():.3f}")
print(f"Test baseline:       {baseline_test_accuracy.item():.3f}")

The trained model should perform clearly better than the majority-class baseline. Do not treat a particular accuracy value as the sole criterion for correctness. Stronger evidence comes from the combination of internally consistent shapes, finite gradients and parameters, decreasing training loss, and test performance that is meaningfully above the test baseline.

Compare the training and test results as well. If training and test performance are similar, the model has learned a relationship that generalizes to unseen examples. If the test loss is much higher than the training loss, the model has overfit the training data and will not perform well on new examples.

4.2 Incorrect predictions#

Aggregate metrics summarize performance but hide individual failures. Inspecting the incorrect examples can reveal repeated confusions, uncertain decisions, or possible limitations of the linear model.

with torch.no_grad():
    test_predictions = predict(test_logits)

incorrect_mask = test_predictions != y_test_t
incorrect_indices = torch.where(incorrect_mask)[0]

print("Number of incorrect test predictions:", len(incorrect_indices))

Analysis

Display the original measurements, true class, predicted class, and logits for every incorrect test example. Look for repeated class pairs and for cases where the two largest logits are close.

for index in incorrect_indices.tolist():
    true_class = int(y_test_t[index].item())
    predicted_class = int(test_predictions[index].item())

    print("\nExample", index)
    print("Measurements:", X_test[index])
    print("True class:   ", class_names[true_class])
    print("Prediction:   ", class_names[predicted_class])
    print("Logits:       ", test_logits[index].tolist())

Distinguish observations from explanations. Seeing several versicolor–virginica confusions is an observation. Claiming that the species overlap in the selected feature space is an explanation that would require additional evidence, such as a visualization or a comparison with a more flexible model.

5. Experimentation#

A working model is the beginning of an investigation, not the end. Controlled experiments help determine which behaviours come from the model, the optimization procedure, or an implementation mistake. In this section, you will vary one factor at a time while keeping the data and initial parameters fixed.

5.1 Investigate the learning rate#

Train the model with several learning rates and compare their training curves, while keeping the initial parameters identical through the seed argument. Do not evaluate every run on the test set: doing so would use the reserved test data to guide an experimental choice. This experiment is about training dynamics, so compare the loss, training accuracy, and numerical stability of the runs.

learning_rates = [0.001, 0.01, 0.1, 50.0]
results = {}

for rate in learning_rates:
    weights_for_rate, bias_for_rate, rate_history = train_linear_classifier(
        X_train_t,
        y_train_t,
        learning_rate=rate,
        epochs=500,
        seed=SEED,
    )

    losses = np.asarray(rate_history["loss"], dtype=np.float64)
    accuracies = np.asarray(rate_history["accuracy"], dtype=np.float64)

    results[rate] = {
        "weights": weights_for_rate,
        "bias": bias_for_rate,
        "history": rate_history,
        "all_losses_finite": bool(np.isfinite(losses).all()),
        "final_train_accuracy": float(accuracies[-1]),
    }

Plot the loss curves on the same axes. A logarithmic vertical scale can make large differences easier to inspect, especially when one learning rate diverges or improves much more slowly than the others.

plt.figure(figsize=(9, 5))

for rate, result in results.items():
    plt.plot(
        result["history"]["loss"],
        label=f"learning rate = {rate}",
    )

plt.xlabel("Training step")
plt.ylabel("Cross-entropy loss")
plt.yscale("log")
plt.title("Effect of the learning rate")
plt.legend()
plt.show()

Summarize the runs in a compact table containing the initial loss, final loss, final training accuracy, and whether all recorded losses remained finite. The table provides a concise endpoint comparison, while the curves reveal how each run reached that endpoint.

print(
    f"{'rate':>10} "
    f"{'initial loss':>15} "
    f"{'final loss':>12} "
    f"{'train acc.':>12} "
    f"{'finite':>8}"
)

for rate, result in results.items():
    losses = result["history"]["loss"]
    print(
        f"{rate:>10g} "
        f"{losses[0]:>15.4f} "
        f"{losses[-1]:>12.4f} "
        f"{result['final_train_accuracy']:>12.3f} "
        f"{str(result['all_losses_finite']):>8}"
    )

Interpret each run using the evidence. A small learning rate may move in the correct direction but fail to make enough progress within 500 steps. A larger value may improve quickly but produce an irregular curve because updates repeatedly overshoot. The final metric alone cannot reveal this behaviour.

The largest value is intentionally aggressive. Depending on the implementation and initialization, it may oscillate, produce non-finite values, or recover after unstable updates. Describe what actually happens rather than forcing the result into a predetermined label.

5.2 Diagnose a broken update#

The following loop contains a subtle but common problem. Read it before running it and predict what will happen to the stored gradients. Then, run the faulty loop, use the printed gradient norms as evidence, then write a corrected three-step version. Explain why the original program does not implement the independent gradient-descent updates intended in this lab.

broken_weights = torch.zeros(n_features, n_classes, requires_grad=True)
broken_bias = torch.zeros(n_classes, requires_grad=True)

for step in range(3):
    logits = linear_forward(X_train_t, broken_weights, broken_bias)
    loss = F.cross_entropy(logits, y_train_t)
    loss.backward()

    with torch.no_grad():
        broken_weights -= 0.1 * broken_weights.grad
        broken_bias -= 0.1 * broken_bias.grad

    print(step, broken_weights.grad.norm().item(), broken_bias.grad.norm().item())

Your repair should clear both parameter gradients after every update. In the explanation, distinguish the fact that accumulation is PyTorch’s default behaviour from the reason it is incorrect for this particular training loop.

# TODO: write the repaired three-step loop

6. Conclusion#

You should now be able to trace a complete learning system from raw data to an evaluated classifier. You prepared the data without leaking test information, defined a parameterized model, interpreted its logits, calculated a classification loss, obtained gradients with automatic differentiation, updated the parameters manually, and examined whether the learned relationship transferred to unseen examples. Later activities will preserve this learning cycle while introducing PyTorch’s neural-network abstractions.

Final Quiz — Explain the complete learning cycle#

Complete the following passage in your own words. Aim for one connected paragraph rather than a list of separate definitions. Your explanation should connect the features, parameter shapes, logits, cross-entropy loss, gradients, learning rate, parameter update, and test evaluation.

The model receives a batch of flower measurements and produces …

Then answer the following questions.

What evidence shows that the model learned from the training data?
What evidence shows that the result is more useful than the baseline?
Why does the test result provide different information from the training result?
Which operations will later be handled by nn.Module and torch.optim?
Which decisions will remain the user’s responsibility after those abstractions are introduced?

The final question is the most important. PyTorch can organize parameters, calculate gradients, and apply an optimizer update, but it cannot decide whether the data is valid, whether the output formulation matches the task, whether the evaluation is meaningful, or whether the experiment supports the conclusion.

Build a Linear Classifier

Contents

Build a Linear Classifier#

1. Data Preparation#

1.1 Dataset inspection#

Exercise 1#

1.2 Train-test split#

1.3 Feature standardization#

Exercise 2#

1.4 Conversion to PyTorch#

1.5 Baseline#

Exercise 3#

2. Model Definition#

2.1 Model parameters#

2.2 Forward pass#

Exercise 4#

2.3 Class prediction#

Exercise 5#

2.4 Training objective#

2.5 Classification accuracy#

Exercise 6#

Exercise 7#

3. Training#

3.1 Parameter gradients#

3.2 One parameter update#

Exercise 8#

3.3 Training loop#

Exercise 9#

4. Evaluation#

4.1 Test Performance#

4.2 Incorrect predictions#

5. Experimentation#

5.1 Investigate the learning rate#

5.2 Diagnose a broken update#

6. Conclusion#

Final Quiz — Explain the complete learning cycle#