Model Development

Model Development#

(Click the button in the top right corner to download the lab.)

A training workflow can produce loss curves, accuracies, and checkpoints. Those measurements become useful only when we can interpret them. A weak validation result does not identify its own cause. The model may be unable to optimize, unable to represent the task, too specialized to the training examples, or evaluated on data drawn from different conditions.

In this lab, you will reuse the training workflow developed earlier and treat model development as a sequence of experiments. You will begin with a baseline, construct several distinct failure modes, and test one intervention under controlled conditions. The goal is not to discover the best possible FashionMNIST model. The goal is to learn how to make a claim such as “this model underfits” or “weight decay improved generalization” only when the available evidence supports it.

Throughout the lab, keep four concepts separate.

Capacity: What relationships the model can represent.
Optimization: Whether the training procedure can find useful parameters.
Training fit: How well the resulting model performs on training examples.
Generalization: How well does the model perform on never-seen examples.

NOTE: Use these tutorials as focused references when you need to review an API or operation.

Tutorial	Relevance to this lab
MLP — Quick Recap	Pipeline checklist, common symptoms, and end-to-end review
MLP — Training a Neural Network	Loss, optimizer configuration, and parameter updates
MLP — Evaluating a Neural Network	Held-out measurements and evaluation procedure
MLP — Dataset & DataLoader	Reproducible partitions, batching, and stable evaluation loaders
MLP — Building a Neural Network	Configurable hidden width and model outputs
PyTorch — Automatic Differentiation	Gradient inspection and the tiny-batch diagnostic

1. Experimental Setup#

The experiments in this lab use FashionMNIST. By default, we will use the complete official training portion of the dataset, dividing it into separate training and validation partitions. The training partition will update the model parameters, while the validation partition will guide checkpoint selection and experimental comparisons. The official test set will remain untouched until the final configuration has been selected.

Running every experiment on the complete training partition provides the most representative results, but it also increases the execution time considerably. The setup will therefore include an optional limit on the number of training examples. When this limit is enabled, the lab will select a reproducible subset from the training partition while preserving the roles of the validation and test sets. This reduced mode is intended for slower hardware, classroom demonstrations, or preliminary checks.

Later in the lab, one experiment will deliberately use a much smaller training subset to create conditions in which overfitting is easier to observe. That reduction serves a diagnostic purpose and should not be confused with the optional MAX_TRAINING_EXAMPLES setting.

1.1 Imports#

Run the following cell to import the libraries used in this lab.

Random initialization and data shuffling affect neural-network training. A fixed seed makes one comparison reproducible, but it does not eliminate randomness as a scientific concern. Near the end of the lab, you will repeat one comparison with several seeds.

SEED = 7

set_seed(SEED)

if torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

print("PyTorch version:", torch.__version__)
print("Selected device:", device)

1.2 Data partitions#

FashionMNIST contains 28 × 28 grayscale images from ten clothing categories. The transform pipeline composed of ToImage and ToDtype converts each image into a floating-point tensor with shape (1, 28, 28) and values between zero and one. The leading dimension represents the single grayscale channel.

No fitted preprocessing is used in this lab, so there are no statistics to estimate from the data. If we later introduced standardization or another fitted transformation, it would have to be fitted using training data only.

The official FashionMNIST test set is loaded so that the data source is complete, but no test loader will be created until the final section.

# The complete FashionMNIST development data contains 60,000 examples.
# We reserve 6,000 for validation and use the remaining 54,000 for training.
VALIDATION_SIZE = 6_000

# Use None to keep the complete training partition.
# Set an integer for a faster reduced run.
MAX_TRAINING_EXAMPLES = 10_000

# This smaller subset is used only in the deliberate overfitting experiment.
OVERFIT_TRAINING_SIZE = 512

The deliberately reduced overfitting subset is derived from the selected training partition.

Report the active configuration clearly.

Checkpoint#

The following assertions encode the partitioning assumptions on which every later result depends.

if MAX_TRAINING_EXAMPLES is None:
    expected_train_size = len(full_training_dataset) - VALIDATION_SIZE
else:
    expected_train_size = MAX_TRAINING_EXAMPLES

# TODO: verify the expected partition sizes
# TODO: verify that the training and validation index sets are disjoint
# TODO: verify that the reduced overfitting set is contained in training_indices
# TODO: verify that one image has shape (1, 28, 28)
# TODO: verify that one target is an integer class index in [0, 10)

Quiz

Suppose the images were standardized using a mean and standard deviation estimated from data. From which partition should those statistics be calculated?

Answer

They should be calculated from the training set and then applied unchanged to validation and test images. Estimating them from all partitions would allow held-out information to enter the fitted pipeline.

Inspect a few training and validation examples. This is a cheap check for corrupted data, incompatible shapes, and obviously incorrect labels.

1.3 Training workflow#

The functions below provide the infrastructure established in a previous lesson. You are not asked to rebuild mini-batch aggregation, evaluation mode, or checkpointing here. The new work begins when you decide what to change, what to measure, and how to interpret the resulting evidence.

For diagnosis, the workflow measures both training and validation data after each epoch using fixed parameters. This produces directly comparable curves. The update loop still processes mini-batches normally, but its changing-model loss is not used as the diagnostic training curve.

1.4 Model architecture#

The experiments in this lab use a small multilayer perceptron for FashionMNIST classification. Each 28 × 28 grayscale image is flattened into 784 input values, projected into a hidden representation, passed through a ReLU activation, and mapped to ten output logits (one per class). The final layer returns raw logits rather than probabilities. nn.CrossEntropyLoss expects logits and applies softmax internally.

The model is configurable because later experiments will investigate two factors:

hidden_width controls the capacity of the hidden representation;
dropout_probability optionally applies dropout between the hidden activation and the output layer.

All other architectural choices remain fixed unless an experiment explicitly states otherwise. This allows us to change one planned factor while keeping the rest of the model comparable.

class FashionMLP(nn.Module):

    def __init__(self, hidden_width: int = 128, dropout_probability: float = 0.0):
        super().__init__()
        layers = [
            nn.Flatten(),
            nn.Linear(28 * 28, hidden_width),
            nn.ReLU(),
            nn.Linear(hidden_width, 10)
        ]
        
        if dropout_probability > 0:
            layers.insert(-1, nn.Dropout(dropout_probability))

        self.network = nn.Sequential(*layers)

    def forward(self, images):
        return self.network(images)

1.5 Experiment configuration#

The class ExperimentConfig makes the independent variables and controlled conditions explicit. Each experiment creates a fresh model, optimizer, and shuffled loader from a known seed. The best model state is selected by validation loss and restored before the result is returned.

BATCH_SIZE = 128

@dataclass(frozen=True)
class ExperimentConfig:
    name: str
    hidden_width: int = 128
    learning_rate: float = 1e-3
    weight_decay: float = 0.0
    dropout_probability: float = 0.0
    epochs: int = 8
    seed: int = SEED
    batch_size: int = BATCH_SIZE

Show code cell source

Hide code cell source

def run_experiment(config:          ExperimentConfig, 
                   training_data:   Dataset, 
                   validation_data: Dataset, 
                   verbose:         int=2):
    set_seed(config.seed)

    update_loader  = make_loader(training_data, True, config.seed, config.batch_size)
    monitor_loader = make_loader(training_data, False,config.seed, config.batch_size)
    validation_loader = make_loader(validation_data, False, config.seed, config.batch_size)

    model = FashionMLP(config.hidden_width, config.dropout_probability).to(device)
    optimizer = torch.optim.Adam(model.parameters(), config.learning_rate, weight_decay=config.weight_decay)
    loss_fn = nn.CrossEntropyLoss()

    history = {
        "train_loss": [],
        "train_accuracy": [],
        "valid_loss": [],
        "valid_accuracy": [],
    }

    initial_train_metrics = evaluate(model, monitor_loader, loss_fn, device)
    initial_valid_metrics = evaluate(model, validation_loader, loss_fn, device)
    history["train_loss"].append(initial_train_metrics["loss"])
    history["train_accuracy"].append(initial_train_metrics["accuracy"])
    history["valid_loss"].append(initial_valid_metrics["loss"])
    history["valid_accuracy"].append(initial_valid_metrics["accuracy"])

    best_state = None
    best_epoch = None
    best_valid_loss = float("inf")
    best_train_metrics = None
    best_valid_metrics = None

    for epoch in tqdm(range(1, config.epochs + 1), config.name, unit="epoch", disable=verbose==0, leave=verbose==2):

        update_result = train_epoch(model, update_loader, loss_fn, optimizer, device)
        train_metrics = evaluate(model, monitor_loader, loss_fn, device)
        valid_metrics = evaluate(model, validation_loader, loss_fn, device)

        history["train_loss"].append(train_metrics["loss"])
        history["train_accuracy"].append(train_metrics["accuracy"])
        history["valid_loss"].append(valid_metrics["loss"])
        history["valid_accuracy"].append(valid_metrics["accuracy"])

        if valid_metrics["loss"] < best_valid_loss:
            best_valid_loss = valid_metrics["loss"]
            best_state = copy.deepcopy(model.state_dict())
            best_epoch = epoch
            best_train_metrics = train_metrics.copy()
            best_valid_metrics = valid_metrics.copy()

        assert update_result["n_examples"] == len(training_data)
        assert update_result["n_updates"] == len(update_loader)

    model.load_state_dict(best_state)

    return {
        "config": config,
        "model": model,
        "history": history,
        "checkpoint": {
            "epoch": best_epoch,
            "valid_loss": best_valid_loss,
            "train_loss": best_train_metrics["loss"],
            "train_accuracy": best_train_metrics["accuracy"],
            "valid_accuracy": best_valid_metrics["accuracy"],
        },
        "parameter_count": count_parameters(model),
        "training_size": len(training_data),
        "validation_size": len(validation_data),
    }

The plotting and summary helpers keep later cells focused on experimental reasoning rather than repeated display code.

Checkpoint#

Verify the workflow contract.

sample_config = ExperimentConfig(name="contract check", epochs=1)

# TODO: assert that the default learning rate, width, and weight decay
#       match the intended baseline configuration

sample_model = FashionMLP(hidden_width=128).to(device)
sample_images, sample_targets = next(
    iter(make_loader(training_dataset, shuffle=False, seed=SEED))
)
sample_logits = sample_model(sample_images.to(device))

# TODO: assert that sample_logits has shape (batch_size, 10)
# TODO: assert that every model parameter is on device

2. Establish a Baseline#

An experiment needs a reference. Without one, a result such as 80% validation accuracy has no context. We will use two baselines with different purposes:

a majority-class rule, which ignores the images and always predicts the most common training label;
a small MLP, which becomes the current validated neural-network system against which later interventions can be compared.

The majority-class rule answers whether the neural network has learned anything beyond class frequency. The MLP baseline answers whether a proposed modification improves the current system under the same evaluation procedure.

2.1 Majority-class baseline#

The majority class must be determined exclusively from training targets. Validation labels are used only to evaluate that already defined rule.

Using validation targets to choose the predicted class would allow validation information to influence the system being evaluated.

Exercise 1#

Implement the trivial baseline. Count the number of training examples in each class, identify the majority class, and calculate the validation accuracy when every prediction is that majority class.

training_targets = torch.tensor([full_training_dataset.targets[index].item() for index in training_indices])
validation_targets = torch.tensor([full_training_dataset.targets[index].item() for index in validation_indices])

# TODO: identify the most common class using training_targets only
class_counts = None # YOUR CODE HERE
majority_class = None # YOUR CODE HERE

# TODO: calculate validation accuracy when every prediction is majority_class
majority_validation_accuracy = None # YOUR CODE HERE

print("Majority class:", class_names[majority_class])
print(f"Majority validation accuracy: {majority_validation_accuracy:.2%}")

2.2 Neural-network baseline#

The baseline configuration uses 128 hidden units, Adam with learning rate 1e-3, no weight decay, no dropout, and eight epochs. These choices are not claimed to be optimal. They establish a reproducible point of comparison.

Exercise 2#

Run the training process for the neural-network baseline.

baseline_config = ExperimentConfig(
    name="baseline",
    hidden_width=128,
    learning_rate=1e-3,
    weight_decay=0.0,
    dropout_probability=0.0,
    epochs=8,
)

# TODO: run the baseline on training_dataset and validation_dataset
baseline_result = None # YOUR CODE HERE

summarize_results([baseline_result])

Plot both training and validation histories for the same run. A diagnostic claim should consider their evolution, not only the selected checkpoint.

Analysis#

Offer a provisional interpretation.

Is optimization progressing?
Is training fit improving?
Is there evidence of a sustained generalization gap?

3. Diagnose Optimization#

A model with weak training performance may lack capacity, but it may also be using an unsuitable optimization procedure. Increasing the model size before verifying optimization would confuse these explanations. The learning-rate experiment changes one factor while holding the data split, architecture, optimizer type, seed, batch size, and training budget fixed. It therefore asks a focused question:

Can learning-rate choice explain a large difference in training fit?

3.1 Compare learning rates#

For each learning rate in the table below, predict the likely training-loss behaviour over four epochs. Describe the expected curve rather than merely labelling the value “good” or “bad.”

Learning rate	Prediction
`1e-5`
`1e-3`
`1e-1`

Exercise 3#

Complete the loop so that learning rate is the only intentional difference between the three configurations.

learning_rates = [1e-5, 1e-3, 1e-1]
learning_rate_results = []

for learning_rate in learning_rates:
    config = ExperimentConfig(
        name=f"lr={learning_rate:g}",
        hidden_width=128,
        learning_rate=learning_rate,
        weight_decay=0.0,
        dropout_probability=0.0,
        epochs=4,
    )

    # TODO: run the experiment and append its result
    result = None # YOUR CODE HERE
    learning_rate_results.append(result)

summarize_results(learning_rate_results)

plot_histories(learning_rate_results, metric="train_loss", title="Training loss under different learning rates")

Checkpoint#

Confirm the controlled conditions.

configs = [result["config"] for result in learning_rate_results]

# TODO: assert that every run used the same hidden width
# TODO: assert that every run used the same weight decay and dropout
# TODO: assert that every run used the same seed and epoch budget

Analysis#

Identify the run with the weakest training fit and explain why that result does not establish insufficient model capacity. Compare it with the same architecture trained using another learning rate. What does the comparison reveal about the role of optimization?

3.2 Fit a Tiny Batch#

A tiny-batch test asks whether the complete learning pipeline can fit a small, fixed collection of examples. It is stronger than checking one gradient or one update because it tests whether repeated updates can drive the loss down and the accuracy up. Success does not establish generalization. It establishes that the forward computation, loss, gradients, optimizer, and model capacity can cooperate on those examples.

tiny_training_dataset = Subset(training_dataset, list(range(32)))

tiny_loader = make_loader(tiny_training_dataset, False, batch_size=32)

tiny_images, tiny_targets = next(iter(tiny_loader))

print("Tiny image batch:", tiny_images.shape)
print("Tiny target batch:", tiny_targets.shape)
print("Class distribution:", np.bincount(tiny_targets, minlength=len(class_names)).tolist())

Exercise 4#

Complete the repeated update. Use the same 32 examples at every step and record the loss and accuracy before the optimizer changes the model again.

set_seed(SEED)

tiny_model = FashionMLP(hidden_width=128).to(device)
tiny_loss_fn = nn.CrossEntropyLoss()
tiny_optimizer = torch.optim.Adam(tiny_model.parameters(), lr=1e-2)

tiny_losses = []
tiny_accuracies = []

images = tiny_images.to(device)
targets = tiny_targets.to(device)

for step in range(250):
    tiny_model.train()

    # TODO: clear gradients
    # TODO: calculate logits and loss
    # TODO: backpropagate and update parameters
    # TODO: calculate batch accuracy

    tiny_losses.append(loss.item())
    tiny_accuracies.append(accuracy)

print(f"Final tiny-batch loss: {tiny_losses[-1]:.6f}")
print(f"Final tiny-batch accuracy: {tiny_accuracies[-1]:.3f}")

Checkpoint#

Verify that the pipeline can fit.

# TODO: assert that every recorded loss and accuracy is finite
# TODO: assert that the final loss is substantially lower than the initial loss
# TODO: assert that the final accuracy is at least 0.95

Quiz

The model reaches 100% accuracy on the tiny batch. Which conclusion is justified?

Answer

The model and optimization pipeline can fit those 32 examples. The result does not show that the model will generalize to other FashionMNIST images, nor that the model has sufficient capacity for the task.

4. Diagnose Underfitting#

The tiny-batch test has provided evidence that the pipeline can learn. We can now investigate capacity more credibly.

A model underfits when it cannot achieve adequate performance even on the training data after optimization has been given a reasonable opportunity to work. We will compare two models that differ only in hidden width. The width-four model has a severe representational bottleneck; the width-128 model matches the baseline architecture.

4.1 Compare hidden widths#

Predict how reducing hidden width from 128 to 4 should affect training loss, training accuracy, and validation accuracy. Which measurement is most important when diagnosing underfitting?

Exercise 5#

Run the capacity experiment.

widths = [4, 128]
capacity_results = []

for width in widths:
    config = ExperimentConfig(
        name=f"width={width}",
        hidden_width=width,
        learning_rate=1e-3,
        weight_decay=0.0,
        dropout_probability=0.0,
        epochs=8,
        seed=SEED,
    )

    # TODO: run the experiment and append the result
    result = None # YOUR CODE HERE
    capacity_results.append(result)

summarize_results(capacity_results)

plot_histories(capacity_results, metric="train_loss", title="Training fit under different hidden widths")

plot_histories(capacity_results, metric="valid_loss", title="Validation loss under different hidden widths")

Analysis#

Use the complete chain of evidence:

Did the learning-rate experiment identify a setting that optimizes the width-128 architecture successfully?
Did the tiny-batch test show that the pipeline can fit a small collection of examples?
Does the width-four model stabilize at weaker training performance than the width-128 model?
Does increasing width improve training fit under otherwise comparable conditions?

Explain why this evidence makes insufficient capacity a plausible diagnosis for the narrow model. Also identify any remaining uncertainty: eight epochs do not prove that no optimizer or longer budget could improve it.

5. Diagnose Overfitting#

The next experiment deliberately changes the relationship between capacity and available training data. A width-512 model will be trained on only 512 examples while validation data remains unchanged.

This experiment is not a controlled comparison with the baseline because both width and training-set size differ. Its purpose is to create a clear behaviour for diagnosis. Later, the weight-decay experiment will hold this overfitting setup fixed and change only regularization strength.

5.1 Generalization gap#

Predict the likely relationship between training and validation curves. At what point would the evidence become more persuasive than a single worse validation epoch?

Exercise 6#

Train the high-capacity model on the deliberately reduced training set.

overfit_config = ExperimentConfig(
    name="wide model, 512 examples",
    hidden_width=512,
    learning_rate=1e-3,
    weight_decay=0.0,
    dropout_probability=0.0,
    epochs=30,
)

# TODO: run this configuration using overfit_training_dataset and the unchanged validation_dataset
overfit_result = None # YOUR CODE HERE

summarize_results([overfit_result])

Analysis#

Identify the epoch with the lowest validation loss. Then examine what happens afterward. Does training loss continue to improve? Does validation loss stagnate or increase over several epochs? Is the divergence sustained?

Explain why the restored checkpoint is preferable to the final epoch under the predefined validation-loss selection rule.

5.2 Distribution Mismatch#

A large train–validation gap can also result from distribution mismatch. To make this limitation concrete, evaluate the already trained baseline model on a darkened version of the validation images. The model parameters remain unchanged; only the input distribution changes.

Inspect matched normal and darkened examples.

Analysis#

The same checkpoint was evaluated under two validation conditions. Explain why weaker performance on the darkened images is evidence of sensitivity to a changed input distribution, not evidence that additional training caused the model to memorize its original training set.

What additional evidence would you inspect before deciding whether a real train–validation gap is caused primarily by overfitting, distribution mismatch, label problems, or leakage?

6. Test Regularization#

The overfitting run created a setting with high capacity and limited training data. We can now test whether weight decay changes generalization under those same conditions.

A useful experiment must state its logic before producing results.

Hypothesis: a proposed explanation and expected outcome.
Independent variable: the factor deliberately changed.
Outcome: the measurement used for comparison.
Controlled conditions: factors held fixed to isolate the change.

6.1 State the experiment#

Complete the experimental statement before training:

Factor	Value
Hypothesis:	Moderate weight decay will …
Independent variable:	…
Primary outcome:	…
Controlled conditions:	…

The primary outcome should be chosen in advance. Use the best validation loss, because checkpoint selection throughout the lab is based on validation loss.

6.2 Investigate weight decay#

The unregularized result already exists. Reuse it rather than rerunning the same configuration. Train the two non-zero weight-decay values with the same width, data, learning rate, epoch budget, seed, and checkpoint rule.

Exercise 7#

Run the weight-decay experiment.

decay_results = [overfit_result]

for weight_decay in [1e-4, 1e-2]:
    config = ExperimentConfig(
        name=f"weight decay={weight_decay:g}",
        hidden_width=512,
        learning_rate=1e-3,
        weight_decay=weight_decay,
        dropout_probability=0.0,
        epochs=30,
    )

    # TODO: run the experiment using overfit_training_dataset and append its result
    result = None # YOUR CODE HERE
    decay_results.append(result)

summarize_results(decay_results)

Analysis#

Compare the regularized model with the unregularized baseline. Consider the following questions:

Did weight decay improve the predefined outcome?
Did weight decay reduce training fit substantially?
Is any improvement large enough to justify further investigation?
If neither non-zero value improved validation loss, what explanations remain plausible?

A negative result is still informative when the comparison is controlled. It may indicate that the tested values were unsuitable, the baseline was not helped by this intervention, or the training budget interacted with regularization.

6.3 Investigate dropout#

Dropout is another regularization technique. It randomly sets a fraction of the hidden activations to zero during training. The dropout probability is the fraction of activations that are zeroed. Before using it, verify its mode-dependent behaviour.

Exercise 8#

Run the dropout experiment.

dropout_results = [overfit_result]

for dropout_prob in [0.5, 0.75]:
    config = ExperimentConfig(
        name=f"dropout={dropout_prob:g}",
        hidden_width=512,
        learning_rate=1e-3,
        weight_decay=0.0,
        dropout_probability=dropout_prob,
        epochs=30,
    )

    # TODO: run the experiment using overfit_training_dataset and append its result
    result = None # YOUR CODE HERE
    dropout_results.append(result)

summarize_results(dropout_results)

Analysis#

Compare the regularized model with the unregularized baseline. Consider the following questions:

Did dropout improve the predefined outcome?
Did dropout reduce training fit substantially?
Is any improvement large enough to justify further investigation?
If neither non-zero value improved validation loss, what explanations remain plausible?

7. Select a Final Model#

Controlled experiments answer focused causal questions. Final model selection asks a different question: among the candidate systems developed without test feedback, which one has the strongest predefined validation evidence?

Assemble the important candidates. The candidate list may include runs trained with different data amounts or capacities, so this table is not itself a controlled experiment. It is a model-selection table evaluated on one unchanged validation set.

candidate_results = [
    baseline_result,
    capacity_results[0],
    overfit_result,
    dropout_results[1],
]

candidate_table = summarize_results(candidate_results)
candidate_table.sort_values("valid loss")

Analysis#

State the final selection rule before using the test set. Then identify the selected candidate.

A valid statement might be the following.

Select the candidate with the highest validation accuracy.

If there is a tie, select the candidate with the lowest capacity.

Explain why selecting the lowest training loss would answer the wrong question. Also explain why the test set cannot be used to break a close validation tie.

Select the candidate programmatically using validation loss.

selected_result = max(candidate_results, key=lambda result: result["checkpoint"]["valid_accuracy"])

print("Selected experiment:", selected_result["config"].name)
print("Selected validation loss:", selected_result["checkpoint"]["valid_loss"])
print("Selected validation accuracy:", selected_result["checkpoint"]["valid_accuracy"])

8. Evaluate the Test Set Once#

The model configuration and checkpoint have now been selected entirely from training and validation evidence. Only now will the test loader be created.

The guard below prevents accidental repeated evaluation within the intended notebook workflow. It cannot stop someone from deliberately bypassing the function, but it makes the methodological rule executable and visible.

test_evaluation_count = 0


def evaluate_test_once(model):
    global test_evaluation_count

    if test_evaluation_count != 0:
        raise RuntimeError("The test set has already been evaluated in this notebook.")

    test_loader = make_loader(test_dataset, False)

    result = evaluate(model, test_loader, nn.CrossEntropyLoss(), device)

    test_evaluation_count += 1
    return result

Exercise 9#

Final test evaluation.

# TODO: evaluate selected_result["model"] exactly once

final_test_result = None # YOUR CODE HERE

print("Selected experiment:", selected_result["config"].name)
print(f"Validation loss: {selected_result['checkpoint']['valid_loss']:.4f}")
print(f"Validation accuracy: {selected_result['checkpoint']['valid_accuracy']:.3f}")
print(f"Test loss: {final_test_result['loss']:.4f}")
print(f"Test accuracy: {final_test_result['accuracy']:.3f}")
print("Test evaluations performed:", test_evaluation_count)

Reflection#

Suppose the test result is lower than expected.

Explain why changing the model and evaluating on the test set again would compromise its original role.
What would be required to obtain a new independent final estimate after further development?

Model Development

Contents

Model Development#

1. Experimental Setup#

1.1 Imports#

1.2 Data partitions#

Checkpoint#

1.3 Training workflow#

1.4 Model architecture#

1.5 Experiment configuration#

Checkpoint#

2. Establish a Baseline#

2.1 Majority-class baseline#

Exercise 1#

2.2 Neural-network baseline#

Exercise 2#

Analysis#

3. Diagnose Optimization#

3.1 Compare learning rates#

Exercise 3#

Checkpoint#

Analysis#

3.2 Fit a Tiny Batch#

Exercise 4#

Checkpoint#

4. Diagnose Underfitting#

4.1 Compare hidden widths#

Exercise 5#

Analysis#

5. Diagnose Overfitting#

5.1 Generalization gap#

Exercise 6#

Analysis#

5.2 Distribution Mismatch#

Analysis#

6. Test Regularization#

6.1 State the experiment#

6.2 Investigate weight decay#

Exercise 7#

Analysis#

6.3 Investigate dropout#

Exercise 8#

Analysis#

7. Select a Final Model#

Analysis#

8. Evaluate the Test Set Once#

Exercise 9#

Reflection#