{ "cells": [ { "cell_type": "markdown", "id": "c0ae4376", "metadata": {}, "source": [ "# Build a Training Workflow\n", "\n", "*(Click the button in the top right corner to download the lab.)*\n", "\n", "A neural network learns through a short cycle: calculate predictions, measure a loss, calculate gradients, and update the parameters. In the previous labs, the complete training set was small enough to process as one tensor. That arrangement made the learning cycle easy to inspect, but it does not provide the organization required for larger datasets or longer experiments.\n", "\n", "In this lab, you will build the system around that cycle. You will use a `Dataset` to represent individual examples, a `DataLoader` to form mini-batches, separate functions for training and evaluation, and a coordinating function that records progress and retains the best model observed on validation data. The neural network itself will remain deliberately simple. The new subject is the **training workflow**.\n", "\n", "You will work with FashionMNIST, a dataset of grayscale clothing images. The original training portion will be divided into training and validation subsets, while the official test set will remain untouched until the workflow and checkpoint-selection rule have been finalized.\n", "\n", "By the end of the lab, you should be able to explain how information moves through the training workflow, and how the components of that workflow are organized.\n", "\n", ":::{admonition} Technical references\n", "Consult these tutorials when you need to review an API or a technical operation.\n", ":::\n", "\n", "| Tutorial | Relevance to this lab |\n", "|---|---|\n", "| MLP: Dataset & DataLoader | Primary reference for datasets, batching, iteration, and shuffling |\n", "| MLP: Training a Neural Network | Primary reference for loss, optimizers, and parameter updates |\n", "| MLP: Evaluating a Neural Network | Primary reference for evaluation mode and held-out metrics |\n", "| MLP: Building a Neural Network | Review of `nn.Module`, `nn.Sequential`, and model outputs |\n", "| PyTorch: GPU Support | Moving models and batches to a common device |\n", "| PyTorch: Tensors | Shapes, dtypes, indexing, and tensor operations |" ] }, { "cell_type": "markdown", "id": "fdc6e7cf", "metadata": {}, "source": [ ":::{caution}\n", "\n", "Watch out for potential confusion: the word **checkpoint** has two different meanings in this course.\n", "\n", "- A **Checkpoint** heading marks a small verification task, usually completed with assertions. \n", "- A **model checkpoint** is different: it is a saved copy of a model state retained during training.\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "5446aa3b", "metadata": {}, "source": [ "\n", "---\n", "\n", "## 1. Data Organization\n", "\n", "A training loop should not need to know where examples are stored or how they are retrieved. It should receive batches with a predictable structure and concentrate on learning. PyTorch separates these responsibilities through two abstractions:\n", "\n", "- a `Dataset` defines how many examples exist and how one example is retrieved;\n", "- a `DataLoader` retrieves those examples, groups them into batches, and controls their order.\n", "\n", "This separation allows the same training functions to operate on different datasets, provided that the resulting batches have compatible structures." ] }, { "cell_type": "markdown", "id": "3312d608", "metadata": {}, "source": [ "### 1.1 Import the libraries\n", "\n", "Run the setup cell below. The random seeds make the split and parameter initialization reproducible. Device selection is kept explicit because the model and every batch must occupy the same device during a forward pass." ] }, { "cell_type": "code", "execution_count": 1, "id": "6fd773a0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "PyTorch version: 2.13.0\n", "Selected device: cpu\n" ] } ], "source": [ "import copy\n", "import math\n", "import random\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import torch\n", "from torch import nn\n", "from torch.utils.data import DataLoader, random_split\n", "from torchvision import datasets\n", "from torchvision.transforms import v2\n", "\n", "SEED = 7\n", "\n", "random.seed(SEED)\n", "np.random.seed(SEED)\n", "torch.manual_seed(SEED)\n", "\n", "if torch.cuda.is_available():\n", " device = torch.device(\"cuda\")\n", "else:\n", " device = torch.device(\"cpu\")\n", "\n", "print(\"PyTorch version:\", torch.__version__)\n", "print(\"Selected device:\", device)" ] }, { "cell_type": "markdown", "id": "a8ec7f0b", "metadata": {}, "source": [ "### 1.2 Load FashionMNIST\n", "\n", "FashionMNIST contains 28 × 28 grayscale images from ten clothing categories. The transform pipeline composed of `ToImage` and `ToDtype` converts each image into a floating-point tensor with shape `(1, 28, 28)` and values between zero and one. The leading dimension represents the single grayscale channel.\n", "\n", "The first execution may download the dataset. The official test set is loaded now, but it will not be inspected for model-selection decisions or evaluated until the end of the lab." ] }, { "cell_type": "code", "execution_count": null, "id": "e412f27b", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 26.4M/26.4M [00:00<00:00, 75.6MB/s]\n", "100%|██████████| 29.5k/29.5k [00:00<00:00, 1.96MB/s]\n", "100%|██████████| 4.42M/4.42M [00:00<00:00, 35.5MB/s]\n", "100%|██████████| 5.15k/5.15k [00:00<00:00, 16.0MB/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Original training examples: 60000\n", "Test examples: 10000\n", "Classes: ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']\n" ] } ], "source": [ "DATA_DIRECTORY = \".data\"\n", "\n", "transform = v2.Compose([\n", " v2.ToImage(),\n", " v2.ToDtype(torch.float32, scale=True)\n", "])\n", "\n", "training_source = datasets.FashionMNIST(\n", " root=DATA_DIRECTORY,\n", " train=True,\n", " transform=transform,\n", " download=True,\n", ")\n", "\n", "test_dataset = datasets.FashionMNIST(\n", " root=DATA_DIRECTORY,\n", " train=False,\n", " transform=transform,\n", " download=True,\n", ")\n", "\n", "class_names = training_source.classes\n", "\n", "print(\"Original training examples:\", len(training_source))\n", "print(\"Test examples:\", len(test_dataset))\n", "print(\"Classes:\", class_names)" ] }, { "cell_type": "markdown", "id": "38835fca", "metadata": {}, "source": [ "Retrieve one example directly from the dataset. This operation does not create a batch: it justreturns one image and one target." ] }, { "cell_type": "code", "execution_count": 3, "id": "21e9d735", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Image shape: torch.Size([1, 28, 28])\n", "Image dtype: torch.float32\n", "Target: 9 Ankle boot\n" ] } ], "source": [ "sample_image, sample_target = training_source[0]\n", "\n", "print(\"Image shape:\", sample_image.shape)\n", "print(\"Image dtype:\", sample_image.dtype)\n", "print(\"Target:\", sample_target, class_names[sample_target])" ] }, { "cell_type": "markdown", "id": "2c8951ba", "metadata": {}, "source": [ "Visualizing examples before training is a basic validity check. It confirms that the images are readable and that the target names correspond to the displayed content." ] }, { "cell_type": "code", "execution_count": 4, "id": "933647f9", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA8kAAAHtCAYAAAAqbjyQAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAX/lJREFUeJzt3QeYVNX9//GLsGwvwC4d6YiIiIINxQLWKDasqNEYjaJ/NfaaxPyMiYmmGDUmauwRo7GLWIOVpmIBQdEVlrJLWWDZXoD5P9/7O7O/Ydn7PbPnMmx7v56HJ3HOnJk7s3Puud/bPh0ikUjEAwAAAAAA3i58BwAAAAAA/C+KZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIrkZrJy5UrvnXfe8WpqatTnrV692n9eZWXlTn1fAOG8//773jfffBPXcz/++GPv66+/5isHWrivvvrKmz17dlzPXbRokffRRx8lfJkAADteh0gkEknA67Y5FRUV/sSYnp7uHXjggaFf77777vMuv/xyb8WKFV7fvn0Dn/fUU0955557rrd48WJv+PDhO+19E6GwsNDfaDjooIO81NTUnfreQCzZURSPIUOGeAMGDHD68nJycrwzzzzT+/vf/259rrzHwQcf7I/3eHz44Ydet27dvBEjRgQ+p7S01Js3b5633377eVlZWXH1AVqzzz77zNu4caP1eTIO9t57b6f3OOmkk/ydX/HsADv//PP9dY3snI7HwoULvU2bNvlzpGbmzJneoEGDvP79+8fdB8D2O6erqqr8/7/LLrt4GRkZXs+ePb1dd92Vrwq+Tv/7P7B54oknvEsvvdTr2LGjV1BQ4PXp04cvrYlef/1176KLLvK+++47v/gAmsudd965zX/LTijZiXPIIYd4SUlJ9Y//5Cc/cS6Sm0IK5JEjR8b9/NNOO8075phjvMceeyzwOU8//bR31VVXecXFxXH3AVoz+c1/+eWX25yJJWdo7Lnnnl737t3rH99nn32ci+Sm2GOPPZp01tZvfvMbb86cOd6yZcsCnyPF+YQJE7x3333XL5Lj6QNge2effba3YcMGf0dy9GBYfn6+t3XrVn++vPXWW9nWb+cokuP00EMPeQcccID3xRdfeI888oj3i1/8IrF/GQA77UjyhRde6P3zn//0nn/+eS83N3enf/PxHkFuipdfftk74ogj/LNfgPbgj3/8Y6NnYv3yl7/0Tj311J2+PNddd11CxrWcpSI79ACEM2zYsO22B+Ssq0suucTfcS1nbYwePZqvuZ2iSI7zFK7PP//ce/HFF/1/sjF9yy23+KdnxJJTqmQv7/jx473k5GT/NCg55XGvvfaKe0N16dKl/p6soUOH+nuJNXKm/LfffuutXbvW69Wrl9+nKaS/XF8lp5vISiAlJaXR59XV1XkLFizwr4sePHiw/15NfZ58JjlaJ2bNmlW/11vetzmKEiCskpIS/3fdoUMHb7fddlPHuOytlt9/jx49Gj2LQk77kg1fOfIUe02zPF8us5BTSOVSBflvGTu1tbVeUVFR/eSenZ3t7bvvvvV9y8rK/MldLq+QveL//e9/rX2i6zBZB2VmZvobCJ06bTtFxC7TunXrvCVLlni9e/f2Bg4cGOKbBFqO6upq7/vvv/fHkMyp2vwkz5Uj1zLfy9FqOdMsloxZGftypkiUzLlyxEou25KjzDJnyjpExumaNWv814zdaJejxrHbGlIkH3fccf7Y/OSTT+LqI6djyxF1eUzGtZxWGit2mWT+lv9OS0vzP5MsG9CeyDa8zJlyaZIcbZYxGh1PMs7k7DKZx+UsLZnX5b/79etX31/OlpS5Vs5eCbpM8ocffvCfI9vJ0r9hPRHvc5Bgck0ydJdcckmkX79+kc2bN0fmzJkj13BH3nzzze2ed++99/ptn332WeTQQw+NjBkzJtKrV69IdnZ2ZMaMGY0+d8WKFfWPyXOysrIiJ510UqSiosJ/7Mknn/Sft3jx4m36v/HGG5FBgwZFunXrFtl///0jXbp08d/vu+++Uz9L9H1nz54d2W+//fw+PXr0iOTk5ESee+657Z7/0EMPRbp27Rrp06dPZJ999okkJSVFJk2aFCkuLm7S8x599NHI7rvv7r/3uHHjIhMnTvT/yfcJNLef/vSn/m9z3bp11udu3bo1cuWVV0ZSUlIie+21lz+OcnNz/cdkHREl4/7iiy+O3H///ZGhQ4f646JTp07++K6rq9vmNfv37x85++yzt3ks2v/uu+/2x/rw4cMjt956qz9uOnfu7K9bouPo8ssv36bvs88+G+nQoUNk9erVkdraWmufVatWRY444ohIcnKyv07o2bNnpHv37pFp06Y1ukx33nmnv0yjR4/2P5OM9dLS0iZ/70AiRefPxua2xjz++OP+nDp48ODIQQcd5I+D448/3h8fUSeeeGJkt912i3z88cf+mJTxn5GR4f//5cuXb/N65513nj8nxor2/+9//+uvF/bee+/IhAkT/PEoc7GsV6JjVP7J+I2S8bzLLrv441vY+tTU1EQuu+wyf+zL/CvvJ8+96aabIlu2bNlumd5++23/s++7777+Z5I+ixYtcvz2gZZN5l2Z74JcffXV/vrjgw8+qH9M/vuGG26I/OpXv4oMGTLE//fnP//Zb5PnyZiR7WnZLpftgpEjR0YWLFhQ3//bb7/1txtkm13WMcOGDfPH3ltvvdWk52DnoEi2kGJVCtfbb7+9/jEZVKeeempgASobjEuWLPEfq66ujhx22GH+YIyd7BoWyffdd1+kY8eOkWuuuWabyauxInnWrFl+Efqzn/3Mf31RVlbmb+TKgI0+1pjo+x511FH+QBSyYX/BBRf4G7uff/55/XNffvll/7nXX3+9XxgIGewyKY8fP77Jz5NCWp5nK+SBllwkv/jii/5z33vvvfrHZGzfc889kaqqqm0Kyj333NOfTKNee+01v+8///nPuIrkESNG+Bu0UdHJVsaWbIAHkdc68MADt3ksqI+Mfyl2d9111/qxKeugqVOn+hvksjHfcJluueWW+sc++eQT//EzzzwzcHmAll4ky9iXOfC2227b5vHp06dH5s+fv01BKcWzjKXy8nL/sYKCAr+4Pvfcc+MqkqX/GWec4c/b4quvvvL/Vx6TdUEQmUOl4I3dIaX1kSJaPpN8hqh//etf/ncS+zmjyyRjOPqZ1q5d668XBg4cGKmsrFS/O6AtFsmy3pCxIjuqo+S/99hjD39HsZBt3oULF/pjODU1NXLWWWfVH+SS7QHZKS7rgOiYlZ1YY8eO3WZbIT8/P/L000/X/3c8z8HOQZFs8cgjj/gFaVFRUf1jDz/8sP+YTCKNFaAykcX6z3/+4z8+b9687Z67bNmy+omsYb+gIlkK3N69e/t7iWPJQJXn/vvf/w78PNH3laNbsTZt2hTJzMzcZiNajvjKEfSGR71kr5m8xkcffdSk51EkozUVyVKQypGV6L/oWQ9//OMf/efGrhMaI4WjHG2N3ekl5IjTCSecEFeRLJNr7M61KK1IlnEoe6Cjk7itz+uvv+5/HlmvxZINY9kTfswxx2yzTI2N9Ztvvtk/ct3wSBrQ0orklStXbjOu5Z/8nuUMMHluw7MnGpKCUp63dOnS7c44kzk0niJZ+jd2hNZWJMtR7djxqPUpKSnxt1POOeec7dqOO+44f1mj65boMkV3nEfJkSt5XL5HoK2xFcmyg1h+/7E7quW/5ahu9IBQ1Omnn+7Pj9EdX7HrG+nzj3/8w/9v2ek0efJkdbnieQ52Dk5wt3j44Yf962bl+mK5FkH+yXUGMlYef/zxRvvsv//+2/x39O64y5cv3+65Z5xxhn/n7DfeeMO/eVA85KYCcg2gXNsr1x3KP7l+Qq5dkGui5PppG7kJWSyJiJHrIT/99NP6x+T/y2dpeF1i9PoquR6qKc8DWhO5a+yRRx5Z/+/iiy/2Hz/22GP96/fljphybwJZJ0RjJBoaO3bsdtcRyfqgsXVBUP/Yu23H44MPPvCvYT7xxBPjen50zMdeNykkpm3MmDHbjV9ZpoZjfdy4cf46Ue7fALRkr7766jbjWv6Vl5f785/cW0DmYRnrchO/6J3hG5JtgIZ3vZf/luuY5fpfG7kfwO67796k5ZZrhWVdE++4luuK5T4hDce1kMdkWeWeJlFdunTxb2LUcFyL2O0CoL2I3plers9vuP3c8Fp92S6XdYCMlffee69+u1zuUyT3+Ihul59yyin+ukWSJiQeMnqvnljxPAc7BzfuUshNN6QQlZtgNIyMkQv6pYC+9tprt+snk00suamHaGxDWjY25aYbcoOveGzZssV/Hbl5j2zEN3TYYYfFdct6GbSNPSY3AYm+j9zoR4rnhqKPyXLE+zygtZGb1sRuJEdvuCUbt7IBKpPX9OnT/XVD586dvZ/97Gfen/70p21u3tNwXRBdH8Q7JuQmWU31yiuv+Bu78eaqR5claAw3XNagdUd0Qx5oyfr27etNnDhxm8dkR5SMy7lz53oPPPCAN2PGDO/JJ5/052bZKSY365T8VNu4FjJepAje0eP6rbfe8jfaTzjhhB0yrmOfEzSuZUeZbKMwrtEeyY0phdyI1jZ+ZUeb3Hyvse1y2aEevRHvXXfd5f/3s88+6912223+NrdsW/ztb3/zd9jF+xzsHBTJltgn2bMseYQNyd0v5c6XctQmTBSDRFRIPIVksslEfN5556nPlw1wuYuebADLpOlKiuyGd8OWu9pGB7K8jxTbcne9hqKPSeB6vM8T3CUTrYkcJZZ/jZGxE42bkbtcy6T229/+1r+T/QUXXLDDliFozGhjSYpkWZ/E2yc6PmW8NrxzvTwWbY9qLI9V1h3Cdkd+oLkdf/zx/r/GSHF74403+v9k5+9LL73kR0hdf/31/hlfzTmu5a7Wcjd6uZt8U8e1bW4WciaafGbZ4Re1YsUKb/PmzYxrtEv/+c9//LPGJErRNubkKLI8t2GcVEPS9/TTT/f/CYmV/fGPf+xH1K1fv97fKRXPc7BzcLp1ANljK3uS5XSHxsheHfknR5PDkB+7nLZ92WWXeT/5yU+8e+65x9rn/PPP90/naOy0apnkZI+WTcNTxeXUECn8Tz755G1O+ZCdAHJEPZbszZLTT6LfTbzPi+59j2f5gJZKJqlYEt10+eWX129U7gwylhobR3KEWwrWxk7JDOojBYMcSZMj47HkqJqcPj158uRtHp83b55/ClmUnGYt60HZWSZ7v4HWSE6TloIwSopF2UiVS5uae1zLEarXXnutSeNadvDLGW+PPvpo/WmjQk6zlm2bgw46aLsjYg23Cx588EF/g/2kk04K+cmA1kV2fMsp1DfccIOXl5cX13a5zJeyLdyQrFeiZ4s23H6Qyznl7BBpj16qEc9zsHOwOyKA5CHLDzWoSBZyGpZsHP71r3/1N5RdySQkxbFMdj//+c/96wnlFIsgt956q18gy6nVsnG+zz77+KeFyXXT06ZN8/d+yxEtjRwJO+ecc/yiuKCgwLv99tv965SmTp1a/5xf//rX/jUVcmqaHFGTCfXf//639/rrr3uPPfaY161btyY9T65bltO35IibrFBkI4ScZLQ2UkzKdY2yV1fGkZyKKGedyPiVTMWd4fDDD/eefvppf6NWitNo5rEcbZLrJRvec0DrI6ef/vnPf64v9GVDXIoCOW1M2m+66aZtXkeK5ksuucQvICRDVja4paCW9U5Tr58GWgqZU+V65ClTpvhZwnL6tJx2LZmnMj/uDDJG5XRvGXuyw0nuZyCXe0mOulz60ViRHNRH/lfOTjv66KP9bYVLL73UvzxKxrrs2JICONagQYP895GsdJmXZWNftktuvvnmbfLbgbZEdhpFj/7KXJ6fn+9fDyz34vjVr37l/4vHlVde6V+PLHWBHPSSsSjFsRw8knlXtoVlG1vmVNlWljlaLuGQ+wLcf//93llnnVW/rRzPc7BzUCQHkCMlcu7/oYceGvjlyUaiDAAZTPJcOQ1afthyykWsjIwM//HYUxkbe64UxrKB+8ILL/hFuhSw0keel56eXv88KS7llErZsyz/ZCLs2rWrfw3lnDlz1Oudou8rg3j27Nnev/71L/+6JDnlWwrk2I1c2eiXo0by+u+//76/ApFrM2RFIJNoU58nG+ZSOMtGuky+MmHfcccd/oY20JzkiIuMi9hTDYPIjiDZ8JTrhWRPs5wtcdRRR/k7qGLHnmyYNnZzHtmBJeM1lkyesmEeK6i/kOugo+sKGW/yvGiRPGnSpO1uFqb1EbI+kBtyySmljzzyiH99YnRnVvRay9j1mXx2Od1cxrOc/inrHdlZB7Qk0fkznmuAZbzJGVoyP8lGsmzgyjz25ZdfbjM2Gxu/0VOX5b1ix4sUl7FHcbX+Qi6TkB1ucimVzKVSzMo2iIxrWZbGitWgPrIOkI1s2Xn+j3/8w3vuuef8HfKyk0tuTNbYdyKvc9999/kb9LJek9NN5UwxoC2SeXf16tX+3CjjRbazpSiVyx5lG7yxI8gyxhteqijk0kMphmUcyg5jOYNDDp7JmJVxGb1XkNQMzzzzjPfRRx/51xrLOkrGm8zbUfE8BztHB7nF9U56LwBAgsgRINkJJjvQEjWZyqR/5plnbndqNoDEkRvxyZiO3gdhR5PTqeXAQOxlFADQ3nFNMgC0AbL3WY5oN7zJCIDWS44kyVFq2TkFANh5ON0aANoAKZDlH4C2Q06Ltt0xFwCw41EkAwDiol0nDaB10q6TBoD2imuSAQAAAAAwuCYZAAAAAACDIhkAAAAAAIMiGQAAAACApt64S0LoAYTXkqLJW+q41parOb+/4cOHB7bdd999at/nnnsusO3zzz9X+9bW1ga21dXVqX1HjhwZ2HbyySerffPz8wPb7rrrLrVvSUmJ1560pHHdksd2c+jevbvafv755we2PfHEE2rf1atXey3R6NGjndZj4vnnn3de37RFLWlst7dxPWDAAOvNJIOceOKJat/169cHtj311FNq3/nz5zuPr8mTJwe2TZw4Ue1bWVnpvMwPPvig2t7eROIY1xxJBgAAAADAoEgGAAAAAMCgSAYAAAAAwKBIBgAAAADAoEgGAAAAAMCgSAYAAAAAwKBIBgAAAADA6BCJMwCuvWWzAYnSXjIXmyvrWMsHFWeeeaZTfqHYsmVLYFt6erraNzU1NbCtW7duXnNYsmSJ2r5169bAtt12203tu2bNGrX9zTffDGy7++671b4LFy70WpqWNK7b45ydkZHhNObFlVde6ZRRLoqLi5372tozMzMD25KTk9W+ffv2DWx7+eWX1b6zZ892yntvq1rS2G6N4/rYY49V26+66qrAtqqqKrVv586dA9uqq6udx9fIkSPVvj169AhsW7Zsmdp38+bNgW1FRUVq302bNjmvE/r06aO2v/vuu4FtV1xxhdfWkJMMAAAAAEATcLo1AAAAAAAGRTIAAAAAAAZFMgAAAAAABkUyAAAAAAAGRTIAAAAAAAYRUMBORpyEXVZWVmDbE088ofYdNWqU2r7LLsH7BsvKytS+WqREXV2dc3xUUlKS2jc7OzuwraKiwjnGKZG/xZSUFOdILC3WQ3z44YeBbeeee67X3sd1a42KSZTTTjtNbddiZm655Ra1b+/evZ1iYuKJbNm4cWNgW3l5udr37bffDmybNm2ac5zWSy+95LU3LWlst8RxPXjwYLX9tttuc44LTEtLc57PtbnPFsXUr18/z5XtfbV2LeLJtsy2bZANGzY4R0SVlJSofa+99lqvtSECCgAAAACAJuB0awAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAIxO0f+D1pGBFyavLzMzU20/+OCDA9tmzJiRsM/UsWNHp0y4lpxF2JJyFVujF154IbCtf//+at+1a9c6ZxR26qSvErXfo+03o722rW9xcbHT+LHRMibD0rJnbZnTtvFzyCGHBLYNHz5c7fvNN9+o7Wh7bLnbWgbofffdp/a94oorAttqampC5SRry/XZZ5+pfR999NHAtoEDB6p9161bp7YDsa655pqE/Z5sc1RKSorz9qPWvnTpUrWvlmesLZNtG8S2TtBs2bJFbbdt3xQUFAS2jRw5Uu173HHHBbZNnz7da604kgwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBBFQLY7vdve0W70OGDAlsu/DCC50jWyoqKpzjXObNm6f2DRPzpEXn2L5LrW/Y6KkwsTztwZgxY9R2LeZJi0OKJ+ZA+9vYohv69OkT2JaWlqb21X6PdXV1zp/Jtk7QfudJSUlqX20clJWVqX1Xrlzp/No22me2reeuvfZa5/dF61ReXq625+bmOsWiiKuvvjqwrW/fvmrfvLw8tV2LoVm/fr3zZ7KtI8NGIKJ9eeyxx9T2q666yjkias2aNc7RprZ5VVNbW+s8vmxKS0udoxPDsH2m7OzswLYVK1aofVtzzJOGI8kAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABjkJLcwtnxdWybqhAkTAtuOOOII51zT5ORkta+WEXvkkUeqfR9++GHnjLxIJOL8XWkyMjLU9q1bt6rtlZWVzu/dHhx++OFqu/Z7s/0WbX8bbYzV1NSofW+44YbAtsLCQufx1bt3b7VvUVGRcx64lo1o+y61cbDPPvuofS+//HK1Xcu7tuW4an/jU089Ve1LTnL7EyaTO0weqi3TffXq1c7zqpbZbpv/tHkznnYg1rx589QvZPbs2Wr7CSecENg2d+5cta82V2jjx5Y1bssU1sZ2dXW12ldbLtvcp2Us23LXbbTluvHGG732iCPJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGB0icd7rv0OHDvE8Dc3soYceCmw7+eST1b4rVqxwahNvvvlmYNvee++t9k1KSgps+/TTT9W+CxYsCGxbvHix2ne//fYLbNt3333VvrNmzXKOPCgpKfFaiuYa13PmzFHbu3fvHthWVlam9rVFN2ixRps2bVL7HnDAAYFtRx11lNpXi2x59NFH1b4XX3xxYNvChQvVvqmpqc6Rc1oE2xdffKH2/e6779R27e+YkpLiHOkzfPhwte/IkSMD25YsWeK5ammxOczZ/2fSpEnqd5Wenh7YVlFRofbVxpBtfCWSFpOWlZXlHI3z2muvee1NSxrbbXFc5+fnB7a9//77at9169Y5x0GWl5c7b2dobOO+rq7OOQJK2162RV5lZ2er7TNnzgxse/XVV732OK45kgwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgKEHcmGn59zZcruOPPJItX3s2LHOuW9aVuSwYcPUvlr7J598ovb9/vvvnTJtxYEHHhjYdsoppzhn1dmW+cILL1Tba2pq1Pb2bq+99lLbtVzuXXbR9+0lJyc7L5ctP1TzxhtvqO1a3uqIESPUvtdee21g24svvuicEWvLZJw/f35g25gxY5yzjG3rmy1btqh9tfzL5cuXO68zwuQko+WyzSPaOqO6uto5E9WW02rLUw2TiautJ23rUFtOOdCUecQ2Fxx88MGBbXfccYfzl11ZWem8XKmpqWrfqqoq5+9Da7dtO9rGbpi+bTELOSyOJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGAQAeUoTDRDGLfffrva3qtXL+fXTktLc76Ff21trdPt/W2xVbYIDS2iRouWsn2myy67TO07aNAgtf3UU0/12ruRI0cGtq1bt875bxM2NkWLdli/fr2XiM9ri3awjVstBsP2ebWoM1tfLS7JprCwUG3v06dPQiKgtGgOMX78+MC2xx9/XO2L1skWyaKNA9sY0WJVbH0T+draOtQWBWNbxwLx/tbiUVRUFNiWn5+v9h04cKBzfJsWi2rb9tRe2za+ysvLA9vy8vISNq4LCgrUdmyPI8kAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABjkJDuKRCJec9i4caParuWt2vJDk5OTnXMmMzIynLPqtNxaW1adlnk6btw4ta+WKde9e3e17xtvvKG2w/NuuOEGp7+5LUfQlqFre23t92jLe9Qyvbt166b27dq1a2BbUlKS2rdHjx5OOci2z9u5c2e1b05OTmDbGWecofbt0qWL2q6tj7Kzs5372j6T9jdE22TLD62srHTODA6TZWxblyVqG0TLbAda09jNzMx03n7UtnlLS0vVvto8Y9vmra2t9Zojk3rt2rXOfdsrjiQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgEAHVyqSlpTnfLj9MDMamTZvUvuvXrw9sGzBggHOUhS1CQ/tMtu9Ki9+wRQf069dPbYfnzZo1K/Br6Nmzp/oVDRkyJLAtKytL7Zuenq62f/fdd86RLHPmzHH+zWjttvfVYmhs8WzaGLK9rza+ysrK1L5LlixR27XxGSZ2p7CwUO370ksvqe1oe2xzn8b2W9TGdZjfcVjaesEWAWWLQASawvY718bQypUr1b6jRo1yfl9tHNgi1rTYRtu8mpKS4hzVqsVL5ebmqn1XrVrluepk2c4IE03VknEkGQAAAAAAgyIZAAAAAACDIhkAAAAAAIMiGQAAAAAAgyIZAAAAAACDIhkAAAAAAIMiGQAAAAAAg5xkR1r2qC2bTctQy8jIUPv27t3bOffNlo2YnJwc2FZbW+ucsZyTk+OcsWzLOu7cubNzjmt2dnZg21dffaX2tf2dxo4d67V3DzzwgFOb6NKlS2Db0KFD1b5Tp05V2w899NDAtg0bNqh9Fy5cGNhWUlLinKtoy1NNlDA55Fpeo2182cbY2WefrfYFmrLOsI0vbRzY8lITmXWsseWya7mmtrGrZc1rGa/xvDbQFMuWLXMef9r2oW2dYXtfLRe4W7duat+NGzc6va5tO962LmqrWcaJxJFkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMcpIdadmJtkxGLSf5jDPOUPv27NlTbV+3bl1gW2pqqnPuopabKPr16+ecsazlM9fV1TlnQdo+r5Zld//996t9R48e7bxcsNNyBOfNm6f2teWBT5gwwTkTVctdtI0Rbb1gyzwNk3WstdveN0x2ui1PddasWWo70BTauLetE2zj3lXY19XGbph8Zts2yqZNmwLbyEHGzlRVVaW2h5k7tb62MaLNb7Zl0rZvcnNz1b6ZmZmeq6SkJOe+7RVHkgEAAAAAMCiSAQAAAAAwKJIBAAAAADAokgEAAAAAMCiSAQAAAAAwKJIBAAAAADDIqXGkRfzYolE0CxcuVNttURbaLd7DRFN1795d7avFQqxfv955mW0xMlrsjnabfbFy5crAtilTpqh977rrLrV9zpw5ant7Z4st0n4TtvFli10pLS1NyBgJE/di+z4SFVEThu27sikpKUnIe9viN1rid4nmjWVsb9+HFu0G7GhhYpo2b97sHHtq21awbSO69rW9rxZPunbtWrVvXl5eYFt5ebnaF03HkWQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAHZmTrItA1TLMNxll12cX7uurq7Zsttcvf7662p7RUWF2l5VVRXY1rlzZ+dcRS2LzvY3tGUd2/5Orn1tf19tmUeNGqX23bRpUxxLB9es2jC/ifz8fOecZC3/PGwGuvaZE5mTbHtt18+rZVnHQ/s72GjzgpZljbYrTBayNlfYtkHCsM1RiXpv2+tqY8jWN8x2FdqmML+ZzMxMtW+XLl0C2yorK9W+Xbt29VwVFxcHtqWlpal9s7OzE7KNYZvr+/fv3+JqnpaOI8kAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAADs6AgoLX7BFsnRGm8tfsghhwS2TZ48We170EEHOd+yfv369Wq7FvNki7fR/k625dL+/snJyWpfLSLKFn1jWy7X76q8vFzte8opp6jtr776qvNyIVzEjxaDZotYsP1WtXWVbXxp8Qy237nW1xb7oH2XtvetqalxjrmwLVdrXO+j5Qozj4SJZwsTlxQmtsomzPpGa7dFSVZXV8exdGhPwsSC2eJHFy5cGNi2YsUKta82h9l+xz169HCOcVq2bJnz+2rxUUVFRWrf3r17q+3YHkeSAQAAAAAwKJIBAAAAADAokgEAAAAAMCiSAQAAAAAwKJIBAAAAADAokgEAAAAAMCiSAQAAAADY0TnJtuxSV127dnXO/Ro6dKhzX1sO7rBhw5yyRW3Zibbc327duqnthYWFzvlrWv5h9+7d1b5aLpwtT3XWrFmBbRkZGc551bZsvk2bNgW21dXVqX0POOAAtR3h2HI8Nba/u7auCpMfastEDbPMYfJUtbxU2zJrn9e2zGFe2yZMX7RNYbLEw2QKu75ucwqzXGHWc0BTjR8/Xm3/4YcfAtsKCgrUvto2cWlpqdo3KyvLKctYVFVVOWcs9+rVy3PVs2dPtV3bzl+7dq3zeiFMTnZzY20HAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAADAjo6A0iJxbr/9drVvXl5eYFtOTo5znIstNqWkpCSwbfPmzWrfsrIy51u4a/EL2q3hbXFJ4vTTTw9s+/TTT9W+mZmZzrFWAwYM8FztueeeTsskVqxY4RynlZqa6hw91b9/f7UdLVefPn0C2zZu3Kj21dYptqgYLSKhpUbFaMtsi0mzfaYwsVZAa/g92dYJYca9ra/23rbvSmvv1GmHbTaijbDFgtkigPr16xfYNmLECOcIKFv9kJubG9j2/fffq33T09MD2wYOHOhce2jRUmGVl5er7VOmTAls+8tf/qL2bc0xTxqOJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYMQdeGfL1fvrX/8a2NarVy/nrGOtLZ4sXE3nzp2d39eWZ6zJzs52zt+98847nZdr6tSpat/CwsLAturqarXvu+++65RjJ4YOHRrY1q1bN7WvlkmdlJSUsAzYdevWqe1IbL5oGLYM9EStM7RcU1vmqdYeJovVlm2ojSFbdrptuWzjM8xro/3Rfue2san9nmxj05YR6/q+YfuGWS7tM2vbL6K0tNT5fdE6hc3IPfroowPbFi1apPZNSUlx/i0OGDAgsG3VqlVq3+HDhzt/HytXrgxsGzVqlNp3zZo1ztvLGzduVNv79OkT2DZkyBC1ry1XurXiSDIAAAAAABTJAAAAAABsiyPJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAA0NQLqxz/+sdquRRfl5+erfTMyMpzaRNeuXT1XWgSJLeZgxYoVTlFKIi0tzen27uLxxx9X20866aTAtldffdX5dvi2v8OYMWMC2w4//HDnqAot4kkkJyc7xfXY2CJDbPE1/fr1c35vJJYWXWSLutPio2x9tVgIW5yL9tq2MaK9dqdOnZz7honfEzk5OaH6A/Guk21xSLaYJ9e+LTWqLEwkljbnAi602KOvvvrKeW60bQOG+S3b5nvXbQFbfJQWx2rb7rRFYmntA5T6QBABBQAAAABAG8fp1gAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAADQ1J3nt2rXOucGZmZnOuaXa69rye20ZaVlZWYFtGzZsUPsWFBQ4LZOoqqpyykCz5bSKF198MbBtwYIFal8tB82WR61ltZaUlKh96+rqnD+vlilnyzLW+tpyM22/rWHDhqntaD62HEJXtt9MmMxULec1TMarbZnCZMDaxm5qaqrnqqXmz6L5aJnftjGiZZ621t+abfy5zsm2zGmgqRm7RUVFgW0pKSlq3/Lycqd1gm2MhJmfwmy3hslurqysVNt79Oihtq9atSqwLS8vz2uPWNsBAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAADQ1Ago7dbgtpiElStXqn3T09MD23Jzc9W+WrxQcXGx2nfdunXOt47XbtNuix7Sbmlvi8uyxS9on3n33XdX+1ZUVDhHcW3cuNH5lvbaMmtRFLZb7dv6arf479mzp9p306ZNavvo0aPVdjSfREWYJDIqprkioLT3DRsBlZaWZlk6IH62WD6N9lu2Rca1xkgk29jV5k7GLZpq1113Vdu1MWbbFtfGvS0+asuWLc7vq+nSpYvz3Gh7X6196dKlat+hQ4eq7WvWrAlsy87OVvtqMbG2SN2WrPWt3QEAAAAASBCKZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAIy4g8C++OILtf2FF14IbLvgggvUvoWFhYFtP/zwg9q3uro6sC0jI0Ptq+UZaxm6tmy2jh07qn1ramqcctviyTesrKwMbCsqKnJ+bdtyadlt2t/I9neqra11zsnW2mxZkLaM14EDBzrnzaF5M4c1trGbqM8UJus4zDKH+Z5t+bC2dUYiv2u0P9qcHCbTO8zYbE7a+LSNTW1uHDJkSKhtRbQ/tnW99lvVtmltud3aNr5t+9KWj66tU2y1h7a+0eoD0adPn8C2Tz/9VO17yCGHqO1ajdDJkt+sZUOTkwwAAAAAQBvA6dYAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAANDUCyuZ3v/udcyTAtddeG9g2YMAAtW9xcbFzBFBFRYXzLeu1uAnbrdK117bFTdiiLLRb3ttuh699JlvfMDEZWl9blJJ2q/2uXbuqfbVb/Pfs2VPt+9VXX6ntTz31VGDbk08+qfaF/psIGw+lxT5ocRJhab832/omTERNc8VpJTICqrk+E1qu3r17O/fVImhsv7Uw4zqREWzactnWGdr6RtvmAhqTm5vrvO25bt06te/IkSMD21JSUtS+paWlTstkGyOZmZlqX+21bZGpo0aNCmybPn262tdWE2nL1UWJeIqn7mmtOJIMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIDRaWdk8s2YMUPtq7UffvjhzvnM/fv3V/tmZ2c7f14t/9CWF2bLD9WsXbvWOXdx1apVat+amprAtvLy8mbJPK2rq1P7VlZWOv8N33777cC2xYsXq31nzZqltqN1sv1mtLFryx7VXtv2vlq7tu6NZ7lcx6ZtmW3CrDOApuSLJiUlOf/Obb9TbXwlMivcNjdqr21bZ2RkZAS2FRQUxLF0QPw5ydpcsn79eufteNu2eFFRkXNO8saNGwPbKioq1L5h507X7XRtmW3rhQrLZ+rVq1dg27fffuu1VhxJBgAAAADAoEgGAAAAAMCgSAYAAAAAwKBIBgAAAADAoEgGAAAAAMCgSAYAAAAAwKBIBgAAAACgqTnJtly9RJk5c6bafsABBzi/9vDhw51z3UpKSgLb+vbtq/ZdtmyZc/Zhfn6+2g60BVpuaViFhYWBbcOGDVP7bt682XkdqbXbcly1vrb31b5LW46rLWfS9X0Tma2O9mnevHnO4zonJyewraqqynmZbBnl2vokkb9zLdPUtl5YsmRJApYIbZmWuy0qKysD27p06eL8vikpKWp7bW2t89yXl5cX2LZu3Tq1b3p6utPr2mqTwYMHq31t2wpafvNWS9/MzEyvLeJIMgAAAAAABkUyAAAAAAAGRTIAAAAAAAZFMgAAAAAABkUyAAAAAAAGRTIAAAAAAIZ7vkcb8M033yTkdRcuXJiQ1wUQnhb3okUz2GIhbLFxWryC1hZPRJQrWwSUFtO0YsUKtW9aWprabourSFRUBdomLUbmiSeeUPsefvjhzuNaW2fYYs5sEVAa2zpDG9tLly51jt7UvmegMUOHDlW/GO33aItxCjNGtDmqurpa7Ttr1qzAtilTpjhvR7z77rsJ247Qtn1ERUVFQtYZrRlHkgEAAAAAMCiSAQAAAAAwKJIBAAAAADAokgEAAAAAMCiSAQAAAAAwKJIBAAAAADAokgEAAAAAMDpEIpGIF4cOHTrE8zQAFnEOuZ2ipY5rbbnCfn933XVXYFtycrLat6SkJCFZxrZ8w/LycufvQ/subTmtWuZwbW2t2rdLly5q+7x58wLbXnvtNa+1aUnjuiWP7da4ztB07do1sK1nz55q36ysLOf3Xb16tXO7LQM2zO+qpY2DHaElfabWOK61XGDbPGSbG7U5avDgwWrfgoKCwLa+ffuqfZctW6a2o22Ma44kAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAA0NQIKAAAAAAA2jqOJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkN7MNGzZ4Dz/8sPfDDz9Yn1teXu4/99tvv90pywYgMV577TVv+vTp1scAtG8ffPCB9/TTTzf3YgDtVmFhob/tXVRUpD6GtqdDJBKJNPdCtFTy4493o3XSpElejx49mvweX3zxhbf33nt706ZN884880z1ucuWLfMGDhzoPfTQQ96FF15ofe1NmzZ5zz33nHfooYd6Q4cOVZfhq6++8n784x/H3QdoL7755hvvo48+qv/vTp06eXl5ed6BBx7ode3a1ek1Dz74YP913nvvPfUxAIm1dOlSb/Hixf7cJ/PrXnvt5aWmpraYr/3888/33njjDW/16tXNvShAqyA7lSorK/3/36FDBy8tLc3bbbfdvH322cfp9d555x3vyCOP9GbOnOkddthhgY+h7enU3AvQkpWWlnpz5szZ5jEpIDt27Oidcsop2zwug8SlSG6KzMxM76c//ak3fPjwuIv8iy66yHv00UfVgveXv/ylV1NT4xfJ8fYB2gspWqdOneode+yxXu/evb3q6mpv3rx53vLly73bb7/du+6665p7EQE00bp167yf/exn/hkchxxyiNezZ0/v888/94vRiy++2Pvd737Hdwq0QldffbX/v8cff3z9UV8pakeOHOm98MIL3oABA5p5CdFaUCQrZM+TnE7RcIM5JSVlu8d3hm7duu3w95W9bbLy+OMf/7hDXxdoixPvEUcc4f//zZs3e6eddpp3/fXXe/vvv7+/kQ2g9bjgggu8//73v/4OLzmbK+qxxx7zdxxTJAOt15AhQ7bZXp47d6530EEH1Y97IB4UyQm2detW/2h0QUGB16VLF38yDjriLGe+f/jhh/4Rqt13390bM2bMdtckP/PMM9748eP9Aj56TbPsGZswYYJ/qpicFpqfn+/tueee3owZM+qvaZKNeiGnku277771r/nWW295VVVV3gknnOCtXLnS+89//mPtI8shyyl74nv16uWfJhp7elrsMskeO9mxIHvyZJnktYDWTk6Lvuyyy7yXXnrJH2dSJD/55JPeoEGD/Ik41rvvvuuPCSmqXch64dNPP/XvRSA76A444ACvb9++9e2yDLKeaXh2i/jss8/8o2Nnn332NmP0+++/9+bPn+/V1dV5o0eP9vbYY49t+sn4zcjI8I466ij/dFTZwJD3lLEOtHZbtmzxx+0xxxyzTYEcPb254Twl86HMj1OmTPHWr1/vb2Tvsssu3uGHHx54ycWaNWu82bNne2VlZd7gwYP9yzPk1M8oucRJCnQhj2dlZfmng8pz4yFHwOU9TjrpJH8HupDxLO8p2xvZ2dn+toJsd0TJnP3yyy/7p4nuuuuu/ueS8X3cccf5l5AAbZXszJa5U7ZHZZtX7gMkY6Xh3Lhx40bv+eef98d2vGOxIVlHyLa4nI0q28Ay9mWbQcj2vWx3yw73hke0ZR6XnXSyfR+7HSHL+/HHH/vb0bm5uf7lkOnp6dtciikHu2RdkJOT458CvmLFCv8STjnVHO4okhNIfqQyGcnR2nHjxvk/dJkY5XriW265ZZvnynPkumYZJPJPBpGc4nn//ffXP6e4uNg/FVquSY4WyTLg5LF//OMffoErp4LLxHn55Zd7X375pf8cKZplQhcyoccWvDJhjh071uvTp4/39ddfW/vIe8gpat27d/dGjRrlb4RL0fzUU0/5nzV2mf7+9797zz77rL9hX1tb66+cZNDKqdzRFQbQWkUnVrlUQciYk993wyL5gQce8K/7dymSZcNcit/vvvvOv6RDJt+zzjrLu/LKK7277rrL37iWnXB33323P+7kdPBYUshXVFTU38NAJu2f/OQn3uuvv+4X9jKBynpGioUnnnjCH6tCjqTJBC6FtIx5eV1ZR1Akoy2QeVI2MteuXdtoe8PC+ZFHHvGvC5ZCUi6vkPlX5j6Zk99+++1t5lSZv2+88UbvL3/5i79hLmNHNpjlf2WHlowjsWrVqvrLuaRol+2F999/3y/EH3/88cBll+fKWS1/+9vf/HVLtECWgvfcc8/1/7+8r6w7pACQeVj+Nzqvy9wsn0d26sk6TJ4nl3BRJKOtk9+77HSWnUkybq+66ir/lOzYIlnGpYwRGR8uRbKM+5tuuskbMWKEv9Ncxr7scJZ5VHa+yc4w2VaQnXEyfmPJdr9cUhk9WCVefPFF//KP6EE22Vku6wq57lp2YgvZiS7LLOuC3//+9/6BuEWLFvntFMkhyY27EL/BgwdH9thjj7iee9FFF0V23XXXSGVlZf1jNTU1kZdeeqn+vz///HO5cVpkxIgRkS+++KL+8bvvvtt//LPPPqt/bOnSpf5jDz300Hb9hw0bVv/curq6SGFhYWTx4sV+26OPPtro8m3ZsiWSm5sbuf322+sf0/osXLgwkpSUFJkyZUpk8+bN/mPV1dWR4447LpKenh4pKCjYbpnmz59f3//111/3H7/jjjvi+v6AluCBBx7wf7dvv/32No9ff/31/uPPPPOM/9/Z2dmRiy++eLv+kydP9tcbsQ466KDIoYcean1sv/32i/Tu3TuybNmy+sdkbMr73nPPPf5/L1mypNFx9fXXX/uP//nPf65/bNKkSZGuXbtGFi1aVP/Y999/H8nJyYlcddVV9Y/JOq5v376RW265pf6x5cuXW78roLW4+uqr/fFx5plnRt55551IWVlZ4HPPO++8SEZGRuTCCy/053BRXl4e2W233SKHHHLINs/99a9/HenQoUPklVdeqX+spKQkMnr06Mj48ePVZZo7d26kY8eOkSeeeGKb9+7Ro4f//0tLSyM/+tGP/PEqyxyVn5/vL9+pp57qz8lRv//97yOdOnWKfPnll/5/z5492//Mu+++e+Srr77yH6utrY2sXr067u8NaOlkvMh8Gkt+41lZWZHhw4f7/y3zooyFoqKibZ63YMEC//Enn3yy/jGZ++WxmTNnqo9Nnz7df+zmm2+uf2zDhg2RvffeO9KnT59IRUWF/9jZZ5/tby/E1gZCxm9eXp4/JqPrAxm/l112Wf0299atWyNXXHGFP95XrVrlP/bcc8/577v//vvXbyvIOmfTpk2hv8v2jgioHUD26Mi1D9F/0b3DspdajpjGnmLVuXNn78QTT9zuNeTUjthTvOS6ieipmvGQIzzRO/fJe8pp0DZy+obsCW9seRoje6RlL7YctZI98SI5Odn/bzla9c9//nOb58uRqtg98nLjI9mzFXt0HGgt5PRMGd/y+z3nnHP8I7lymcKpp56akPeTU5zldMxrrrnG69+/f/3jsgdaLsW45557/P+WG+zJWJMzNGLDCmQ8yvpGllUsXLjQe/XVV/0jYXI5R5TsLZezQ+QMleglFkJOE40946Vfv34J+ZxAc/jDH/7gjyE54iJnQcmNMeWygxtuuME/Y6MhOWNK2mRMCTkSfcYZZ/iXHkXPJpEzpmQ+lLM/5MywKDn1+eabb/afK+MwSo5oyWNyJpaMVznTTM7ckqPCDcnRI5nn5W77s2bN8iZOnFjf9te//tWfg2XdJHNylKw75AjUgw8+uN32hlz+JJKSkhJ+01FgZ5MzKqPb5L/5zW/8U61lfpQzMBLl3nvv9U+HljOxomT8/c///I9/hFpO4xZytFjuph/9byHrnFdeecU/G0TGZHQdJeuZ2G1uqSfk88iZqXL2VyzZlo9uK8g6R45aIxzOed0B5NQnGZBRciqkDEg5xVEmS7mBwMknn+xfRyDX6TZ2DVPDa6BkYMlpEjIxxkOuK2wqGZByHXN0srSRCVxOGWtYgMupWnI6ibRrp6wJKeTllBIpzmVlArQW0ZgYmazkVOQ333yz/hKDRIiOp4b3JhByiYRcYiEbxjKJyrpG7k4vG9eynpGNbzldTK5Rio6z6PWPMhnLdU+ywRAtqmX9JUWAnLItp4gJOaW0JUXhADuSjOMrrrjC/yfjOroTSTZ0ZeNTruWXO15HyXwsc3ksuU5fxpBsAMu4kQJWdi7Jhmx0jAn5XxlbQopyucuuXNcsp0HLcsjp0bJRK5c4yY6qhnFP8pr77befvzyyE77hqdEytmW7Qi6jiL5f9J8U//KeYbcXgNZExoyMFRmLMo9de+21/g7tRO4QkjlbxnbsjqrofB1tF3LplKxLZMdYdCe2zNeyk00K6NhxLWNd7kXUcFxLAcy4TjyK5B1AJjqZZKPkIn0h1zrIxCsZyHI9rgwIORIr1yvcdttt27yGTGQNyd4kGTTxcLmeSK5HjvcospBlbzj4o2TvumyYx2rsudHHYo9YAa3t7taNkY1duR6xIYmMciHjzTaOomNOJn+5zkmuNZQiWTb25SY9sRNu9GiX3LRLbk4SS84+kedGr0kWXKOI9kIKVLmXgPyTHblyhPi+++7zj9jY5mgRnaejY0yiFGOz1aNkjElhLWNbthvkukXZ2RZ7j47p06dvc0ZIdLzL2R+ffPKJv6EdexQ59n0be8/GbkDE2EZ7u7t1Q9Ejsw3nbNf5WttGbjhfS+EuZ4vKmVpynwAZn3ImmNQOsk6IHdfy3MbGtRyAixbfUYzrHY8ieQfQ4pOGDRvm/epXv/L/yc25Lr30Uu/Xv/61v1Ere5wSKfY078aOisnNgORIU7x9ZCDLJC2fI/ZmALIxLnfvbbiXfcmSJdu9htx0QDY2GMxoa+Qoj4yFhmScuYhu2MqYkTNTYskRKzlCLHeyFLKnPHrDHzkSJjvk5O61sUV99GZ/p59+un/zL6A9k8uh5AaUDUVvwiV3iG4qmQOjR4a17QK5G60cLZaN5NgCWc7oiD0rLXYntFzuIcW73IladrzL2WmxY1vWM3J2SXTjH0Cw6FkiMmfH3vDSdb6OztmNbffKfC1it5Hlsqlf/OIXfnEsB6tk51fDol7GdfS0cTQPrklOoIanQkhhKadbR+8ym2jRQrThUaPoUWQ5Pavh3Wq1PrIRLnvMo9dCRsnd9KS4jt5BM0ru0Bf7OhI1IXfqkzsAM5GjrZG9unJ3WtmJFDvO5KiSCzkiLHfClesNo0eKhNxxWuIeGo43OeVa3ls2zuXolNzFOnqH+ujryTWXkv8au4xB6yugLZMdT9FLEGLJmA26zMFGLpOS+U12Usl815Ds8JKjTbKBLsWx/Hcs2YEedDdaORol1zDK68ud8uV07ii5Q71cLiHXMDYkY12KcgD/R84YkfkxGpUqZPtWzsZyJdvIMu6fe+65+sfkrBC5f4nsyJ48eXL943LZ4o9+9CN/HMv9QKL3OIglB9Wk6JZTsRsqKSnxI6GQWBxJTqDf/va3/h4kOTVKjurIdUtyAw3ZE9zwyFAiSBEs0VNyZEnIEdxo5rFsCMjp4A2LVa3P0Ucf7d/0R/Z+y5FoueZYbiAiE/edd965TQxG9NQy+ayyIpDTTOSzy540eS7Q1shlFNF8RZnsZLKUiUxuVicRUE0lR4/kiJHcAEjGpLymXE8sN9CT07JiTwWNTvoyJuVxmZilSI4lY112UsnryR5quRZK9qBLBIxcDiL3J4he+wS0dRKXIuNI5ii5RlBOu5RrGCWmSR6TwtOFxLrI0R+JSDzvvPP8sSZHqySmRY5OS8yiXNZw/fXX+zuY5dIjOeNMdnxJn4bZqQ3HsBx5kjNI5HRNWb/8/Oc/9/OQ5X3l+moZy/J5ZP0hRbjcA0R2bGuvC7Q3ctRX5ki5yZbcI0fOKpGCWeZFGYsuZJ0hO8plB7b8r9ynQO4TIDfnk7O8Gt7PR7aR5dIo2akmR5bl3j6xZIeYjGFZTlkvSd0g6ynZ/pYsZJmvG8Y+YseiSG4iOVUxeh2SjdyxUiZEmaTkRy2nR8oeJtmIjp2oZaA0lscmN+KRm3VEScEqz5UbZcXTX0gxLDchkVNIZC+ZFMFSsMudc6XgjbdPtACWPdWyF1sGtuzhkpt+yUqmsZt/yQpCPq/sKZONeznlXDYauBkQWhO5FlDGWDTfNIiMSymGZexIgSynXMppzbKXWDaCY0mhGnuUN+gx2fiVcSjFskyWsnEtG8lymUTD50aPRMn4lTtcxt4RO0ruhL1gwQJ//Mo6QMawPE82sKN3xxeyx5sb66Etk0uHZF6WG2j98MMP/t1iZQzIvQdkp1QsOQsjmkccS8a1rBuilz1E52nJYJViVV5b3kPWHZLJKpc/RC9puuOOO/wzuWRjV3agS4ErO6KluI0dew3fW/pLFqvM+XKKpvyT4lqyVKW4f+GFF/xxLUekZR0khbgc4RZSCMjyUjCjLZMitbFLKRqSyxNke3z27Nn+mRhyWrNsn8oYiT01WsavPBZb5Db2mMzJzz77rF9kyz+5/4fc3FNet7ExF90ZJ9vZck+Rxsh2s3weKZJlu0LunyDrEbmTvRx9FrKDW5aFyxh3vA6SA5WA10ULJhvtMiHL3rPoINuRpFCQI1qyYS97wgAAAACgteCa5HZI9nbJ3bUTUSADAAAAQGvG6dbtUGwsDAAAAADg/3AkGTuc7TppAAAAAGipuCYZAAAAAACDI8kAAAAAABgUyQAAAAAAGBTJAAAAAAA09e7WEmCP9qt3795qe2Fh4U5bltauJUWTN9e4tr1vc31H3bt3V9snTJgQ2HbhhReqfUtKSgLbFi9erPatra0NbMvJyVH7jhs3LrBtzpw5at+bb745sK2qqsprb7+P1rRMzNlA2xvbLXFch12m5vp+Dz300MC2/Px8te/KlSsTsESeN2DAALV93333DWx77rnnErBEbVc8vzuOJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGB0iMR57/WWeNv5RHr33XfV9i5dugS2rV+/Xu170UUXBbYtW7bMa64Yp5kzZwa2paamqn0LCgoC24455hi1b0VFhdeetJc4Ce21w3wHubm5avuVV16pth9xxBGBbcnJyc6/VVvf4cOHB7ZlZmZ6rurq6pyjKoqKitS+2rjfsGGD2veDDz5Q2++9997Ato0bN3qtTUsa1+1xzgbaw9huieN6l130421bt251fu2+ffuq7RdccEFg2zXXXKP2zcrK8lqbLVu2BLZt3rxZ7XvDDTeo7ffcc4/X2n4fYRABBQAAAABAE3C6NQAAAAAABkUyAAAAAAAGRTIAAAAAAAZFMgAAAAAABkUyAAAAAAAGRTIAAAAAAAY5yQHee+89TzN48GDnvFQte7SsrEzt+/zzz6vt55xzTmBbx44d1b7V1dWBbSUlJWrfqqqqwLa99tpL7dvetJfMxTA5ydr4evXVV9W+a9ascf6d2zKHtYzCmpoata+WK5yRkZGw9+3cuXNgW15entq3U6dOTq8bT3tlZWVg29///ne174svvui1NC1pXLfUPFWgNWpJY7u5xrWWdRs253b+/PmBbUOHDlX7pqSkOM0xoqKiwul1xcaNG523l3v16hXYlpaWpvbVPpNWW8SznaFto7zzzjtq37PPPttrib8tDTnJAAAAAAA0AadbAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYJCT7JhHPHbsWOdstq5duzrnlmp5YuKDDz4IbBs1apRzvqyWlyoKCgoC2yZMmKD2bW/IXLR79tlnA9tyc3Ods/5EUlKS899Gy1G25flpeca2rGMt29mWy56dne30XYTN5LStq7QcZdtynXTSSYFt5eXlXnsf14KcZKDtje1EjWvb64b5DmbPnu28Pb169Wq1rzb/2Za5Y8eOzn21PGPb3KfVCFu2bFH7anNjVVWVF4b22rmW7a6XX37Zab5uzt8lOckAAAAAADQBp1sDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYOi5Pu3YDz/8oLYfcMABgW2bN29W+2pxL2Fv779s2bLAtvHjx6t9V61aFdiWmprqfDt8oKFevXqpX0rPnj0D2zZt2uQcLWQbn7bfcXp6unPsgxYRZYt90NpTUlKcl9n2vtp3Zetri2LSYq20ZRaTJk0KbJs2bZraFwCw46J0Tj75ZLV9//33V9tXrlzpvE2sxRbZYhm1z2z7PsrKypyXWdtWsPXV5l3bdrrt+9Dm++XLl6t9jzrqqMC2Y489Vu07Y8aMFhu/xpFkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMcpIDLFq0yNN07NjRc1VRURHYVltbq/YdNWqU8/tWVVWp7Vo+W6dO+k+ltLTUebnQ/nTp0sU5J9mWz2vLSdYyeG0Z58nJyc4ZhNr4CpOPblsXaa8dZpltf4e8vDy1vbi42PlveOSRRwa2kZMMAE2bK2zrc80LL7zgvK4XmZmZgW0lJSVq37q6OuftVi2D1zavalnHicz21V7b9je0LZc23ycpedRi06ZNgW2vv/662rdXr16BbatXr1b7an9j2/ZcPDiSDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEEEVIBVq1Z5rred124Nb7uVelFRkdp3/vz5antZWZnzZ9JueW+LqNFu/w40NcpM+y1q8VDxjD+tvbq6Wu1bWFgY2Jafn6/2XbZsmVMsnG25bH21dZUtakn7Ox1//PFqX9t3mZOTE9iWkZHhHOMFANixMU8vv/yyc0xTeXm52t6/f3/n19ZiDMNEANm2I1oiW8STrV37fXS0RGJp2yG2+NnDDjsssO2ZZ55J2G86Hq3vVwAAAAAAQIJQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGCQk+yQh2rLHrVlCmu5brZs0UWLFjlnMNty37Ss4+TkZLWv7TMDTcm++/DDDwPbzj77bLXvyJEj1fbf/va3gW3ffPONlyhpaWmBbampqWpfrd2WGZySkuKcsTxt2rTAtptuuknt+8knn6jtPXr0CGyrrKxU+w4aNEhtBwDsOAceeKBz386dOztvP4bJwQ2bG9zatnnDft4wf4ckpfbQtkHE2LFjnbcVw/wN48GRZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAAwioAIUFxd7mgEDBjjHyGgxT7bbynfq5P4nq62tTdjt37VILKChP/zhD84xaTNnzlT7fv7552p7VlaW89jVxkhpaanad/369YFtJSUlzuMrTKxDdna22nePPfYIbMvPz1f72qK6ysvLnb4rUVNTo7YDOyquxTa+Onbs6LQei+e1tfl+8+bNXqJocZG2z5QoWsSM7ftIdExMe1BVVeUc8RQmxsk2drW50fab0fratrW135Tt82rjyxbVGuZ9bbTvq8Yy52q/AVvUpLatcO2113rNiSPJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAAY5CQHWL16tefKlnOmZZHZ+tpoGWqJzIzbuHFjHEsH/K8333xT/SomTpwY2DZ58mS171FHHaW2P/7444FtU6dOVfvm5OQEtg0ZMkTtm5GRkZAsVltGpZaPbss8feqppwLbysrK1L433HCD83LZ1iennHJKYNu4cePUvhs2bFDb0fYkMidXy3EN+76JykK2reduvfXWwLY+ffp4zUHbPkF4e+21l9qem5sb2FZaWqr2TUlJcZ4LbH2rq6udt6e1+c82N2rttnEf5n3DsH0f2hjrYMmr7tKli9PfN9GZ72FxJBkAAAAAAIMiGQAAAAAAgyIZAAAAAACDIhkAAAAAAIMiGQAAAAAAgyIZAAAAAACDCChHNTU1rl1DxUKEubX8li1b1L5au+3277YIACDWnXfe6RxFUFhYqPZdvHix2j5p0qTAtl/+8pdeoiJKtHWGbWxq494Wn6DFR9li4bTYKltM07x585xj9mbOnKn2/e677wLbiHjCzoxxSmR8yVlnnRXYtvfee6t9TzvttMC2qqoqtW9xcXFg27Rp05yXOQxb1N31118f2Pab3/wmAUvUtthiPrV5xDZG0tPTE7btqc1htr5aJFKYvrYYpzDva9tWcH1f27qso/L3t/W1LXPfvn29loojyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGOQkO7LloLmy5c3ZMtRs7a59bctVUVHh/L5of1544QW1feLEiYFtY8eOVfvOmDFDbX/llVcC27p37672Xb58uXOOoJbnmJKSEirD0jW/sLKyUu1bW1sb2JaVlaX27d+/v9r+85//3LnvYYcdFtj2+eefq32/+OILtR2tU5j5y9auGTJkiFNWsRg3bpzaftRRRwW25efnq31XrlwZ2FZaWqr2HTBgQGDbj370I685nHnmmWr7/vvvv9OWpS3aZ599nOcv2/ix5fNq84wt0zsjI8PpdW1snylMDaD1tW1HaGx9w7z2Lpa/YWpqamBbWVmZ2re8vNx5XM+dO9dLJI4kAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYBABlaDbobuyRTgl8hbu2i3vt2zZova1RecAsUaMGKF+IVrsw+rVq9W+c+bMUdsPOuigwLaRI0c6j5EwY9MWJ6G9b5hYONsya8tl+zs8/fTTzlFMP/zwg9p3xYoVgW1LlixR+yIc2zyi/WY6d+6s9k1kZIsmJycnsO2OO+5Q+55xxhnOEWtFRUVq+7x585wieWyRLN98843at2/fvoFtt99+u+fKtp2gfZd/+tOf1L7Dhw8PbBszZoza97PPPvPaO9s8oo172/xVV1fnNcdya/GHIjk52XmbV4tltH0fiaofbOtA7fOKTZs2Bbalp6erfbVtiTB/By0qUpx11lleInEkGQAAAAAAgyIZAAAAAACDIhkAAAAAAIMiGQAAAAAAgyIZAAAAAACDIhkAAAAAAIMiGQAAAAAAg5zkBGXKufYNk1VnyyOzvbaW+2bLORswYIDaDsQaNGiQ829Ry/CMJ79Xyy61/c7LysoSMjZtecW2zEZXtuxDLd8yLy9P7WvLiM3MzHT+G2u5tj179lT72jKYEW6OSlQOss3EiRMD2yZPnqz2nTJlSmDb+vXr1b6LFi1yXp9kZWWp7d26dXPKkreNv7FjxzqvQ7XvSlx33XXOy7xgwQLnjNeUlBSn9TbCf0e2uc827rV5Juw2sUZ77URlGSeSLY9a266y5Swnh8hYtn2XNTU1TuN6Z2h9vwIAAAAAABKEIhkAAAAAAIMiGQAAAAAAgyIZAAAAAACDIhkAAAAAAIMiGQAAAAAAgwgoR2FiMLTboYd53bD9tRgaWwQNEVBoClskQHV1tfNv0RZlkZaWFti2detW5zFii3EKEzcRZp2hfSbb+3bu3Nn58xYXF3uuunbt6hxl0bt3b7UvEVB2WhRIouLIxBVXXBHYdskll6h9e/ToEdi2cuVK5+gh2+fV3tfGtr7R/g62sau99rp160JFU2lmzZoV2HbyySc7v+6tt96qtl966aWBbcuXL1f7nnPOOV57d/PNNzvHC9mizmzxQdr63jaPhN1mbm20edcWtWVb32h/p6SkJOftrtTUVLWvFg130kknOf/9tfVnvDiSDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQU5ygGHDhnmu+aG2LDIt49PGlo2oZYbZ8uS0dlsOXm5urtoO7KjfsW18bdiwQW3XMvtsr60td5hMPltfrd02rrV8S1t+pbausv0NV69enbAsbC0rMjMzU+0Lz9tnn33Ur+HII48MbNttt93UvikpKc4Z1hkZGYFtJSUlat9Vq1YFtmVnZzsvs9ZmG5uVlZVqX1v2qDa2bWNEG7u29ZyWW6qNW7HffvsFthUWFjr//W1Z1999911gW1pamtr3oosu8tq7QYMGqe01NTXO84itvaCgwDljN9E5ua2JbVvAlqOsjb9OlrpF+661+dr22suWLXN+3x2BI8kAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABgUyQAAAAAAGBTJAAAAAAAYFMkAAAAAABjkJAfYfffdPY2W2aflksaTjaix5Y3ZctI0Wu6plpEnevToEdg2btw4te+sWbPiWDq0J9rv3JbxuWbNGrXdlrvoyjb2tOW2ZRBqYzNM5nSYPGIbWyajxvaZtOUOs8xtyf/7f/8vsO2UU05xHiNhsjhtc5+WK2x7Xy3j07bOqKiocM5nDpNHbMtg1j6zLXtWGwe2daC2XLa/YWlpaWDb5s2b1b4bN2507qt9JrLT/1efPn2cs6SLi4ud+9rmAm2c2NbnWl/bPKL1tY1dbdzb3ldjm5O1dtv72rbjtTz5Oktdo+WnZ2VlqX21sd2vXz+vOXEkGQAAAAAAgyIZAAAAAACDIhkAAAAAAIMiGQAAAAAAgyIZAAAAAACDIhkAAAAAAIMIqAATJ070NJFIJCGRLNrrxiNMf+1W+7bXzc/PD2ybOnWq2pcIqPYnzO/UFgWjxYjYIkxsy6XFQtiWS4s5sK0ztOUK813aYlW05bJ9XlvMjBatY4vG0YTp25Y8+eSTgW2ffPKJ2leL7Rs5cqTat3///s5RPF26dHGOSQsTjZKXl+fUFja+pnPnzs0SM1NeXu4ciWWL89HWKbbPq8XI2Ppqy2yLvpk+fbrafv3113ttwfjx4537auPL9rex/Wa0v3vXrl3Vvlo0UZj5PMy8GnY7PlFsfwctgm+rJRJLW7fb1t3a37+5Ix05kgwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEFOcoADDjjAc81ms+V6hclJtuWNhaHlLtqyR7WcswMPPDDUcgE7kvZbtmUBamM3TD66TaIyG22vq+Uq2j6vLSf5+++/D2wbPXq083KF+Z7bEu17WLhwodp37ty5zu+bnJwc2DZw4EC175AhQwLbBgwYoPbt3bu38/wVZlxr64zi4uJQecXr1693yhm3tdv6VlVVOWWp2tjydMOMXe271jKUW3Ku7Y6mbbfaaFnTYee+nJwc59fWPlOYsWvrq7Xb8ojDZJyHyQ0Ok1dda+mr5VnbllnLVm9uHEkGAAAAAMCgSAYAAAAAwKBIBgAAAADAoEgGAAAAAMCgSAYAAAAAwKBIBgAAAADAIALKMW5i48aNzrd3DxM3YLuVeqKiDGzvm5aWFtjWs2dP58gQLXYArVdZWZnanp6enpD4BFs0kS3mQBtftvgo19e1RWjY4jW0sWt7Xy1ew/a+tr/T8uXLA9vGjh2r9tXWC2EiMtoSLeZHG1+iV69eCYnp2bBhg9r+3nvvOcc4hYm3CTNGwkQn2n6rWmSSLQ5Se++MjAy1b15eXmBbVlaW2jcpKcn5b6R9Jm0bwzan2N63oKDAaw/ef/99575h5r4tW7Y4jyFbPFCYuUD7TLbxpb227fNq61BbX+19w8592t+hk+X70Nptf8OWHMHGkWQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAIx2nZPcpUuXwLbc3Fy175o1a5yzEbVMMFsGpS1PTMtYs+WWau+t5TWKt956K7DttNNOU/uOGTMmsG3WrFlqX7Rc2m8mTPZoaWlpqOUKk+OpsX0m7fsIk6too+UX2t5Xy7+0fV5bruKyZcuc/ka25bb1hedVVFSEak9ERrntb2f7rWrZv8nJyc7va6Nlk9rmXFt+qOv7hs2pLywsdF4XaePe9j1r34dtfaL1raysdP68bclxxx3n3Le2ttapzZa7bduetr12mGxfbX6zjU1tHITZjrcts/Z5bXOybfxVV1c7r286hchJtq3bmxNHkgEAAAAAMCiSAQAAAAAwKJIBAAAAADAokgEAAAAAMCiSAQAAAAAwKJIBAAAAADDadQTU6NGjnW/hrt2yPMzt322REbZ4KS1mRrvdvW25bLdw32233Zxvab/77rsHthEB1Xppv6cw8UGrVq0KtVxalIFtuWxjKFGREVq7bZm0dZUt1kH7PmyxDZmZmWr7kiVLEhLdESYuC4lVVVUVql2zceNG575Ae3DMMcc499XiEWtqakLNBVOnTg1se+qpp5y3eW1RZ9o8YoueStS8GmYbxBZ1Z6sfsrOzA9vef/99tW///v0D20pKSrxE6dGjR2CbFi0WL44kAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgUCQDAAAAAGBQJAMAAAAAYFAkAwAAAABgtOuc5EmTJgW2FRcXO2fG2XJLtfaMjAy1ry0DNCkpyTkjrbS01Onzip49ezpnLO+5555qO9oeWxaglhceNidZe23bcmnjy5ZxrmUnJip/OWzWcZjMYS1zUXz99dfO36XWTk4yAOzYTOH09PSEzF/ixRdfDGy799571b5Tpkxxzmfu1q1bYFthYaHa15ZJrNG+L9s2iJbfnJubq/a1zfdz584NbLvnnnvUvoceemhCaiKbE044IbDtoYce8sLiSDIAAAAAAAZFMgAAAAAABkUyAAAAAAAGRTIAAAAAAAZFMgAAAAAABkUyAAAAAABGu46AGjx4sPOt47XII1t8yYYNG5xe1xZbJV577bXAtqqqKrVvWlqaczyAa3SA2GOPPZxfG+0vAmr58uWh3rumpiawbd26dWpfbRzYos40YaKYbOsbra8tLkmLubBFytnGvRblZVsuLTKiU6d2Pa0BQJPnXds2b0lJSbN8qzfeeGOodle2+U37vsLEMoaJgNJiXJtTB8v3oc3ZtrpFq4mIgAIAAAAAYAfidGsAAAAAAAyKZAAAAAAADIpkAAAAAAAMimQAAAAAAAyKZAAAAAAADIpkAAAAAACMdh0oqWUKH3bYYc6vq2V4itTUVOfXLi8vd+5ry3HV8tfC5LxWV1erfRcsWOD8vmi5wuTzasJmAWrZv1qbqKurC2zr2rWr8xixjc0w31eYjGXtu7blIPfu3Vtt19YLnTt3ds5VtPUFgPbowgsvDGybPHmy2jctLc15HtHmvpbKtt1qa29vli5dGtiWl5fnnMFty6v++OOPvUTiSDIAAAAAAAZFMgAAAAAABkUyAAAAAAAGRTIAAAAAAAZFMgAAAAAABkUyAAAAAABGu46AeuihhwLbHnzwQedYleLi4lARUYnqa1uu7Oxsp+gbkZmZGdiWlZWl9r3nnnvUdrROHTt2dI4b0yKRbHETNs8//7zzb3Xt2rVOsUTxxDxptNe2xUNp7bb1ibbMmzZtUvt++umnarvr+yb69wEAbZEWtdO/f3/nqB1t21FMmzbNa4m0ucI2j2jtkUjEeZnC9LXN57Z2bVshYlmuN9980yl6zFY/TJ8+Xe37+9//3ksktiYAAAAAADAokgEAAAAAMCiSAQAAAAAwKJIBAAAAADAokgEAAAAAMCiSAQAAAAAwKJIBAAAAADDadU6yZs8991TbFyxY4PzaNTU1zn27d+/u3LdHjx5qe2pqqnMGrJZzdvTRR6t9CwoK1Ha0TtrvyZbtq2UQ5uTkhFqu3/3ud6H6I/FsmYyJ/H0AQHuzfPlytT05Odlp+0/07dvXebnS09PV9oqKCufX1nKDbZnCbVHHjh0D2zZv3qz2/eKLLwLb6urq1L4ZGRmBbffff7/XnDiSDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEEEVICFCxd6Gi3C5uCDD1b7jhgxIrBtwoQJat+PP/7Yc2W7lboWL/XMM8+ofWfMmOG8XGibNmzYENi2ZMkSte/KlSsD2+bOnRtquWzxU2GiibBj/Otf/1LbBw0aFNg2f/58/gwAsAPnxeuuu85prhdFRUXNEpmKnbd9s3bt2sC2qqoqtW9tbW2LjeLiSDIAAAAAAAZFMgAAAAAABkUyAAAAAAAGRTIAAAAAAAZFMgAAAAAABkUyAAAAAAAGRTIAAAAAAEaHCMGfAAAAAAD4OJIMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACAQZEMAAAAAIBBkQwAAAAAgEGRDAAAAACA97/+P6thHH/DXOjeAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "figure, axes = plt.subplots(2, 4, figsize=(10, 5))\n", "\n", "for index, axis in enumerate(axes.flat):\n", " image, target = training_source[index]\n", " axis.imshow(image.squeeze(0), cmap=\"gray\")\n", " axis.set_title(class_names[target])\n", " axis.axis(\"off\")\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "db511674", "metadata": {}, "source": [ ":::{admonition} Quiz\n", "\n", "Does the FashionMNIST `Dataset` decide that training batches should contain 128 examples?\n", "\n", "
Answer\n", "No. The dataset defines access to individual examples. Batch size is a responsibility of the `DataLoader`.\n", "
\n", ":::" ] }, { "cell_type": "markdown", "id": "57f4eb15", "metadata": {}, "source": [ "### 1.3 Split the dataset\n", "\n", "The original FashionMNIST training portion contains 60,000 examples. We will use 54,000 for parameter updates and reserve 6,000 for validation. The validation examples will not contribute gradients, but their loss will guide model-checkpoint selection.\n", "\n", "### Exercise 1\n", "\n", "Complete the split. The two subset sizes must add up to the size of `training_source`.\n", "\n", "*Hint:* Use `torch.utils.data.random_split` to create the two subsets. Pass the seeded generator to make the split reproducible." ] }, { "cell_type": "code", "execution_count": null, "id": "acb40e70", "metadata": {}, "outputs": [], "source": [ "training_size = 54_000\n", "validation_size = 6_000\n", "\n", "split_generator = torch.Generator().manual_seed(SEED)\n", "\n", "# TODO: divide training_source into training and validation subsets\n", "training_dataset, validation_dataset = ...\n", "\n", "print(\"Training examples:\", len(training_dataset))\n", "print(\"Validation examples:\", len(validation_dataset))\n", "print(\"Test examples:\", len(test_dataset))" ] }, { "cell_type": "markdown", "id": "de096b66", "metadata": {}, "source": [ "### Checkpoint\n", "\n", "Encode the assumptions on which the remainder of the lab depends. In addition to checking sizes and sample structure, verify that the index collections of the training and validation subsets are disjoint." ] }, { "cell_type": "code", "execution_count": null, "id": "cf15adec", "metadata": {}, "outputs": [], "source": [ "training_image, training_target = training_dataset[0]\n", "validation_image, validation_target = validation_dataset[0]\n", "\n", "# TODO: assert that the subset sizes have the expected values\n", "# TODO: assert that the two subset sizes add up to len(training_source)\n", "# TODO: assert that the training and validation index sets are disjoint\n", "# TODO: assert that an image has shape (1, 28, 28) and dtype torch.float32\n", "# TODO: assert that a target is an integer in the range [0, 10)" ] }, { "cell_type": "markdown", "id": "3da82116", "metadata": {}, "source": [ "The test set is already separate because FashionMNIST distributes it independently. Nothing in this lab should use its performance to change the model, optimizer, training duration, or checkpoint rule.\n", "\n", "### 1.4 Construct data loaders\n", "\n", "A data loader forms batches and provides an iterator over them. The training loader should shuffle before each epoch so that the batch composition changes. Validation and test loaders should use a stable order because no parameter updates occur and a consistent order helps later inspection.\n", "\n", "We will use a batch size of 128. None of the three dataset sizes is divisible by 128, so each loader will produce a smaller final batch. This gives us a concrete reason to aggregate metrics by example count rather than taking an unweighted mean of batch means.\n", "\n", ":::{admonition} Reflection\n", "\n", "Before constructing the loaders, calculate the expected number of batches and the expected final-batch size for each partition. Record your predictions, then compare them with the observed values.\n", "\n", "Recall that the number of batches is the dataset size divided by batch size and rounded upward.\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "8b7b38fc", "metadata": {}, "source": [ "### Exercise 2\n", "\n", "Create the validation and test loaders." ] }, { "cell_type": "code", "execution_count": null, "id": "b3bf2b01", "metadata": {}, "outputs": [], "source": [ "BATCH_SIZE = 128\n", "\n", "shuffle_generator = torch.Generator().manual_seed(SEED)\n", "\n", "training_loader = DataLoader(\n", " training_dataset,\n", " batch_size=BATCH_SIZE,\n", " shuffle=True,\n", " generator=shuffle_generator,\n", " num_workers=0,\n", ")\n", "\n", "# TODO: create a non-shuffled validation loader\n", "validation_loader = ...\n", "\n", "# TODO: create a non-shuffled test loader\n", "test_loader = ...\n", "\n", "print(\"Training batches:\", len(training_loader))\n", "print(\"Validation batches:\", len(validation_loader))\n", "print(\"Test batches:\", len(test_loader))" ] }, { "cell_type": "markdown", "id": "4b3794a4", "metadata": {}, "source": [ "Inspect the first training batch." ] }, { "cell_type": "code", "execution_count": null, "id": "943a86fb", "metadata": {}, "outputs": [], "source": [ "first_training_batch = next(iter(training_loader))\n", "batch_images, batch_targets = first_training_batch\n", "\n", "print(\"Image batch shape:\", batch_images.shape)\n", "print(\"Target batch shape:\", batch_targets.shape)\n", "print(\"Image dtype:\", batch_images.dtype)\n", "print(\"Target dtype:\", batch_targets.dtype)" ] }, { "cell_type": "markdown", "id": "9b36648d", "metadata": {}, "source": [ "Now iterate through one validation epoch and record the size of every batch. Because validation is not shuffled, this inspection does not affect any later training order." ] }, { "cell_type": "code", "execution_count": null, "id": "df0dea6d", "metadata": {}, "outputs": [], "source": [ "validation_batch_sizes = []\n", "\n", "for images, targets in validation_loader:\n", " validation_batch_sizes.append(targets.shape[0])\n", "\n", "print(\"First validation batch:\", validation_batch_sizes[0])\n", "print(\"Final validation batch:\", validation_batch_sizes[-1])\n", "print(\"Validation examples observed:\", sum(validation_batch_sizes))" ] }, { "cell_type": "markdown", "id": "69de6d64", "metadata": {}, "source": [ "### Checkpoint\n", "\n", "Verify the batch contract." ] }, { "cell_type": "code", "execution_count": null, "id": "e40bb7c3", "metadata": {}, "outputs": [], "source": [ "# TODO: assert that the first image batch has shape (BATCH_SIZE, 1, 28, 28)\n", "# TODO: assert that the first target batch has shape (BATCH_SIZE,)\n", "# TODO: assert that the target dtype is torch.int64\n", "# TODO: assert that len(validation_loader) equals the rounded-up batch count\n", "# TODO: assert that the final validation batch is smaller than BATCH_SIZE\n", "# TODO: assert that the observed validation examples equal len(validation_dataset)" ] }, { "cell_type": "markdown", "id": "4a33844f", "metadata": {}, "source": [ ":::{admonition} Quiz\n", "\n", "If the training loader contains 422 batches, how many parameter updates occur during one epoch? How many occur during five epochs?\n", "\n", "
Answer\n", "Each training batch produces one parameter update, so one epoch contains 422 updates. Five epochs contain 2,110 updates.\n", "
\n", ":::" ] }, { "cell_type": "markdown", "id": "907b5c2a", "metadata": {}, "source": [ "\n", "---\n", "\n", "## 2. Model Definition\n", "\n", "The data pipeline is now able to produce batches with a stable contract. Each image batch has shape `(batch_size, 1, 28, 28)`, and each target batch contains one class index per image.\n", "\n", "The model will flatten each image into 784 pixel values, construct a hidden representation, and return ten logits. This architecture is intentionally modest. The aim is to train it through a reliable workflow, not to optimize FashionMNIST performance." ] }, { "cell_type": "markdown", "id": "e92ec1ae", "metadata": {}, "source": [ "### Exercise 3\n", "\n", "Implement the model. The final layer must return raw logits because `nn.CrossEntropyLoss` performs the required normalization internally." ] }, { "cell_type": "code", "execution_count": null, "id": "74017863", "metadata": {}, "outputs": [], "source": [ "class FashionClassifier(nn.Module):\n", " \n", " def __init__(self, hidden_width: int = 128):\n", " super().__init__()\n", "\n", " self.network = nn.Sequential(\n", " # TODO: flatten each image into 784 values\n", " ...,\n", " # TODO: map 784 inputs to hidden_width values\n", " ...,\n", " # TODO: apply ReLU\n", " ...,\n", " # TODO: produce 10 class logits\n", " ...,\n", " )\n", "\n", " def forward(self, images):\n", " return self.network(images)" ] }, { "cell_type": "markdown", "id": "50c04cf3", "metadata": {}, "source": [ "Create a model on the selected device and inspect one forward pass." ] }, { "cell_type": "code", "execution_count": null, "id": "0f9a9c9d", "metadata": {}, "outputs": [], "source": [ "torch.manual_seed(SEED)\n", "model = FashionClassifier(hidden_width=128).to(device)\n", "\n", "sample_images = batch_images[:8].to(device)\n", "sample_logits = model(sample_images)\n", "\n", "print(model)\n", "print(\"Input shape:\", sample_images.shape)\n", "print(\"Logit shape:\", sample_logits.shape)\n", "print(\"Model device:\", next(model.parameters()).device)" ] }, { "cell_type": "markdown", "id": "242154f4", "metadata": {}, "source": [ "### Checkpoint\n", "\n", "Verify the model contract." ] }, { "cell_type": "code", "execution_count": null, "id": "0b35286f", "metadata": {}, "outputs": [], "source": [ "# TODO: assert that sample_logits has shape (8, 10)\n", "# TODO: assert that sample_logits is floating point\n", "# TODO: assert that every model parameter is on device\n", "# TODO: assert that the model has at least one trainable parameter" ] }, { "cell_type": "markdown", "id": "b2a2ed11", "metadata": {}, "source": [ "The forward pass preserves the batch dimension. Every row of `sample_logits` describes one image, while every column corresponds to one FashionMNIST class." ] }, { "cell_type": "markdown", "id": "1e8dba73", "metadata": {}, "source": [ "\n", "---\n", "\n", "## 3. Training\n", "\n", "Training with mini-batches preserves the same learning cycle used in the previous labs. The difference is that one update now reflects only the examples in the current batch. The next batch is therefore processed by a slightly different model.\n", "\n", "We will first implement and inspect one update. Only after that operation is trustworthy will we place it inside an epoch loop.\n", "\n", "### 3.1 Train one batch\n", "\n", "The batch-level function should perform one complete update and return the information required for epoch aggregation. The returned loss is a **mean over the current batch**. The number of correct predictions and the batch size are counts, which can be added directly across batches.\n", "\n", "We clear gradients before backpropagation so that each call to `train_batch` is self-contained and cannot accidentally reuse gradients left by an earlier operation.\n", "\n", "### Exercise 4\n", "\n", "Implement the `train_batch` function." ] }, { "cell_type": "code", "execution_count": null, "id": "6582f639", "metadata": {}, "outputs": [], "source": [ "def train_batch(model, batch, loss_fn, optimizer, device):\n", " images, targets = batch\n", "\n", " # TODO: move images and targets to device\n", "\n", " # TODO: clear old gradients\n", "\n", " # TODO: calculate logits and mean batch loss\n", "\n", " # TODO: calculate gradients and update the parameters\n", "\n", " # TODO: count correct predictions in this batch\n", " correct = ...\n", "\n", " return {\n", " \"loss\": ...,\n", " \"correct\": ...,\n", " \"n_examples\": ...,\n", " }" ] }, { "cell_type": "markdown", "id": "2afef902", "metadata": {}, "source": [ "Use a fresh model to inspect one update without changing the model that will later be trained for several epochs." ] }, { "cell_type": "code", "execution_count": null, "id": "7010710d", "metadata": {}, "outputs": [], "source": [ "torch.manual_seed(SEED)\n", "\n", "update_model = FashionClassifier(hidden_width=128).to(device)\n", "update_loss_fn = nn.CrossEntropyLoss()\n", "update_optimizer = torch.optim.Adam(\n", " update_model.parameters(),\n", " lr=1e-3,\n", ")\n", "\n", "parameters_before = {\n", " name: parameter.detach().clone()\n", " for name, parameter in update_model.named_parameters()\n", "}\n", "\n", "update_model.train()\n", "\n", "update_result = train_batch(\n", " update_model,\n", " first_training_batch,\n", " update_loss_fn,\n", " update_optimizer,\n", " device,\n", ")\n", "\n", "print(update_result)" ] }, { "cell_type": "markdown", "id": "796fd4aa", "metadata": {}, "source": [ "### Checkpoint\n", "\n", "A successful function call should produce a finite loss, gradients for trainable parameters, and at least one changed parameter." ] }, { "cell_type": "code", "execution_count": null, "id": "9983f3bc", "metadata": {}, "outputs": [], "source": [ "parameters_changed = any(\n", " not torch.equal(parameters_before[name], parameter.detach())\n", " for name, parameter in update_model.named_parameters()\n", ")\n", "\n", "gradients_exist = all(\n", " parameter.grad is not None\n", " for parameter in update_model.parameters()\n", ")\n", "\n", "gradients_are_finite = all(\n", " torch.isfinite(parameter.grad).all().item()\n", " for parameter in update_model.parameters()\n", " if parameter.grad is not None\n", ")\n", "\n", "# TODO: assert that the returned loss is finite\n", "# TODO: assert that n_examples equals the size of first_training_batch\n", "# TODO: assert that correct is between zero and n_examples\n", "# TODO: assert that gradients exist and are finite\n", "# TODO: assert that at least one parameter changed" ] }, { "cell_type": "markdown", "id": "13517c08", "metadata": {}, "source": [ "This checkpoint does not establish that the model has learned the complete task. It establishes something more local: one batch can produce a connected loss, gradients, and an optimizer update.\n", "\n", "### 3.2 Train one epoch\n", "\n", "An epoch processes every batch from the training loader. Because the model changes after each batch, the training loss for an epoch summarizes measurements made at several parameter states. This is appropriate for monitoring the optimization process, but it differs from evaluation, where all batches are measured using one fixed model state.\n", "\n", "The epoch function must weight each mean batch loss by the number of examples in that batch:\n", "\n", "$$\n", "L_{\\text{epoch}}\n", "=\n", "\\frac{\\sum_b B_b L_b}{\\sum_b B_b}.\n", "$$\n", "\n", "Accuracy can be accumulated by counting every correct prediction and dividing by the total number of examples." ] }, { "cell_type": "markdown", "id": "860b6655", "metadata": {}, "source": [ "### Exercise 5\n", "\n", "Implement the `train_epoch` function. It should call `train_batch` to process a batch." ] }, { "cell_type": "code", "execution_count": null, "id": "5738a4e0", "metadata": {}, "outputs": [], "source": [ "def train_epoch(model, loader, loss_fn, optimizer, device):\n", " # TODO: place the model in training mode\n", "\n", " total_loss = 0.0\n", " total_correct = 0\n", " total_examples = 0\n", " total_batches = 0\n", "\n", " for batch in loader:\n", " # TODO: perform one batch update\n", " batch_result = ...\n", "\n", " # TODO: add the weighted loss contribution\n", " # TODO: add correct predictions and example count\n", " # TODO: count the batch\n", "\n", " return {\n", " \"loss\": ...,\n", " \"accuracy\": ...,\n", " \"n_examples\": ...,\n", " \"n_batches\": ...,\n", " }" ] }, { "cell_type": "markdown", "id": "67e2c824", "metadata": {}, "source": [ "Test the function on a fresh model for one complete epoch." ] }, { "cell_type": "code", "execution_count": null, "id": "17016ef4", "metadata": {}, "outputs": [], "source": [ "torch.manual_seed(SEED)\n", "\n", "epoch_model = FashionClassifier(hidden_width=128).to(device)\n", "epoch_loss_fn = nn.CrossEntropyLoss()\n", "epoch_optimizer = torch.optim.Adam(\n", " epoch_model.parameters(),\n", " lr=1e-3,\n", ")\n", "\n", "one_epoch_result = train_epoch(\n", " epoch_model,\n", " training_loader,\n", " epoch_loss_fn,\n", " epoch_optimizer,\n", " device,\n", ")\n", "\n", "print(one_epoch_result)" ] }, { "cell_type": "markdown", "id": "19ce4fa9", "metadata": {}, "source": [ "### Checkpoint\n", "\n", "Verify epoch accounting." ] }, { "cell_type": "code", "execution_count": null, "id": "0f6f86db", "metadata": {}, "outputs": [], "source": [ "# TODO: assert that n_examples equals len(training_dataset)\n", "# TODO: assert that n_batches equals len(training_loader)\n", "# TODO: assert that loss is finite\n", "# TODO: assert that accuracy lies between zero and one" ] }, { "cell_type": "markdown", "id": "8cf0d94b", "metadata": {}, "source": [ "### 3.3 Compare aggregation\n", "\n", "The smaller final batch creates a subtle problem. Each batch loss is already a mean. If we calculate the ordinary mean of those batch means, the final batch receives the same influence as a full batch even though it represents fewer examples.\n", "\n", "The controlled example below uses three batches. The last batch contains only 16 examples and has a relatively high loss." ] }, { "cell_type": "markdown", "id": "3cf65743", "metadata": {}, "source": [ "\n", "### Exercise 6\n", "\n", "Calculate both epoch losses. The first is the ordinary mean of the three batch means, while the second is the weighted mean that accounts for the different batch sizes." ] }, { "cell_type": "code", "execution_count": null, "id": "2b4f1f4f", "metadata": {}, "outputs": [], "source": [ "example_batch_losses = np.array([0.42, 0.51, 1.35])\n", "example_batch_sizes = np.array([128, 128, 16])\n", "\n", "# TODO: calculate the unweighted mean of the three batch losses\n", "unweighted_mean = ...\n", "\n", "# TODO: calculate the mean weighted by batch size\n", "weighted_mean = ...\n", "\n", "print(f\"Unweighted mean: {unweighted_mean:.4f}\")\n", "print(f\"Weighted mean: {weighted_mean:.4f}\")" ] }, { "cell_type": "markdown", "id": "006e4099", "metadata": {}, "source": [ ":::{admonition} Analysis\n", "\n", "Explain why the two values differ. Identify which calculation gives every example equal influence, and connect that reasoning to the `total_loss += batch_loss * batch_size` operation in `train_epoch`.\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "11d902dc", "metadata": {}, "source": [ "\n", "---\n", "\n", "## 4. Evaluation\n", "\n", "Evaluation processes batches without changing the model. It therefore measures one fixed parameter state across the complete validation set. The function resembles `train_epoch`, but the missing operations are as important as the shared ones: there is no gradient clearing, no backpropagation, and no optimizer step.\n", "\n", "A correct evaluation procedure uses both `model.eval()` and `torch.inference_mode()`. Evaluation mode changes the behaviour of layers such as dropout and batch normalization. Inference mode disables gradient recording and avoids storing the intermediate information required for backpropagation.\n", "\n", "### Exercise 7\n", "\n", "Implement the `evaluate` function." ] }, { "cell_type": "code", "execution_count": null, "id": "5b9d27af", "metadata": {}, "outputs": [], "source": [ "def evaluate(model, loader, loss_fn, device):\n", " # TODO: place the model in evaluation mode\n", "\n", " total_loss = 0.0\n", " total_correct = 0\n", " total_examples = 0\n", " total_batches = 0\n", "\n", " # TODO: disable gradient recording\n", " with ...:\n", " for images, targets in loader:\n", " # TODO: move the batch to device\n", " # TODO: calculate logits and loss\n", " # TODO: accumulate weighted loss, correct predictions, and counts\n", " pass\n", "\n", " return {\n", " \"loss\": ...,\n", " \"accuracy\": ...,\n", " \"n_examples\": ...,\n", " \"n_batches\": ...,\n", " }" ] }, { "cell_type": "markdown", "id": "dba2dae2", "metadata": {}, "source": [ "Test the function with a fresh model. No optimizer is required because evaluation must not update parameters." ] }, { "cell_type": "code", "execution_count": null, "id": "e6e3ad1e", "metadata": {}, "outputs": [], "source": [ "torch.manual_seed(SEED)\n", "\n", "evaluation_model = FashionClassifier(hidden_width=128).to(device)\n", "evaluation_loss_fn = nn.CrossEntropyLoss()\n", "\n", "parameters_before_evaluation = {\n", " name: parameter.detach().clone()\n", " for name, parameter in evaluation_model.named_parameters()\n", "}\n", "\n", "for parameter in evaluation_model.parameters():\n", " parameter.grad = None\n", "\n", "validation_result = evaluate(\n", " evaluation_model,\n", " validation_loader,\n", " evaluation_loss_fn,\n", " device,\n", ")\n", "\n", "print(validation_result)" ] }, { "cell_type": "markdown", "id": "7225fe1a", "metadata": {}, "source": [ "### Checkpoint\n", "\n", "Verify the evaluation behaviour." ] }, { "cell_type": "code", "execution_count": null, "id": "9d434b74", "metadata": {}, "outputs": [], "source": [ "parameters_unchanged = all(\n", " torch.equal(parameters_before_evaluation[name], parameter.detach())\n", " for name, parameter in evaluation_model.named_parameters()\n", ")\n", "\n", "gradients_remain_absent = all(\n", " parameter.grad is None\n", " for parameter in evaluation_model.parameters()\n", ")\n", "\n", "# TODO: assert that every validation example and batch was counted\n", "# TODO: assert that loss and accuracy are finite\n", "# TODO: assert that the parameters remained unchanged\n", "# TODO: assert that evaluation did not create gradients" ] }, { "cell_type": "markdown", "id": "1b408414", "metadata": {}, "source": [ ":::{admonition} Quiz\n", "\n", "Why is `torch.inference_mode()` not a replacement for `model.eval()`?\n", "\n", "
Answer\n", "Inference mode disables gradient tracking. It does not change the operating mode of layers such as dropout or batch normalization. Evaluation mode and inference mode have different responsibilities.\n", "
\n", ":::" ] }, { "cell_type": "markdown", "id": "6cf05f26", "metadata": {}, "source": [ "\n", "---\n", "\n", "## 5. Training and Monitoring\n", "\n", "The lower-level functions now have separate responsibilities. `train_batch` performs one update, `train_epoch` processes the full training loader, and `evaluate` measures a loader without updates. The final workflow should coordinate these functions rather than reproduce their internal code.\n", "\n", "After each epoch, the workflow will record training and validation metrics. Whenever validation loss improves, it will preserve an independent copy of the model state. At the end, it will restore the best state rather than leaving the model at the final epoch automatically." ] }, { "cell_type": "markdown", "id": "05cefe96", "metadata": {}, "source": [ "### Exercise 8\n", "\n", "Implement the `train_model` function. It should call `train_epoch` and `evaluate` to process the training and validation loaders." ] }, { "cell_type": "code", "execution_count": null, "id": "7b68b9ee", "metadata": {}, "outputs": [], "source": [ "def train_model(\n", " model,\n", " training_loader,\n", " validation_loader,\n", " loss_fn,\n", " optimizer,\n", " n_epochs,\n", " device,\n", "):\n", " history = {\n", " \"train_loss\": [],\n", " \"train_accuracy\": [],\n", " \"valid_loss\": [],\n", " \"valid_accuracy\": [],\n", " }\n", "\n", " best_valid_loss = float(\"inf\")\n", " best_epoch = None\n", " best_state = None\n", "\n", " for epoch in range(1, n_epochs + 1):\n", " # TODO: train for one epoch\n", " train_result = ...\n", "\n", " # TODO: evaluate on validation data\n", " valid_result = ...\n", "\n", " # TODO: append all four metrics to history\n", "\n", " # TODO: save an independent state copy when validation loss improves\n", "\n", " print(\n", " f\"Epoch {epoch:02d}/{n_epochs} | \"\n", " f\"train loss {train_result['loss']:.4f} | \"\n", " f\"train acc {train_result['accuracy']:.3f} | \"\n", " f\"valid loss {valid_result['loss']:.4f} | \"\n", " f\"valid acc {valid_result['accuracy']:.3f}\"\n", " )\n", "\n", " # TODO: restore the best state\n", "\n", " checkpoint = {\n", " \"epoch\": best_epoch,\n", " \"valid_loss\": best_valid_loss,\n", " }\n", "\n", " return history, checkpoint" ] }, { "cell_type": "markdown", "id": "afd1753f", "metadata": {}, "source": [ ":::{admonition} Quiz\n", "\n", "Why is `best_model = model` insufficient when validation loss improves?\n", "\n", "
Answer\n", "The assignment creates another reference to the same model object. Later optimizer updates would continue changing it. `copy.deepcopy(model.state_dict())` preserves an independent copy of the parameter state at that epoch.\n", "
\n", ":::" ] }, { "cell_type": "markdown", "id": "9c93e6dc", "metadata": {}, "source": [ "### 5.1 Train the model\n", "\n", "Create a fresh model, loss function, and optimizer. Five epochs are enough to exercise the workflow and produce meaningful curves, but they are not intended to exhaustively optimize FashionMNIST performance." ] }, { "cell_type": "code", "execution_count": null, "id": "74e5f24c", "metadata": {}, "outputs": [], "source": [ "torch.manual_seed(SEED)\n", "\n", "model = FashionClassifier(hidden_width=128).to(device)\n", "loss_fn = nn.CrossEntropyLoss()\n", "optimizer = torch.optim.Adam(\n", " model.parameters(),\n", " lr=1e-3,\n", ")\n", "\n", "N_EPOCHS = 5\n", "\n", "history, checkpoint = train_model(\n", " model,\n", " training_loader,\n", " validation_loader,\n", " loss_fn,\n", " optimizer,\n", " N_EPOCHS,\n", " device,\n", ")\n", "\n", "print(\"Selected checkpoint:\", checkpoint)" ] }, { "cell_type": "markdown", "id": "667eb4ca", "metadata": {}, "source": [ "### 5.2 Plot the curves\n", "\n", "A final metric would tell us where training ended, but it would not show how the model reached that point. Plotting both training and validation measurements preserves the evolution of the process." ] }, { "cell_type": "code", "execution_count": null, "id": "c22884f1", "metadata": {}, "outputs": [], "source": [ "epochs = np.arange(1, N_EPOCHS + 1)\n", "\n", "plt.figure(figsize=(8, 5))\n", "plt.plot(epochs, history[\"train_loss\"], marker=\"o\", label=\"training\")\n", "plt.plot(epochs, history[\"valid_loss\"], marker=\"o\", label=\"validation\")\n", "plt.xlabel(\"Epoch\")\n", "plt.ylabel(\"Cross-entropy loss\")\n", "plt.title(\"Loss history\")\n", "plt.xticks(epochs)\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "067283dd", "metadata": {}, "outputs": [], "source": [ "plt.figure(figsize=(8, 5))\n", "plt.plot(epochs, history[\"train_accuracy\"], marker=\"o\", label=\"training\")\n", "plt.plot(epochs, history[\"valid_accuracy\"], marker=\"o\", label=\"validation\")\n", "plt.xlabel(\"Epoch\")\n", "plt.ylabel(\"Accuracy\")\n", "plt.title(\"Accuracy history\")\n", "plt.xticks(epochs)\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "f9272a5b", "metadata": {}, "source": [ ":::{admonition} Analysis\n", "\n", "Describe what happened without yet trying to diagnose every possible cause. Did training and validation performance improve? At which epoch was the selected checkpoint saved? Was the final epoch also the best validation epoch? Did loss and accuracy tell a consistent story?\n", "\n", "Lesson 4 will develop the deeper distinction between optimization failure, underfitting, overfitting, and distribution mismatch. Here, the objective is to produce trustworthy evidence for that later analysis.\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "6c001aef", "metadata": {}, "source": [ "### Checkpoint\n", "\n", "The `train_model` function should have restored the model state associated with `checkpoint[\"valid_loss\"]`. Re-evaluate the validation set and compare the result with the stored value." ] }, { "cell_type": "code", "execution_count": null, "id": "56855348", "metadata": {}, "outputs": [], "source": [ "restored_validation_result = evaluate(\n", " model,\n", " validation_loader,\n", " loss_fn,\n", " device,\n", ")\n", "\n", "history_lengths = {\n", " key: len(values)\n", " for key, values in history.items()\n", "}\n", "\n", "# TODO: assert that every history list contains N_EPOCHS values\n", "# TODO: assert that checkpoint[\"epoch\"] lies between 1 and N_EPOCHS\n", "# TODO: assert that the restored validation loss matches checkpoint[\"valid_loss\"]\n", "# TODO: assert that all recorded values are finite" ] }, { "cell_type": "markdown", "id": "5014e962", "metadata": {}, "source": [ "\n", "---\n", "\n", "## 6. Final Evaluation\n", "\n", "The training and validation results have now determined the model parameters and selected checkpoint. Only at this point should the test set enter the workflow.\n", "\n", "The test result provides information about the selected model on examples that did not influence parameter updates or checkpoint choice. It should be evaluated once and reported, not used to restart model development within this lab.\n", "\n", "### Exercise 9\n", "\n", "Perform the final test evaluation. Report the loss and accuracy." ] }, { "cell_type": "code", "execution_count": null, "id": "7144bfcb", "metadata": {}, "outputs": [], "source": [ "# TODO: evaluate the restored model on test_loader\n", "final_test_result = ...\n", "\n", "print(f\"Selected epoch: {checkpoint['epoch']}\")\n", "print(f\"Validation loss at selection: {checkpoint['valid_loss']:.4f}\")\n", "print(f\"Test loss: {final_test_result['loss']:.4f}\")\n", "print(f\"Test accuracy: {final_test_result['accuracy']:.3f}\")" ] }, { "cell_type": "markdown", "id": "110ab3c9", "metadata": {}, "source": [ "Aggregate metrics summarize thousands of decisions. Inspecting individual predictions reconnects those numbers to the underlying task." ] }, { "cell_type": "code", "execution_count": null, "id": "8ef61f04", "metadata": {}, "outputs": [], "source": [ "model.eval()\n", "\n", "test_images, test_targets = next(iter(test_loader))\n", "\n", "with torch.inference_mode():\n", " test_logits = model(test_images.to(device))\n", " test_predictions = test_logits.argmax(dim=1).cpu()\n", "\n", "figure, axes = plt.subplots(2, 5, figsize=(12, 5))\n", "\n", "for index, axis in enumerate(axes.flat):\n", " image = test_images[index].squeeze(0)\n", " true_name = class_names[test_targets[index].item()]\n", " predicted_name = class_names[test_predictions[index].item()]\n", "\n", " axis.imshow(image, cmap=\"gray\")\n", " axis.set_title(f\"true: {true_name}\\npred: {predicted_name}\")\n", " axis.axis(\"off\")\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "a7d30192", "metadata": {}, "source": [ "Find several incorrect predictions from the first portion of the test set. The code below collects examples without changing the model." ] }, { "cell_type": "code", "execution_count": null, "id": "4bc28c50", "metadata": {}, "outputs": [], "source": [ "incorrect_examples = []\n", "\n", "model.eval()\n", "with torch.inference_mode():\n", " for images, targets in test_loader:\n", " logits = model(images.to(device))\n", " predictions = logits.argmax(dim=1).cpu()\n", "\n", " incorrect_indices = torch.nonzero(\n", " predictions != targets,\n", " as_tuple=False,\n", " ).flatten()\n", "\n", " for index in incorrect_indices:\n", " incorrect_examples.append(\n", " (\n", " images[index],\n", " targets[index].item(),\n", " predictions[index].item(),\n", " )\n", " )\n", "\n", " if len(incorrect_examples) == 10:\n", " break\n", "\n", " if len(incorrect_examples) == 10:\n", " break\n", "\n", "print(\"Incorrect examples collected:\", len(incorrect_examples))" ] }, { "cell_type": "code", "execution_count": null, "id": "b8e83224", "metadata": {}, "outputs": [], "source": [ "figure, axes = plt.subplots(2, 5, figsize=(12, 5))\n", "\n", "for axis, (image, target, prediction) in zip(axes.flat, incorrect_examples):\n", " axis.imshow(image.squeeze(0), cmap=\"gray\")\n", " axis.set_title(\n", " f\"true: {class_names[target]}\\n\"\n", " f\"pred: {class_names[prediction]}\"\n", " )\n", " axis.axis(\"off\")\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "90de4944", "metadata": {}, "source": [ ":::{admonition} Analysis\n", "\n", "Describe several mistakes visible in the grid. Are any pairs of categories visually similar? Do the mistakes suggest that the workflow is broken, or are they plausible classification errors for a simple MLP? Keep the conclusion limited to the evidence available here; a systematic diagnosis belongs to the next lesson.\n", "\n", "::" ] }, { "cell_type": "markdown", "id": "0eec9367", "metadata": {}, "source": [ "\n", "---\n", "\n", "## 7. Conclusion\n", "\n", "You have implemented the responsibilities that a higher-level training abstraction must coordinate. You can now implement a complete training workflow without repeating the same operations in multiple places. Alternatively, you can use the course [`Trainer`](../code/training.py) to handle the same responsibilities. \n", "\n", "The following table summarizes the key components and their responsibilities.\n", "\n", "| Function or object | Responsibility |\n", "|---|---|\n", "| `Dataset` | Define access to individual examples |\n", "| `DataLoader` | Form batches, shuffle training data, and provide iteration |\n", "| `train_batch` | Forward pass, loss, backward pass, and optimizer update |\n", "| `train_epoch` | Process all training batches and aggregate metrics |\n", "| `evaluate` | Measure one fixed model state without gradients or updates |\n", "| `train_model` | Coordinate epochs, record history, and retain the best state |\n", "| `checkpoint` | Identify the validation-selected model state |" ] } ], "metadata": { "kernelspec": { "display_name": "deep-learning-book (3.14.5)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.5" } }, "nbformat": 4, "nbformat_minor": 5 }