{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3f50cbd9",
   "metadata": {},
   "source": [
    "# Topic 5: Engineering a training run\n",
    "\n",
    "So far our datasets fit comfortably in memory and every training run finished in a minute.\n",
    "Real work is messier: datasets are large, runs take hours and get interrupted, and months later you need to know exactly which settings produced a model.\n",
    "Here we take the Topic 3 MLP (3 previous tokens → next token) to **words** on **WikiText-103** (~100M words of Wikipedia) and add the practical tooling: a config dict, cached on-disk data, `Dataset`/`DataLoader`, progress bars, and checkpoints that let us resume training and reload a model for inference."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5890686",
   "metadata": {},
   "source": [
    "## 1. Setup, seed and device\n",
    "Same as before: fix the seed and use a GPU if one is available.\n",
    "`tqdm` gives us progress bars for long loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "50a188bf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:19.440210Z",
     "iopub.status.busy": "2026-10-01T19:42:19.440049Z",
     "iopub.status.idle": "2026-10-01T19:42:21.347612Z",
     "shell.execute_reply": "2026-10-01T19:42:21.347089Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch 2.13.0+cu132 | device: cpu\n"
     ]
    }
   ],
   "source": [
    "import json, math, os, time\n",
    "from collections import Counter\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from tqdm import tqdm\n",
    "\n",
    "torch.manual_seed(1337)\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(\"torch\", torch.__version__, \"| device:\", device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6584531e",
   "metadata": {},
   "source": [
    "## 2. One config dict for every setting\n",
    "Instead of scattering magic numbers through the notebook, we collect every hyperparameter and path in a single `config` dict.\n",
    "This makes experiments easy to change, and (as we'll see) we can save the dict inside the checkpoint so a model always carries the recipe that produced it.\n",
    "The knobs marked **(knob)** are the ones to turn up if you have more time, cores, or a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0c520f69",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:21.349740Z",
     "iopub.status.busy": "2026-10-01T19:42:21.349525Z",
     "iopub.status.idle": "2026-10-01T19:42:21.354906Z",
     "shell.execute_reply": "2026-10-01T19:42:21.354379Z"
    }
   },
   "outputs": [],
   "source": [
    "config = {\n",
    "    # data\n",
    "    \"data_dir\": \"/project/scv/examples/machine_learning/tutorials/LLM/data/datasets/WikiText103/wikitext-103\",\n",
    "    \"train_mb\": 50,             # (knob) MB of wiki.train.tokens to use; the full file is ~539 MB\n",
    "    \"vocab_size\": 10_000,       # (knob) most frequent words kept, incl. <unk> and <eos>\n",
    "    \"cache_dir\": \"cache\",\n",
    "    # model\n",
    "    \"block_size\": 3,            # number of previous words used as context (4-gram)\n",
    "    \"n_embd\": 64,               # embedding dimension per word\n",
    "    \"n_hidden\": 256,            # hidden layer width\n",
    "    # training\n",
    "    \"batch_size\": 256,\n",
    "    \"lr\": 3e-3,\n",
    "    \"max_steps\": 4500,          # (knob) total optimizer steps\n",
    "    \"eval_interval\": 500,       # evaluate + checkpoint every this many steps\n",
    "    \"eval_batches\": 20,         # (knob) validation batches per evaluation (cap keeps it fast)\n",
    "    \"num_workers\": 0,           # (knob) DataLoader worker processes; try 2-4 outside a notebook\n",
    "    \"ckpt_dir\": \"checkpoints\",\n",
    "}\n",
    "os.makedirs(config[\"cache_dir\"], exist_ok=True)\n",
    "os.makedirs(config[\"ckpt_dir\"], exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12f5ff63",
   "metadata": {},
   "source": [
    "## 3. Vocabulary and tokenization, with caching\n",
    "WikiText-103 is already split into words, so \"tokenizing\" is just `line.split()`; we add an `<eos>` token at the end of each line and concatenate everything into one long stream of words.\n",
    "The full vocabulary has ~267k distinct words, far too many for an output layer on a CPU, so we keep the `vocab_size` most frequent ones and map the rest to `<unk>` (which WikiText already uses for rare words).\n",
    "Let's peek at the raw text first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1a435e1a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:21.357075Z",
     "iopub.status.busy": "2026-10-01T19:42:21.356915Z",
     "iopub.status.idle": "2026-10-01T19:42:21.365328Z",
     "shell.execute_reply": "2026-10-01T19:42:21.364836Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wiki.train.tokens: 539 MB\n",
      "' \\n'\n",
      "' = Valkyria Chronicles III = \\n'\n",
      "' \\n'\n",
      "' Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield'\n",
      "' The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chr'\n",
      "' It met with positive sales in Japan , and was praised by both Japanese and western critics . After '\n"
     ]
    }
   ],
   "source": [
    "train_path = os.path.join(config[\"data_dir\"], \"wiki.train.tokens\")\n",
    "print(f\"wiki.train.tokens: {os.path.getsize(train_path) / 1e6:.0f} MB\")\n",
    "with open(train_path, encoding=\"utf-8\") as f:\n",
    "    for _ in range(6):\n",
    "        print(repr(f.readline()[:100]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74ee2b9",
   "metadata": {},
   "source": [
    "Reading and converting text is slow and gets repeated every time the notebook restarts, so we do it **once** and save the result: the vocabulary as JSON and the token ids as a NumPy `.npy` file.\n",
    "The cache file names include the settings they depend on, so changing `train_mb` or `vocab_size` automatically triggers a rebuild."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e5ecf8e5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:21.367178Z",
     "iopub.status.busy": "2026-10-01T19:42:21.366976Z",
     "iopub.status.idle": "2026-10-01T19:42:24.710702Z",
     "shell.execute_reply": "2026-10-01T19:42:24.709638Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building vocab and token ids (only happens once) ...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  9,529,733 words, 133,971 distinct\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done in 3.3 s; vocab size 10000; first words: ['<unk>', '<eos>', 'the', ',', '.', 'of', 'and', 'to', 'in', 'a', '=', '\"']\n"
     ]
    }
   ],
   "source": [
    "def read_words(path, max_bytes=None):\n",
    "    \"\"\"Return a list of words (with <eos> at each line end), reading at most ~max_bytes of the file.\"\"\"\n",
    "    words = []\n",
    "    with open(path, encoding=\"utf-8\") as f:\n",
    "        lines = f.readlines(max_bytes) if max_bytes else f.readlines()\n",
    "    for line in lines:\n",
    "        toks = line.split()\n",
    "        if toks:\n",
    "            words.extend(toks)\n",
    "            words.append(\"<eos>\")\n",
    "    return words\n",
    "\n",
    "def encode(words, stoi):\n",
    "    unk = stoi[\"<unk>\"]\n",
    "    return np.array([stoi.get(w, unk) for w in words], dtype=np.int32)\n",
    "\n",
    "tag = f\"{config['train_mb']}MB_v{config['vocab_size']}\"\n",
    "vocab_file = os.path.join(config[\"cache_dir\"], f\"vocab_{tag}.json\")\n",
    "train_file = os.path.join(config[\"cache_dir\"], f\"train_{tag}.npy\")\n",
    "val_file = os.path.join(config[\"cache_dir\"], f\"val_{tag}.npy\")\n",
    "\n",
    "t0 = time.time()\n",
    "if os.path.exists(vocab_file) and os.path.exists(train_file) and os.path.exists(val_file):\n",
    "    print(\"Found cached vocab and token ids, skipping tokenization.\")\n",
    "    itos = json.load(open(vocab_file))\n",
    "else:\n",
    "    print(\"Building vocab and token ids (only happens once) ...\")\n",
    "    train_words = read_words(train_path, config[\"train_mb\"] * 1_000_000)\n",
    "    counts = Counter(train_words)\n",
    "    print(f\"  {len(train_words):,} words, {len(counts):,} distinct\")\n",
    "    specials = [\"<unk>\", \"<eos>\"]\n",
    "    itos = specials + [w for w, _ in counts.most_common() if w not in specials][: config[\"vocab_size\"] - len(specials)]\n",
    "    stoi = {w: i for i, w in enumerate(itos)}\n",
    "    np.save(train_file, encode(train_words, stoi))\n",
    "    np.save(val_file, encode(read_words(os.path.join(config[\"data_dir\"], \"wiki.valid.tokens\")), stoi))\n",
    "    json.dump(itos, open(vocab_file, \"w\"))\n",
    "    del train_words, counts\n",
    "stoi = {w: i for i, w in enumerate(itos)}\n",
    "print(f\"done in {time.time() - t0:.1f} s; vocab size {len(itos)}; first words: {itos[:12]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc75014a",
   "metadata": {},
   "source": [
    "Now we open the cached arrays with `mmap_mode=\"r\"`: this **memory-maps** the file, so the data stays on disk and the operating system reads only the pieces we actually index.\n",
    "That is how you work with datasets larger than RAM; the array still behaves like a normal NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d491b65a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:24.712852Z",
     "iopub.status.busy": "2026-10-01T19:42:24.712687Z",
     "iopub.status.idle": "2026-10-01T19:42:24.721879Z",
     "shell.execute_reply": "2026-10-01T19:42:24.721195Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "memmap int32\n",
      "train: 9,529,733 tokens (38 MB on disk) | val: 216,347 tokens\n",
      "fraction of train tokens that are <unk>: 0.107 (first 1M)\n",
      "decoded: = <unk> Chronicles III = <eos> <unk> no <unk> 3 : <unk> Chronicles ( Japanese : <unk> , lit . <unk> of the <unk> 3 ) , commonly referred to\n"
     ]
    }
   ],
   "source": [
    "train_ids = np.load(train_file, mmap_mode=\"r\")\n",
    "val_ids = np.load(val_file, mmap_mode=\"r\")\n",
    "print(type(train_ids).__name__, train_ids.dtype)\n",
    "print(f\"train: {len(train_ids):,} tokens ({train_ids.nbytes / 1e6:.0f} MB on disk) | val: {len(val_ids):,} tokens\")\n",
    "print(f\"fraction of train tokens that are <unk>: {(np.asarray(train_ids[:1_000_000]) == stoi['<unk>']).mean():.3f} (first 1M)\")\n",
    "print(\"decoded:\", \" \".join(itos[i] for i in train_ids[:30]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc2f9eb6",
   "metadata": {},
   "source": [
    "## 4. `Dataset` and `DataLoader`\n",
    "In Topic 3 we built every (context, target) pair up front and sampled random indices by hand.\n",
    "A `torch.utils.data.Dataset` instead describes the data with two methods: `__len__` (how many examples) and `__getitem__(i)` (build example *i* on demand).\n",
    "Here example *i* is the window `ids[i : i+3]` as context and `ids[i+3]` as target, so nothing is copied up front; windows that cross an `<eos>` are fine, since we treat the text as one long stream."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ca85199c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:24.723556Z",
     "iopub.status.busy": "2026-10-01T19:42:24.723415Z",
     "iopub.status.idle": "2026-10-01T19:42:24.728309Z",
     "shell.execute_reply": "2026-10-01T19:42:24.727911Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9529730 training examples\n",
      "example 0: ['=', '<unk>', 'Chronicles'] -> III\n"
     ]
    }
   ],
   "source": [
    "class WindowDataset(Dataset):\n",
    "    def __init__(self, ids, block_size):\n",
    "        self.ids = ids\n",
    "        self.block_size = block_size\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.ids) - self.block_size\n",
    "\n",
    "    def __getitem__(self, i):\n",
    "        chunk = torch.from_numpy(self.ids[i : i + self.block_size + 1].astype(np.int64))\n",
    "        return chunk[:-1], chunk[-1]          # (context of 3 ids, next id)\n",
    "\n",
    "train_ds = WindowDataset(train_ids, config[\"block_size\"])\n",
    "val_ds = WindowDataset(val_ids, config[\"block_size\"])\n",
    "x, y = train_ds[0]\n",
    "print(len(train_ds), \"training examples\")\n",
    "print(\"example 0:\", [itos[i] for i in x], \"->\", itos[y])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9aac5c0",
   "metadata": {},
   "source": [
    "A `DataLoader` wraps a `Dataset` and handles the bookkeeping: `batch_size` groups examples and stacks them into tensors, `shuffle=True` visits examples in a fresh random order each epoch, and `drop_last=True` discards a final incomplete batch so every batch has the same shape.\n",
    "`num_workers > 0` prepares batches in background processes while the model trains, which matters when `__getitem__` is expensive (e.g. loading images from disk); in a notebook `0` (load in the main process) is the safest choice.\n",
    "We shuffle the training data but not the validation data, so every evaluation sees the same validation batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e27f8dbd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:24.729995Z",
     "iopub.status.busy": "2026-10-01T19:42:24.729845Z",
     "iopub.status.idle": "2026-10-01T19:42:25.390168Z",
     "shell.execute_reply": "2026-10-01T19:42:25.389471Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch shapes: torch.Size([256, 3]) torch.Size([256]) torch.int64\n",
      "  of a <unk> -> who\n",
      "  occasional featured lead -> vocal\n",
      "  <unk> agent of -> <unk>\n",
      "37,225 batches per epoch\n"
     ]
    }
   ],
   "source": [
    "g = torch.Generator().manual_seed(1337)\n",
    "train_loader = DataLoader(train_ds, batch_size=config[\"batch_size\"], shuffle=True, drop_last=True,\n",
    "                          num_workers=config[\"num_workers\"], generator=g)\n",
    "val_loader = DataLoader(val_ds, batch_size=512, shuffle=False, num_workers=config[\"num_workers\"])\n",
    "\n",
    "xb, yb = next(iter(train_loader))\n",
    "print(\"batch shapes:\", xb.shape, yb.shape, xb.dtype)\n",
    "for i in range(3):\n",
    "    print(\" \", \" \".join(itos[t] for t in xb[i]), \"->\", itos[yb[i]])\n",
    "print(f\"{len(train_loader):,} batches per epoch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7425f883",
   "metadata": {},
   "source": [
    "## 5. The model\n",
    "The architecture is exactly Topic 3's MLP, only with words instead of characters: a shared embedding table, the 3 context embeddings concatenated, one `tanh` hidden layer, and an output layer with one logit per word.\n",
    "All sizes come from `config`, which is what will let us rebuild the model later from a checkpoint alone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "31f2faff",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:25.391925Z",
     "iopub.status.busy": "2026-10-01T19:42:25.391778Z",
     "iopub.status.idle": "2026-10-01T19:42:27.407082Z",
     "shell.execute_reply": "2026-10-01T19:42:27.406505Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3,259,408 parameters\n",
      "uniform-guess loss: 9.21 (perplexity 10000)\n"
     ]
    }
   ],
   "source": [
    "class MLP(nn.Module):\n",
    "    def __init__(self, vocab_size, block_size, n_embd, n_hidden):\n",
    "        super().__init__()\n",
    "        self.emb = nn.Embedding(vocab_size, n_embd)\n",
    "        self.hidden = nn.Linear(block_size * n_embd, n_hidden)\n",
    "        self.out = nn.Linear(n_hidden, vocab_size)\n",
    "\n",
    "    def forward(self, x):                      # x: (B, block_size) word ids\n",
    "        e = self.emb(x).flatten(1)             # (B, block_size * n_embd)\n",
    "        h = torch.tanh(self.hidden(e))         # (B, n_hidden)\n",
    "        return self.out(h)                     # (B, vocab_size) logits\n",
    "\n",
    "def build_model(cfg):\n",
    "    return MLP(cfg[\"vocab_size\"], cfg[\"block_size\"], cfg[\"n_embd\"], cfg[\"n_hidden\"]).to(device)\n",
    "\n",
    "config[\"vocab_size\"] = len(itos)               # actual size (smaller if the text has fewer distinct words)\n",
    "model = build_model(config)\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=config[\"lr\"])\n",
    "print(f\"{sum(p.numel() for p in model.parameters()):,} parameters\")\n",
    "print(f\"uniform-guess loss: {math.log(len(itos)):.2f} (perplexity {len(itos)})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d76ed78a",
   "metadata": {},
   "source": [
    "## 6. Evaluation on a capped number of validation batches\n",
    "Evaluating on the full validation set every time would be slow, so `evaluate()` averages the loss over the first `eval_batches` batches, in eval mode and without tracking gradients.\n",
    "Alongside the loss we report **perplexity** = exp(loss): roughly, the number of words the model is \"choosing between\" at each step (a uniform guess would have perplexity equal to the vocabulary size)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2a7a3ede",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:27.408999Z",
     "iopub.status.busy": "2026-10-01T19:42:27.408793Z",
     "iopub.status.idle": "2026-10-01T19:42:28.586361Z",
     "shell.execute_reply": "2026-10-01T19:42:28.585665Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "untrained val loss 9.245, perplexity 10350\n"
     ]
    }
   ],
   "source": [
    "@torch.no_grad()\n",
    "def evaluate(model, loader, max_batches):\n",
    "    model.eval()\n",
    "    losses = []\n",
    "    for i, (xb, yb) in enumerate(loader):\n",
    "        if i >= max_batches:\n",
    "            break\n",
    "        xb, yb = xb.to(device), yb.to(device)\n",
    "        losses.append(F.cross_entropy(model(xb), yb).item())\n",
    "    model.train()\n",
    "    return sum(losses) / len(losses)\n",
    "\n",
    "val_loss = evaluate(model, val_loader, config[\"eval_batches\"])\n",
    "print(f\"untrained val loss {val_loss:.3f}, perplexity {math.exp(val_loss):.0f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9beadbb9",
   "metadata": {},
   "source": [
    "## 7. Training with progress bars and checkpoints\n",
    "A **checkpoint** is a dict with everything needed to continue or reuse the run: model and optimizer `state_dict`s (the optimizer has its own state, e.g. Adam's running averages), the `config`, the vocabulary `itos`, the current `step` and the best validation loss so far.\n",
    "We overwrite `latest.pt` at every evaluation (for resuming) and also write `best.pt` whenever validation loss improves (the model you'd actually use)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "018f4890",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:28.588402Z",
     "iopub.status.busy": "2026-10-01T19:42:28.588249Z",
     "iopub.status.idle": "2026-10-01T19:42:28.591897Z",
     "shell.execute_reply": "2026-10-01T19:42:28.591464Z"
    }
   },
   "outputs": [],
   "source": [
    "def save_checkpoint(path, model, optimizer, step, best_val):\n",
    "    torch.save({\n",
    "        \"model\": model.state_dict(),\n",
    "        \"optimizer\": optimizer.state_dict(),\n",
    "        \"config\": config,\n",
    "        \"itos\": itos,\n",
    "        \"step\": step,\n",
    "        \"best_val\": best_val,\n",
    "    }, path)\n",
    "\n",
    "latest_path = os.path.join(config[\"ckpt_dir\"], \"latest.pt\")\n",
    "best_path = os.path.join(config[\"ckpt_dir\"], \"best.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8120da32",
   "metadata": {},
   "source": [
    "The training loop runs from `start_step` to `stop_step`, pulling batches from the `DataLoader` (restarting it if an epoch ends) and showing a `tqdm` progress bar.\n",
    "Every `eval_interval` steps it evaluates, prints a line, and saves checkpoints; `stop_step` lets us stop early to simulate an interruption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "88339b1b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:28.593517Z",
     "iopub.status.busy": "2026-10-01T19:42:28.593368Z",
     "iopub.status.idle": "2026-10-01T19:42:28.598226Z",
     "shell.execute_reply": "2026-10-01T19:42:28.597789Z"
    }
   },
   "outputs": [],
   "source": [
    "def train(model, optimizer, start_step, stop_step, best_val=float(\"inf\")):\n",
    "    batches = iter(train_loader)\n",
    "    pbar = tqdm(range(start_step, stop_step), initial=start_step, total=config[\"max_steps\"], mininterval=10)\n",
    "    for step in pbar:\n",
    "        try:\n",
    "            xb, yb = next(batches)\n",
    "        except StopIteration:                   # end of an epoch: start a new shuffled pass\n",
    "            batches = iter(train_loader)\n",
    "            xb, yb = next(batches)\n",
    "        xb, yb = xb.to(device), yb.to(device)\n",
    "\n",
    "        loss = F.cross_entropy(model(xb), yb)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        pbar.set_postfix(loss=f\"{loss.item():.3f}\", refresh=False)\n",
    "\n",
    "        if (step + 1) % config[\"eval_interval\"] == 0 or step + 1 == stop_step:\n",
    "            val_loss = evaluate(model, val_loader, config[\"eval_batches\"])\n",
    "            is_best = val_loss < best_val\n",
    "            best_val = min(best_val, val_loss)\n",
    "            save_checkpoint(latest_path, model, optimizer, step + 1, best_val)\n",
    "            if is_best:\n",
    "                save_checkpoint(best_path, model, optimizer, step + 1, best_val)\n",
    "            tqdm.write(f\"step {step + 1:5d} | train loss {loss.item():.3f} | val loss {val_loss:.3f} \"\n",
    "                       f\"(ppl {math.exp(val_loss):.0f}){'  * saved best' if is_best else ''}\")\n",
    "    pbar.close()\n",
    "    return best_val"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5476286",
   "metadata": {},
   "source": [
    "We pretend the job gets killed partway: train for the first two thirds of `max_steps` only.\n",
    "On 1 CPU core this takes a few minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f4e48db0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:42:28.600074Z",
     "iopub.status.busy": "2026-10-01T19:42:28.599929Z",
     "iopub.status.idle": "2026-10-01T19:46:24.595524Z",
     "shell.execute_reply": "2026-10-01T19:46:24.594931Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/4500 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  3%|▎         | 122/4500 [00:10<05:59, 12.19it/s, loss=6.073]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  6%|▌         | 254/4500 [00:20<05:32, 12.77it/s, loss=6.332]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  9%|▊         | 387/4500 [00:30<05:17, 12.97it/s, loss=5.372]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                              "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  9%|▊         | 387/4500 [00:40<05:17, 12.97it/s, loss=5.430]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step   500 | train loss 5.430 | val loss 5.649 (ppl 284)  * saved best\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 12%|█▏        | 520/4500 [00:41<05:21, 12.37it/s, loss=5.352]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 14%|█▍        | 652/4500 [00:51<05:04, 12.65it/s, loss=5.769]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 17%|█▋        | 784/4500 [01:01<04:50, 12.78it/s, loss=5.342]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 916/4500 [01:11<04:37, 12.91it/s, loss=5.523]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                              "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 916/4500 [01:19<04:37, 12.91it/s, loss=5.446]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step  1000 | train loss 5.446 | val loss 5.455 (ppl 234)  * saved best\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 23%|██▎       | 1048/4500 [01:22<04:34, 12.55it/s, loss=5.699]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|██▌       | 1180/4500 [01:32<04:21, 12.72it/s, loss=5.154]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 29%|██▉       | 1313/4500 [01:42<04:07, 12.88it/s, loss=5.317]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|███▏      | 1446/4500 [01:53<03:55, 12.98it/s, loss=5.558]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                               "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|███▏      | 1446/4500 [01:58<03:55, 12.98it/s, loss=5.580]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step  1500 | train loss 5.580 | val loss 5.336 (ppl 208)  * saved best\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 35%|███▌      | 1579/4500 [02:04<03:51, 12.62it/s, loss=5.891]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 38%|███▊      | 1712/4500 [02:14<03:37, 12.81it/s, loss=5.428]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 41%|████      | 1845/4500 [02:24<03:25, 12.93it/s, loss=5.213]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 44%|████▍     | 1978/4500 [02:34<03:13, 13.01it/s, loss=5.235]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                               "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 44%|████▍     | 1978/4500 [02:37<03:13, 13.01it/s, loss=5.286]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step  2000 | train loss 5.286 | val loss 5.260 (ppl 192)  * saved best\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|████▋     | 2111/4500 [02:45<03:09, 12.61it/s, loss=5.394]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|████▉     | 2243/4500 [02:55<02:56, 12.77it/s, loss=5.455]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|█████▎    | 2376/4500 [03:05<02:44, 12.90it/s, loss=5.408]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                               "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|█████▎    | 2376/4500 [03:16<02:44, 12.90it/s, loss=5.517]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step  2500 | train loss 5.517 | val loss 5.189 (ppl 179)  * saved best\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 56%|█████▌    | 2509/4500 [03:17<02:38, 12.58it/s, loss=4.909]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 59%|█████▊    | 2641/4500 [03:27<02:25, 12.74it/s, loss=6.002]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 62%|██████▏   | 2773/4500 [03:37<02:14, 12.86it/s, loss=5.541]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 65%|██████▍   | 2905/4500 [03:47<02:03, 12.90it/s, loss=5.333]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                               "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 65%|██████▍   | 2905/4500 [03:55<02:03, 12.90it/s, loss=5.468]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 67%|██████▋   | 3000/4500 [03:55<01:57, 12.72it/s, loss=5.468]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step  3000 | train loss 5.468 | val loss 5.174 (ppl 177)  * saved best\n",
      "'interrupted' at step 3000 after 236 s\n",
      "checkpoint size: 39.3 MB (about 3x the weights: AdamW keeps 2 running averages per parameter)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "interrupt_at = 2 * config[\"max_steps\"] // 3\n",
    "t0 = time.time()\n",
    "train(model, optimizer, 0, interrupt_at)\n",
    "print(f\"'interrupted' at step {interrupt_at} after {time.time() - t0:.0f} s\")\n",
    "print(f\"checkpoint size: {os.path.getsize(latest_path) / 1e6:.1f} MB (about 3x the weights: AdamW keeps 2 running averages per parameter)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51dde452",
   "metadata": {},
   "source": [
    "## 8. Resuming after an interruption\n",
    "To mimic a fresh start (new kernel, new batch job) we throw away the model and optimizer and build new ones, then load `latest.pt`.\n",
    "`map_location=device` lets a checkpoint saved on a GPU be loaded on a CPU (and vice versa).\n",
    "Recent PyTorch versions default to `torch.load(..., weights_only=True)`, which refuses to unpickle arbitrary Python objects for safety; plain dicts, lists, strings, numbers and tensors are allowed, so our checkpoint loads fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a6de7ff9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:46:24.597840Z",
     "iopub.status.busy": "2026-10-01T19:46:24.597653Z",
     "iopub.status.idle": "2026-10-01T19:46:25.819838Z",
     "shell.execute_reply": "2026-10-01T19:46:25.819336Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "resuming from step 3000, best val loss so far 5.174\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "val loss of restored model: 5.174\n"
     ]
    }
   ],
   "source": [
    "del model, optimizer\n",
    "model = build_model(config)\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=config[\"lr\"])\n",
    "\n",
    "ckpt = torch.load(latest_path, map_location=device, weights_only=True)\n",
    "model.load_state_dict(ckpt[\"model\"])\n",
    "optimizer.load_state_dict(ckpt[\"optimizer\"])\n",
    "start_step, best_val = ckpt[\"step\"], ckpt[\"best_val\"]\n",
    "print(f\"resuming from step {start_step}, best val loss so far {best_val:.3f}\")\n",
    "print(f\"val loss of restored model: {evaluate(model, val_loader, config['eval_batches']):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f51e8f3",
   "metadata": {},
   "source": [
    "The restored model scores exactly what it did before the \"crash\", so training simply picks up from `start_step` until `max_steps`.\n",
    "(The data order after resuming differs from an uninterrupted run, which is harmless with random shuffling.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9fef059b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:46:25.821670Z",
     "iopub.status.busy": "2026-10-01T19:46:25.821531Z",
     "iopub.status.idle": "2026-10-01T19:48:26.129033Z",
     "shell.execute_reply": "2026-10-01T19:48:26.128419Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 67%|██████▋   | 3000/4500 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 69%|██████▉   | 3123/4500 [00:10<01:52, 12.29it/s, loss=5.373]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 72%|███████▏  | 3251/4500 [00:20<01:39, 12.58it/s, loss=4.786]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 75%|███████▌  | 3380/4500 [00:30<01:28, 12.73it/s, loss=5.639]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                               "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 75%|███████▌  | 3380/4500 [00:40<01:28, 12.73it/s, loss=5.391]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step  3500 | train loss 5.391 | val loss 5.131 (ppl 169)  * saved best\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 78%|███████▊  | 3509/4500 [00:41<01:21, 12.18it/s, loss=5.223]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 81%|████████  | 3636/4500 [00:51<01:09, 12.35it/s, loss=4.605]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|████████▎ | 3764/4500 [01:01<00:58, 12.50it/s, loss=4.939]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 87%|████████▋ | 3895/4500 [01:11<00:47, 12.66it/s, loss=5.629]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                               "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 87%|████████▋ | 3895/4500 [01:20<00:47, 12.66it/s, loss=4.891]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step  4000 | train loss 4.891 | val loss 5.085 (ppl 162)  * saved best\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|████████▉ | 4026/4500 [01:22<00:38, 12.27it/s, loss=5.293]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|████████▉ | 4026/4500 [01:32<00:38, 12.27it/s, loss=5.419]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 92%|█████████▏| 4153/4500 [01:32<00:28, 12.38it/s, loss=5.006]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|█████████▌| 4286/4500 [01:42<00:16, 12.64it/s, loss=5.099]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 98%|█████████▊| 4419/4500 [01:52<00:06, 12.83it/s, loss=5.283]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                               "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 98%|█████████▊| 4419/4500 [02:00<00:06, 12.83it/s, loss=5.107]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 4500/4500 [02:00<00:00, 12.49it/s, loss=5.107]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step  4500 | train loss 5.107 | val loss 5.040 (ppl 155)  * saved best\n",
      "finished in 120 s; best val loss 5.040 (perplexity 155)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "t0 = time.time()\n",
    "best_val = train(model, optimizer, start_step, config[\"max_steps\"], best_val)\n",
    "print(f\"finished in {time.time() - t0:.0f} s; best val loss {best_val:.3f} (perplexity {math.exp(best_val):.0f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df6710ea",
   "metadata": {},
   "source": [
    "## 9. Loading a checkpoint for inference\n",
    "A colleague (or you, next month) only has `best.pt`. Because the checkpoint holds the config and vocabulary, it is self-contained: we rebuild the model from the **saved** config, load the weights, and switch to eval mode, without touching any of the variables above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7e4ee32e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:48:26.131243Z",
     "iopub.status.busy": "2026-10-01T19:48:26.131087Z",
     "iopub.status.idle": "2026-10-01T19:48:26.204272Z",
     "shell.execute_reply": "2026-10-01T19:48:26.203776Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded best.pt from step 4500, val loss 5.040 (perplexity 155)\n"
     ]
    }
   ],
   "source": [
    "ckpt = torch.load(best_path, map_location=device, weights_only=True)\n",
    "cfg, vocab = ckpt[\"config\"], ckpt[\"itos\"]\n",
    "word_to_id = {w: i for i, w in enumerate(vocab)}\n",
    "\n",
    "net = MLP(cfg[\"vocab_size\"], cfg[\"block_size\"], cfg[\"n_embd\"], cfg[\"n_hidden\"]).to(device)\n",
    "net.load_state_dict(ckpt[\"model\"])\n",
    "net.eval()\n",
    "print(f\"loaded best.pt from step {ckpt['step']}, val loss {ckpt['best_val']:.3f} (perplexity {math.exp(ckpt['best_val']):.0f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af4fde27",
   "metadata": {},
   "source": [
    "First, the model's top-5 guesses for the next word after a few 3-word contexts (unknown words map to `<unk>`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a19558df",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:48:26.205947Z",
     "iopub.status.busy": "2026-10-01T19:48:26.205777Z",
     "iopub.status.idle": "2026-10-01T19:48:26.212796Z",
     "shell.execute_reply": "2026-10-01T19:48:26.212362Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'the United States'    -> , (0.22), . (0.12), and (0.09), in (0.05), <unk> (0.05)\n",
      "'he was born'          -> in (0.15), by (0.13), on (0.09), as (0.06), , (0.05)\n",
      "'the film was'         -> <unk> (0.12), a (0.04), shot (0.03), written (0.03), \" (0.03)\n",
      "'at the end'           -> of (0.60), , (0.10), . (0.07), in (0.03), and (0.02)\n"
     ]
    }
   ],
   "source": [
    "@torch.no_grad()\n",
    "def top_k_next(context, k=5):\n",
    "    ids = [word_to_id.get(w, word_to_id[\"<unk>\"]) for w in context.split()][-cfg[\"block_size\"]:]\n",
    "    probs = F.softmax(net(torch.tensor([ids], device=device)), dim=-1)[0]\n",
    "    p, ix = probs.topk(k)\n",
    "    return \", \".join(f\"{vocab[i]} ({pi:.2f})\" for pi, i in zip(p.tolist(), ix.tolist()))\n",
    "\n",
    "for context in [\"the United States\", \"he was born\", \"the film was\", \"at the end\"]:\n",
    "    print(f\"{context!r:22} -> {top_k_next(context)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c87c449a",
   "metadata": {},
   "source": [
    "Finally, we generate text by repeatedly sampling the next word and sliding the 3-word window forward, printing `<eos>` as a line break.\n",
    "We forbid `<unk>` (set its logit to `-inf`) so the text is readable; with only 3 words of context the output is locally plausible but wanders, and the point of this notebook is the engineering, not the prose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b3471a5f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:48:26.214692Z",
     "iopub.status.busy": "2026-10-01T19:48:26.214532Z",
     "iopub.status.idle": "2026-10-01T19:48:26.251329Z",
     "shell.execute_reply": "2026-10-01T19:48:26.250909Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The city of A. Highway System\n",
      "=\n",
      "The more than 6 of the second units . Following this outbreak . When the local flag ) . The interchange for the new constitution were based from a opening @-@ appearance in the\n"
     ]
    }
   ],
   "source": [
    "@torch.no_grad()\n",
    "def generate(context, n_words=40, temperature=0.8):\n",
    "    gen = torch.Generator(device=device).manual_seed(1337)\n",
    "    ids = [word_to_id.get(w, word_to_id[\"<unk>\"]) for w in context.split()]\n",
    "    for _ in range(n_words):\n",
    "        logits = net(torch.tensor([ids[-cfg[\"block_size\"]:]], device=device))[0]\n",
    "        logits[word_to_id[\"<unk>\"]] = -float(\"inf\")\n",
    "        probs = F.softmax(logits / temperature, dim=-1)\n",
    "        ids.append(torch.multinomial(probs, 1, generator=gen).item())\n",
    "    return \" \".join(vocab[i] for i in ids).replace(\" <eos> \", \"\\n\")\n",
    "\n",
    "print(generate(\"The city of\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b352f3ed",
   "metadata": {},
   "source": [
    "## 10. Bonus: the model learned word vectors\n",
    "The embedding table `net.emb.weight` holds one 64-number vector per word, the same idea as word2vec (which is a stripped-down version of this very model).\n",
    "The output layer `net.out.weight` holds a second vector per word, used to score it as the *next* word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "134a819c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:48:26.253275Z",
     "iopub.status.busy": "2026-10-01T19:48:26.253101Z",
     "iopub.status.idle": "2026-10-01T19:48:26.256234Z",
     "shell.execute_reply": "2026-10-01T19:48:26.255832Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([10000, 64]) torch.Size([10000, 256])\n"
     ]
    }
   ],
   "source": [
    "emb_vectors = net.emb.weight.detach().cpu()   # (vocab, n_embd): input word vectors\n",
    "out_vectors = net.out.weight.detach().cpu()   # (vocab, n_hidden): output word vectors\n",
    "print(emb_vectors.shape, out_vectors.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cde43606",
   "metadata": {},
   "source": [
    "Words used in similar contexts should end up with similar vectors, which we can measure with **cosine similarity**.\n",
    "Comparing the two tables, common words (months, numbers, pronouns) already have sensible neighbours, while rarer words are still noisy after this short training run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a53cc675",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:48:26.257992Z",
     "iopub.status.busy": "2026-10-01T19:48:26.257849Z",
     "iopub.status.idle": "2026-10-01T19:48:26.280783Z",
     "shell.execute_reply": "2026-10-01T19:48:26.280342Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "January   emb: February, acquired, Brad, promote, August      out: August, June, November, February, October\n",
      "three     emb: consecutive, +, Marine, Three, seven           out: four, two, six, a, one\n",
      "he        emb: who, she, He, She, effectively                 out: they, she, it, there, when\n",
      "France    emb: friends, 300, reflected, move, Voice           out: hybrids, La, Jerusalem, Apple, Norway\n",
      "war       emb: Powell, closer, candidacy, sequences, Dallas   out: game, city, population, previous, latter\n"
     ]
    }
   ],
   "source": [
    "def nearest(vectors, word, k=5):\n",
    "    v = F.normalize(vectors, dim=1)              # unit length, so a dot product = cosine similarity\n",
    "    sims = v @ v[word_to_id[word]]\n",
    "    ix = sims.topk(k + 1).indices[1:]            # skip the word itself\n",
    "    return \", \".join(vocab[i] for i in ix.tolist())\n",
    "\n",
    "for word in [\"January\", \"three\", \"he\", \"France\", \"war\"]:\n",
    "    print(f\"{word:8}  emb: {nearest(emb_vectors, word):45}  out: {nearest(out_vectors, word)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa918495",
   "metadata": {},
   "source": [
    "To *see* this structure we project the output vectors of a few groups of words onto 2 dimensions with PCA.\n",
    "Even after this short run, months, numbers and pronouns land in clearly separate clusters: the model discovered these word categories purely from predicting the next word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5443a1da",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T19:48:26.282674Z",
     "iopub.status.busy": "2026-10-01T19:48:26.282519Z",
     "iopub.status.idle": "2026-10-01T19:48:27.849220Z",
     "shell.execute_reply": "2026-10-01T19:48:27.848830Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjoAAAHBCAYAAABg9RGHAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAhA1JREFUeJzt3Xd8zdf/B/DXzc28kUm2iIQYMYNE7BXEilm7Ni1qa1FaVZR+tarVVquo2rs1a8RIESTEihENEkEW2Xvc8/sjv3zqypCQK8n1ej4e91Gf8znnfM7nE3XfOZ8zZEIIASIiIiINpFXWDSAiIiJSFwY6REREpLEY6BAREZHGYqBDREREGouBDhEREWksBjpERESksRjoEBERkcZioENEREQai4EOEZWKwYMHo2XLlmXdDCIiFQx0qMK6c+cOPvzwQ9SrVw+WlpaoVasW3n//fVy6dOmN6v3ss89QpUqVUmpl+b1maUtOTkZiYmJZN6NA7u7uGDZsWFk3o1iCgoIwffp0uLm5wdbWFs2bN8eiRYuQlJSUL2/Pnj1hamoKU1NTmJmZoWrVqnB1dcWoUaPw119/IScnp1jXfLEeU1NTWFtbo0WLFli1ahWys7Pz5T916hSGDx+O2rVrw8rKCvXr14e3tze2bt2KjIyMAq/Rq1cvmJqa4ssvvyzZAyF6Qwx0qELatGkTGjVqhOjoaKxbtw5BQUHYuXMnDA0N0aJFC3z11VevXXdaWhri4+NLr7Hl9JrvksTERKSkpJR1M14pOjoaDRo0wNWrV7Fs2TJcvnwZCxYswIYNG9CsWbN8gWRycjLkcjlCQ0Px8OFDBAQEYM2aNbC3t8f777+Pli1bIjIy8pXXfbGe0NBQnD9/Hp6enpgxYwbGjRsn5VMqlRg3bhy6dOkCS0tLbN26FTdv3sTOnTvh5uaGDz74AIsWLcpX/4MHD3D48GHo6upi7dq1xQ7AiEqFIKpgLl68KLS1tcXw4cMLPD9//nwBQOzZs+e16p81a5aQy+Vv0sQKcc3S1qNHD1GvXr2ybkaBateuLXr37l3WzXilyMhIMXfuXKFUKlXSjx8/LgCIb775RiW9Xbt2onLlygXWde3aNWFoaChatGjxyusWVk+nTp0EAPHkyRMhhBCLFi0SAMTWrVsLrCc4OFisXr06X/q8efOEkZGROHHihAAgDhw48Mo2EZUWBjpU4XTr1k3o6uqK6OjoAs9nZGQIS0tLUbduXSlt6NChws3NLV/etWvXChMTE+kf8hEjRgg9PT0BQJiYmEifBw8eCCGE+Pjjj4WdnZ1ISUkRkyZNEtWqVRPVq1cXU6dOFQkJCSp1l9Y1X+bj4yNMTEzEhQsXpLTAwEBhYmIiXF1dVfLOmDFDODo6qnxxBgQEiL59+4qqVasKGxsb0blzZ3Hs2DGVcnn3mZqaKmbMmCGcnJykupVKpfj2229FvXr1hI2NjfD29hYhISHFCnS8vb1FgwYN8n2RCyHEqFGjhJOTk8jMzJTSTp48KXr06CHs7OyEjY2N6Nmzp/D3989X9uTJk8Lb21tUq1ZNODg4iIEDB4pbt24JIYSwsrISWlpaQltbW3q2L/9c/vjjD9GqVSthZWUlHB0dxahRo/I9/wYNGogxY8aI4OBg0adPH2FjYyPmzp0rhBDCz89PeHt7CycnJ+Hg4CC6d+8uTpw4UeSzKIknT54IAGLChAkq6UUFOkIIMXv2bAFA+Pj4FFl/YfXMnDlTABDnz58XCQkJwtDQULRt27ZEbc/KyhI2NjZi0qRJQgghmjRpIry9vUtUB9Gb4KsrqlAyMjJw6tQpeHh4wMLCosA8urq66Nq1K+7cuYPQ0FAAQEpKSoHjRzIyMpCQkAClUgkAWLNmDcaPH6/SjR8aGgoHBwcAQGpqKuLj4/Hhhx+iRYsWOHv2LFatWoUdO3age/fuKuMZSuuaL/Pw8EBaWhqOHj0qpR0/fhxpaWm4evUqwsLCpPSDBw+iSZMmkMlkAIAzZ86gVatWyMnJwcGDB3Hq1CnUrFkTXl5e2LRpk1Tuxfts3LgxfHx8MHLkSADAxx9/jLlz52LcuHHw8/PD9OnT8cEHHxQ4huRlAwYMwM2bN3HmzBmV9Li4OGzfvh09e/aEjo4OAGDt2rXo3LkzateujWPHjsHX1xeOjo5o06YNzp8/L5X9+eef4enpCRsbG/z555/w9fXFgAEDMGfOHABAcHAwatSogS5dukjP9uTJk1L5+fPnY/To0ejWrRv8/PywdetWBAUFoVmzZrh//76ULzExEeHh4Zg8eTJmzJiBU6dOoV69eggJCUGnTp1gY2ODI0eO4Pz585g2bRpWrFiBJ0+evPKZFMeJEycAAE5OTiUq161bNwBQud+SyLt/Gxsb+Pr6IiUlBb169SpRHYcPH0ZERAQmTZoEAJg4cSIOHz6Mp0+fvlabiEqsrCMtopJ48OCBACBGjhxZZL7PPvtMABCnT58WQgjRu3dvUbt27Xz5Vq9eLQCI8PBwKa2o10iTJ08WAMQvv/yikv7XX38JAGLLli1SWmldsyDt2rVTeSXRqVMnMXr0aGFiYiLWrl0rhBDi4cOH+drauHFj4eDgoNJrIoQQ7du3F5UrVxZpaWkq9/nzzz+r5AsNDRVaWlpi9uzZKuk3b94UMpnslT06qampwsTERAwbNkwl/YcffhAAxPXr14UQQkRHRwsDAwMxYsSIfHW0adNGtGzZUgghREREhNDX1xdDhgwp8rqFvboKDQ0Vcrk8X09JdHS0UCgUYuDAgVKag4ODkMvl+Xp61q9fLwCIp0+fFtmG1xUdHS1sbW2Fqalpvl7MV/Xo3Lt3TwAo8DkWVU9GRob4/fffhZaWlujcubMQ4r+f0e7du0vU/h49eqj0AqWkpAhTU1OxZMmSEtVD9LrYo0MVSt4gRi2tov/qyuVylfylrX///irH3t7e0NfXx/Hjx9VyvZd17twZ/v7+iI+PR1paGs6dO4du3bqhY8eO0m//ef/t3LkzgNyBrteuXUP//v2lXpM8Q4YMwfPnz3H58mWV9Jfv8+TJk1AqlfnS69evj7p1676y3QYGBhg6dCj27duHhIQEKX3Dhg1wc3NDw4YNAQDHjh1DWloahg4dmq8OLy8vXLx4ESkpKTh+/DjS09MxevToV167ICdPnkROTg4GDx6skm5hYYFOnTrh2LFjKulNmjSBo6OjSlrt2rUBAFOmTEFAQIDUU1ca0tLS0K9fP0RGRmLDhg2F9mIWJu//EyHEK/PGxsZKs64UCgXmzp2LSZMmYc+ePQCK///ei548eYKjR49KvTkAoFAoMHLkSGzYsKFY7SJ6Uwx0qEKxsbGBXC5HeHh4kfnyztvb2xeZ73X+odXW1s43FVwmk8HKygrR0dGvLF8a/7h37twZOTk5OHnyJM6ePYusrCx06tQJnTt3loKREydOwMnJSXrdERMTAwCwtbXNV19e2ovt19XVhaWlpUq+vDpsbGzy1VFQWkHGjh2LtLQ0bNu2DQAQGBiIa9euYezYsVKevNcagwcPRpUqVVC5cmWYm5vD3NwcS5YsgVKpxPPnzxEREQHg1T/nwrzqmSQkJCAzM1NKq1q1ar58rVq1wm+//YZr167B3d0dZmZm6N27d74gqaQyMzPRt29fnD9/HmvXrkXfvn1LXEfec7S2tn5lXjMzM+nVXlJSEiIjI7F69WoYGxsD+O8ZP378uNjX37BhA3JycjBhwgSV6eu//fYbHjx48Nqv1IhKgoEOVSh508cvXLhQ6JgQpVIJHx8f2Nvbo1atWgAAY2PjAqcXF2fq7cuys7MLnAr+7NkzmJmZScelec2XNWvWDGZmZjh+/DhOnDiBJk2awNzcHJ07d0ZsbCz8/f1x6tQpqTcHAExNTQH89+X+orwA58X2v9zr86o6CkorSNOmTeHq6ooNGzYAyP0yVCgUGDJkSL7r7N27FyEhIbh//z4ePHiABw8e4OnTp4iLi0PVqlVRuXJlAK//TF/1TBQKBXR1daW0F//8onHjxiEkJAQPHz7Ejz/+iNjYWHh5eeHQoUOv1a6srCy89957OH78OH755ReVILAk8gKJtm3bvjKvTCaTAhEDA4N859u1awcdHR2pp/BVhBDYsGEDPvvsM4SFhamMP3vy5Al69uyJ3377rWQ3RPQaGOhQhTNv3jykpKRg8eLFBZ5fs2YNwsLCMG/ePCmtevXqiIyMzBegFPSqSaFQQKlUFvna6+Xf1v/55x+kpKSgXbt2arvmi7S0tNChQwcp0OnSpQsAoGbNmnB0dMTy5cvx/PlzeHp6SmXs7OxQo0YNHDlyJF+v0qFDh6BQKNCsWbMir5v3hfny/YeFheH27dvFajuQ26tz+fJl+Pv7Y9u2bRgwYIDUcwDk9ljJ5XKcPHlSpSfgxY+WlhY8PT0hl8uxe/fuIq+nUCiQlZVV6P0cPnxYJT05ORm+vr4qP8/iqF69Ot5//30cPXoUMpms2EHBi3JycjBs2DAcPHgQP//8MyZMmFDiOoDcnpcff/wRLi4u0qDkN1GlShV88MEHOHLkCHx9fQvMExMTg61btwLIfXUaGhoKb2/vAn9+PXv2xF9//YVnz569cduIilSWA4SIXtcXX3whAIgZM2ZIg0Dj4uLE8uXLhba2thgzZoxK/nv37kmDTlNTU0V8fLyYNWuW6NmzZ76BwVu3bhUAxKlTp/Jdd/LkydIg1fPnz4ucnBxx/fp1Ubt2bVGrVi2RkpJS6tcszJo1awQAlUHXQggxYcIEAUBoaWmJ58+fq5TZuXOnACCmTp0q4uLiREpKilixYoWQyWTiq6++UrlPQ0PDAq87ePBgYWxsLPbv3y+ysrJEaGio6NWrl2jWrFmx19GJi4sT+vr6ombNmgKA8PX1zZdn4cKFQltbW3zzzTciNjZWCJG7zszWrVvFhx9+KOX79NNPhba2tli+fLmIjY0VWVlZIiAgQGWA8ZAhQ0TVqlWlel40YsQIYWBgILZt2yYyMjJERESE6N27t9DV1RUBAQFSPgcHBzFo0KB85X/++WexePFiERISInJyckRmZqb0s3lxvZm8ZQX2799f6HNRKpVixIgRQiaTiTVr1rziKRY8GDkyMlKsX79e2NraiurVq4t79+69Vj0FSUtLE126dBGVKlUSq1evFs+ePRNCCJGcnCw2bdok7OzsxJw5c4QQQgwYMEBYWFgUuJSAEEKEhYUJAOLbb7995XWJ3gQDHaqwjh07Jjp37iwMDAyEQqEQurq6omXLlmLbtm0F5t+0aZOwtbUVcrlc2NnZiU2bNhU4AyorK0sMHDhQ6OnpiUqVKqmsaZMXAERERIiuXbsKPT09IZfLRc+ePcWjR4/Ucs3ChISECADC0NBQZGRkSOl79uwRAApcw0cIIfbt2ycaN24stLS0hFwuFzVr1sz3pVpUoJOWliYmT54sKlWqJHR0dET9+vWFn59fiRcMHDZsmAAgnJ2dC82zbds24ebmJuRyudDT0xM2NjZixIgR4tq1ayr51q1bJ+rVqydkMpkwNDQUzZs3F8ePH5fOBwcHiyZNmgi5XJ5vHZ2srCyxcOFCYWtrK7S1tYVcLhdt27YVfn5+KtcoLNCJiooSn3/+uXB2dhb6+vpCX19fNGzYUKxbt04l39dffy0AiDNnzhR6v0FBQVKQ+uKaSnmfoUOHquRv166dkMlk0nk9PT1hYmIi2rZtK7777juRnJxc6LVerqc4gY4QQmRnZ4t169aJli1bCl1dXaFQKESlSpVEw4YNxZIlS0RkZKSIjo4WOjo6+WbXvczFxUVlvSsidZAJwWHvVPGlpqbCwMBAWi+mKGlpadIYhMzMTKSmpsLExCRfWaVSieTkZCiVShgbG0NLSwsfffQRNm7ciOTkZAC5YylkMhm0tbXVds2iJCQkQEtLC0ZGRip1JCYmQldXFwqFotCyWVlZUCqV0NPTK7C9mZmZMDExKbS8EAKZmZlS+ZSUFCiVSpW2FCXvObyqnXn3lJWVVWBbX/Sqn0dmZibS0tLyPbM8aWlp0NXVlWbtvSgxMRHa2tpFtjUzMxM6OjoF/j3s1asXnj9/Dj8/v0LL5/3sCqOjowNDQ0PpODk5WVq7SSaTQaFQFDi26lVe/DtXUqmpqfmeSXZ2NpKTk2FgYFDkzyw1NRWZmZnSWCkidWCgQ1QCLwc6RMWRnZ0Nc3NzHDx4sMTjfojozRT9aygREb2xvCURiuohIyL14KwrIiI1k8lkDHKIyghfXRGVQHHGrhARUfnBQIeIiIg0Fl9dERERkcZ6ZwcjK5VKPH36FEZGRsWakkxERES5hBBISkqCra1tiTZ6LQvvbKDz9OnT194IkIiIiHI3UC5os9vy5J0NdPIWCwsPD3+tRbKIiIjeVYmJibC3ty/2AqFl6Z0NdPJeVxkbGzPQISIieg0VYehH+X6xRkRERPQGGOgQERGRxmKgQ0RERBrrnR2jQ0RE6pGTk4OsrKyybga9AR0dHcjl8rJuRqlgoENERKVCCIHIyEjEx8eXdVOoFJiamsLa2rpCDDguCgMdIiIqFXlBjqWlJRQKRYX/gnxXCSGQmpqK6OhoAICNjU0Zt+jNMNAhjRQXFwc9PT0oFIqybgrROyEnJ0cKcipXrlzWzaE3ZGBgAACIjo6GpaVlhX6NxcHIpJFmzJiBXbt2lXUziN4ZeWNy+MuF5sj7WVb08VYMdEjjpKWlITU1FXFxcXj8+DFSUlKkcykpKUhPT1fJHxcXh9TUVAC5q30S0evj6yrNoSk/SwY6pHFOnDiBw4cPY/HixfDw8MCff/6JJ0+eoGPHjnBwcICFhQUGDx4s/ZYyZcoUTJkyBdWqVYOtrS26deuGnJycMr4LInpXZWVl4fHjxxBClHVTNAIDHdIcyhzg4Vl4O2bivW5tsfKbb/D48WMMHz4cEydOxIABA3Dt2jXcuHEDCQkJ2LRpk1Q0PDwcd+/exfPnzxEVFYVTp06V4Y0Q0bsiOzsbjx8/hlKplNKuX78Oe3t7JCQklGHLNAcHI5NmuH0AODoHSHyaexySBhy/BnhUgajbC6dOnUJgYCC++uorqUibNm2kPw8ePFh6H92sWTOEh4e/zdYT0Tvq7t27aNCgASIiImBtbV3WzdFIDHSo4rt9ANg1AsB/3bwyGYD0BGDXCMgGboKuri7OnDmDmjVrFliFrq6u9GctLS2V366I6O3KUQr4P4xFdFI6LI304e5oDrmWesaLZGVlISoqCnZ2dpDJZIiNjYWJiYk0yyg7OxuJiYkwNzcvtI5nz55BoVDkG4j9ct3x8fEwNTWVziuVSkRFRQEAIiIikJ2dLc12etHL5V6UmZmJ9PR0bk5dBL66oopNmZPbkwPVd9mVDWQIjMhBeIISKfs/wZDBgzFu3DhcunQJjx8/xuPHj/MNSiaisnc0KAKtvz6FIb9dxLQd1zDkt4to/fUpHA2KUMv18l4TzZs3D1ZWVnBwcICNjQ3+/vtvLF26FJaWlqhWrRpq1qyJoKAglbJHjhyBk5MTqlevDlNTU/Ts2VNae+bFupcvXw5ra2s4ODigSpUq2L9/PwDg+fPnGDZsGACgR48e8PDwwJw5c6Tya9asKbAcACQkJMDb2xvGxsaoXr06XFxccPr0abU8o4qOgQ5VbGF+/72uesGEpjq4EZWD1r8n48+AMHw7tR86dOiASZMmoUWLFvDw8ICvry8AwNzcHIaGhlJZMzMzlWMiejuOBkVg4pZARCSo/hISmZCOiVsC1RbsAMCtW7cQEhKChIQEeHl5oU+fPggMDER4eDgSEhLQuHFjzJw5U8r/5MkTvPfee5gwYQISEhLw9OlTxMTEYPz48fnqPnv2LIKDg5GQkICpU6dizJgxSE9Ph4WFBXx8fAAAgYGBePz4MdatW/fKcgCwYsUKREREIDo6GrGxsdi/fz8uXbqktudTkcnEOzqsOzExESYmJkhISGCXX0V2cw+wd+yr8/VfDzQYoP72EL2j0tPT8fDhQzg6OkJfX7/E5XOUAq2/PpUvyMkjA2Btoo9zczqW6musy5cvw83NDTdu3ECDBg0AAEePHkW3bt1w7949ODs7AwD27NmDCRMmIDY2FgCwaNEibN26FcHBwdI07H/++Qft2rVDaGgoHBwcCqw7IiICtra2uHXrFlxcXBAUFJRvjE5xyk2cOBFPnz5V6eUpbUX9TCvSdyh7dKhiq2RVuvmIqEz4P4wtNMgBcl9ORySkw/9hrFqub29vL/25UqVKBaa9uM5WcHAwGjVqpLLWTJMmTQAA9+7dK7RuIyMjAMVbs6uocpMmTUJAQADq1auHjz/+GD4+PpyOXggGOlSxObQEjG2R+/teQWSAsV1uPiIqt6KTijdmrrj51E1PTw+ZmZkqaRkZGdI5dWvQoAFCQ0OxatUqaGtrY+TIkfD29lb7dSsiBjpUsWnJAa+v///g5WDn/4+9lufmI6Jyy9KoeK+7iptP3Ro1aoRLly5JwQ0A+Pr6QkdHB3Xr1i12PXlBUXZ2domur1Qqoauri86dO2PZsmXYtWsXDh06pDIYmnIx0KGKz8UbGLgJMH5ph11j29x0F/6WQ1TeuTuaw8ZEv6i+WdiY5E41Lw/Gjx8PuVyOUaNG4fr16zh69CimTZuGSZMmwcLCotj1VK1aFQYGBvjzzz8RHh6O58+fF6vc2LFjsXz5cly5cgV37tzBxo0bYW9vjypVqrzuLWmstxLoZGRkICQkBMnJycUuk56ejvDw8HxdgyXNQ+8IF29gehAw8lDuwOORh4DpNxnkEFUQci0ZFvZyAVBo3ywW9nIp9fV0dHV1YWdnBy2t/74O9fT0pLVv8ujr66Nq1arSsaGhIf755x8AQP/+/TF37lxMmDAB33zzTZF1a2lpwc7OTurJMTAwwPr167Flyxa0bt0ac+bMKVa5lStXIjExER988AEGDBiA9PR0nDx5UqUM5VLrrKvHjx9j9erV2LJlC54+fYrNmzdj+PDhryy3cOFCrFixAgYGBsjKysLSpUsxZcqUEucpSkUaMU5EVN696ayrPEeDIrDo4G2Vgck2JvpY2MsFXvVtiihJpU1TZl2pdWVkX19fmJmZ4erVq7CyKt6sl82bN2PFihXw8fFBy5YtceDAAfTr1w916tRB586di52HiIgqHq/6NujsYv3WVkYmzffW1tGRyWTF6tFp2bIlatSogc2bN0tpHTp0gJmZGfbt21fsPK9SkaJRIqLyrrR6dKj80JQenXL1Mk+pVCIwMBAtWrRQSW/dujUuX75c7DxEREREQDnb1DMpKQkZGRmoXLmySnqVKlXw7NmzYucpSEZGhso0wOIs1kREREQVW7nq0ckbLZ6VlaWSnpmZKe0kW5w8BVm2bBlMTEykz4srThIREZFmKleBjpGREUxMTBAZGamSHhkZKU3rK06egsybNw8JCQnSJzw8vPRvgIiIiMqVMg90YmNj8fDhQ+m4ffv2OHbsmEqev//+G+3bty9Rnpfp6enB2NhY5UNERESaTa2BTlpaGkJCQhASEgIAiIqKQkhICGJiYqQ8P/zwA1xdXaXj+fPn459//sHnn3+OgIAATJ48GeHh4Zg1a1aJ8hARERGpNdC5evUqvLy84OXlhRo1amDNmjXw8vLCd999J+UxNzeHk5OTdOzm5objx48jICAAo0aNwpMnT+Dr64uaNWuWKA8RERHRW1tHp7ypSGsAEBGVd1xH5/XcunULgwYNwoULF2BkZFTWzVHBdXSIylBkZCSuX79e1s0gInojaWlpuHXrFnJycsq6KRqLgQ5VSMHBwfj777/LuhlERFTOMdChcu/Jkyc4deoUjh49Kg1sr127Nrp16wYA8Pf3V1ku4NatWwgODi6TthJRKVDmAA/PAjf35P5Xqb7ejlu3bqF+/fq4ePEi3n//fXh4eGDYsGEqs4HPnz+Pxo0bq5R7+vQp6tevL/3bk1fP8ePHMWzYMLi7u2PYsGF49OgR/Pz80LdvX7i7u+Ojjz4qcMHa69evY+jQoXB3d8eIESPw+PFjlfOJiYn47LPP0KpVK7Rr1w4LFixASkpKvvvw8/ND//794erqivPnz0MIgZ9//hmdO3dGq1atMHPmTMTGxpbiEyz/GOhQubZx40a4urriq6++wqpVq3Dx4kUAwJkzZ7Bs2TIAuUsUdO3aFcnJybh//z66desGHR2dsmw2Eb2u2weAVfWBP3oCe8fm/ndV/dx0Nch7dTRy5Ej07t0bK1euRHx8PLy9vaFUKgHkrsgfFBSkUi4zMxO3bt2SVtzPq2fKlCl477338O233yI4OBgdO3bExIkTMXr0aKxYsQK+vr6YOXNmvnYMGjQI3bt3x4oVKxATE4OOHTtKC+Omp6ejQ4cOuH37NpYsWYLFixfjypUr6NatG/KG2eZdf/jw4Rg4cCA2bdqERo0aYe3atViyZAkmTZqElStXwtHREdOnT1fLsyyvytUWEEQSZQ4Q5oe/Nv2MXxZNRb8P5gFaBa987eXlBV9fX0yYMAH37t3Dt99+qzKTj4gqiNsHgF0jALw0RyYxIjd94CbAxVstl/7xxx/RuXNnAMB3332H2rVrIywsDI6OjiWq56effoKnpycAYO7cuXjvvfdw7tw5tGrVCgAwa9YszJ8/P1+5pUuXSpteN2nSBPb29ti2bRtGjhyJTZs2ISkpCbt27ZJ2AGjSpAmqVKkCPz8/qW4AWLlyJfr06SMdBwQEwMvLC3379gUANG/ePN/OApqOPTpU/rzwG91oyyB8MPtztKphhC8/Goq4uLgCiyxevBj//PMPqlWrhvfee+8tN5iI3pgyBzg6B/mCHOC/tKNz1fYa68X13GxtbQEA0dHRJa7nxVdc1tbWAIBGjRqppBVUb4cOHaQ/GxkZoVmzZrh69SoA4OzZs4iOjkbTpk3RqFEjNGzYEC1btoRSqcS9e/dU6nF3d1c59vLywo4dOzBr1iycPHkS6enp71yPN3t0qHx56Te63nV00M1ZG37hSvx+bg/GDnyIfScu5Cu2bt06ODg44PLly7h9+zZcXFzecsOJ6I2E+QGJT4vIIIDEJ7n5HNuU+uW1tfN/Hb7O6isF1fNyWkH1GhgYqBwrFAppDE5qaiqaNWuGVatW5StnZ2eXr9yLBgwYAAcHB+zYsQOzZ89GSEgIli5diqlTpxbrfjQBAx0qPwr4je7i42zEpwPZSgFtGaAdcyvfb3TXr1/HsmXLcPHiRVy5cgWDBg3CpUuX8v0PT0TlWHJU6eYrRZUqVUJOTg7S09Ol9WSePHlSqte4ffs2bGxsAOQGQrdv30bLli0BALVq1cLWrVtRu3bt1+qNcXNzg5ubGwBgy5YtGDlyJEaMGAFTU9NSa395xldXVH4U8BvdzqBsrLqYgd8Cs2BVSYZfuubms7GxkbqIN2zYgHXr1sHGxgY9e/bE0KFDsWfPnrfffiJ6fZWsSjdfKapbty709PSwfft2AEBycjIWL15cqtdYuHAhkpKSAOSO83n8+DHef/99AMD48eMRGxuLmTNnIjMzE0DuAOnFixe/8vXaypUrce3aNek4KysLOjo60NPTK9X2l2fs0aHyo4Df1L7zKmCF1eQotGs3AO3atQMAfP/99yqn582bV+xLZmZm4ocffkBwcDAWL14svVMnorfMoSVgbJs78LjAcTqy3PMOLd92y1C5cmWsWrUKkyZNwqJFi5CamooRI0bk21z6Tbi7u8Pe3h4GBgZISUnBhg0bpNdSTk5OOHr0KD788EOYmprC0tISiYmJmDJlCszMzIqs19XVFWPGjEF4eDgUCgUyMzOxadOmfK/KNBm3gKgAy1e/Mx6ezZ1K+iojD5XaO/qlS5fi3Llz6N+/P/r164f58+djzZo1pVI30bukVLaAkMboAarBjiz3P2qYdZWeno6QkBC4uLhASyv3JYdSqcTt27dRo0YNlYAgPT0dkZGRsLe3h1KpRHBwMGrVqgVdXd0C60lNTcWDBw9Qr149yGS595CcnIywsDDUq1cv3/VzcnLw6NEj2NraFhqIREREICMjAw4ODlKdhd3Hi549e4a0tDTY2tpKM7eK82w0YQsIBjoV4If0zlDm5M62etVvdNNvFjrVvKT69euH8ePHS+tR/PHHHxg1alSp1E30Lim1va5uH8gdq/fia2xjO8BrudqmllPBNCXQ4asrKj+05IDX1///G50MBf5G57W81IKcX375BQEBAUhPT8fevXsBADo6Ohg1ahSOHDmCxMREDB48WMr/8ccfY/bs2bCysoJSqcTmzZtx9epV2NjYYNy4cahcuXKptIvonebiDdTpkTtmLzkqd0yOQ8tS+/+e3j0cjEzli4t3bve0sY1qurFt6XVb///y8s7yCJgodFC3Tm14eHjA3d0dv//+OwDA3t4eCxYskIpcuXIFBw8ehJVV7kDIwYMHw9fXF7Vq1UJERATatm0rrZBKRG9IS577errBgNz/MsihN8AeHSp/1Pkb3Qvd4p0AVM1JQbfUvfBs2QHZtbpLa0s0aNAAxsbGOHfuHFq3bo2NGzdi9OjRAIC7d+/ixIkT6N+/PwIDAwEAMTExuH79er7FuoiIqGwx0KHyKe83ute0ZMkSnDhxAuPGjZOmaBa6vHxqbG56v99VkkeNGoWNGzfC3d0du3btklYpDQ8Ph5mZGTw8PKS8Hh4eqFq16mu3l4iI1IOBDmmc8+fPY+fOnfjpp59Qs2bN3MTiLC9/7FOV1GHDhmHx4sVo164dmjZtKi0LX7duXTx79gyenp6oXr262u6DiIjeHAMd0jjBwcFo06YN2rZt+19icZaXT3oCvDAJsXLlymjTpg2mTZuGX3/9VUqvWrUqFixYgGbNmqFTp04wMjICkDu4uaDl34mIqOxwMDJplGPHjmHJkiXYt28fPDw8cPfuXWRlZeHrH35F1y0pGLYvFYER/20hYaovQ2KGEgAglwHdWzeWFgFbtmwZ3Nzc0LhxY2zatEnlOp988gkuXbqEfv36wcPDAx4eHiprWhARUfnAXz9JMyhzgDA/NDV4gvc6e+DB80zMmj0b9vb2+PTTT3Hh7A185qGHWzE56PhHCm5NqgQ7Yy1kKwGt/w9QZDIZjM3MERMTAwC4f/8+jh07hq+//hq1a9fOd8kaNWqgRo0ab/U2iYioZBjoUMX3wkyqKgAcn2QiJV4fHsYjAENDbNu2Dcf+PoL6PoPRNTECFx/n4OC9bHzYTPeFSv5/MUJDS5WqJ0yYgKFDh77V2yEiotLDV1dUseXNpHp5/E1mam767QNITEyEhZV17mKEACwNtZCQnjsWRyYDhPhvMcLsHNWd0fN2EyYiooqJgQ5VXMWZSXV0Lpo2bZK767CLN+K9fsaR+wJNbXPX5LEylCEoUQEM3IQ4mzY4e/bsW2s+ERGpH19dUcVVnJlUiU+w6uPZ8J74BX799VdERUVh0MAx8JwzDEiOwji3RHR8fzZ2D14AIQTXwiEqB3KUOQiMDkRMagwsFBZoYtkEcjWtjvz48WOsWrUK8+bNw+HDhxESEoI6depg8ODB0uaYeXnmz5+PPXv24MGDBxg1ahRq166NhIQEbN++HY8ePUK1atUwZMgQmJiYSPUfPXoUDx8+RPfu3XH48GE8e/YMHTp0QJs2quuEvaqevXv3IiUlBSNGjJDSzp49C39/f8yaNatE1zp+/DguXbqESpUqoVu3bqhTp06pP9fyhD06VHElRwEA7sTk4Mi/WVJyv7ramN1STzpuXM0Y9+/fx969e3H37l2s+eUXaXn5xj3GICwsDNu3b8fVq1exadMmeHl5AQA+/fRT9O7d++3eE9E7zifMB133dsWYY2Mw5+wcjDk2Bl33doVPmI9arhcZGYlvv/0WzZs3x/Hjx5GVlYXZs2dj4MCB+fK0aNECfn5+MDc3h56eHp48eYL69etj8+bN0NbWxqZNm1C/fn08ffrfL2Dnzp3DokWL4OXlhfDwcMTExKBz587Ytm2blKc49Zw4cQIHDhxQafuVK1ekbWuKe61Zs2ZhzJgxSE1NRWRkJIYMGYIjR46U6jMtb9ijQxVXpdx9p65E5OBoSA66O+sAyB2D83I+HR0duLi4FFiNkZER6tWrBwAqs6icnJzU0GgiKoxPmA9mnpkJ8dLr6OjUaMw8MxMr26+Ep4OnWq49YMAALF++HEDuquj16tXDqVOn0LFjRynP+PHjpd4TABg7dixsbGzg6+sLbW1tfP7552jevDk+++wzrF+/XsqXkJAAf39/VKtWDQCgp6eHH3/8UZro8PnnnxernuJ41bW2bNmCX375BX379gWQu4r848ePS/q4KhT26FDF5dASaQbWOPJvDu48y8E3fhk4GJyFDVcz/3+wsQznn5thx4Vwqcivv/6KlJQUAEBUVBQ2bdqEzZs3S1PKiahs5ChzsNx/eb4gB4CU9rX/18hR5uQ7XxqGDx8u/bl27dpwd3eHj49qL1JecJDnxIkTGD58uLRQqLa2NkaMGJGvXP369aXAI+/40aNHJa6nOF51LUdHR/zxxx+4c+cOAEBHRweOjo4lvk5FwkCHKi4tOYTnEiRkCKRlAZHJAnHpAsfuZ8PnQe4/hstuWmDm7NnIyclBTEwMFi1aBENDQ/z7779o0KABDh8+jAMHDqB+/fp48OBBGd8Q0bsrMDoQUalRhZ4XEIhMjURgdKBarm9hYaFybGVlhYiICJU0c3NzlePIyEhYWVmppFlbW+crZ2hoqHKsra2N7OzsEtdTHK+61q5du2BmZob27dvDxsYGkydPxvPnz0t8nYqEgQ5VXMocKMytMaSPF5rYK/BNF32MaKSLTo7a8Hmsi+x+v+N+VDI6dOiAy5cv4+TJk+jQoQMA4Ntvv8W4ceOwc+dO7N69G8OGDcN3331XxjdE9O6KSS1er2px85XUi+NhgNwByK+anGBnZ4cnT56opIWHh5d4UkNx6tHR0UFWVpZKnsTExBJdBwCqVauG33//HVFRUfj7779x8eJFTJkypcT1VCQMdKhiun0AWFUf+KMnEOIDZGcAisqAxyR0mvU7Tj6zwKUkS7i7u6NTp044efIkTp48iU6dOgEAHjx4AHd3d6m6Fi1a4P79+2V1N0TvPAuFxaszlSBfSb24n93ly5dx5coV9OjRo8gyvXr1wu+//460tDQAQGpqKn7//Xf06tWrRNcuTj1OTk64fv261DuTmZmJffv2leg6SqUSJ0+elI4bN26Mdu3a5QuyNA0DHap4XlokUFtLBqUQQGoscHENalgqkJmZiQ0bNsDT0xOenp5SoJM3sLBq1aoICgqSqrxx4wanlhOVoSaWTWClsIIMBe8ZJ4MM1gprNLFsopbrBwQEoHPnzhg9ejQ8PT0xbtw4eHh4FFlm4cKFUCqVaNq0KSZMmICmTZsCyB1cXBLFqWfkyJHIzs5Gy5YtMXHiRDRt2hRyecmm3MtkMixfvhxubm748MMPMWjQIKxdu1ZlgLUmeiuzrmJjY/HkyRM4ODjA2Ni4yLy3bt2SotoXmZiYwNnZGQAQERGRLwLV1tZG48aNS63NVE4VsEhg7cpaOPUwB1+dTUcDSzl6HZ2Ljh06YNOmTVi8eDFsbW3x9OlTyGQyVK9eHQAwdepUdOzYEc+ePYNSqcTmzZu5WCBRGZJryTHXfS5mnpkJGWQqg5Lzgp857nPUtp7OsWPHcOXKFYSEhGD48OFS7y8A2NvbY8WKFTAwMFApU7lyZVy9ehVHjhxBWFgYunfvju7du0NX97/tZbp164YGDRqolGvatCkWLlxYonrMzc1x8+ZNHDlyBJmZmZg+fTpSUlJw48aNYl9LJpPhxIkTUo+VoaEhVq1apfErwMuEEAUtK1sqhBCYPn06fvnlF1StWhVPnjzB/Pnz8dlnnxVaZsCAAQgNDVVJu3LlCoYNG4YtW7YAyJ0Ot3z5cpVFjoyMjHD69Olity0xMREmJiZISEh4ZfBF5cjDs7mvq16y/24W/J/koHYVLYxopAu/+stw8s5z6e/a1q1bIYRQmVnx77//4s8//wQA9O/fnxt0Er2B9PR0PHz4EI6OjtDX13/tenzCfLDcf7nKwGRrhTXmuM9Ry9Tyy5cvw83NDXFxcTA1NS31+iuyon6mFek7VK09Or/99ht+//13XL58GQ0aNICvry88PT3RqFEjeHt7F1hmz549Ksd+fn5o1aoVRo4cqZJev359XLx4UW1tp3IqueBZGb3r6KB3HR3puGVtS7QcMEk6HjZsWL4yzs7O+OSTT0q/jUT02jwdPNHBvsNbWxmZNJ/aA50BAwZIXWnt2rVDx44dsW7dukIDnZetX78e1atXh6enaiSvVCoRHBwMfX19VKtWDTJZwe91ScNUsnp1npLkK0du3ryJ9evXIyoqCkIIfP/999KU0127duHUqVNQKBQYM2YM6tevj+vXr+Ovv/5S6QJfvnw5OnTogObNmxdaDgCWLl2K9u3b4+TJk3j69ClGjBiBli1bvv2bJiqAXEsON2u3t3Ktwl5LkeZQ22DknJwcXL9+HW5uqn9ZmzdvjsDA4q2DkJycjF27dmHcuHH5ApmAgAB0794dzZo1g42NDXbt2lVqbadyzKElYGwLFDJgEZABxna5+SoCZQ7w8Cxyru1El07tYW1lCW9vb/Tp00daD2PBggXYtWsXPDw8YGtri+7du+PBgweoVasWfvzxR0RHRwMA4uPjsWLFCtStW7fIcgBw+vRpjB8/HiYmJqhevTp69Oih8WtpEBXEysoKs2fPhp6e3qszU4WktkAnKSkJWVlZqFy5skp6lSpViv0P6s6dO5GWlobRo0erpLu6uiI4OBj3799HdHQ0pk+fjqFDh+Ly5cuF1pWRkYHExESVD1VAWnLA6+v/P3g52Pn/Y6/lufnKuxemyGftGY+s5DjUvvczvKoLDB48GJUqVUJOTg5WrlwJLS0tHD16FJcvX4auri6OHz8OAwMD9O3bF1u3bgUA7NixA927d4exsXGR5fLMmTMH06ZNw9y5c9GsWTNcv369rJ4EEZHaqO3VlY5O7niJjIwMlfS0tDTp3KusX78ePXr0gK2trUr6i2sbyGQyzJ07F7///jt27NiBZs2aFVjXsmXLsGjRopLcApVXLt7AwE25s69e3L3c2DY3yHEp3mvRMpU3Rf7/Z5boa8uwrb8BfjgfhTE7hqF502+wdf8JKJVKZGdno2/fvlKvZp8+feDq6gogd0+eSZMmYcaMGdi4cSO++uorALkzHYsqB+QuUpbH0NAQqampb+POiYjeKrUFOoaGhjA3N8+32uTTp09V9uEozJ07d3DhwgUcPHiwWNezsbEpcmOyefPmYebMmdJxYmIi7O3ti1U3lUMu3kCdHkCYX+4A5UpWua+rKkJPTgFT5AGgSw1tdKmhjWwl0Hfvv9i7ezcmfPghLC0tYWhoWOC4tpYtWyItLQ07duxAVFSUtPKzhYVFkeWI1EWpVJZ1E6iUaMrPUq2DkTt16oTDhw9jzpw5AHIf2pEjR9C9e3cpz9OnTxEdHZ1vDZz169fDzs4O3bp1y1dvenq6ylS32NhYXL9+XfpHviB6enp8B6tptOSAY5uybkXJhflJPVFJGQIXHuegkZUWph1NBwAkZuTuyP6NsxGA3P8Xhg8fjoYNG0r78SxZsgQ1a9YEkLuQ2IQJEzBjxgyVsWyvKkdUmnR1daGlpYWnT5/CwsICurq6nCRSQQkhkJmZiZiYGGhpaams51MRqTXQWbBgATw8PDB16lT06tULmzZtQmxsrMoqjGvXrsWqVasQHx8vpWVlZWHz5s344IMPClz5sX379hg4cCBcXV3x7NkzLF++HCYmJpg0aVK+vERlITU1Fb6+vgUG6i9OkX+SpMTUv9NxeYIh+vz/9HhjPaCVvQG0KwkcPXoUXl5e+PfffxEQECCNbzMzM5Pq+OCDD+Dk5CSt+pyna9euhZZbsGCBNGgZAGbPng0nJ6fSuXl6J2lpacHR0RERERH5evKpYlIoFKhWrRq0tCr2JgpqDXQaNmyIs2fP4ptvvsEXX3wBZ2dn+Pn5qby6srW1VRk3AOTOqLK3t8fYsWMLrPfgwYNYtWoVli5dCoVCgb59+2LatGkwMjJS5+0QFVtKSgoOHjxYcKBTwNT3SroyDK6vOnYtJFWOjz76CCEhITA1NUXnzp0LvFblypUxePDgAs8VVq59+/Yqx61bty7kToiKT1dXF9WqVUN2djZycnLKujn0BuRyObS1tTWiV06tKyOXZxVpVUcq327fvo2wsDDUrVtX2mLi5R6drKwsXLx4EYaGhqjp5IhLcxqhs00i7j7LRp8daQj8wBD+T3JQ2UCGBlbagLEtNprOxrz587F69WooFAqVV75ERGWpIn2HvpW9rog01YgRI3Dz5k1UrVoV165dw9y5czF58mRER0dj4sSJCA0NRXZ2Njp27Ijnz5/DysoKSqUSz5/KEDQMAGRIyBDovDkVJnoyXH6ag0Xt9TBxxXL4rvgTSUlJ2LFjBypXrsxAh4joNTDQISopZQ4Q5ocDBw/h9lV/zP/sS0BLC7169cK8efMwefJklex79uyBTCbDzZs3IZfLsXz58tx92wauBv6Yjri0ZGwfZ4BqJlo4+8wMM85oY6KLN+bPd8HZs2fzbYtCRETFx0CHqCRuH5DW77l6JgMpkVnYsXQcYFUfMLZFhw4dkJWVpVrk9m106NBBGljfpUuX3EDHxRsYWQPVdnmh2pjvgEpWcJRXw/PD7cvgxoiINBMDHaLiemmRP0tDGZzMtLCnrxaA28DAuQUuVlilShVcu3ZNOn706NF/J7Xk0NIzBBoMyD1+/Bh5w+a0tLTwjg6hIyIqNQx0iIqjgEX+hjTQwbJzGRi7PxUdHbWh93ASzN83QEdP1VlOAwYMwBdffAEHBwfY2Njgp59+KtZ0TWtra8TExGDdunXSXlVERFQyFXtyPNHb8sIif3lM9WW4+oEhHM20cCQkCzsConBs9+8AclcGzwtMbG1tcebMGURFReHff//FjBkzYG1tDQAwNjZG165dpTpfnF2lUCiwefNmnD17Fvv3738bd0lEpHE4vbwCTI17F2RkZODmzZuF7lVW5m7uAfYWvK6Tiv7r/3sN9YITJ04gISEBMTEx+O677/Dxxx9j/PjxamgoEZH6VaTvUL66ojKRmZmJ69evw83NDQAQERGBAQMGIDQ0tGwbVpgCFvkrSb6///4bjx49gpmZGZYsWYKBAweWYuOIiKgwDHSoTERHR6Nv375FbsRarji0zN0dPTECL2/GmUuWe96hZYHFV65cqdbmERFRwThGh8rErVu3kJmZiXPnzuHSpUtSuhACd+/eVdkrJyMjA5cvX4YQAsHBwYiK+m+vqPDwcNy4cQMZGRn5rlHUuRLTkgNeX///wctLov//sdfyirF7OhHRO4SBDpWJ33//HQkJCZg7dy4WL14MAEhLS0Pnzp0xdOhQ1K1bFxs3bgSQ+1qrR48e6NixI4YPH46AgACkpqaie/fu6NSpE0aNGgVnZ2dpCndR596IizcwcBNgbKOabmybm17A1HIiIipbHIxcAQZSaZT/X1X4cchteIz4DI+fRABacoSGhqJGjRq4fPkyXF1d4evri2nTpuHatWsIDQ2Fk5MTLl68CHd3dwDAkiVLcO/ePUyYMAEAcOzYMdy8eRN//fVXkedK8x6QHJU7JsehJXtyiOidUpG+QzlGh96eF1YVRqISSEkBVtXPfSWkaKiyk329evVUXlFZWVlJQQ4A+Pn5ISwsDHPnzpXSatWq9cpzpUJLDji2Kb36iIhIbRjo0Nvx0qrC0iiXxIjc9FYrpC0S8rzY2ainp6dyztTUFJ06dcKsWbPyXaqoc0RE9G5hoEPqV8CqwlUUMsSnCxwIzoSVoRasfL8uvHwBPvzwQ/Tp0wd6enpo2LAhtLS0YGtrCycnpyLPERHRu4WDkUn9ClhVWE9bhp976OPngEws/icD+ulRcHNxlM7r6OigefPmAAB9fX1pvZ08bdu2xaFDhxAQEIDPPvsMc+fOlVYPLuocERG9WzgYuQIMpKrw3nBVYSIiKl8q0ncoe3RI/d5wVWEiIqLXxUCH1C9vVeF8C+3lkQHGdoWuKkxERPS6GOiQ+nFVYSIiKiMMdOjt4KrCRERUBji9nN4eF2+gTg+uKkxERG8NAx16u7iqMBERvUV8dUVEREQai4EOERERaSwGOkRERKSxGOgQERGRxmKgQ0RERBqLgQ4RERFpLAY6REREpLEY6BAREZHGeisLBoaFheHRo0eoWbMmbGxsiswbGhqKkJAQlTQdHR20a9fujeolIiKid49aA52cnByMGTMGe/bsQZ06dXD79m1MmjQJ3377baFltmzZghUrVsDNzU1Kq1Spkkqg8zr1EhER0btHrYHOjz/+iAMHDuD69euoWbMmLl++jFatWqF58+YYOHBgoeXq1q0LHx+fUq+XiIiI3i1qHaOzceNGDBw4EDVr1gQANGvWDF26dMHGjRuLLJeZmYlLly7h+vXrSE9PL7V6iYiI6N2itkAnOzsbQUFBaNKkiUp6kyZNcO3atSLLBgUFYdKkSejXrx+sra2xbt26N643IyMDiYmJKh8iIiLSbGoLdJKTk5GdnQ1zc3OV9MqVKyMuLq7Qci1atEBoaCiuXLmC+/fvY9myZZgwYQLOnTv3RvUuW7YMJiYm0sfe3v4N7o6IiIgqArUFOrq6ugCAtLQ0lfTU1FTpXEE6deoEW1tb6XjixImoW7cu9u7d+0b1zps3DwkJCdInPDy8ZDdEREREFY7aBiMrFApYWFjg8ePHKumPHz9G9erVS1RXlSpVEBER8Ub16unpQU9Pr0TXJSIioopNrYORu3Tpgv3790vHWVlZOHToELp06SKlPXjwAL6+vtJxQkKCSh2RkZG4evUqGjVqVKJ6iYiIiGRCCKGuyoODg+Hu7o4+ffqgZ8+e2Lp1Ky5duoSrV6/C2toaAPDFF19g1apViI+PBwC4u7vDy8sLrq6uePbsGb799ltoaWnBz88Ppqamxa73VRITE2FiYoKEhAQYGxur4/aJiIg0UkX6DlVrj07t2rVx6dIlKBQKbNy4EdWrV8elS5dUghEnJye0b99eOj558iSMjIywdetWnD17Fh999BGuXr0qBTnFrZeIiIhIrT065VlFikaJiIjKk4r0HcpNPYmIiEhjMdAhIiIijcVAh4iIiDQWAx0iIiLSWAx0iIiISGMx0CEiIiKNxUCHiIiINBYDHSIiItJYDHSIiIhIYzHQISIiIo3FQIeIiIg0FgMdIiIi0lgMdIiIiEhjMdAhIiIijcVAh4iIiDQWAx0iIiLSWAx0iIiISGMx0CEiIiKNxUCHiIiINBYDHSIiItJYDHSIiIhIYzHQISIiIo3FQIeIiIg0FgMdIiIi0lgMdIiIiEhjMdAhIiIijcVAh4iIiDQWAx0iIiLSWAx0iIiISGMx0CEiIiKNxUCHiIiINJb227jIjRs3EBYWBmdnZ9SpU+eV+VNSUnD16lVkZWWhQYMGqFKlisr5u3fvIigoSCVNV1cX3t7epdpuIiIiqtjUGuhkZmZi4MCBOHv2LBo3boyAgAC89957WLduHWQyWYFlvvzyS/z6669wdHSEXC7H5cuX8cUXX+Djjz+W8uzZswffffcdOnToIKUZGhoy0CEiIiIVag10Vq1ahfPnz+P69euoWrUqbt26hWbNmqF9+/Z4//33CyxjaWmJu3fvwsjICADw119/oW/fvmjfvj3c3NykfM7OztizZ486m09EREQVnFrH6GzevBmDBg1C1apVAQD16tVDt27dsHnz5kLLfPjhh1KQAwB9+vSBrq4url69qpIvLS0Nx44dg6+vL2JjY9VzA0RERFShqS3Qyc7Oxp07d9CwYUOV9IYNG+LGjRvFrufcuXPIzMxEvXr1VNIfPHiA//3vf5g5cyaqVq2KlStXFllPRkYGEhMTVT5ERESk2dQW6CQnJyMnJwdmZmYq6ZUrV0Z8fHyx6oiLi8OYMWPQo0cPtGrVSkpv3749Hj16hJMnT+LKlSvYsGEDZs2ahZMnTxZa17Jly2BiYiJ97O3tX+u+iIiIqOJQW6Cjp6cHAEhNTVVJT05Ohr6+/ivLJyYmolu3bjA1NcW2bdtUzrVu3VolgBo8eDAaNGiAAwcOFFrfvHnzkJCQIH3Cw8NLcjtERERUAaltMLKBgQGsra3x6NEjlfRHjx7BycmpyLJJSUnw8vJCdnY2fHx8YGxs/MrrmZiYIDo6utDzenp6UvBFRERE7wa1Dkbu1q0b9u3bB6VSCQBIT0/HwYMH0a1bNynP7du3VXpikpOT4eXlhczMTJw4cQKmpqb56o2KilI5Dg8PR2BgIJo1a6aeGyEiIqIKSSaEEOqqPDQ0FG5ubmjdujW6d++OHTt24P79+7hy5QoqV64MAPjiiy+watUqadxOhw4dcPnyZXz33XcqQY6LiwtcXFwAAB4eHnB3d4erqyuePXuG1atXw9raGqdOnUKlSpWK1bbExESYmJggISGhWD1GRERElKsifYeqdR2d6tWrIzAwED///DPOnDmDNm3aYMeOHVKQA+QGML1795aObWxs0LVrVxw9elSlroEDB0qBjq+vLzZt2oTz589DoVDgq6++wpAhQyCXy9V5O0RERFTBqLVHpzyrSNEoERFReVKRvkO5qScRERFpLAY6REREpLEY6BAREZHGYqBDREREGouBDhEREWksBjpERESksRjoEBERkcZioENEREQai4EOERERaSwGOkRERKSxGOgQERGRxmKgQ0RERBqLgQ4RERFpLAY6REREpLEY6BAREZHGYqBDREREGouBDhEREWksBjpERESksRjoEBERkcZioENEREQai4EOERERaSwGOkRERKSxGOgQERGRxmKgQ0RERBqLgQ4RERFpLAY6REREpLEY6BAREZHGYqBDREREGouBDhEREWksBjpERESksRjoEBERkcbSVvcFlEolzp07h7CwMDg7O8PDw6NUyrxOvURERPRuUWugk5aWhh49euDevXto3rw5/vnnH7Rr1w47d+6EXC5/7TKvUy8RERG9e9Qa6KxYsQJ3797F9evXYWFhgfv376NRo0bYsGEDxo8f/9plXqdeIiIieveodYzO9u3bMWjQIFhYWAAAatSoge7du2P79u1vVOZ16iUiIqJ3j9oCnaysLNy7dw8uLi4q6fXq1UNQUNBrl3mdegEgIyMDiYmJKh8iIiLSbGoLdJKTk6FUKmFmZqaSbm5uXmiQUZwyr1MvACxbtgwmJibSx97e/nVui4iIqNxLS0tDenp6WTejXFBboGNgYAAASEpKUklPSkqSzr1OmdepFwDmzZuHhIQE6RMeHl6CuyEiIqo4Pv74YyxZsqSsm1EuqG0wsr6+Puzs7BAaGqqS/vDhQ9SsWfO1y7xOvQCgp6cHPT29Et8HERFRRZaSkgI9PT1oa+d+5aempkJHRwc6Ojoqf87KyoKOjk6+8tnZ2ZDL5ZDJZG+76aVCrYORe/Xqhb179yIrKwtA7mungwcPolevXlKewMBAbN68uURlipOHiIiIgP79++PMmTPS8YgRI3D48GEAwJgxYzBjxgzUqFEDRkZGGDx4MIQQAIDIyEj07NkT5ubmMDIywgcffAClUlkWt/BG1BrofPbZZ4iPj0e3bt2wYsUKdOrUCWZmZpg2bZqU58CBA5gyZUqJyhQnDxERkSb58MMPcf78+ULP5ygFLtx/jv3XniAyIR3K/w9YXuX27du4cuUKYmJicPPmTZw9e1a6Xq9evRAWFoZ79+7h4cOHFXJ2s1oDHVtbW1y9ehWdO3dGaGgohgwZgkuXLsHExETK06RJE4wYMaJEZYqTh4iIqCKYOXNmsQKI0NDQfONT8xwNikDrr09hyG8XMW3HNRy/HYlNfqE4GhShkm///v04e/YskpOTpbSRI0fC1NQURkZGaN68OcLCwiCEwPHjxzFnzhw4OjrCxcUF/v7+ePDgwZvdLIBDhw7h+++/f+N6ikvtW0BYWFhgzpw5hZ739vaGt7d3icoUNw8REVF59+jRIzg5Ob12+aNBEZi4JRAv998kZ+Rg4pZAWKblIDs7GwCwcuVKpKam4p9//sHw4cMBQGX8qpaWFnJycqQ/BwUFoWrVqq/dtoI8fvwYd+7cKdU6i8JNPYmIiMqJAQMGICQkRDoeMWIEbty4oZInICAAQ4YMAZD7umrRwduIOfgNMp4Gq+TLC3we5Zhg+/Yd8PPzw+3bt5GRkYGTJ0+q5N26dSs2bdokHf/5559o2LAhPvjgA9y+fRtr1qyBt7c35syZg+3bt2PdunUAgF27dqmU27dvH9auXQsAiIqKwvTp09GtWzd4enpi9+7diIqKwvfff49Dhw7B09MTK1aseLMHVgxq79EhIiIiVTlKAf+HsYhOSsfz5ExpPE1ISIjK+jcPHjxAamqqStlmzZrh8uXLuHv3LuJ0LPDo4X1kRASjss1MKY/IzoK8kh4EAHkjb1y4+Cv2dvWCc80a6NSpE86fP49r167B0NAQurq6uH37NrKzs6FQKKCrq4tHjx6hWbNm0NbWRqdOnRAdHQ09PT1UrVoVU6dOxXvvvQcACAsLU9lj8tGjR3j8+DEAYNq0aahSpQqmT58OuVyOWrVqwcTEBF27dsX9+/cxbdo02NraqusRSxjoEBERvUVHgyKw6OBtRCTkBjQxD58j9MQ9KKqflWY8FUUmk2HixIlYu3Yt2o2YhaRrf8PItSdkMi0IZQ6y4yOR/ugGzJybAwC0jSojw3Mu0v8dC9FxOkaN7oJGBzdiw4YNWL9+PYDcAckA8MMPPwAAVq1aBV1dXXzzzTcIDg7GyJEjMXDgQAC5r7oyMjJe2U6lUgkhBMzMzNC0aVMpIKpTpw7S09Ph6elZwif3evjqioiIqBgSEhJw7949pKSkFLtMcHCwND4G+G88TV6QkycxLQsfDPFGfGq6SrCTt4zKy0aPHo09e/ZAgUyk3ruASg07AwAyox4gasd8GDg1g0ENNyl/+sOryEmNQ/De79C3Zzf8vmUHtm7dKgUsL6+RkzdOBwDi4+NhY2MjHb/455e9WO6nn36CkZERPvzwQ9jZ2eHQoUOFllMnBjpERERFSEpKwuDBg2FtbY0uXbqgSpUqmDZtmkoAU5hOnTohMjISwH/jaQrqs8lLixcKBARcBgC0bt0aN2/eLLBeMzMzdOnSBd8vmIoqdT0g11MAAPRsnFF10kaYd/4QMtl/X/HJN31g5NoDxh7vwcTjPWh7DEONmjWxf/9+ALkTfIKDc8f4KJVK+Pj4SGUbNGiAgwcPAshdPPDIkSPSuSpVqhRazsLCAsuXL0dgYCC+/vpr/PzzzwBydzhIS0t75bMrLQx0iIiIijBmzBjExcUhIiICoaGhCA4OxqlTp/D5558DyN1XqlevXti2bRsePnyIoKAgPH/+HE+fPkV2djaCg4MRFBSE88GRiEhIh8jJQnZCFIQyt/dDCCVk8twViVMT4zFl6lRUr14d58+fV5mNJYRAVFSUNDX8o48+wpEjh/HRiPcglDkQGanITnqer/056clIu+8P4+b9YVC9MfSrN0aqRT106j0EGzZsAAD07dsX//zzD5o2bYr69eur9MzMnTsXu3btQuPGjeHi4gKZTCbN1OrVq1eh5caPHw9PT0+0b98es2fPxrBhwwAALVu2xP79+9G+ffu3MhhZJorzQlADJSYmwsTEBAkJCTA2Ni7r5hARUTn09OlTVK1aFWFhYSqbQZ84cQIDBgxAbGwsduzYgblz5yIyMhJVq1aFoaEhZs+ejYcPH+Krr75CjRo1oK2tjenf/I65P+1E7IlfoKVnCJGVDnOvjxB/+neYd5mIZ4dXQqdyNVTKiEFqYjzS0tLw4MEDWFtb4/Tp0xg7diwMDQ3x/PlzTJs2DR06dMCyZctw7do11G/eFmf/+QcGLh1h2maYyj3kpCUh6/lj6Fetq5L+v961YZEejg4dOgDI3WXg1q1bcHZ2RkpKCtLT0+Hs7AwgN5gLCgpCtWrVMG7cOHTr1g2TJ09GQkICtLS0Cix38eJFJCcnQ19fH3Xq1EGVKlWka0dHR+Pu3buoUqUKXFxc1PXjA8BAh4EOERHlkzcr6uSpk/ju0ymIjoqEXOu/cSxxcXEwNzdHeHg4hg0bhqtXr8LMzAx//fUXXF1dAeSOrzE2NsaWLVtw4MABPE/JxpGDf8JqyDLoWlTH0w0fITv2MWQKU5h3HIu40+sh09HH2ctBqGGUAysrK/j7+6N27dpo1KgR1q9fD0tLSyQnJ6N9+/bQ0dGBj48PevfujWnTpuGTOXPh/zAW50Ni8OPp+6+8x+3jPdCiRuVX5ouLi8N7770HIQRCQkJgb2+PP//8E5aWlhXiO5SzroiIiF7w4qyojCePEJeYhNZfn8LCXi7wqm+DHKXAP7dyp1D7BP6LW7duoUaNGhgyJPdV0OrVqwHkDsxNT0/Hjz/+iEmTJuWufqzMhp6lY+7sqNgngFwHph7vQc8ut7clJz4S7o7mUlAVEBCAjIwMREdHY+rUqVIbq1Wrhh9++AHNm+fOrBoyZAjkWjK0qFEZ7o7m2BuYuw1EQT0ZMgDWJvpwdzQv1vMwNDTE3LlzIZPJYGtrizp16hS6QnN5xECHiIjo/728yrCOpSMA4OGNS5iYkI4JbR1x4HoE7p3eA21zO0z/32/QqdYEEdF3MWzYMLi6uuKbb75RWW34f//7H9zc3ODk5ISDhw4h42kwdK1rAhCQaetCUbsFdIwtpfwv9hwZGhrC0NAQlStXxo0bN6ClVfDQWn19fZXyC3u5YOKWQMgAlWAnr+aFvVxUrlMUXV3dtzYVXB04GJmIiAgFz4rS0tGHaevhiDm4Akk3TuCH7X8j+MR2xPn+AdN2o5EcdAoJT+4jOuYZXJu5IzExEatXr8bz5/8NCg4MDERQUBDq1KkDPV1d1Iz5B5USQwGZFnTM7SA3soC1iT5M9HMHJJ8/f15a36Zr165o2LAhrKysMGbMGPj5+SEoKAhBQUFF7iTuVd8GPw5uCHOZ6lR4axN9rBneBF71C58irmnYo0NERATA/2FsvvVtAMDYrTe0jS2QfOM4clLiIDexhGX/zyGUOZDJtWHWbiSU6cnIuucLc5nAF198gSpVqmDw4MEAgB9//BE//vgjNm3aBF1dXVRRaCPj2nZoyYBfN+1AbcdqcHc0R6djLvD19cXixYulIMba2hoAcPToUSxZsgSzZ89GYmIigNzXWgYGBqhTpw50dHTytdtBOwFhv03CwUvBiE5Kh7G2EtUrKeFY/d0JcgAORq4QA6mIiEj99l97gmk7rhU7/7OD30Cnsj1MWg6S0tYPrY++rRvg1q1bsLCwgLGxMdq3b4/U1FTcvXsXEyZMwFdffYX09HSYmpoiPT0dISEh0NPTg729PWQyGYQQSEtLg0KhQFxcHExNTV/rfoKCgtC+fXs8e/YM2dnZ+PPPP7Fy5UqcPn1a5VVXRkYGEhMTYWFhgaSkJBgaGkqvyNLT05Gamgpzc3Mpb1JSEnR1dWFiYoLbt2/D2NgYlStXLrTOxMRE6OnpITY2FhYWFtDWzu1jycnJQXR0dJELEJYGvroiIiICYGmk/+pML6jUqCsquXZXSUtWauPEiRPSl7mWlhaOHDmC1atXw8/PD1999RWA3HEveYvweXh4oEmTJkhLS8Pff/8NAPj++++hpaWFLVu2vOltAcjdk2rSpEkIDAxElSpV0K9fP2nBwwsXLqBr167o2rUrqlevDhsbG1y+nLto4dGjRzFhwgSpnrNnz0q7ngOAp6cnmjdvXmidnp6eqF27NgICAjB16lRs27ZNKrtjxw5MmTKlVO6vKAx0iIiIALg7msPGRB/FG6IL6FdrALmBkUqapZE+mjdvrtJLoa2tDVdXV9SqVUtK09LSQufOnaXjFi1aYNeuXfDy8oIQAps2bVIZABwSEoK7d+/iyZMn+fbDunv3LoDcNxXPnj1DjlLgwv3nOH4zHBlZ2bh1+w4cHBywe/duNG/eHEFBQYiLi8POnTulOu7cuYNly5bh+fPnmDNnTrEX8rtz5w7u3buHoKAgpKamYt++fdK5oKAgLFu2DBEREWjdujU++ugjaWdzAPj111/fSqDDMTpERER49WylosZ5FDRl+8Vem1f58MMPsXTpUowcORLHjx9H48aNVWZYjR49GjExMUhJSYG+vj6OHDkiLeZXt25djB07FidOnEC3IeNw3aQlbv/5E5KuHobIyUazdl3x3dpN+PGzj3D79m3Url0bWVlZuHTpEhwdc2eVubq6okmTJgByt57Ys2dPsdrdpUsX3LhxA6ampkhOTka7du2kc66urnBz+2+/rXbt2iEhIQEDBw5Ev379kJiYqJJfXdijQ0RE9P+86ttgzfAmsDZRfY1lbaKPD9o6Qgbk6/EpbMr2y702L8rrddl/7QmylQK169SFtrY2bty4gV9++QUffvihSv6zZ8/C398fp06dwuDBg/P1uHh4eODXwxdxDE0QcvE4km+ehLxS7mKAwsAE0yZ9AC19IxgYGODatWuYPHkyOnXqhJUrVwLIDcpebHfeYGgdHR2VjUVfXj/HxcUFSUlJePr0KUaOHKmyBYSBgUG++540aRIuXryInTt3vpXeHIA9OkRERCq86tugs4s1/B/GIjopHZZG+tIifq7VzKTFBPNYm+hLiwkWx4sLEgJAUno2Bvzih649BmHhwoW4f/8+2rZtK73myc7OxrBhw3DkyBFYWVkhMzMTderUUalzyNBh8PzeDwJAevgtyI3MoWtVAzlpiTBpPgAJF3bgZuAlODk64u7du9i0aROGDBmCGzduFNnW2rVrw8/PD35+ftDT08PXX3+tMjg6MjIS9+/fx61bt7Bp0yZ8/PHHRdb3/vvvY+bMmfD391cZr6NODHSIiIhekrfK8MuKCoKK4+UFCfPEJGVgh7CF7r8P8cn0qSrnfH198eDBAzx//hy6urrYtWuXylgXALgRkSoFTlq6+hDKbGjpG0FLzxAGNZpBz64Onm6chocPH2Lnzp0YO3YsLl26hKCgIHz66aeQyf5rvxAC8fHx6NChA+Li4uDq6orx48fDxsYGlpaWuHTpEgYMGAAgtyfI29sb2dnZaNmyJf78808cP34c3t7ekMlk6NatG3JycrB06VK4ubmhUqVKMDc3R4MGDaTXcR988AEGDhwIIHe21tdff41Tp05BoVBg0qRJ6NmzJ4Dc13vt2rXDwYMHoauri40bNxbrmTPQISIiKoHCgqBXKWhBwhfJ5DqoPOxbjB7TUSVdoVAgOjoaly5dQnp6Or744gvY2tqq5IlO+q+HybBuOyQG/IXUHH9om1gh63k4tM1sYdpqCJ4fWYUNGzZg4sSJqFWrFuLj4/Hpp59i5syZuH79Oho1aoRVq1ahWrVqmDt3LszNzWFubo5q1aph5MiRsLOzw6effopHjx7hxIkT+Pjjj9GqVSt4enoiMjIS//vf//DkyRNMmzYNbdq0wccff4yAgACMGDECV65cwb///ov4+HiEhIRg9erVSElJwaRJk+Dg4IDmzZtjwoQJsLOzw/Lly/Hs2TPMnDkT9vb2aNSoEUJCQnD+/HmsWLECNWrUKPZzZ6BDRET0FhS2IKG2mQ2gJYcAEJGQDv+HsWhRozJsbW1hZmaGFi1aYMKECZgxYwbs7e0xffp0XLx4USpfu3ZthD5LlY51rZwg0zWAMiUWOTKB6H1LYT10GbT0KwHIHWezfft2ODk5ISoqCgsWLEBcXBzOnj2LOnXqYPfu3YiOjlZ5RZWZmYnt27fDxcUFR44ckcbwXLhwAa1atQIAzJ8/XxqT9OWXX2LBggXw8PBA27ZtsWDBAuzfvx8LFixA1apVMW/ePHTr1g0AcPv2bezduxeurq4q1wAgtatRo0YAgLlz58LLy6tEz52BDhER0VvwYq/Li2ze/7bAfP/73/+ktPnz52P+/PnS8Ytr26zacxoTtwSq1GFQ3RV6tnVg7NYbQO6AaYWhLnLMzaU1frZu3Qq5XC6VsbOzQ2pqKuRyOUxMTFTqS05OBgCsX78ecrkcycnJaNOmDfr16yfleTEw0tXVlY5lMhm0tbXRq1cvDBkyBJ6enjAzM5PympubIyIiIt81XmxXHisrK5QUZ10RERG9BcVdkLAkCxe+6nVYnozoh0j5ZyPGjRuHypUrw9nZGf/88w8aN24sfSwsLGBmZgZHR0f89ttvKuXNzc1VyjRs2BAAUKVKlWK39UXbtm2DUqlERkYGdu3aBTc3t3zXeLFdb4KBDhER0VvwqgUJZQBsXlqL51UKex0GAIn+e/F0w0cI/3E44nfPx9CB/bFkyRIAuYHG2rVrYWdnJwUUN2/eBABs3rwZy5cvh4ODAxo3bozPP/88X5nWrVsDAG7dulXstr7MwcEBdnZ2MDIywvvvv//Kdr0u7nXFva6IiOgtyZt1BeRfkBBAiXcWL2x/ruyEaCjTkwGZDDJdA/w4oQv6NrHPly88PFzaab1WrVpQKBS5bRMCoaGhSEhIkAYjv1gmLCwMbdq0QUREBKytrXH//n1YWVmhUqXccUB3796Fo6Mj9PT0AAA3b95EvXr1oKWlJeUFcsfg2NsXr10vX6O4GOgw0CEiorfo5XV0gNyenJKsxZPnwv3nGPLbxVfm2z7e47VmihWmIn2HcjAyERHRW/Sma/G8KO91WGRCeoHjdAramuJdw0CHiIjoLXvdtXgKqqeo/bmA/FtTvGs4GJmIiKgCK2p/rpKO+dFE7NEhIiKq4ErzdZimYaBDRESkAUrrdZimUXugk5WVhb///hthYWFwdnZGly5doKVV9Buz0NBQnDt3DllZWWjWrBkaNGigcv7ixYs4d+6cSpq+vj4++uijUm8/ERERVVxqHaOTlJSEli1bYvbs2bh27RomTJgALy8vZGZmFlpm/Pjx6Ny5M44ePYozZ86gRYsW+QIYHx8frFy5EpGRkdInKipKnbdCRERUoYWGhqJNmzZl3Yy3Tq09OsuWLUNUVBRu3LgBU1NTPHnyBC4uLli7dm2hvS99+vTBr7/+KvX6fPjhh2jZsiX69++PDh06SPmqVauGb775Rp3NJyIiKrd+/fVXLF68WCXN19e30J29s7OzERER8TaaVq6otUdn9+7dGDRokLSxl52dHXr27Ildu3YVWqZHjx4qr7Y8PDygq6uLf//9VyVffHw8fv31V/zxxx+4ffu2WtpPRERUXiUlJcHd3R0XL16UPg4ODmXdrHJHbYFOZmYm7t+/j9q1a6uk16lTB3fu3Cl2PYcPH0ZmZibc3NxU0jMyMuDv748DBw7A1dUVU6dOLbKejIwMJCYmqnyIiIgqMoVCgapVq0ofbW1tXLhwAV5eXqhduzb69euHkJAQlTLffPMNGjVqhK5duyI4OBhA7mut1q1bY+XKlXB1dcXu3btx48YNeHt7S+Xu378PT09PAEBqaioAoG7dumjYsCHmzp2LnJycQuuaOXMmNm/eLNV19+5dtG3bVq3PJk+JXl2dP38eFy5cKDLPoEGDYG9vj5SUFAghVLZtB3K3cU9KSirW9cLCwjBu3DiMGTMGrq6uUnqfPn0wZ84c6OjoAADOnj2L9u3bo127dujfv3+BdS1btgyLFi0q1nWJiIjKoxylkKaQhz1PwZ9//omqVasCAOrVq4e1a9di4sSJWLVqFapVqwYfHx8MGjQIV65cAQA8ePAAMTEx2LlzJ/bs2YN+/frh5s2byM7Ohp+fHzp27Ig9e/bAxsYGt2/fRnR0tHTtrKwsREZGAgAMDAwA5I6ZzcnJwbx587B582aMGjWqwLpsbW0xY8YMafPOX375Bb17934rz6xEgU5SUpJ0k4XJG2ictzHYyz0nCQkJMDQ0fOW1nj59Ck9PTzRr1gxr1qxROVe/fn2V4zZt2qBJkybw8fEpNNCZN28eZs6cKR0nJiYWuJEYERFRefTyHlkJlx5Br3oTLPt+FTrUsYKuri527NiB+/fvY/jw4VK5iIgIZGRkAACMjY3x1VdfQS6XY8GCBVi3bh3u378PmUwGMzMzLFq0CDJZ8dfeGTx4MGJjY5GSkgInJyeMGjUKAPLV1apVK2RlZeHatWuoU6cO9u7d+8a7khdXiQIdLy8veHl5FSuvnp4eHBwccP/+fZX0+/fvw9nZuciyERER6NChA2rVqoW9e/dCV1f3ldfT1tYu8nWUnp6etIsqERFRRZK36/nL+1llQhuf+0RijbUtvOpbIi0tDX369MGyZcuwatUq9OnTB9WrV5e+R/X09CCXy6XyCoUC6enpMDAwgJmZmUqQo6WlhRf3/X5xxvRff/0FAPjuu+9Qo0YNbNy4UWWg88t1AcDkyZOxdu1aeHh4oGfPnvne+KiLWgcj9+7dG7t370Z6em70GRsbi4MHD6Jv375SHj8/P/z444/ScWRkJDp06ICaNWti3759BQYnQUFBKsc3b97ElStX3tr7PiIiolWrVmHq1KlQKpVS2g8//JBvnbc3laMUWHTwdoGbduZZdPA2cpQCHTt2xN9//43nz5/j3Llz0NbWRtWqVaWgIzo6Gnv27AGQ+9opNjYWtWrVKrBOOzs7/Pvvv4iOjoZSqcTvv/8unYuJiQEANG3aFEZGRjhw4MAr72Po0KE4cuQIVq9ejSlTphTz7t+cWgOdBQsWQKlUol27dpg/fz7atWsHBwcHlanlx48fx4IFC6TjLl264MmTJ2jZsiVWr16Nb775Bt988w38/PykPDNnzoS3tzcWLlyIyZMno1WrVujZsyfGjBmjztshIiKS+Pj44Ndff8W2bduktFOnTuUb/Pum/B/GSq+rCiIARCSkw/9hLNzc3LB8+XJ4enriypUr8Pb2RteuXaW89vb22L59OypXroyBAwdi/fr1hb7tsLKywqhRo+Dg4ABbW1uV3p1+/foBABwcHNCwYUPUrFnzlfehUCjQv39/mJqawsXFpZh3/+bUuo6OhYUFrl69im3btuHRo0eYNWsWBg8eDH39/zYea9mypTRSG8idXp6VlYW4uDiVupKTk6U/Hz9+HCdOnMClS5dQo0YNHDlyBK1bt1bnrRAREeXzwQcfYOHChRg0aJA0QSZPTk4ONm7ciKtXr8LJyQkffPABDA0N4e/vj/Pnz2PGjBlS3mXLlqFnz57STgB79uzB2bNnYWxsDHuP7lK++PPbYVC9MdIf3UR20jMYufVBVnwkkq8ewbfRf2PN1wsxbtw4jBs3Ds2aNcPUqVNx+/ZtfPzxxxg5ciT8/f1hbW2NlJQUHD58GMePH8fFixcxcuRIqSdq8eLF6NatG06cOAE7Ozs8e/ZMCoaeP38OAKhSpQoA4Pr163BwcEBqaqr0asvR0bHQXq3Q0NBXzpIubTLxYoj2DklMTISJiQkSEhJgbGxc1s0hIqIK4MVZT/+bORqTx43Gzp070LlzZ3z00Ufo06cP+vTpg1GjRmHs2LG4c+cOhg4diuPHjyMmJgZ+fn5ISEhAjRo1EBoaCiMjI8TExKBu3boIDw+HgYEBPvnkE9y/fx9dunRBZGQk1vz6G3T6LYe8khkiNs2CyExDpYadkRX3FOkPAyE3sYSiVks00YuCdnYajh49CiB3Hbq4uDiMHTsWz58/x7p163Dz5k3Y2trmu8b69eulIMjDwwNJSUkYNWoUGjVqhC5duuR7DiX9Dj137hx69eqFZs2a4fjx4yUa8PymuKknERFRMbw86yn6UTwWHriF2aNn4MvJwzB69Ggpb0pKCrZs2YKIiAiYm5tj4sSJqFGjBi5fvgw3Nzd4enpi9+7dGDNmDLZs2YJ+/frBwMAA6enp+OGHHzB48GBpOReFvi7kMbeRXakVAMCk9VAY1mkNIQQefz8YVXp9jOoujbF3sjtsbaxV2vzJJ59g7NixAHIX2t2yZQumTp2a7xra2to4deoUhg4dCiB3pvKLM7felJubG+7evQtLS8u3GuQADHSIiIheqbBZT/GpmVh+MQUuzVpj1apVUnpUVBRMTU1hbm4OAJDL5ahRo4a0RMuoUaOwfPlyjBkzBn/88Ye0jEpERAR0dHTQvn17qa727dsju3INLD2fO7NY2yj3tZFMJoNMRx/aRlWwsJcLTIyNpIX88ry4HUTNmjURERFR6DWaNm1aYLnSoKenBysrq1Kts7gY6BARERWhOLOentfujR9+mCHtBmBvb4/U1FTcunUL9erVQ1xcHK5duyad79KlC8aPH4+9e/ciPT0dLVq0AJC7j2NeUPTyBpxVHSPw3jbV3hC5lgxf9asPr/o2BbbL19dXCmh8fX3h7e1d5DXyvO1eF3VioENERFSE4sx6itUyQfvufbFr468YM2YMdHR08PXXX6NDhw5o27YtAgICMHToUGkqt1wux7BhwzB27FjMnTtXqksul+O3335D79690bp1a6lHaOnSpfCqbwcXWxOM610fljXrw9JIH+9t0UOHOoX3lBw+fBiXLl1CbGws0tLS8P777xd5DTs7u1J4YuULByNzMDIRERVh/7UnmLbjWr70tNBr0Da1ho5p7riYJd0ckP3wMlq3bi1Nt7537x5u3LgBJycnNGnSRKX806dPcfz4cfTs2VOaxZQnMjIS/v7+iI2NBZC79ZGpqSkOHz4Md3d3WFhYAMjdPLtHjx7SbgQbN26UVic+fPgwmjVrhn///RcJCQno0KGDlK8k1yhIRfoOZaBTAX5IRERUdi7cf44hv118Zb7t4z3Qokblt9CisleRvkPVumAgERFRRefuaA4bE30UNmpFBsDGRB/ujuZvs1lUTAx0iIiIiiDXkmFhr9yVfF8OdvKOF/ZygVxLcwbwahIGOkRERK/gVd8Ga4Y3gbWJvkq6tYk+1gxvUuisJyp7nHVFRERUDF71bdDZxVpaGdnSKPd1FXtyyjcGOkRERMUk15K9MwOONQVfXREREZHGYqBDREREGouBDhEREWksBjpERESksRjoEBERkcZioENEREQai4EOERERaSwGOkRERKSxGOgQERGRxmKgQ0RERBqLgQ4RERFpLAY6REREpLEY6BAREZHGYqBDREREGouBDhEREWksBjpERESksRjoEBERkcZioENEREQai4EOERERaSwGOkRERKSxtNV9gdTUVOzZswdhYWFwdnZG//79oaOjU2h+X19fnDhxQiXNwMAA8+fPf6N6iYiI6N2j1h6d2NhYNGvWDN999x3i4+OxcOFCtG3bFmlpaYWWOXv2LP744w/o6+tLHz09vTeul4iIiN49au3RWbp0KTIzMxEQEABDQ0PMnTsXtWvXxk8//YTZs2cXWs7Ozg4LFiwo9XqJiIjo3aLWHp19+/Zh4MCBMDQ0BABYWFigV69e2LdvX5Hlnj17hq+//hrff/89Lly4UGr1EhER0btFbYFORkYGQkNDUbNmTZV0Z2dnBAcHF1nWwMAA0dHRuHHjBjw9PTFixIg3rjcjIwOJiYkqHyIiItJsJXp15ePjgzNnzhSZZ+zYsXB0dERqaioAwNjYWOW8iYkJUlJSCi0/ePBgzJ8/HzKZDAAwefJkNG/eHF5eXhg6dOhr17ts2TIsWrSoyLYTERGRZilRj46Ojo7KIOGCPlpauVUaGhpCJpMhISFBpY74+HhUqlSp0GvUrFlTCnIAoEmTJmjatKkUYL1uvfPmzUNCQoL0CQ8PL8mtExERUQVUoh6ddu3aoV27dsXKq6urC0dHR9y7d08l/d69e6hTp05JLgulUinNqHrdevX09PLN3iIiIiLNptbByP3798euXbuQlJQEAIiMjMTBgwcxYMAAKc+pU6ewdOlS6fjlwceXLl1CYGAgOnXqVKJ6iYiIiGRCCKGuyuPj49GuXTvk5OSgXbt2OHr0KKpWrYrjx49LvStffPEFVq1ahfj4eAC5QUxMTAxcXV3x7Nkz7Nu3D0OGDMFvv/0GuVxe7HpfJTExESYmJkhISMg33oeIiIgKV5G+Q9Ua6ABAeno69u/fj0ePHsHZ2Rm9evWSAhYA+Oeff+Dv76+y/k1AQAAuXboEhUIBNzc3NGjQoMT1vkpF+iERERGVJxXpO1TtgU55VZF+SEREROVJRfoO5aaeREREpLEY6BAREZHGYqBDREREGouBDhEREWksBjrlQFRUFLS11bqRPBER0TuJgU45IIRATk5OWTeDiIhI4zDQKcfatWsHf39/AECrVq0QGBhYxi0iIiKqWPi+pAzlKHMQGB2I4LBg6Viu9d+ihydOnICOjg4AICMjA0qlskzaSUREVFGxR6eM+IT5oOverhhzbAyWXFwCAOi6tyt8wnykPJ07d0ZAQAAWLVqEwMBAtGrVCvr6+vDz8yurZhMREVUoDHTKgE+YD2aemYmo1CiV9OjUaMw8MxO/XP8FRx4cQWxyLLKyszB//ny4urri9OnTiI+PR4sWLcqo5URERBULX129ZTnKHCz3Xw6B/Dtv5KX9dO0nAMD9+PuYfno6ltktg0wmg66uLvT19d9qe4mIiCoyBjpvWWB0YL6enKLEpcdh5pmZSM9MV2OriIiINBNfXb1lMakx+dJ0THVQb329AvPn9fI8E8/w6NEjtbaNiIhI0zDQecssFBYFpsvkskLLCAhUalMJo8aM4mBkIiKiEmCg85Y1sWwCK4UVZCg8sMnj+KkjDGoYAABMmptgR+AOJCQkcDAyERFRMTHQecvkWnLMdZ8LAK8MdrS0tSCT/ZfHQmEBPT09lTQiIiIqHAOdMuDp4ImV7VfCUmFZrPwyyGCtsEYTyyZqbhkREZFm4ayrMuLp4IkO9h0QGB2ImNQYPEp8hJ+u/wQZZCpTz/N6fea4z1FZNZmIiIhejYFOGZJryeFm7SYd1zSrieX+y1Wmn1sprDDHfQ48HTzLoolEREQVGgOdcuTlXh4LhQWaWDZhTw4REdFrYqBTzrzcy0NERESvj4ORiYiISGMx0CEiIiKNxUCHiIiINBYDHSIiItJYDHSIiIhIYzHQISIiIo3FQIeIiIg0FgMdIiIi0lgMdIiIiEhjMdAhIiIijaX2LSDi4uKwefNmhIWFwdnZGSNGjIBCoSg0/5IlSxAZGZkv3cXFBZMmTQIAHDt2DAcPHlQ5r1Ao8L///a90G09EREQVmlp7dKKiotCkSRPs3r0bpqamWLt2LTw8PJCcnFxoGUdHR9SpU0f6VKtWDT/99BOePn0q5QkICMDhw4dV8jk7O6vzVoiIiKgCUmuPzpIlS6Cvrw8fHx/o6elh6tSpcHZ2xg8//IBPP/20wDLDhg1TOf7jjz8gk8kwduxYlXQrKyt89NFHams7ERERVXxq7dHZv38/3nvvPejp6QEATExM0KtXL+zfv7/Ydaxfvx6enp5wdHRUSY+KisK8efOwaNEi/P3336XabiIiItIMagt0MjIyEB4eni9AcXJywr///lusOv7991+cPXsW48ePz3fOzs4OCoUCiYmJGDp0KHr37g2lUllkexITE1U+REREpNlK9Orq0KFDOHr0aJF5pk2bBmdnZ6SlpQEAjIyMVM4bGxsjNTW1WNdbv349LCws0Lt3b5X0kSNHYsGCBdLxuHHj0LhxY/zxxx8YPXp0gXUtW7YMixYtKtZ1iYiISDOUKNCxsLBAnTp1isyTN6PK0NAQMpkM8fHxKufj4uJgbGz8ymtlZ2fjjz/+wMiRI6Grq6tyzt7eXuW4bt26aNq0Kc6dO1dooDNv3jzMnDlTOk5MTMxXDxEREWmWEgU6zZs3R/PmzYuVV0dHB87Ozrhz545K+p07d+Di4vLK8keOHEFkZCTGjRtXrOtlZGQgOzu70PN6enrSWCEiIiJ6N6h1MPKgQYOwc+dOxMbGAgDCwsJw6NAhDBo0SMpz5MgRfPLJJ/nKrlu3Dm3btkXt2rXznTt27JjK8cmTJ3H16lV07dq1lO+AiIiIKjK1Bjpz5sxBtWrV0LRpUwwbNgwtWrRAu3btVHpp/P39sXbtWpVyERER+PvvvwschAwAO3bsQKNGjTBy5Ej06NEDPXr0wPTp0zFkyBB13g4RERFVMDIhhFDnBXJycuDj44NHjx7B2dkZ7dq1g0wmk877+/vj+vXrKkHNvXv3cPz4cYwbNw76+voF1nvv3j34+/tDoVCgSZMmqF69eonalZiYCBMTEyQkJBRrzBARERHlqkjfoWoPdMqrivRDIiIiKk8q0ncoN/UkIiIijcVAh4iIiDQWAx0iIiLSWAx0iIiISGMx0CEiIiKNxUCHiIiINBYDHSIiItJYDHSIiIhIYzHQKUdSU1MRHh5e1s0gIiLSGAx0yhE/Pz+MHj26rJtBRESkMRjoEBERkcZioFNORUZGIiMjI196eno6Hj9+DKVSWQatIiIiqlgY6JQzsbGxaN26Ndzd3WFra4tLly5J52bOnImqVauiTZs2qF69Oi5evFiGLSUiIir/tMu6AQTkKHMQGB2ISxGXEHwvGAEBAXCp64IffvgB3333HXbs2IEdO3bg9u3b8PX1hVwux5kzZzBz5kz4+fmVdfOJiIjKLQY6ZcwnzAfL/ZcjKjUKybeSgarA1KCpmKuYCzc3N+zZswcA4Ovri+vXr6N///6IiYmBXC6Hs7NzGbeeiIiofOOrqzLkE+aDmWdmIio1SkqTyWWITo3GzDMzERAZII3FUSgUGDlyJO7evYvRo0fjk08+wfnz58uq6URERBUCe3RKWVRUFCpVqgRDQ0Okp6fj6dOncHJyAgBER0fDwMAARkZGyFHmYNnFZciMz4S2sTZkWjKpDgEBGWTYcnsLdKADABg2bBg6deoEZ2dnPHv2DFpaWnjy5Ans7OzK5D6JiIgqAvbolLKff/4Z33//PQBg69atqFGjBp48eQIA6N+/P27dugUA+HjJx/Ad54sHXz5A8PRgJF1PgpaeFnQq5wY2AgLhd8Nx/cZ1GBoaomXLlpg9ezYOHjyIQ4cO4bfffkPz5s1hYWGB4cOHIzs7G0DurKxZs2bBwcEBdnZ2mDJlCrKyssrgSRAREZU9BjqlJEeZg4DIACjqKLDvyD7kKHPg4+ODHj164OTJk0hOTsadO3fQrFkzXLp0CQd2HYDjHEdUn1MdtqNt8WTjEyhqKlB1bFWpzriTcfjkh08QExOD+Ph4zJs3D3/99RdGjRqFypUr4/Dhw7hx4wbu3buHv/76CwDwxRdfQF9fHwEBAbh8+TKioqKwZs2aMnoqREREZYuvrkrBiwOKldlKBF8Lhuc2T9z3v48tv2/B+vXrUaVKFbRo0QLa2to4c+YMYp7EIH11ulSHlo4WlJlKaOn+F3vq2uhi1ferEBMeg5nDZ8LR0VE6N2LECDRq1AgA0LVrV4SEhAAA/vzzT4SGhuLbb78FACiVSigUirfxGIiIiModBjpvKG9AsYAAAGhpa8HA0QB3D91Fqlkq0uzS4O/vj8qVK6NTp04AAAMDA3T36o6YfjGITo2Wyr7MfpI9kgKTsPXkVmxbvQ0//fATBg8eDAAqwYtcLpdeXWVnZ+PMmTNwdXVVOU9ERPQu4qurN5CjzMFy/+X5AhXDuoaIORyDSnUr4ZvAb+Dk5IRNmzahY8eOAIA+ffrg2LFjqBVUC+lP0pERlYHM55n56tfS1oKJuwnsRtrByMMI586de2WbevbsiUWLFiE2Nhb6+vrQ19eHjo5O6dwwERFRBcNA5w0ERgeqTA3PU6l+JcgVchjWN0RkaiQad2wMBwcHNGjQAACgr6+PY8eOIeZ2DDI3ZiJ8ZTgiNkfkq+fOR3dwa9wt3JpwC9F3ouH5vucr2/TVV1/B0dERDRo0gJ6eHvT19bFu3bo3v1kiIqIKSCaEKPi9iYZLTEyEiYkJEhISYGxs/Fp1HHlwBHPOznllvnl15+E91/eknpXGjRtj48aNaNy4MQDg0P1DmHduXr5yyiwlIACZtgwyLRm+bvM1ujt1R3Z2NmQymfRK6uXjPBkZGRBCQEdHh6+viIio1JTGd+jbwh6dN2ChsChWvsXjF+P69euFnrcytCowXUtHC1q6WtIaO3nX09bWVglcXj7Ok9ejwyCHiIjeVQx03kATyyawUlhBBlmB52WQoQqqQBe6iI6OxuPHj5GZ+d9YnJycHKSkpBRYT05aDkSO+K8erSpw0nVSqf/58+dIS0tTw50RERFpBgY6b0CuJcdc97kAkC/YyTt2DnbGnTt3MHr0aHh4eEg9O+vXr4ednR2qVKmCWTNnSfWkh6Xj3wX/4t4n93Bn8h1E7ckdA9Rb0RudPTtL9WdnZ6Nx48aIjo5W+30SERFVVAx03pCngydWtl8JS4WlSrqlwhKTGk9Cn3F9UKNuDRw4eACPHz+Gm5sbgNzenCdPniA8PBxbtmyBi44Lvmn7DZ6ufQrrgdaouagmnD53QurVVIw3GY/p/adDCIHLly8DAA4cOIBmzZrBwcHhrd8zERFRRcF1dEqBp4MnOth3QGB0IGJSY/Ao8RF239uNn679BAAITQzFlFNT8JXVV/B0yJ05NWrUKMjlclSpUgW1atXC48ePUbNKTWRGZSJtexqSRTLkMjlMtUxhm2MLAJg8eTLWrl2LZs2a4ZdffsG8efkHMBMREdF/GOiUErmWHG7WbvAJ88HP139WWVtHJpMhNj0WM8/MxMr2KwEAurq60nktLS0olUro6enBwMAAj8MeF7j2zdChQ/Hll1/i2rVriIqKQocOHdR/Y0RERBUYX12VosIWEJQbypF6PxVZsVlY5res0PJ2dnZwdXXFmDFjcPPmTTx+/BiPHz9GTk4OgNzVkN977z30798fH330kVrvhYiISBOoPdCJjY3FypUrMXz4cPj5+RWrTEREBL788kuMHTsWy5cvR1xc3GvledsKW0CwSo8qiD8fj5AlIQi9Gwp9U32VHh1LS0vo6ekBAPbt2wdjY2MMHToUHh4e8PDwQHh4uJR3zJgxiIuLw/Dhw9V/Q0RERBWcWgOdHTt2oEGDBnj06BG2bt2KBw8evLJMeHg4mjRpgosXL6Jx48Y4fPgw3N3dVQKZ4uQpCzGpMQWmV6pbCTUW1kCdlXWgcFLg8w2fw8XFRTq/b98+aW+qypUr46efflLp0alevTqA3Pv+448/MGHCBBgYGKj9foiIiCo6tQY6LVq0QEhICFatWlXsMosXL4alpSUOHjyIKVOm4NixY0hOTlapozh5ykJxFxAsbr6XeXl54dq1a5g7d+5rlSciInrXqDXQcXBwKHHPw6FDh9CvXz9pNV+FQgFvb28cOnSoRHnKQnEWELRWWKOJZZPXqv/WrVs4fvw4TE1N36CVRERE745yNRg5LS0NERER+daGcXBwkF57FSdPQTIyMpCYmKjyKW3FWUBwjvscyLW4JQMREdHbUKLp5Xv27MFff/1VZJ758+ejbt26r9WY9PR0AEClSpVU0o2MjKStDoqTpyDLli3DokWLXqtdJZG3gOBy/+UqA5OtFFaY4z5HWkeHiIiI1K9EgY6zszO8vLyKzGNubv7ajalUqRK0tLTyDSqOjY2FiYlJsfMUZN68eZg5c6Z0nJiYCHt7+9dua1FeXkDQQmGBJpZN2JNDRET0lpUo0GnUqBEaNWqkrrZAR0cHderUQVBQkEr6zZs30aBBg2LnKYienp40hfttyFtAkIiIiMpOmY/R2bdvHz744APpeOjQodi5cyciIyMBAMHBwfj7778xdOjQEuUhIiIiUmugc+vWLQwfPlxa3G7NmjUYPnw41q9fL+W5ceMGdu7cKR3PmjULjRs3RqNGjdCjRw+0aNECffr0wahRo0qUh4iIiEgmhBCvzvZ6IiMj4ePjky+9Zs2a8PDwAJAb6Ny9excDBw6Uzgsh4O/vj0ePHsHZ2RmNGzfOV0dx8hQlMTERJiYmSEhIgLGxcYnKEhERvcsq0neoWgOd8qwi/ZCIiIjKk4r0HVrmY3SIiIiI1IWBDhEREWksBjpERESksRjoEBERkcZioENEREQaq0QrI2uSvMlm6tjck4iISJPlfXdWhInb72ygk5SUBABq2++KiIhI0yUlJRW5z2R58M6uo6NUKvH06VMYGRlBJpOVdXPeWN4mpeHh4eV+TYOKhM9Vffhs1YfPVn34bHMJIZCUlARbW1toaZXvUTDvbI+OlpYWqlatWtbNKHXGxsbv9P986sLnqj58turDZ6s+fLYo9z05ecp3GEZERET0BhjoEBERkcZioKMh9PT0sHDhQujp6ZV1UzQKn6v68NmqD5+t+vDZVjzv7GBkIiIi0nzs0SEiIiKNxUCHiIiINBYDHSIiItJY7+w6OpoiKysLPj4+yMzMRO/evYtVJjExEefPnwcAtGrV6p1fC6IwUVFRuHTpEgwNDdG6desiBx+mp6fjr7/+ypfeunVrjVyvqbju37+PmzdvwtLSEh4eHsVaWOx1yryLrl+/jgcPHsDJyQmNGjUqMu+9e/cQGBiokqalpYWBAweqs4kV1tWrVxEcHIyOHTvC0tLylfmVSiUuXbqEqKgo1K9fHzVr1nwLraRiE1Rhffnll8Le3l5Ur15dWFlZFavM6dOnhZmZmWjWrJlwc3MTZmZm4vTp0+ptaAX0xx9/CIVCIdq2bSvq1KkjHBwcRHBwcKH5IyIiBADh5eUlBg0aJH0uXrz4FltdvsyfP18oFArh6ekp7OzshJubm4iNjS31Mu+arKwsMWDAAGFmZia6dOkizMzMxIABA0RWVlahZb777jthYmKi8ndz6NChb7HVFcOJEydE8+bNhbOzswBQrH8b4+PjhYeHh7C1tRWenp5CoVCIOXPmqL+xVGwMdCqwVatWiadPn4oVK1YUK9BJT08Xtra2Ytq0aVLa5MmThZ2dncjIyFBjSyuW8PBwoaenJ9asWSOEECI7O1t06dJFtG7dutAyeYHOzZs331Yzy7XTp08LAOKff/4RQgiRkJAgatWqJT788MNSLfMuWr16tTA1NRUPHz4UQghx//59YWxsLH788cdCy3z33XeiXr16b6mFFdeff/4p/Pz8RHh4eLEDncmTJ4tatWqJ+Ph4IYQQZ8+eFQCEj4+PmltLxcU+4Qps2rRpsLGxKXb+06dP4+nTp5gxY4aUNmvWLDx58gSnT59WRxMrpD179kBPTw9jxowBAMjlckyfPh3nzp1DaGhokWUDAgJw4MAB3L59+y20tPzasmUL3Nzc0KZNGwC5y+WPGzcO27Ztg1KpLLUy76ItW7agb9++qF69OgDAyckJffr0wZYtW4osl56ejiNHjuDEiROIiop6Cy2tePr06YMWLVoUO78QAlu3bsXYsWOl7RBat24Nd3f3V/486O1hoPMOuXnzJoyMjODg4CClOTo6wtDQEDdv3izDlpUvN2/eRO3ataGrqyulNWjQAAAQFBRUaDmZTIaffvoJP//8M1q2bIlOnTohJiZG7e0tj27evIn69eurpDVo0ACJiYl49OhRqZV5FxX2nF71/3BUVBRWrVqFhQsXolq1avjiiy/U2Mp3w+PHjxEfH/9aPw96ezgYuRzZvXs3cnJyCj1vbm6OLl26vHb9CQkJMDc3z5deuXJlxMfHv3a95V1KSgoOHjxYZJ7q1avDw8MDQMHPqXLlygBQ6HNSKBQ4d+4cWrZsCQCIjo5G69at8dFHH2Hnzp1veAcVz+s8w9cp867Jzs5Gampqgc8pJSUF2dnZ0NbO/896y5YtERoaKj3PI0eOoGfPnmjUqBH69u37VtquiRISEgCgwJ8H/86WHwx0ypGDBw8iMzOz0PM1atR4o0BHT08PycnJ+dKTk5Ohr6//2vWWd6mpqQXOiHpRmzZtpEBHT08PkZGRKufznlthz8nY2FgKcgDA0tISU6ZMwbx58yCEgEwme4M7qHgK+rv2qmf4OmXeNdra2pDL5QU+J21t7QKDHABwd3dXOe7evTtatGiBgwcPMtB5A3kzMQv6efDvbPnBQKcc2bRpk1rrr1GjBuLi4pCUlAQjIyMAuVPN4+Pj4eTkpNZrlyULCwvs2LGj2Plr1KiBs2fPqqSFhYUBQImek7GxMVJSUpCamgpDQ8Nil9MENWrUyPe6KSwsDHK5XOXV6ZuWeRc5OTkV+JwcHR1LVI+xsfE7+2q1tFSrVg3a2toF/jw0+d/UioZjdDTc3r17pYGxnp6e0NbWxr59+6Tzu3fvhra2Njw9PcuqieVO9+7dER4ejkuXLklpO3fuhK2tLRo3bgwASEtLw44dO/D48WMAQERERL569u3bh7p1675zQQ6Q+wxPnz6NZ8+eSWk7d+5Ehw4dYGBgAACIjIzEjh07kJSUVOwylPuc/vrrL2RlZQEAMjMz8ddff6FHjx5SnuDgYOzatUs6fvnvZ2RkJC5cuAA3N7e302gNEhAQgKNHjwLI7dHp1KkTdu/eLZ1/9uwZTp06pfLzoDJW1tO+6PWdOXNGbN++XQwbNkyYmJiI7du3i+3bt4vExEQpj6GhoVi8eLF0/OWXX4pKlSqJpUuXiqVLl4pKlSqpnKdcQ4cOFVWrVhXfffed+Pjjj4W2trbYuXOndD5v+unu3buFEEL88MMPokOHDmLFihXil19+Ed26dRNGRkbixIkTZXULZSojI0O4ubmJxo0bix9//FG8//77QqFQiMuXL0t5/v77bwFA3Llzp9hlKHcpAzs7O9GlSxfx888/i86dOws7OzsREREh5VmxYoWQy+XScceOHcX48ePFr7/+Kr7++mvh4OAgXF1dpSnRlOvBgwdi+/bt4qeffhIAxGeffSa2b9+usmzE2LFjVabqX716VRgaGorhw4eLn376Sbi6uoomTZqI9PT0srgFKgBfXVVgFy5cwLVr1wAAXl5e0jiUtm3bSq+mBgwYgHr16kllPvvsMzRq1AiHDh0CAGzduhXe3t5vtd0VwebNm7Fp0yb8888/UCgU8PX1VRmDo1AoMGjQINjb2wMApkyZAnd3d/z5558IDg5G+/bt8fvvv8PKyqqsbqFM6erq4syZM/jll1/g7+8PKysrBAYGonbt2lIeGxsbDBo0SFqZuzhlCLC2tsaVK1ewZs0aXLhwAS1btsTmzZtV/q7VqVMHgwYNko6PHTuG7du3w8/PD3p6eli8eDGGDh0KuVxeFrdQbj169Ej6d3TQoEG4d+8e7t27B21tbWlmlbu7O8zMzKQyjRs3RmBgINatW4dLly5h6NChmDhxYpErqdPbJRNCiLJuBBEREZE6cIwOERERaSwGOkRERKSxGOgQERGRxmKgQ0RERBqLgQ4RERFpLAY6REREpLEY6BAREZHGYqBDREREGouBDhEREWksBjpERESksRjoEBERkcZioENEREQa6/8Axzesu+iS6pkAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "groups = {\n",
    "    \"months\": [\"January\", \"February\", \"March\", \"April\", \"June\", \"July\", \"August\", \"October\", \"November\", \"December\"],\n",
    "    \"numbers\": [\"two\", \"three\", \"four\", \"five\", \"six\", \"seven\", \"eight\", \"ten\"],\n",
    "    \"pronouns\": [\"he\", \"she\", \"they\", \"it\", \"we\", \"I\"],\n",
    "}\n",
    "words = [w for ws in groups.values() for w in ws if w in word_to_id]\n",
    "X = out_vectors[[word_to_id[w] for w in words]]\n",
    "X = X - X.mean(dim=0)\n",
    "U, S, V = torch.pca_lowrank(X, q=2, center=False)\n",
    "xy = X @ V                                       # (n_words, 2)\n",
    "\n",
    "plt.figure(figsize=(6, 5))\n",
    "for name, ws in groups.items():\n",
    "    idx = [words.index(w) for w in ws if w in words]\n",
    "    plt.scatter(xy[idx, 0], xy[idx, 1], label=name)\n",
    "    for i in idx:\n",
    "        plt.annotate(words[i], xy[i].tolist(), fontsize=8)\n",
    "plt.legend()\n",
    "plt.title(\"Output word vectors, 2D PCA\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdaff382",
   "metadata": {},
   "source": [
    "## Takeaways\n",
    "- A single **config dict** makes runs reproducible and, saved in the checkpoint, documents exactly how a model was made.\n",
    "- **Caching** preprocessed data (here as `.npy`) means you pay the tokenization cost once; **memory-mapping** keeps big arrays on disk instead of in RAM.\n",
    "- **`Dataset` + `DataLoader`** separate \"how to get example *i*\" from batching, shuffling and parallel loading (`num_workers`), and the same loop works for any dataset.\n",
    "- **Checkpoints** with model + optimizer state + step let an interrupted run resume where it left off; keep a `latest` one for resuming and a `best` one for using.\n",
    "- A trained language model is also a source of **word vectors** (`emb.weight` / `out.weight`) that you can reuse in other tasks.\n",
    "- Next (Topic 6) we turn this notebook into a script with command-line hyperparameters and submit it as a batch job, where checkpoint-and-resume lets a job that hits its walltime simply be resubmitted."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}