{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0588441b",
   "metadata": {},
   "source": [
    "# Topic 1: Why PyTorch?\n",
    "\n",
    "We will build a tiny character-level **bigram** model that learns to generate new names, using **only NumPy**.\n",
    "Every piece (forward pass, loss, gradient, update) is written by hand, which gives us a baseline for appreciating what PyTorch automates for us in the next topic."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "943293d1",
   "metadata": {},
   "source": [
    "## 1. Setup and random seed\n",
    "We import NumPy and create a random number generator with a fixed seed (1337) so everyone gets the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fd79208f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:37.907016Z",
     "iopub.status.busy": "2026-10-01T17:08:37.906875Z",
     "iopub.status.idle": "2026-10-01T17:08:38.043443Z",
     "shell.execute_reply": "2026-10-01T17:08:38.042785Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "rng = np.random.default_rng(1337)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21df393d",
   "metadata": {},
   "source": [
    "## 2. Load the dataset\n",
    "`names.txt` contains ~32,000 names, one per line. We read them into a Python list and look at a few."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7202cb7f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.045361Z",
     "iopub.status.busy": "2026-10-01T17:08:38.045177Z",
     "iopub.status.idle": "2026-10-01T17:08:38.050045Z",
     "shell.execute_reply": "2026-10-01T17:08:38.049592Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32033 names\n",
      "['emma', 'olivia', 'ava', 'isabella', 'sophia', 'charlotte', 'mia', 'amelia']\n"
     ]
    }
   ],
   "source": [
    "words = open(\"names.txt\").read().splitlines()\n",
    "print(len(words), \"names\")\n",
    "print(words[:8])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe787f94",
   "metadata": {},
   "source": [
    "Neural networks work with numbers, not letters, so we map each character to an integer.\n",
    "We also add a special `.` token (index 0) that marks the start and end of a name, giving a vocabulary of 27 tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a5aa3a7a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.051778Z",
     "iopub.status.busy": "2026-10-01T17:08:38.051628Z",
     "iopub.status.idle": "2026-10-01T17:08:38.056470Z",
     "shell.execute_reply": "2026-10-01T17:08:38.056113Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27 tokens: .abcdefghijklmnopqrstuvwxyz\n"
     ]
    }
   ],
   "source": [
    "chars = sorted(set(\"\".join(words)))\n",
    "stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
    "stoi[\".\"] = 0\n",
    "itos = {i: ch for ch, i in stoi.items()}\n",
    "vocab_size = len(stoi)\n",
    "print(vocab_size, \"tokens:\", \"\".join(itos[i] for i in range(vocab_size)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19a6aac5",
   "metadata": {},
   "source": [
    "A bigram model predicts the next character from the current one only.\n",
    "We turn every name into (current, next) pairs, e.g. `.emma.` gives `(.,e) (e,m) (m,m) (m,a) (a,.)`: `xs` are the inputs and `ys` the targets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6a1e3c3c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.058229Z",
     "iopub.status.busy": "2026-10-01T17:08:38.058065Z",
     "iopub.status.idle": "2026-10-01T17:08:38.154635Z",
     "shell.execute_reply": "2026-10-01T17:08:38.154216Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "228146 training pairs\n",
      "first pairs: [('.', 'e'), ('e', 'm'), ('m', 'm'), ('m', 'a'), ('a', '.')]\n"
     ]
    }
   ],
   "source": [
    "xs, ys = [], []\n",
    "for w in words:\n",
    "    chs = [\".\"] + list(w) + [\".\"]\n",
    "    for ch1, ch2 in zip(chs, chs[1:]):\n",
    "        xs.append(stoi[ch1])\n",
    "        ys.append(stoi[ch2])\n",
    "xs = np.array(xs)\n",
    "ys = np.array(ys)\n",
    "num = len(xs)\n",
    "print(num, \"training pairs\")\n",
    "print(\"first pairs:\", [(itos[x], itos[y]) for x, y in zip(xs[:5], ys[:5])])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47a1c50d",
   "metadata": {},
   "source": [
    "To feed the integer inputs to a neural network we **one-hot encode** them: each input becomes a length-27 vector of zeros with a single 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ed76db52",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.156300Z",
     "iopub.status.busy": "2026-10-01T17:08:38.156150Z",
     "iopub.status.idle": "2026-10-01T17:08:38.174745Z",
     "shell.execute_reply": "2026-10-01T17:08:38.174306Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(228146, 27)\n",
      "[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "xenc = np.zeros((num, vocab_size))\n",
    "xenc[np.arange(num), xs] = 1.0\n",
    "print(xenc.shape)\n",
    "print(xenc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f47248",
   "metadata": {},
   "source": [
    "## 3. Initialize the model\n",
    "Our \"neural network\" is a single linear layer with no bias: just a 27x27 weight matrix `W`, initialized with random numbers.\n",
    "Row *i* of `W` will learn the (unnormalized) scores for which character follows character *i*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2caa9130",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.176520Z",
     "iopub.status.busy": "2026-10-01T17:08:38.176363Z",
     "iopub.status.idle": "2026-10-01T17:08:38.179067Z",
     "shell.execute_reply": "2026-10-01T17:08:38.178704Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(27, 27)\n"
     ]
    }
   ],
   "source": [
    "W = rng.standard_normal((vocab_size, vocab_size))\n",
    "print(W.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63ec46f5",
   "metadata": {},
   "source": [
    "## 4. Forward pass: logits and softmax\n",
    "Multiplying the one-hot inputs by `W` gives the **logits** (raw scores) for every possible next character.\n",
    "**Softmax** exponentiates and normalizes them into probabilities that sum to 1; subtracting the max first keeps `exp` numerically stable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8b823fa2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.180809Z",
     "iopub.status.busy": "2026-10-01T17:08:38.180629Z",
     "iopub.status.idle": "2026-10-01T17:08:38.376394Z",
     "shell.execute_reply": "2026-10-01T17:08:38.375920Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(228146, 27) 1.0\n"
     ]
    }
   ],
   "source": [
    "logits = xenc @ W                                     # (num, 27)\n",
    "counts = np.exp(logits - logits.max(axis=1, keepdims=True))\n",
    "probs = counts / counts.sum(axis=1, keepdims=True)    # (num, 27), each row sums to 1\n",
    "print(probs.shape, probs[0].sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b91ecca",
   "metadata": {},
   "source": [
    "## 5. Cross-entropy loss\n",
    "The **cross-entropy** loss is the average negative log-probability the model assigns to the correct next character.\n",
    "A model that guesses uniformly would score `-log(1/27) ≈ 3.30`; lower is better (our random initial weights actually start out a bit worse than uniform)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "222483c2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.378305Z",
     "iopub.status.busy": "2026-10-01T17:08:38.378168Z",
     "iopub.status.idle": "2026-10-01T17:08:38.390162Z",
     "shell.execute_reply": "2026-10-01T17:08:38.389790Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 3.7448   (uniform guess: 3.2958)\n"
     ]
    }
   ],
   "source": [
    "loss = -np.log(probs[np.arange(num), ys]).mean()\n",
    "print(f\"loss: {loss:.4f}   (uniform guess: {np.log(vocab_size):.4f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2257c17c",
   "metadata": {},
   "source": [
    "## 6. Backward pass: the gradient, derived by hand\n",
    "To improve `W` we need the gradient of the loss with respect to `W`.\n",
    "Working through the chain rule for softmax + cross-entropy gives a compact result: `dlogits = (probs - onehot(ys)) / num`, and since `logits = xenc @ W`, `dW = xenc.T @ dlogits`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "429eaae5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.391898Z",
     "iopub.status.busy": "2026-10-01T17:08:38.391754Z",
     "iopub.status.idle": "2026-10-01T17:08:38.464334Z",
     "shell.execute_reply": "2026-10-01T17:08:38.463899Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(27, 27)\n"
     ]
    }
   ],
   "source": [
    "dlogits = probs.copy()\n",
    "dlogits[np.arange(num), ys] -= 1.0\n",
    "dlogits /= num\n",
    "dW = xenc.T @ dlogits                                 # same shape as W\n",
    "print(dW.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84483a54",
   "metadata": {},
   "source": [
    "## 7. Gradient descent step (SGD)\n",
    "We nudge `W` a small step in the direction that *decreases* the loss: `W ← W − lr · dW`.\n",
    "Recomputing the forward pass shows the loss went down a little."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f5d86d17",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.466198Z",
     "iopub.status.busy": "2026-10-01T17:08:38.466044Z",
     "iopub.status.idle": "2026-10-01T17:08:38.664951Z",
     "shell.execute_reply": "2026-10-01T17:08:38.664478Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss after one step: 3.3715\n"
     ]
    }
   ],
   "source": [
    "lr = 50.0\n",
    "W -= lr * dW\n",
    "\n",
    "logits = xenc @ W\n",
    "counts = np.exp(logits - logits.max(axis=1, keepdims=True))\n",
    "probs = counts / counts.sum(axis=1, keepdims=True)\n",
    "loss = -np.log(probs[np.arange(num), ys]).mean()\n",
    "print(f\"loss after one step: {loss:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40a874d8",
   "metadata": {},
   "source": [
    "## 8. Generate names with the barely-trained model\n",
    "To generate a name we start from `.`, turn that character's row of `W` into probabilities, sample the next character, and repeat until we sample `.` again.\n",
    "After a single step the model is still close to random, so expect gibberish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "039bae15",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.666829Z",
     "iopub.status.busy": "2026-10-01T17:08:38.666668Z",
     "iopub.status.idle": "2026-10-01T17:08:38.674999Z",
     "shell.execute_reply": "2026-10-01T17:08:38.674552Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jygduyxkrvbdpkicgvcrhicdmb\n",
      "ofwsnaintenigcdymfvbidqevm\n",
      "aysavbbrtedi\n",
      "fddldudggbtjtulusiqcrvrrdkqsenxijouycurrhlzszkflsilpdufqelcbsztulnahltevbrdmpktfwjhenlhiesqzmzmpkeig\n",
      "ohjkthisqstuj\n"
     ]
    }
   ],
   "source": [
    "def generate(W, n=5):\n",
    "    for _ in range(n):\n",
    "        out, ix = [], 0\n",
    "        while True:\n",
    "            row = W[ix]                               # same as onehot(ix) @ W\n",
    "            p = np.exp(row - row.max())\n",
    "            p /= p.sum()\n",
    "            ix = rng.choice(vocab_size, p=p)\n",
    "            if ix == 0:\n",
    "                break\n",
    "            out.append(itos[ix])\n",
    "        print(\"\".join(out))\n",
    "\n",
    "generate(W)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb6b0a66",
   "metadata": {},
   "source": [
    "## 9. Train for 100 epochs and generate again\n",
    "Now we repeat forward pass → loss → hand-written gradient → SGD step for 100 epochs (each epoch uses all training pairs at once).\n",
    "The loss should drop to about 2.5 and the generated names start to look more name-like (a bigram model only sees one character of context, so don't expect miracles)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "02483a46",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-01T17:08:38.676691Z",
     "iopub.status.busy": "2026-10-01T17:08:38.676543Z",
     "iopub.status.idle": "2026-10-01T17:09:05.751581Z",
     "shell.execute_reply": "2026-10-01T17:09:05.750946Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch   0  loss 3.3715\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  10  loss 2.6557\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  20  loss 2.5621\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  30  loss 2.5268\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  40  loss 2.5085\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  50  loss 2.4971\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  60  loss 2.4892\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  70  loss 2.4836\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  80  loss 2.4793\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  90  loss 2.4761\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch  99  loss 2.4737\n",
      "flamayara\n",
      "reazach\n",
      "ayleeya\n",
      "brerelle\n",
      "talielar\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(100):\n",
    "    # forward\n",
    "    logits = xenc @ W\n",
    "    counts = np.exp(logits - logits.max(axis=1, keepdims=True))\n",
    "    probs = counts / counts.sum(axis=1, keepdims=True)\n",
    "    loss = -np.log(probs[np.arange(num), ys]).mean()\n",
    "\n",
    "    # backward (hand-derived)\n",
    "    dlogits = probs.copy()\n",
    "    dlogits[np.arange(num), ys] -= 1.0\n",
    "    dlogits /= num\n",
    "    dW = xenc.T @ dlogits\n",
    "\n",
    "    # update\n",
    "    W -= lr * dW\n",
    "\n",
    "    if epoch % 10 == 0 or epoch == 99:\n",
    "        print(f\"epoch {epoch:3d}  loss {loss:.4f}\")\n",
    "\n",
    "generate(W)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1a93f3c",
   "metadata": {},
   "source": [
    "## Takeaways\n",
    "This worked, but notice what we had to do ourselves: derive the gradient with the chain rule, write the backward pass by hand, and keep it in sync with the forward pass.\n",
    "That is manageable for one matrix, but for a model with many layers it becomes error-prone very quickly, and NumPy also can't run on a GPU.\n",
    "In Topic 2 we rewrite this same model in PyTorch and let **autograd** compute the gradients for us."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}