exercise_720.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backpropagation Again\n",
    "\n",
    "We run through the backpropagation procedure again.\n",
    "\n",
    "This time, we assume that the features are $\\begin{bmatrix}8 & -2\\end{bmatrix}$ and the correct category is $0$ which is one-hot encoded as $\\begin{bmatrix}1 & 0\\end{bmatrix}$.\n",
    "\n",
    "In this exercise, we compute those partial derivatives manually, so we can learn the nuts and bolts of the backpropagation algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters (weights): tensor([[ 1.,  2.],\n",
      "        [ 2.,  1.]])\n",
      "Parameters (bias): tensor([[ 0.,  0.]])\n",
      "Features: tensor([[ 8., -2.]])\n",
      "One hot encoding of the correct class: tensor([[ 1.,  0.]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "weights = torch.nn.Parameter(torch.Tensor([[1,2],[2,1]]))\n",
    "print(\"Parameters (weights): \"+str(weights.data))\n",
    "\n",
    "bias = torch.nn.Parameter(torch.Tensor([[0,0]]))\n",
    "print(\"Parameters (bias): \"+str(bias.data))\n",
    "\n",
    "features = torch.autograd.Variable(torch.Tensor([[8, -2]]))\n",
    "print(\"Features: \"+str(features.data))\n",
    "\n",
    "target = torch.autograd.Variable(torch.LongTensor([0]))\n",
    "#print(\"The correct class: \"+str(target.data))\n",
    "\n",
    "one_hot_target = torch.autograd.Variable(torch.Tensor([[1, 0]]))\n",
    "print(\"One hot encoding of the correct class: \"+str(one_hot_target.data))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume the weights are $\\begin{bmatrix}1 & 2 \\\\ 2 & 1\\end{bmatrix}$ at start.\n",
    "\n",
    "The training is assumed to proceed one data point at a time (batch size of 1).\n",
    "\n",
    "For this iteration of training, the features are $\\begin{bmatrix}-10 & 20\\end{bmatrix}$ and the correct category is $1$ which is one-hot encoded as $\\begin{bmatrix}0 & 1\\end{bmatrix}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c: tensor([[  4.,  14.]])\n",
      "Softmax of c: tensor([[ 0.0000,  1.0000]])\n",
      "Log softmax of c: tensor([[-1.0000e+01, -4.5399e-05]])\n",
      "NLL + log_softmax loss: tensor(10.0000)\n",
      "Cross entropy loss: tensor(10.0000)\n",
      "transpose of features: tensor([[ 8.],\n",
      "        [-2.]])\n",
      "grad of loss wrt to c: tensor([[-1.0000,  1.0000]])\n",
      "grad of loss wrt to weights: tensor([[-7.9996,  7.9996],\n",
      "        [ 1.9999, -1.9999]])\n",
      "grad of loss wrt to bias: tensor([[-1.0000,  1.0000]])\n",
      "\tThe manually computed gradient of the loss with respect to weights is tensor([[-7.9996,  7.9996],\n",
      "        [ 1.9999, -1.9999]])\n",
      "\tThe manually computed gradient of the loss with respect to bias is tensor([[-1.0000,  1.0000]])\n",
      "\tThe weights are now tensor([[ 1.0800,  1.9200],\n",
      "        [ 1.9800,  1.0200]])\n",
      "\tThe bias is now tensor(1.00000e-03 *\n",
      "       [[ 9.9995, -9.9995]])\n"
     ]
    }
   ],
   "source": [
    "if weights.grad is not None:\n",
    "    weights.grad.data.zero_()\n",
    "\n",
    "# Forward pass\n",
    "\n",
    "result = torch.mm(features, weights) + bias\n",
    "print(\"c: \"+str(result.data))\n",
    "\n",
    "softmax_result = F.softmax(result, dim=1)\n",
    "print(\"Softmax of c: \"+str(softmax_result.data))\n",
    "\n",
    "log_softmax_result = F.log_softmax(result, dim=1)\n",
    "print(\"Log softmax of c: \"+str(log_softmax_result.data))\n",
    "\n",
    "loss_nll_softmax = F.nll_loss(log_softmax_result, target)\n",
    "print(\"NLL + log_softmax loss: \"+str(loss_nll_softmax.data))\n",
    "\n",
    "loss = F.cross_entropy(result, target)\n",
    "print(\"Cross entropy loss: \"+str(loss.data))\n",
    "\n",
    "# Backward pass\n",
    "\n",
    "print(\"transpose of features: \"+str(features.data.t()))\n",
    "\n",
    "grad_c = (softmax_result.data - one_hot_target.data)\n",
    "print(\"grad of loss wrt to c: \"+str(grad_c))\n",
    "\n",
    "grad_weights = features.data.t().mm(grad_c)\n",
    "print(\"grad of loss wrt to weights: \"+str(grad_weights))\n",
    "\n",
    "grad_bias = grad_c\n",
    "print(\"grad of loss wrt to bias: \"+str(grad_bias))\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to weights is \"+str(grad_weights))\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to bias is \"+str(grad_bias))\n",
    "\n",
    "# You can now update the weights and bias\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "weights.data = weights.data - learning_rate * grad_weights\n",
    "\n",
    "bias.data = bias.data - learning_rate * grad_bias\n",
    "\n",
    "print(\"\\tThe weights are now \"+str(weights.data))\n",
    "\n",
    "print(\"\\tThe bias is now \"+str(bias.data))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Parameters\n",
    "\n",
    "At the end of the first pass, the weights (after being nudged around) should look something like this\n",
    "\n",
    "$$\\begin{bmatrix}1.08 & 1.92 \\\\ 1.98 & 1.02\\end{bmatrix}$$\n",
    "\n",
    "and the bias should look like this\n",
    "\n",
    "$$\\begin{bmatrix}0.01 & -0.01\\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-Check Results\n",
    "\n",
    "You can check that our computations are correct by verifying the same with Pytorch's automatic backward pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "grad of loss wrt to weights: tensor([[-7.9996,  7.9996],\n",
      "        [ 1.9999, -1.9999]])\n",
      "grad of loss wrt to bias: tensor([[-1.0000,  1.0000]])\n"
     ]
    }
   ],
   "source": [
    "# We can uncomment the following two lines to check our gradient against the automatically computed gradients\n",
    "\n",
    "loss.backward(retain_graph=True)  # Setting retain_graph=True allows us to call loss.backward repeatedly in a local scope\n",
    "\n",
    "grad_weights = weights.grad.data\n",
    "print(\"grad of loss wrt to weights: \"+str(grad_weights))\n",
    "\n",
    "grad_bias = bias.grad.data\n",
    "print(\"grad of loss wrt to bias: \"+str(grad_bias))\n",
    "\n",
    "if weights.grad is not None:\n",
    "    weights.grad.data.zero_()\n",
    "if bias.grad is not None:\n",
    "    bias.grad.data.zero_()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}