bag_of_tricks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "bag-of-tricks.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "JZ_zmZhTyOSN",
        "ZiaIqUQKWHgT",
        "5HShJN2xh22e"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/davidcpage/cifar10-fast/blob/master/bag_of_tricks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JZ_zmZhTyOSN",
        "colab_type": "text"
      },
      "source": [
        "### Recommended: running on GCP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XUJEUs3ND0eq",
        "colab_type": "text"
      },
      "source": [
        "You can run this notebook using a free Colab GPU instance (Tesla T4) but timings will be considerably slower than on a V100. \n",
        "If you have a GCP account and want to  use a faster V100 GPU you can follow the instructions [here](https://blog.kovalevskyi.com/gce-deeplearning-images-as-a-backend-for-google-colaboratory-bc4903d24947) to use that as an alternative Colab backend. Don't forget to shut down the GCP instance (not just the Colab notebook) once you've finished!\n",
        "\n",
        "FWIW, I use the following (preemptible) instance type:\n",
        "\n",
        "```\n",
        "export IMAGE_FAMILY=\"pytorch-latest-gpu\"\n",
        "export ZONE=\"europe-west4-a\"\n",
        "export INSTANCE_NAME=\"pytorch-colab-backend\"\n",
        "\n",
        "gcloud compute instances create $INSTANCE_NAME \\\n",
        "  --zone $ZONE \\\n",
        "  --machine-type n1-standard-4 \\\n",
        "  --accelerator type=nvidia-tesla-v100,count=1 \\\n",
        "  --image-family $IMAGE_FAMILY \\\n",
        "  --image-project=deeplearning-platform-release \\\n",
        "  --metadata install-nvidia-driver=True \\\n",
        "  --maintenance-policy TERMINATE \\\n",
        "  --preemptible \n",
        "  ```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KjOIoRO9ZNE2",
        "colab_type": "text"
      },
      "source": [
        "On GCP you will need to install altair for plotting. This is pre-installed on Colab instances."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8m1LCN_bZIrs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#GCP only\n",
        "!python -m pip install -q vega altair"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ns4mKklxZGIT",
        "colab_type": "text"
      },
      "source": [
        "You can check the details of your setup (free Colab GPU or V100 on GCP) with this nice utility from fastai:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OpxpGgl_yJ8r",
        "colab_type": "code",
        "outputId": "81784a4d-6135-49af-fdb0-e7fbb2875338",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 656
        }
      },
      "source": [
        "from fastai.utils.show_install import show_install \n",
        "show_install()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "```text\n",
            "=== Software === \n",
            "python        : 3.7.3\n",
            "fastai        : 1.0.55\n",
            "fastprogress  : 0.1.21\n",
            "torch         : 1.1.0\n",
            "nvidia driver : 410.104\n",
            "torch cuda    : 10.0.130 / is available\n",
            "torch cudnn   : 7501 / is enabled\n",
            "\n",
            "=== Hardware === \n",
            "nvidia gpus   : 1\n",
            "torch devices : 1\n",
            "  - gpu0      : 16130MB | Tesla V100-SXM2-16GB\n",
            "\n",
            "=== Environment === \n",
            "platform      : Linux-4.9.0-9-amd64-x86_64-with-debian-9.9\n",
            "distro        : #1 SMP Debian 4.9.168-1+deb9u4 (2019-07-19)\n",
            "conda env     : base\n",
            "python        : /opt/anaconda3/bin/python\n",
            "sys.path      : /home/jupyter\n",
            "/opt/anaconda3/lib/python37.zip\n",
            "/opt/anaconda3/lib/python3.7\n",
            "/opt/anaconda3/lib/python3.7/lib-dynload\n",
            "\n",
            "/opt/anaconda3/lib/python3.7/site-packages\n",
            "/opt/anaconda3/lib/python3.7/site-packages/IPython/extensions\n",
            "/home/jupyter/.ipython\n",
            "```\n",
            "\n",
            "Please make sure to include opening/closing ``` when you paste into forums/github to make the reports appear formatted as code sections.\n",
            "\n",
            "Optional package(s) to enhance the diagnostics can be installed with:\n",
            "pip install distro\n",
            "Once installed, re-run this utility to get the additional information\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6IGml3SMd6ID",
        "colab_type": "text"
      },
      "source": [
        "# How to Train Your ResNet - 8: Bag of Tricks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tac7LB2td_Cs",
        "colab_type": "text"
      },
      "source": [
        "In the [final post of the series](https://myrtle.ai/how-to-train-your-resnet-8-bag-of-tricks/) we come full circle, speeding up our single-GPU training implementation to take on a field of multi-GPU competitors. Whilst we've been otherwise occupied - investigating [hyperparameter tuning](https://myrtle.ai/learn/how-to-train-your-resnet-5-hyperparameters/), [weight decay](https://myrtle.ai/learn/how-to-train-your-resnet-6-weight-decay/) and [batch norm](https://myrtle.ai/learn/how-to-train-your-resnet-7-batch-norm/) - our entry for training CIFAR10 to 94% test accuracy has slipped five (!) places on the DAWNBench leaderboard:\n",
        "\n",
        "![](https://drive.google.com/uc?id=1MzB0-FR_oIPVe9PCffvRI9DB13m2YI21)\n",
        "\n",
        "The top six entries all use 9-layer ResNets which are cousins - or twins - of the network we developed  [earlier in the series](https://myrtle.ai/learn/how-to-train-your-resnet-4-architecture/). First place is a 4-GPU implementation from Kakao Brain which completes in an impressive 37s. The single-GPU version of the same comes in third with 68s, an apparent 7s improvement over our single-GPU entry from last year, although close inspection shows that these submissions are using test-time augmentation (TTA). We shall discuss the validity of this approach towards the end of the post (our conclusion is that any reasonable restriction should be based on total inference cost and that the form of mild TTA used here, along with a lightweight network, passes on that front.) Note that our earlier submission, allowing the same TTA, would achieve a time of 60s on a 19 epoch training schedule without further changes.\n",
        "\n",
        "By the end of the post our single-GPU implementation surpasses the top multi-GPU times comfortably, reclaiming the coveted DAWNBench crown with a time of 34s and achieving a 10× improvement over the single-GPU state-of-the-art at the start of the series! Using the same TTA employed by the Kakao Brain submission, this drops to 26s. We achieve these times by accumulating a series of small (typically 0.1-0.3% in absolute test accuracy) improvements, which can be traded for shorter training times. These improvements are based on a collection of standard and not-so-standard tricks. \n",
        "\n",
        "Our main weapon is statistical significance. The standard deviation in test accuracy for a single training run is roughly 0.15% and when comparing between two runs we need to multiply this by $\\sqrt{2}$. This is larger than many of the effects that we are measuring. Given that training times soon drop below a minute, we can afford to run experiments 10s-100s of times to make sure that improvements are real and this allows us to make consistent progress.\n",
        "\n",
        "Sharp experimental results are essential to advancing the field but if a baseline is poorly-tuned or the number of runs too few, experimental validation holds little value. The main goal of today's post is to provide a well-tuned baseline on which to test novel techniques, allowing one to complete a statistically significant number of training runs within minutes on a single GPU. We confirm, at the end of the post, that improvements in training speed translate into improvements in final accuarcy if training is allowed to proceed towards convergence.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SWyjxzQb_yMA",
        "colab_type": "text"
      },
      "source": [
        "### Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7WeTS0FF4iQE",
        "colab": {},
        "cellView": "form"
      },
      "source": [
        "#@title Lib (RUN ME) - double-click to show/hide code\n",
        "####################\n",
        "## CORE\n",
        "#####################\n",
        "\n",
        "import inspect\n",
        "from collections import namedtuple, defaultdict\n",
        "from functools import partial\n",
        "import functools\n",
        "from itertools import chain, count, islice as take\n",
        "\n",
        "#####################\n",
        "## dict utils\n",
        "#####################\n",
        "\n",
        "union = lambda *dicts: {k: v for d in dicts for (k, v) in d.items()}\n",
        "\n",
        "make_tuple = lambda path: (path,) if isinstance(path, str) else path\n",
        "\n",
        "def path_iter(nested_dict, pfx=()):\n",
        "    for name, val in nested_dict.items():\n",
        "        if isinstance(val, dict): yield from path_iter(val, pfx+make_tuple(name))\n",
        "        else: yield (pfx+make_tuple(name), val)  \n",
        "            \n",
        "map_values = lambda func, dct: {k: func(v) for k,v in dct.items()}\n",
        "\n",
        "def map_nested(func, nested_dict):\n",
        "    return {k: map_nested(func, v) if isinstance(v, dict) else func(v) for k,v in nested_dict.items()}\n",
        "\n",
        "def group_by_key(seq):\n",
        "    res = defaultdict(list)\n",
        "    for k, v in seq: \n",
        "        res[k].append(v) \n",
        "    return res\n",
        "\n",
        "reorder = lambda dct, keys: {k: dct[k] for k in keys}\n",
        "\n",
        "#####################\n",
        "## graph building\n",
        "#####################\n",
        "\n",
        "def identity(value): return value\n",
        "\n",
        "def build_graph(net, path_map='_'.join):\n",
        "    net = {path: node if len(node) is 3 else (*node, None) for path, node in path_iter(net)}\n",
        "    default_inputs = chain([('input',)], net.keys())\n",
        "    resolve_path = lambda path, pfx: pfx+path if (pfx+path in net or not pfx) else resolve_path(net, path, pfx[:-1])\n",
        "    return {path_map(path): (typ, value, ([path_map(default)] if inputs is None else [path_map(resolve_path(make_tuple(k), path[:-1])) for k in inputs])) \n",
        "            for (path, (typ, value, inputs)), default in zip(net.items(), default_inputs)}\n",
        "\n",
        "#####################\n",
        "## network visualisation (requires pydot)\n",
        "#####################\n",
        "import IPython.display\n",
        "\n",
        "class ColorMap(dict):\n",
        "    palette = (\n",
        "        'bebada,ffffb3,fb8072,8dd3c7,80b1d3,fdb462,b3de69,fccde5,bc80bd,ccebc5,ffed6f,1f78b4,33a02c,e31a1c,ff7f00,'\n",
        "        '4dddf8,e66493,b07b87,4e90e3,dea05e,d0c281,f0e189,e9e8b1,e0eb71,bbd2a4,6ed641,57eb9c,3ca4d4,92d5e7,b15928'\n",
        "    ).split(',')\n",
        " \n",
        "    def __missing__(self, key):\n",
        "        self[key] = self.palette[len(self) % len(self.palette)]\n",
        "        return self[key]\n",
        "\n",
        "def make_pydot(nodes, edges, direction='LR', sep='_', **kwargs):\n",
        "    from pydot import Dot, Cluster, Node, Edge\n",
        "    class Subgraphs(dict):\n",
        "        def __missing__(self, path):\n",
        "            *parent, label = path\n",
        "            subgraph = Cluster(sep.join(path), label=label, style='rounded, filled', fillcolor='#77777744')\n",
        "            self[tuple(parent)].add_subgraph(subgraph)\n",
        "            return subgraph\n",
        "    g = Dot(rankdir=direction, directed=True, **kwargs)\n",
        "    g.set_node_defaults(\n",
        "        shape='box', style='rounded, filled', fillcolor='#ffffff')\n",
        "    subgraphs = Subgraphs({(): g})\n",
        "    for path, attr in nodes:\n",
        "        *parent, label = path.split(sep)\n",
        "        subgraphs[tuple(parent)].add_node(\n",
        "            Node(name=path, label=label, **attr))\n",
        "    for src, dst, attr in edges:\n",
        "        g.add_edge(Edge(src, dst, **attr))\n",
        "    return g\n",
        "\n",
        "class DotGraph():\n",
        "    colors = ColorMap()   \n",
        "    def __init__(self, graph, size=15, direction='LR'):\n",
        "        self.nodes = [(k, {\n",
        "            'tooltip': '%s %.1000r' % (typ, value), \n",
        "            'fillcolor': '#'+self.colors[typ],\n",
        "        }) for k, (typ, value, inputs) in graph.items()] \n",
        "        self.edges = [(src, k, {}) for (k, (_,_,inputs)) in graph.items() for src in inputs]\n",
        "        self.size, self.direction = size, direction\n",
        "\n",
        "    def dot_graph(self, **kwargs):\n",
        "        return make_pydot(self.nodes, self.edges, size=self.size, \n",
        "                            direction=self.direction, **kwargs)\n",
        "\n",
        "    def svg(self, **kwargs):\n",
        "        return self.dot_graph(**kwargs).create(format='svg').decode('utf-8')\n",
        "\n",
        "    try:\n",
        "        import pydot\n",
        "        def _repr_svg_(self):\n",
        "            return self.svg()\n",
        "    except ImportError:\n",
        "        def __repr__(self):\n",
        "            return 'pydot is needed for network visualisation'\n",
        "\n",
        "\n",
        "#####################\n",
        "## Layers\n",
        "##################### \n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch import nn\n",
        "import torch.nn.functional as F\n",
        "from collections import namedtuple\n",
        "import copy\n",
        "\n",
        "torch.backends.cudnn.benchmark = True\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "cpu = torch.device('cpu')\n",
        "    \n",
        "class Network(nn.Module):\n",
        "    def __init__(self, net, loss=None):\n",
        "        super().__init__()\n",
        "        self.graph = {path: (typ, typ(**params), inputs) for path, (typ, params, inputs) in build_graph(net).items()}\n",
        "        self.loss = loss or identity\n",
        "        for path, (_,node,_) in self.graph.items(): \n",
        "            setattr(self, path, node)\n",
        "    \n",
        "    def nodes(self):\n",
        "        return (node for _,node,_ in self.graph.values())\n",
        "    \n",
        "    def forward(self, inputs):\n",
        "        outputs = dict(inputs)\n",
        "        for k, (_, node, ins) in self.graph.items():\n",
        "            outputs[k] = node(*[outputs[x] for x in ins])\n",
        "        return outputs\n",
        "    \n",
        "    def half(self):\n",
        "        for node in self.nodes():\n",
        "            if isinstance(node, nn.Module) and not isinstance(node, nn.BatchNorm2d):\n",
        "                node.half()\n",
        "        return self\n",
        "\n",
        "build_model = lambda network, loss: Network(network, loss).half().to(device)\n",
        "show = lambda network, size=15: display(DotGraph(network.graph if isinstance(network, Network) else build_graph(network), size=size))\n",
        "    \n",
        "class Add(namedtuple('Add', [])):\n",
        "    def __call__(self, x, y): return x + y \n",
        "    \n",
        "class AddWeighted(namedtuple('AddWeighted', ['wx', 'wy'])):\n",
        "    def __call__(self, x, y): return self.wx*x + self.wy*y \n",
        "    \n",
        "class Identity(namedtuple('Identity', [])):\n",
        "    def __call__(self, x): return x\n",
        "\n",
        "class BatchNorm(nn.BatchNorm2d):\n",
        "    def __init__(self, num_features, eps=1e-05, momentum=0.1, weight=True, bias=True):\n",
        "        super().__init__(num_features, eps=eps, momentum=momentum)\n",
        "        self.weight.data.fill_(1.0)\n",
        "        self.bias.data.fill_(0.0)\n",
        "        self.weight.requires_grad = weight\n",
        "        self.bias.requires_grad = bias\n",
        "\n",
        "class GhostBatchNorm(BatchNorm):\n",
        "    def __init__(self, num_features, num_splits, **kw):\n",
        "        super().__init__(num_features, **kw)\n",
        "        self.num_splits = num_splits\n",
        "        self.register_buffer('running_mean', torch.zeros(num_features*self.num_splits))\n",
        "        self.register_buffer('running_var', torch.ones(num_features*self.num_splits))\n",
        "\n",
        "    def train(self, mode=True):\n",
        "        if (self.training is True) and (mode is False): #lazily collate stats when we are going to use them\n",
        "            self.running_mean = torch.mean(self.running_mean.view(self.num_splits, self.num_features), dim=0).repeat(self.num_splits)\n",
        "            self.running_var = torch.mean(self.running_var.view(self.num_splits, self.num_features), dim=0).repeat(self.num_splits)\n",
        "        return super().train(mode)\n",
        "        \n",
        "    def forward(self, input):\n",
        "        N, C, H, W = input.shape\n",
        "        if self.training or not self.track_running_stats:\n",
        "            return F.batch_norm(\n",
        "                input.view(-1, C*self.num_splits, H, W), self.running_mean, self.running_var, \n",
        "                self.weight.repeat(self.num_splits), self.bias.repeat(self.num_splits),\n",
        "                True, self.momentum, self.eps).view(N, C, H, W) \n",
        "        else:\n",
        "            return F.batch_norm(\n",
        "                input, self.running_mean[:self.num_features], self.running_var[:self.num_features], \n",
        "                self.weight, self.bias, False, self.momentum, self.eps)\n",
        "        \n",
        "class Mul(nn.Module):\n",
        "    def __init__(self, weight):\n",
        "        super().__init__()\n",
        "        self.weight = weight\n",
        "    def __call__(self, x): \n",
        "        return x*self.weight\n",
        "    \n",
        "class Flatten(nn.Module):\n",
        "    def forward(self, x): \n",
        "        return x.view(x.size(0), x.size(1))\n",
        "\n",
        "# Losses\n",
        "class CrossEntropyLoss(namedtuple('CrossEntropyLoss', [])):\n",
        "    def __call__(self, log_probs, target):\n",
        "        return torch.nn.functional.nll_loss(log_probs, target, reduction='none')\n",
        "    \n",
        "class KLLoss(namedtuple('KLLoss', [])):        \n",
        "    def __call__(self, log_probs):\n",
        "        return -log_probs.mean(dim=1)\n",
        "\n",
        "class Correct(namedtuple('Correct', [])):\n",
        "    def __call__(self, classifier, target):\n",
        "        return classifier.max(dim = 1)[1] == target\n",
        "\n",
        "class LogSoftmax(namedtuple('LogSoftmax', ['dim'])):\n",
        "    def __call__(self, x):\n",
        "        return torch.nn.functional.log_softmax(x, self.dim, _stacklevel=5)\n",
        "\n",
        "    \n",
        "# node definitions   \n",
        "from inspect import signature    \n",
        "empty_signature = inspect.Signature()\n",
        "\n",
        "class node_def(namedtuple('node_def', ['type'])):\n",
        "    def __call__(self, *args, **kwargs):\n",
        "        return (self.type, dict(signature(self.type).bind(*args, **kwargs).arguments))\n",
        "\n",
        "conv = node_def(nn.Conv2d)\n",
        "linear = node_def(nn.Linear)\n",
        "batch_norm = node_def(BatchNorm)\n",
        "pool = node_def(nn.MaxPool2d)\n",
        "relu = node_def(nn.ReLU)\n",
        "    \n",
        "def map_types(mapping, net):\n",
        "    def f(node):\n",
        "        typ, *rest = node\n",
        "        return (mapping.get(typ, typ), *rest)\n",
        "    return map_nested(f, net) \n",
        "\n",
        "#####################\n",
        "## Compat\n",
        "##################### \n",
        "\n",
        "def to_numpy(x):\n",
        "    if isinstance(x, torch.Tensor):\n",
        "        return x.detach().cpu().numpy()  \n",
        "    return x\n",
        "  \n",
        "def flip_lr(x):\n",
        "    if isinstance(x, torch.Tensor):\n",
        "        return torch.flip(x, [-1]) \n",
        "    return x[..., ::-1].copy()\n",
        "  \n",
        "trainable_params = lambda model: {k:p for k,p in model.named_parameters() if p.requires_grad}\n",
        "\n",
        "#####################\n",
        "## Optimisers\n",
        "##################### \n",
        "\n",
        "from functools import partial\n",
        "\n",
        "def nesterov_update(w, dw, v, lr, weight_decay, momentum):\n",
        "    dw.add_(weight_decay, w).mul_(-lr)\n",
        "    v.mul_(momentum).add_(dw)\n",
        "    w.add_(dw.add_(momentum, v))\n",
        "\n",
        "norm = lambda x: torch.norm(x.reshape(x.size(0),-1).float(), dim=1)[:,None,None,None]\n",
        "\n",
        "def LARS_update(w, dw, v, lr, weight_decay, momentum):\n",
        "    nesterov_update(w, dw, v, lr*(norm(w)/(norm(dw)+1e-2)).to(w.dtype), weight_decay, momentum)\n",
        "\n",
        "def zeros_like(weights):\n",
        "    return [torch.zeros_like(w) for w in weights]\n",
        "\n",
        "def optimiser(weights, param_schedule, update, state_init):\n",
        "    weights = list(weights)\n",
        "    return {'update': update, 'param_schedule': param_schedule, 'step_number': 0, 'weights': weights,  'opt_state': state_init(weights)}\n",
        "\n",
        "def opt_step(update, param_schedule, step_number, weights, opt_state):\n",
        "    step_number += 1\n",
        "    param_values = {k: f(step_number) for k, f in param_schedule.items()}\n",
        "    for w, v in zip(weights, opt_state):\n",
        "        if w.requires_grad:\n",
        "            update(w.data, w.grad.data, v, **param_values)\n",
        "    return {'update': update, 'param_schedule': param_schedule, 'step_number': step_number, 'weights': weights,  'opt_state': opt_state}\n",
        "\n",
        "LARS = partial(optimiser, update=LARS_update, state_init=zeros_like)\n",
        "SGD = partial(optimiser, update=nesterov_update, state_init=zeros_like)\n",
        "  \n",
        "class PiecewiseLinear(namedtuple('PiecewiseLinear', ('knots', 'vals'))):\n",
        "    def __call__(self, t):\n",
        "        return np.interp([t], self.knots, self.vals)[0]\n",
        "     \n",
        "class Const(namedtuple('Const', ['val'])):\n",
        "    def __call__(self, x):\n",
        "        return self.val\n",
        "\n",
        "#####################\n",
        "## DATA\n",
        "##################### \n",
        "\n",
        "import torchvision\n",
        "from functools import lru_cache as cache\n",
        "\n",
        "@cache(None)\n",
        "def cifar10(root='./data'):\n",
        "    download = lambda train: torchvision.datasets.CIFAR10(root=root, train=train, download=True)\n",
        "    return {k: {'data': torch.tensor(v.data), 'targets': torch.tensor(v.targets)} \n",
        "            for k,v in [('train', download(True)), ('valid', download(False))]}\n",
        "  \n",
        "cifar10_mean, cifar10_std = [\n",
        "    (125.31, 122.95, 113.87), # equals np.mean(cifar10()['train']['data'], axis=(0,1,2)) \n",
        "    (62.99, 62.09, 66.70), # equals np.std(cifar10()['train']['data'], axis=(0,1,2))\n",
        "]\n",
        "cifar10_classes= 'airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck'.split(', ')\n",
        "\n",
        "#####################\n",
        "## data preprocessing\n",
        "#####################\n",
        "mean, std = [torch.tensor(x, device=device, dtype=torch.float16) for x in (cifar10_mean, cifar10_std)]\n",
        "\n",
        "normalise = lambda data, mean=mean, std=std: (data - mean)/std\n",
        "unnormalise = lambda data, mean=mean, std=std: data*std + mean\n",
        "pad = lambda data, border: nn.ReflectionPad2d(border)(data)\n",
        "transpose = lambda x, source='NHWC', target='NCHW': x.permute([source.index(d) for d in target]) \n",
        "to = lambda *args, **kwargs: (lambda x: x.to(*args, **kwargs))\n",
        "\n",
        "def preprocess(dataset, transforms):\n",
        "    dataset = copy.copy(dataset)\n",
        "    for transform in reversed(transforms):\n",
        "        dataset['data'] = transform(dataset['data'])\n",
        "    return dataset\n",
        "\n",
        "#####################\n",
        "## Data augmentation\n",
        "#####################\n",
        "\n",
        "chunks = lambda data, splits: (data[start:end] for (start, end) in zip(splits, splits[1:]))\n",
        "\n",
        "even_splits = lambda N, num_chunks: np.cumsum([0] + [(N//num_chunks)+1]*(N % num_chunks)  + [N//num_chunks]*(num_chunks - (N % num_chunks)))\n",
        "\n",
        "def shuffled(xs, inplace=False):\n",
        "    xs = xs if inplace else copy.copy(xs) \n",
        "    np.random.shuffle(xs)\n",
        "    return xs\n",
        "\n",
        "def transformed(data, targets, transform, max_options=None, unshuffle=False):\n",
        "    i = torch.randperm(len(data), device=device)\n",
        "    data = data[i]\n",
        "    options = shuffled(transform.options(data.shape), inplace=True)[:max_options]\n",
        "    data = torch.cat([transform.apply(x, **choice) for choice, x in zip(options, chunks(data, even_splits(len(data), len(options))))])\n",
        "    return (data[torch.argsort(i)], targets) if unshuffle else (data, targets[i])\n",
        "\n",
        "class Batches():\n",
        "    def __init__(self, batch_size, transforms=(), dataset=None, shuffle=True, drop_last=False, max_options=None):\n",
        "        self.dataset, self.transforms, self.shuffle, self.max_options = dataset, transforms, shuffle, max_options\n",
        "        N = len(dataset['data'])\n",
        "        self.splits = list(range(0, N+1, batch_size))\n",
        "        if not drop_last and self.splits[-1] != N:\n",
        "            self.splits.append(N)\n",
        "     \n",
        "    def __iter__(self):\n",
        "        data, targets = self.dataset['data'], self.dataset['targets']\n",
        "        for transform in self.transforms:\n",
        "            data, targets = transformed(data, targets, transform, max_options=self.max_options, unshuffle=not self.shuffle)\n",
        "        if self.shuffle:\n",
        "            i = torch.randperm(len(data), device=device)\n",
        "            data, targets = data[i], targets[i]\n",
        "        return ({'input': x.clone(), 'target': y} for (x, y) in zip(chunks(data, self.splits), chunks(targets, self.splits)))\n",
        "    \n",
        "    def __len__(self): \n",
        "        return len(self.splits) - 1\n",
        "    \n",
        "#####################\n",
        "## Augmentations\n",
        "#####################\n",
        "\n",
        "class Crop(namedtuple('Crop', ('h', 'w'))):\n",
        "    def apply(self, x, x0, y0):\n",
        "        return x[..., y0:y0+self.h, x0:x0+self.w] \n",
        "\n",
        "    def options(self, shape):\n",
        "        *_, H, W = shape\n",
        "        return [{'x0': x0, 'y0': y0} for x0 in range(W+1-self.w) for y0 in range(H+1-self.h)]\n",
        "    \n",
        "class FlipLR(namedtuple('FlipLR', ())):\n",
        "    def apply(self, x, choice):\n",
        "        return flip_lr(x) if choice else x \n",
        "        \n",
        "    def options(self, shape):\n",
        "        return [{'choice': b} for b in [True, False]]\n",
        "\n",
        "class Cutout(namedtuple('Cutout', ('h', 'w'))):\n",
        "    def apply(self, x, x0, y0):\n",
        "        x[..., y0:y0+self.h, x0:x0+self.w] = 0.0\n",
        "        return x\n",
        "\n",
        "    def options(self, shape):\n",
        "        *_, H, W = shape\n",
        "        return [{'x0': x0, 'y0': y0} for x0 in range(W+1-self.w) for y0 in range(H+1-self.h)]  \n",
        "\n",
        "#####################\n",
        "## TRAINING\n",
        "#####################\n",
        "\n",
        "import time\n",
        "\n",
        "class Timer():\n",
        "    def __init__(self, synch=None):\n",
        "        self.synch = synch or (lambda: None)\n",
        "        self.synch()\n",
        "        self.times = [time.perf_counter()]\n",
        "        self.total_time = 0.0\n",
        "\n",
        "    def __call__(self, update_total=True):\n",
        "        self.synch()\n",
        "        self.times.append(time.perf_counter())\n",
        "        delta_t = self.times[-1] - self.times[-2]\n",
        "        if update_total:\n",
        "            self.total_time += delta_t\n",
        "        return delta_t\n",
        "\n",
        "default_table_formats = {float: '{:{w}.4f}', str: '{:>{w}s}', 'default': '{:{w}}', 'title': '{:>{w}s}'}\n",
        "\n",
        "def table_formatter(val, is_title=False, col_width=12, formats=None):\n",
        "    formats = formats or default_table_formats\n",
        "    type_ = lambda val: float if isinstance(val, (float, np.float)) else type(val)\n",
        "    return (formats['title'] if is_title else formats.get(type_(val), formats['default'])).format(val, w=col_width)\n",
        "\n",
        "every = lambda n, col: (lambda data: data[col] % n == 0)\n",
        "\n",
        "class Table():\n",
        "    def __init__(self, keys=None, report=(lambda data: True), formatter=table_formatter):\n",
        "        self.keys, self.report, self.formatter = keys, report, formatter\n",
        "        self.log = []\n",
        "        \n",
        "    def append(self, data):\n",
        "        self.log.append(data)\n",
        "        data = {' '.join(p): v for p,v in path_iter(data)}\n",
        "        self.keys = self.keys or data.keys()\n",
        "        if len(self.log) is 1:\n",
        "            print(*(self.formatter(k, True) for k in self.keys))\n",
        "        if self.report(data):\n",
        "            print(*(self.formatter(data[k]) for k in self.keys))\n",
        "            \n",
        "    def df(self):\n",
        "        return pd.DataFrame([{'_'.join(p): v for p,v in path_iter(row)} for row in self.log])     \n",
        "            \n",
        "def reduce(batches, state, steps):\n",
        "    #state: is a dictionary\n",
        "    #steps: are functions that take (batch, state)\n",
        "    #and return a dictionary of updates to the state (or None)\n",
        "    \n",
        "    for batch in chain(batches, [None]): \n",
        "    #we send an extra batch=None at the end for steps that \n",
        "    #need to do some tidying-up (e.g. log_activations)\n",
        "        for step in steps:\n",
        "            updates = step(batch, state)\n",
        "            if updates:\n",
        "                for k,v in updates.items():\n",
        "                    state[k] = v                  \n",
        "    return state\n",
        "  \n",
        "#define keys in the state dict as constants\n",
        "MODEL = 'model'\n",
        "VALID_MODEL = 'valid_model'\n",
        "OUTPUT = 'output'\n",
        "OPTS = 'optimisers'\n",
        "ACT_LOG = 'activation_log'\n",
        "WEIGHT_LOG = 'weight_log'\n",
        "\n",
        "#step definitions\n",
        "def forward(training_mode):\n",
        "    def step(batch, state):\n",
        "        if not batch: return\n",
        "        model = state[MODEL] if training_mode or (VALID_MODEL not in state) else state[VALID_MODEL]\n",
        "        if model.training != training_mode: #without the guard it's slow!\n",
        "            model.train(training_mode)\n",
        "        return {OUTPUT: model.loss(model(batch))}\n",
        "    return step\n",
        "\n",
        "def forward_tta(tta_transforms):\n",
        "    def step(batch, state):\n",
        "        if not batch: return\n",
        "        model = state[MODEL] if (VALID_MODEL not in state) else state[VALID_MODEL]\n",
        "        if model.training:\n",
        "            model.train(False)\n",
        "        logits = torch.mean(torch.stack([model({'input': transform(batch['input'].clone())})['logits'].detach() for transform in tta_transforms], dim=0), dim=0)\n",
        "        return {OUTPUT: model.loss(dict(batch, logits=logits))}\n",
        "    return step\n",
        "\n",
        "def backward(dtype=torch.float16):\n",
        "    def step(batch, state):\n",
        "        state[MODEL].zero_grad()\n",
        "        if not batch: return\n",
        "        state[OUTPUT]['loss'].to(dtype).sum().backward()\n",
        "    return step\n",
        "\n",
        "def opt_steps(batch, state):\n",
        "    if not batch: return\n",
        "    return {OPTS: [opt_step(**opt) for opt in state[OPTS]]}\n",
        "\n",
        "def log_activations(node_names=('loss', 'acc')):\n",
        "    logs = []\n",
        "    def step(batch, state):\n",
        "        if batch:\n",
        "            logs.extend((k, state[OUTPUT][k].detach()) for k in node_names)\n",
        "        else:\n",
        "            res = map_values((lambda xs: to_numpy(torch.cat(xs)).astype(np.float)), group_by_key(logs))\n",
        "            logs.clear()\n",
        "            return {ACT_LOG: res}\n",
        "    return step\n",
        "\n",
        "def update_ema(momentum, update_freq=1):\n",
        "    n = iter(count())\n",
        "    rho = momentum**update_freq\n",
        "    def step(batch, state):\n",
        "        if not batch: return\n",
        "        if (next(n) % update_freq) != 0: return\n",
        "        for v, ema_v in zip(state[MODEL].state_dict().values(), state[VALID_MODEL].state_dict().values()):\n",
        "            ema_v *= rho\n",
        "            ema_v += (1-rho)*v\n",
        "    return step\n",
        "\n",
        "train_steps = (forward(training_mode=True), log_activations(('loss', 'acc')), backward(), opt_steps)\n",
        "valid_steps = (forward(training_mode=False), log_activations(('loss', 'acc')))\n",
        "\n",
        "epoch_stats = lambda state: {k: np.mean(v) for k, v in state[ACT_LOG].items()}\n",
        "\n",
        "def train_epoch(state, timer, train_batches, valid_batches, train_steps=train_steps, valid_steps=valid_steps, on_epoch_end=identity):\n",
        "    train_summary, train_time = epoch_stats(on_epoch_end(reduce(train_batches, state, train_steps))), timer()\n",
        "    valid_summary, valid_time = epoch_stats(reduce(valid_batches, state, valid_steps)), timer(update_total=False) #DAWNBench rules\n",
        "    return {\n",
        "        'train': union({'time': train_time}, train_summary), \n",
        "        'valid': union({'time': valid_time}, valid_summary), \n",
        "        'total time': timer.total_time\n",
        "    }\n",
        "\n",
        "summary = lambda logs, cols=['valid_acc']: logs.df().query('epoch==epoch.max()')[cols].describe().transpose().astype({'count': int})[\n",
        "    ['count', 'mean', 'min', 'max', 'std']]\n",
        "\n",
        "#on_epoch_end\n",
        "def log_weights(state, weights):\n",
        "    state[WEIGHT_LOG] = state.get(WEIGHT_LOG, [])\n",
        "    state[WEIGHT_LOG].append({k: to_numpy(v.data) for k,v in weights.items()})\n",
        "    return state\n",
        "\n",
        "def fine_tune_bn_stats(state, batches, model_key=VALID_MODEL):\n",
        "    reduce(batches, {MODEL: state[model_key]}, [forward(True)])\n",
        "    return state\n",
        "\n",
        "#misc\n",
        "def warmup_cudnn(model, batch):\n",
        "    #run forward and backward pass of the model\n",
        "    #to allow benchmarking of cudnn kernels \n",
        "    reduce([batch], {MODEL: model}, [forward(True), backward()])\n",
        "    torch.cuda.synchronize()\n",
        "\n",
        "\n",
        "#####################\n",
        "## Plotting\n",
        "#####################\n",
        "\n",
        "import altair as alt\n",
        "alt.renderers.enable('colab')\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from IPython.display import SVG\n",
        "\n",
        "def empty_plot(ax, **kw):\n",
        "    ax.axis('off')\n",
        "    return ax\n",
        "\n",
        "def image_plot(ax, img, title):\n",
        "    ax.imshow(to_numpy(unnormalise(transpose(img, 'CHW', 'HWC'))).astype(np.int))\n",
        "    ax.set_title(title)\n",
        "    ax.axis('off')\n",
        "\n",
        "def layout(figures, sharex=False, sharey=False, figure_title=None, col_width=4, row_height = 3.25, **kw):\n",
        "    nrows, ncols = np.array(figures).shape\n",
        "\n",
        "    fig, axs = plt.subplots(nrows=nrows, ncols=ncols, sharex=sharex, sharey=sharey, figsize=(col_width*ncols, row_height*nrows))\n",
        "    axs = [figure(ax, **kw) for row in zip(np.array(axs).reshape(nrows, ncols), figures) for ax, figure in zip(*row)]\n",
        "    fig.suptitle(figure_title)\n",
        "    return fig, axs\n",
        "\n",
        "#####################\n",
        "## Network\n",
        "#####################\n",
        "\n",
        "conv_block = lambda c_in, c_out: {\n",
        "    'conv': conv(in_channels=c_in, out_channels=c_out, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False), \n",
        "    'norm': batch_norm(c_out), \n",
        "    'act':  relu(),\n",
        "}\n",
        "\n",
        "conv_pool_block = lambda c_in, c_out: dict(conv_block(c_in, c_out), pool=pool(2))\n",
        "conv_pool_block_pre = lambda c_in, c_out: reorder(conv_pool_block(c_in, c_out), ('conv', 'pool', 'norm', 'act'))\n",
        "\n",
        "residual = lambda c, conv_block: {\n",
        "    'in': (Identity, {}),\n",
        "    'res1': conv_block(c, c),\n",
        "    'res2': conv_block(c, c),\n",
        "    'out': (Identity, {}),\n",
        "    'add': (Add, {}, ['in', 'out']),\n",
        "}\n",
        "\n",
        "def build_network(channels, extra_layers, res_layers, scale, conv_block=conv_block, \n",
        "                  prep_block=conv_block, conv_pool_block=conv_pool_block, types=None): \n",
        "    net = {\n",
        "        'prep': prep_block(3, channels['prep']),\n",
        "        'layer1': conv_pool_block(channels['prep'], channels['layer1']),\n",
        "        'layer2': conv_pool_block(channels['layer1'], channels['layer2']),\n",
        "        'layer3': conv_pool_block(channels['layer2'], channels['layer3']),\n",
        "        'pool': pool(4),\n",
        "        'classifier': {\n",
        "            'flatten': (Flatten, {}),\n",
        "            'conv': linear(channels['layer3'], 10, bias=False),\n",
        "            'scale': (Mul, {'weight': scale}),\n",
        "        },\n",
        "        'logits': (Identity, {}),\n",
        "    }\n",
        "    for layer in res_layers:\n",
        "        net[layer]['residual'] = residual(channels[layer], conv_block)\n",
        "    for layer in extra_layers:\n",
        "        net[layer]['extra'] = conv_block(channels[layer], channels[layer])     \n",
        "    if types: net = map_types(types, net)\n",
        "    return net\n",
        "\n",
        "channels={'prep': 64, 'layer1': 128, 'layer2': 256, 'layer3': 512}\n",
        "network = partial(build_network, channels=channels, extra_layers=(), res_layers=('layer1', 'layer3'), scale=1/8)   \n",
        "\n",
        "x_ent_loss = Network({\n",
        "  'loss':  (nn.CrossEntropyLoss, {'reduction': 'none'}, ['logits', 'target']),\n",
        "  'acc': (Correct, {}, ['logits', 'target'])\n",
        "})\n",
        "\n",
        "label_smoothing_loss = lambda alpha: Network({\n",
        "        'logprobs': (LogSoftmax, {'dim': 1}, ['logits']),\n",
        "        'KL':  (KLLoss, {}, ['logprobs']),\n",
        "        'xent':  (CrossEntropyLoss, {}, ['logprobs', 'target']),\n",
        "        'loss': (AddWeighted, {'wx': 1-alpha, 'wy': alpha}, ['xent', 'KL']),\n",
        "        'acc': (Correct, {}, ['logits', 'target']),\n",
        "    })\n",
        "\n",
        "#####################\n",
        "## Misc\n",
        "#####################\n",
        "\n",
        "lr_schedule = lambda knots, vals, batch_size: PiecewiseLinear(np.array(knots)*len(train_batches(batch_size)), np.array(vals)/batch_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "izQpvJHoBnAx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#####################\n",
        "## Config\n",
        "#####################\n",
        "\n",
        "N_RUNS = 5 #number of times to run each experiment"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tofkr8ecBqgF",
        "colab_type": "text"
      },
      "source": [
        "### Baseline (75s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kngp4RTwBKNl",
        "colab_type": "text"
      },
      "source": [
        "First we should check that timings haven't changed since November and our submission still runs in 75 seconds (note that this requires a V100 GPU, see instructions at the top of the notebook for how to use one from Colab.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "975cee77-3b99-4a75-ffb1-ef463a33a5ea",
        "id": "n___bs94Rvm2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 605
        }
      },
      "source": [
        "!git clone -q https://github.com/davidcpage/cifar10-fast.git\n",
        "!cd cifar10-fast && python -m dawn --data_dir=~/data "
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading datasets\n",
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n",
            "Warming up cudnn on random inputs\n",
            "Starting timer\n",
            "Preprocessing training data\n",
            "Finished in 2.9 seconds\n",
            "Preprocessing test data\n",
            "Finished in 0.13 seconds\n",
            "       epoch           lr   train time   train loss    train acc    test time    test loss     test acc   total time\n",
            "           1       0.0800       3.0059       1.6135       0.4145       0.2104       2.3074       0.3942       6.0854\n",
            "           2       0.1600       2.9737       0.9336       0.6671       0.2104       1.7186       0.5166       9.0591\n",
            "           3       0.2400       2.9913       0.7203       0.7468       0.2106       0.7480       0.7410      12.0504\n",
            "           4       0.3200       2.9765       0.6227       0.7832       0.2119       0.6620       0.7625      15.0268\n",
            "           5       0.4000       2.9793       0.5505       0.8092       0.2109       0.7297       0.7482      18.0061\n",
            "           6       0.3789       2.9949       0.4957       0.8285       0.2107       0.6034       0.7990      21.0010\n",
            "           7       0.3579       2.9771       0.4414       0.8482       0.2118       0.4500       0.8484      23.9781\n",
            "           8       0.3368       2.9722       0.4117       0.8586       0.2111       0.5800       0.7999      26.9504\n",
            "           9       0.3158       2.9904       0.3791       0.8705       0.2112       0.4200       0.8566      29.9407\n",
            "          10       0.2947       2.9782       0.3664       0.8741       0.2101       0.4473       0.8469      32.9189\n",
            "          11       0.2737       2.9823       0.3392       0.8845       0.2097       0.3835       0.8729      35.9012\n",
            "          12       0.2526       2.9931       0.3204       0.8912       0.2116       0.4211       0.8571      38.8943\n",
            "          13       0.2316       2.9808       0.2995       0.8989       0.2108       0.4225       0.8608      41.8751\n",
            "          14       0.2105       2.9800       0.2886       0.9015       0.2344       0.5089       0.8359      44.8550\n",
            "          15       0.1895       2.9777       0.2685       0.9096       0.2118       0.4559       0.8448      47.8327\n",
            "          16       0.1684       2.9780       0.2483       0.9166       0.2115       0.3886       0.8731      50.8108\n",
            "          17       0.1474       2.9953       0.2297       0.9226       0.2148       0.3245       0.8923      53.8061\n",
            "          18       0.1263       2.9814       0.2085       0.9307       0.2098       0.3315       0.8908      56.7875\n",
            "          19       0.1053       2.9788       0.1849       0.9388       0.2116       0.3156       0.8921      59.7663\n",
            "          20       0.0842       2.9910       0.1652       0.9457       0.2110       0.2541       0.9158      62.7573\n",
            "          21       0.0632       2.9801       0.1362       0.9565       0.2116       0.2433       0.9194      65.7374\n",
            "          22       0.0421       2.9867       0.1134       0.9655       0.2125       0.2116       0.9288      68.7241\n",
            "          23       0.0211       2.9996       0.0937       0.9712       0.2124       0.1935       0.9371      71.7237\n",
            "          24       0.0000       2.9813       0.0730       0.9793       0.2113       0.1852       0.9395      74.7050\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6woN6tKcCLLy",
        "colab_type": "text"
      },
      "source": [
        "Training indeed reaches ~94% test accuracy and completes in about 75s. We will need to cut this in half and then a bit more....\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOPAnn6orgLn",
        "colab_type": "text"
      },
      "source": [
        "### Preprocessing on the GPU (70s)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WyIgaE0Yr6wl",
        "colab_type": "text"
      },
      "source": [
        "We start with the practical matter of some code optimisation. The logs above show three seconds wasted on data preprocessing, which counts towards training time. Recall that we are normalising, transposing and padding the dataset before training to avoid repeating the work at each epoch.\n",
        "\n",
        "We can do better by transferring the data to the GPU, preprocessing there and then transferring back to the CPU for random data augmentation and batching:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W-y1gWp1zsWA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "6652754d-bbb4-4d8e-b4fe-941b82b8037a"
      },
      "source": [
        "#####################\n",
        "## timings\n",
        "#####################\n",
        "dataset = cifar10() #downloads dataset\n",
        "print('Starting timer')\n",
        "t = Timer(synch=torch.cuda.synchronize)\n",
        "dataset = map_nested(to(device), dataset)\n",
        "print(f'Transfer to GPU:\\t{t():.3f}s')\n",
        "train_set = preprocess(dataset['train'], [partial(pad, border=4), transpose, normalise, to(torch.float16)])\n",
        "valid_set = preprocess(dataset['valid'], [transpose, normalise, to(torch.float16)])\n",
        "print(f'Data preprocessing:\\t{t():.3f}s')\n",
        "map_nested(to(cpu), {'train': train_set, 'valid': valid_set})\n",
        "print(f'Transfer to CPU:\\t{t():.3f}s')"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n",
            "Starting timer\n",
            "Transfer to GPU:\t0.046s\n",
            "Data preprocessing:\t0.009s\n",
            "Transfer to CPU:\t0.400s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDVW85kFex8c",
        "colab_type": "text"
      },
      "source": [
        "Not bad! We've reduced the preprocessing time to about half a second. Actual preprocessing now takes a negligible amount of time and the bulk of time is spent transferring data back to the CPU. This is a bit silly, since the data will need to cross to the GPU again after batching and augmentation, incurring a further delay at each training step. Can we remove this by doing data augmentation on the GPU?\n",
        "\n",
        "The answer is yes, but it requires a little care. If we naively apply augmentation to individual training examples, as on the CPU, we will incur substantial overhead launching multiple GPU kernels to process each item. We can avoid this by applying the same augmentation to groups of examples and we can preserve randomness by shuffling the data beforehand.\n",
        "\n",
        "For example, consider applying 8×8 cutout augmentation to CIFAR10 images. There are 625 possible 8×8 cutout regions in a  32×32 image, so we can achieve random augmentation by shuffling the dataset and splitting into 625 groups, one for each of the possible cutout regions. If we choose evenly-sized groups, this is not quite the same as making a random choice for each example (which leads to irregular group sizes) but it's close enough. As a further optimisation, if the number of groups for an augmentation becomes too large, we can consider capping it at a reasonable limit - say 200 randomly selected groups per epoch.\n",
        "\n",
        "Our basic implementation takes about 35 lines of code and doesn't use Pytorch DataLoaders."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oNz7-dE7fE5J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_batches = partial(Batches, dataset=train_set, shuffle=True,  drop_last=True, max_options=200)\n",
        "valid_batches = partial(Batches, dataset=valid_set, shuffle=False, drop_last=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LwHDoIF807rV",
        "colab_type": "text"
      },
      "source": [
        "As a sanity check that we're doing things correctly - here are two random augmentations of the same 8 images:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4hF3Grkd1OVW",
        "colab_type": "code",
        "outputId": "c97aee8f-700b-402a-fb15-94b7b1cf4394",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        }
      },
      "source": [
        "batches = train_batches(batch_size=8, transforms=(Crop(32, 32), FlipLR(), Cutout(8, 8)), shuffle=False)\n",
        "\n",
        "layout([[partial(image_plot, img=x, title=cifar10_classes[y]) for x,y in zip(*next(iter(batches)).values())]], col_width=2, row_height=2)\n",
        "layout([[partial(image_plot, img=x, title=cifar10_classes[y]) for x,y in zip(*next(iter(batches)).values())]], col_width=2, row_height=2);"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1152x144 with 8 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA54AAACOCAYAAABZsdfhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvXmYZNdZ5vmd2PeM3PeqrF2lpSRLsizLkixZwuCtsTFuwxh6mm6YZuhuuukeoDE0zUOb6Zl5aDDGDQym2ZseGxvvBhtjS9iSbO1LqfaqzKqs3DMyY9/uvXHmj4i675siSyXJEVVy6Xufpx4d3bxx77ln+c6NOL/zHmOtFZVKpVKpVCqVSqVSqXqlwJXOgEqlUqlUKpVKpVKprm7pF0+VSqVSqVQqlUqlUvVU+sVTpVKpVCqVSqVSqVQ9lX7xVKlUKpVKpVKpVCpVT6VfPFUqlUqlUqlUKpVK1VPpF0+VSqVSqVQqlUqlUvVUr+kvnsaYA8aYp4wxJWPMT1/p/KheHTLGWGPM3iudD9V3Jq3HKy9jzB8bYz50pfOhevXJGPOAMebHL/K3HcaYsjEmeKlzVVdWWo+XR8aYOWPM/dscv8sYc/xlXkvj8mXQxersta7X9BdPEfk5EXnAWpu21n7kSmdG9dKlHfrqkNajSnXl9Gr9ImCtPWetTVlrvSudl+8GaT2+dmWt/Ya19sCVzodK9VL1Wv/iuVNEnt/uDxd+oVN998kYE7rSeVB959J6VG0nbRcqlUp1aWms/O7X1ViHr9kvnsaYr4nIvSLy0Q4K8hfGmN81xnzJGFMRkXuNMX3GmD81xqwZY84aY37JGBPofD5ojPmvxph1Y8ysMeZfddC+q66RvNpkjPkzEdkhIp/v1N3Pdcr+nxtjzonI14wx9xhjzr/gc/7sWqf+PmiMOd1BrZ8wxkxvc687jTHzxph7L8vDvYak9Xj1yRjzOmPMk526+LiIxOhv7zTGPG2MyRtjHjbGHKK/TRhjPtWJtbO89MEY8yvGmE8aY/7cGFMUkX96WR/qu0DGmP9AfeCIMeY9neO/Yoz5czpv5sI4ZYz5NRG5SzAGfrRzzh3GmMeMMYXOf++gzz9gjPlQp/7KxpjPG2MGjTH/wxhT7Jw/Q+df9Fod7THGPNr5+2eNMQMvzOdFnvefGWOOGmM2jTFfNsbs7FJRXlFpPV4d9dhDvb7TLjaNMX9kjIm9cIzsjI8/b4x5VkQqnTZy0bis6rluMsY82+kbHzfGxEREjDE/YYw5ZYzZMMZ8zhgzceEDnT7zL40xJ0XkpGnrN40xq53rPGuMub5zbtQY8+vGmHPGmBVjzO8ZY+JX6Flfmqy1r9l/IvKAiPx4J/3HIlIQkTdJ+wt5TET+VEQ+KyJpEZkRkRMi8s875/+kiBwRkSkR6ReRr4qIFZHQlX6u18I/EZkTkfs76ZlO2f+piCRFJC4i94jI+Rf5zM+KyHMickBEjIjcKCKDnb9ZEdkrIt8rIvMictuVft6r9Z/W49XzT0QiInJWRH5GRMIi8oMi4ojIh0TkZhFZFZE3iEhQRP7XTj1GO/H2CRH55c41dovIGRH53s51f6VznXd3zo1f6Wd9tf0TkfeJyESnfN4vIhURGe+U3Z/TeRf6WKjz/w9IZwzs/P+AiGyKyI+KSEhEfrjz/4N0/ikR2SMifdIeA0+IyP2d8/9URP7oZVxrQUSu7/T3T13I64vls9MOTonIwc51f0lEHr7SdaD1qPXY47YxJyKHRWS6UycPSTu23iM0RnbOe7pzXlxeJC5f6We62v916uLRTp8eEJGj0v7u8BYRWZf2uBgVkd8Wkb+nz1kR+dvOZ+LSfod5QkSy0n7POSgi451zPywin+ucmxaRz4vIf7nSz/5i/16zM54X0WettQ9Za1vS7pjvF5FfsNaWrLVzIvJfpR18RUT+sYj8lrX2vLV2U0T+ryuSYxXrV6y1FWtt7SWc++Mi8kvW2uO2rWestTn6+/tE5PdF5O3W2kd7klvVxaT1+N2p26X9YvNha61jrf2kiDzW+dtPiMj/a639trXWs9b+iYg0Op95vYgMW2t/1VrbtNaeEZGPicgP0bUfsdZ+xlrbeont4jUla+1fWmsXO+XzcRE5KSK3vYJLvUNETlpr/8xa61pr/6eIHBORd9E5f2StPW2tLYjIX4vIaWvtV621roj8pYi87mVc68+stYettRUR+Y8i8o/NpZe5/Atpv1gd7dzz/5T2rMJ3/WyZ1uPVUY891EettfPW2g0R+TVp/wiwnT7SOa8mLx6XVb3XRzp9ekPaXwpvEpEPiMgfWmuftNY2ROQXROSNTBlIu29sdOrQkfaXymtExHT6zJIxxkh7bP2ZzrklafcjHjtfddIvnls1T+khwS9FF3RWRCY76YkXnM9p1ZXRy6mDaRE5/SJ//7ci8glr7XPfWZZUr0Baj9+dmhCRBWvbP8N2dCF+7hSRf2/amG3eGJOXdt1NdP428YK/fVBERuk6Gl9fRMaYf2KAMeelPfs09AouNSFbxzyRreOeiMgKpWvb/H/qZVxr/gV/C8ul871TRH6LnnVD2rMAky/+sVe/tB6vjnrsoV5YzhMv4bwXi8uq3muZ0lVp96stfcpaWxaRnFykT1lrvyYiHxWR/yYiK8aY3zfGZERkWEQSIvIE9aO/6Rx/1Uq/eG4Vd8x1af/KwL++7ZA2UiIisiRtzPaC/sG6MlVPZS9xrCLtDikivlkUd8Z5aWNGF9P7ROTdxph/+51kUnVJaT1ePVoSkcnOr7AXtKPz33kR+TVrbZb+JTqzJ/MiMvuCv6WttW+n62zXTlQi0pkh+piI/Ctp449ZaSN5Rl7Qf0Rk7AUff2G5LsrWMU9k67j3cvRSrjX9gr850h57X0zzIvIvXtBe4tbah19BHl810nq8Ouqxx3phOS9e5DxuDy8Wl1VXRlv6lDEmKSKDsrVPbenT1tqPWGtvEZHrRGS/tJcZrUv7R6LrqA/1WWtT8iqWfvG8iGzb/vsTIvJrxph0Z1D4dyJyYYH/J0Tk3xhjJo0xWRH5+SuU1deqVqS9FuxiOiEiMWPMO4wxYWmvH4nS3/9ARP6zMWZfZ+H2IWPMIP19UUTuE5GfNsb8VLczr/Kl9Xj16BERcaVd1iFjzA8IMMGPichPGmPe0KmnZKdO09JeA1PsGGLETdsw6npjzOuv0HN8tykp7ZeUNRERY8yPSXumTKS91utu095PsU/aSBfrhf3vSyKy3xjzv3Tq8P0icq2IfOEV5OulXOtHjDHXGmMSIvKrIvJJe+mtN35PRH7BGHOdiIhpmwC+7xXk79Umrcerox57qX9pjJkybfOmD4rIx1/CZ14sLquujP5CRH7MGHOTMSYqbTz2250lff9AxpjXd8bOsLR/hKqLiGfbywI/JiK/aYwZ6Zw7aYz53svyFK9Q+sXzxfWvpV3JZ0Tkm9JuLH/Y+dvHROQrIvKsiDwl7eDsiojuV3V59F9E5Jc6aMEPvvCPnXUrPyXtLyYL0q5Hdkf9DWn/ePAVESmKyH+X9iJuvsY5aX9p+XnzKtwj7SqR1uNVImttU0R+QNqus5vSXiP/V52/PS7ttSgf7fztVOe8Cz/yvUvaa19mpf0r7h9I2/REdQlZa49I23/gEWl/AblB2sYjYq39W2m/nD4rbXOKF37x+C0R+UHTdsn8SGd99DtF5N9LG/36ORF5p7X2UrNX2+XrpVzrz6Rt7LcsbUO/n5ZLyFr7aRH5v0Xk/zNtl+PDIvK2l5u/V5u0Hq+Oeuyx/kLaY92Zzr8PXeoDLxaXVVdG1tq/k/Za6E9Je0Z6j7z4usyMtL9zbEob0c2JyK93/vbz0h5Pv9XpR1+Vttniq1ZmK/ateqUyxrxNRH7PWqsL41UqlUqlUqlUKpWKpDOer1AdJOztHXRhUkT+k4h8+krnS6VSqVQqlUqlUqlebdIZz1eozlqGB6Vtb1wTkS+KyL+x1havaMZUKpVKpVKpVCqV6lUm/eKpUqlUKpVKpVKpVKqeSlFblUqlUqlUKpVKpVL1VPrFU6VSqVQqlUqlUqlUPVXoct7sl3/x3/WU611//PN+OhbCrVy35aeDATxyNIrtAKv1Cs5vNf300FB7S8BAEPdxGg0/HQ/F/HSjgc95nuunE4mknzaBMNJBpCWA3wCqdQd5cZAOhpBfz0UeQmFcJxGja7bw3B/8k4d48+DvSD/8I9/rF265hGcul1GG0WjET3O5pFIoi1Qa58Tj2Bv7/PklPx2L4fxMZsRPB4OoR89DGe2Y3uen777zfj890N/eT7fp1P1j2X5ce2gAWz/+zkd/U3qp9/7vP+unHQdlszqPvaDPnTrrp3PL2D3kFz/4f3StHuUfbjreM7UoHdhyW+w+VFyHO/9DX/s7P92XQT0l0+0dNnbshVt4/+hE9zJ6+dS1evzx77/DL9B6He07FkNsClB8cSNoc9EEzqlVUBe59RLOiSL4pftwPov7OPdNp4lrcryNRNp9v0mfMw7y2GqhjSTi2B2n0UTcawo+G47gnpUK4lBfNuuni4Uq5Rf5Wllfw/n9OL9OcX5kEPFhdXXVTz/4rZNdq8d3/+R/9h86GEBsjAjSdeovVUpvVvHMpoUsRQ3qLpVA+VMVSZjuVS+j/bTo+qEQrpPNYJeb6akpEREZHkL5ZPv78TkaOIMe6vRiS3zolC1tgM+3FD9alsd5jLmhEB4wYFAehrqdoeM/9v3Xdq0ef/tt9/mZitN7yMLAuJ8+WcOY9aMHsf3m3PwJP/34WbTLIp3PYZufoUrt1Q3geNCgHgcH0n76Az+ELTOdatuaYpX6wugk4urRkyf99LeeOOynW1SeI30ZP3397r1+em11wU/Xaogr5XIZj0TvRVFqm+kx5NeJYPw/dWTOT++9dsZP/9qH/7Rr9fizf3KzX9CHn0CsmR67GflLIjal4mjrydCAnx7KTvnpkVH0jdQQ6rEYfMZP12LH/XS9lffTrRr6adRDX06mqW+02jE0FEz5xzIpFEkigs85lTHcs4LziznYlJw6gfeQYBR5lzCe+/zCsp9Op5DH/Cbquumi/ETQTvOFgp/O9iGu/M7P/HXX6vEzz3zYLyCOBRxTuB9xTGla9LuWQ3EHQ4kEm2E6jr5m6V0zEkOjjiRRB+EkyisQQfu58N7fos0ZTYuuEcU70ewc3hsPPzeHD9A4sHtmxk/3pdCnDj91xE8//dgpP12l5yhTn41TW9qxZweucxTvqMXlDT+9dPbotvWoM54qlUqlUqlUKpVKpeqp9IunSqVSqVQqlUqlUql6qsuK2vZcQXyP7hsAuunSFHmEEAGPprGDhCA0mpgid9z2TDEjCiHCK2J03DXAnQIWcKErjL3gnqkk8liuYO7ecTG9T8SMlIrAEpLJ2Lbney1Cq+j63VRuA0hkQHAPxpuyWWATudymn3Yc4FC5DSAd42PAPoaHgalUK8AyvBY+G00ADYkFUR+hMOrjyaee9dN333lbO1+EgY2NDflpcxnNnR9+6jk/nUmg/PZPAcnZPTntp+fPAHPqpi6nozXfqUV9I0AQbo3a9/rCvJ9eOo72E+4g2RFCuftHRuna26N0rzZ1M28LS0DTU0kgOLlNlFskAhwokkGMKJTQB6MR9CnOHqNHq2vo+4MD6EsBg3osF4GHRePIT7FExztx0xJO2apTQKYGk6JYFw4hYx4toQiFEfvjhJTm1oHFDmSGKI1+Z6kNDowM++lqveanGyXEdkZ/uynHxf1MEGXBsYmXCjiMJFI8LBUxlhSb9Ay09OIC6iwiko7gBsEgjldoOUewhfJtrKOf5vNtXDKZQpmMjwPR3LMLGGmKxsoo3d+h5SQOMflW0E45ZmzBbqlsvC1BBp/ltsz9LtCj392fKQIrjBjUl3gYEwMGz79+dsVPP7EIZO3YKvqvbWDs42dgnN5xqf8QWh+Lo9zzNZTjo89hXBkfbOet4XJcQoEy/hoOc3BA8sCePX56ZsdOP51No68tL83ho9SWU/3AkD16R0tEgeNODKGNzwdxzYPDON5NTexCfAsGgScOpHbTWWi7C7Nn/PTsAmLy5AT6Y8XiOv0h1K+bOeanA6mcn244iNulPNrAQAjPH6H+m+lrl0U6jneJBvWvpku7/VH8LKwg7m2eQWWfePxpP52cxv0n9wJ7jiWRxyKNJ406zheDc9ZzwLm3xLNGb8br7AD6Woti4Jb3EIP+wstSjKWYZXGdUA15ra0Sd0tfqVpBikFRXDNKY4/EcDxMCK7pfFGo01IVWpUiQUG/H59A3dUryNfSeSDQR57FuzB/Z9lcARZbWUO91KooGxNCHhNZ4PSG8tMfRX5itITtYtIZT5VKpVKpVCqVSqVS9VT6xVOlUqlUKpVKpVKpVD3VVYXaknGUDBDeUSoCk2qSe22N3GMt4bCMqzkdVCng4eJhcmdkR9UQcbSNBo5HwuRcRbhoowzUgjkhMpEUl9CAAmGnKXIPixIaIC7QhUYZ53dT0RCwiRrhcZkMpuFD5MCbTuN4sQgUI04OvOx+2CB3vlaTWJ4w0k1CvyamCJlt4Poze6/x09cearvdbvFTdfF/Djki9lqPfxPYg0P1fngCyOjAIPChvtT2TqLfqS4nispI5RasmWzbcktwPwwQ5jc9jnKpNtv1tDgPt7291x/y08Eo+r0xr5Hf1QjpSZDjaJkcbhuEVeUWEXf6CIlnJC5AQ4PLaGMA6WoNdRQj1DVKWG+L8D9OOx1HWkvxLUTPEaYYk8sDAeI2G6KlFRs5IEOFMlz40jFge3Vyu60Jyoavs5kH5ubSWgyHlkL0ClEnE0KxTDNSh+F0pQIMMRYnJ9swytFzcLzeAHbrEhpt6ZoRtm/f0n1wDjvcXvhsqYq8FE4e9dPrOaDZ6Rja2tQkUMB+Wv4QiTLGTE6TFJ+pKYtLmfQs6itokccQIXVEjUkg2JvXn40AYdIe+sggLUVJZfDM9QqhyyWcX+T3E7qmR+0ySOeEuMJoeVGFnKBT1HYffQbj0P69bRfaa8ipMhRBPJiZAUZbaaFvriyR824J7UvIjf7WuxGfn37sQT9dozotObhXroKyGaihn04G0a/rZVpSQe6p3VS9iXeYXQdQLmdOwv2zQu0+ye+cNdTp4eNYWpOagOv+YBqsohtAGz1/BjFILK7ZHwHCbgVlF4sAtRzoa4+V5QLeD48dxbn9SSxpSmfQXpxB9JfKAs5ZXoHL964pGmdS5JLeQh6bdZRHKEJxdQN1VK2gTsl0W6q0xK2bsjTuGFq7Fg4wym+3Pb+VR3pjFfXS2kRbXz+BZUFxoSUtaby3ZUaoTc9M4pwAuSKTe38k3T7eoIGgSihybgnta/U43puOfAN9mlHb9RUsOQnRe5HXpO8jtENFo4FzbAh5zFP5bazhPSJOY8LuSbSZi+k18mamUqlUKpVKpVKpVKorJf3iqVKpVCqVSqVSqVSqnuqqQm2dJrCT1VU4KHoOpstLVSBTVQ9Ty6k4cFChzcWDHdu2AOFIwShvug48IBEm1JSm7ut13KdGrq4twpfytHF3vornKFdxfp02WM8VgDSkEuQmm8BUf7ZHiGaG0NloDFPsTSr/KpUzb+zNbrfRMG1SW2RcBs/QIFzQdWgj2zDc4WpFYHbpIJCVvgxQmUDHDZOd1nI54AonjmPj7l5rfXHWT3uEDNfKQDmCUcIVqc38b+99e49z1z1tQRK34IlIb6wCATn2zFN+ukzOya0m+k+p05eiDdTz9Rsot9EJYCEX2yD6alMsRSg7IaF8PBxGjGiuErbXJLfSKpC8EOGaoTDaYp36skMIX4xw1f4+YEWWkMc69eUL8SEcQx+tN2jDbdpM25DVaZPaQrlAaB/1lwjF5yA5o8YI33UJ/XaonRQJxw1SmYWI0exVW6pReUqUNjqnsm0FUP5EbvpLQkREIiE8fyoO5I5RNpdQvQZ1zQYtP4gGcAMuR0u/VzudpSOuoE2xK+TyBvr3YgP99NTZc356eBhLJSYm4Oadoo3OY1SnlhA5xxI2xggqY9uEFRtC+7m9ieySbmk8QeMdjWUD/YhNsxZjWTKO/EXpPSNhUP4OOV6yi32dlqV4VC9xckuPUN8Ym4Z77MQUynq98/6xXEQ7esMbbvPTGyvA9n7gvW/y01/6wpf99CMPf8tP77j+Zj/9lkO3+OnTC+T8+tBjfrrQRF2XiaU++Hpcp+YA7RsaonewEOJNNzV7BmVRO4u6iMrItmnWDdmxbY83QN3KsefwDNWb0S6zQ8BxhdpJOIr22j8A5/9UAv2nVGzHjfUVXq6EvMcyKOdiE+X2XB1OvY2BQT8dGMGSlkQM/Xczj3eupUXk0SX3ZYf6V7mCJVD8PsgOq9yWu6lmGf0lTI7njkeuthRjnSrGmCNfR4VVz5ELMb07mjw5L2eAmcZo2UBkjN2tcTy9A+0kaVEu4Y6bLX93OXdszk8/8JWH/fTmeeQrUEb8GHPwrGN9wLSFUOK8h+fIxVAeFYqfhRrOL+RQ10F6106RU75xLz2fqTOeKpVKpVKpVCqVSqXqqfSLp0qlUqlUKpVKpVKpeqqrCrW1hNSUmkCmPA/TwFWaXncpXarQRsAb+Gy44zaWIRc1ZxkYYK0AtG3H0F4/PTIC1z6TxnR2YxO4QrmM+xRKmIpfJ4Rsbh6f9ciFL+hhWn5mGviqoWn8UI9cPSsV5K+vn1ETco2s4Zw4bbieTAIBKmwCw2oSZjE2BiwgTbhgQPDZaBqYyM5BXP/QNXCf6x+H25vbQchmZ4GOPPcc3BcLeXIY7rGeefKbfjqbhnutJUQtlUG59iV7gxL1XIQkBskNrUpY9XOPA7daOoe6aZJb4AnaYH291D4+vRdo0MJ5bLo+Mg6UbOvvalcvauvRLu5r63CZHB5G+49HgFzGY8CtPMb2aElAIgA8jxE+Rmf7CKkNE2pJ5qISo77vkYPzBcKWEadUFvmqVBFLQkHkJTOAWFdYAu4VpX60xcUzhOceG0SfOr8EdNAj9IiXAjS3IGGExFcJ8e2iynVc16PmGgqgXtjtNhwmh18eyskpmC2kU+Q2zDRUi81QWzw+4r68wbolFNLr1LsXJF6X6p8Je0ObyLuETxfJZfns0pyfjkaAIiYIHY3FYnQO6jdMSLjjoF03CM/eDvcWEZGfuF26pYPjcK3cZVEvffQ8UkDMSmSBxFUiyFOLlqLcehOQ09ERtOMzp+CwOn+OXMGpz1hyuo+RO+4b34BrrnVu++iDD/jHjh/HWOrVqKxoPMqT036ZlgKdWsJ7TqVFjqnU8Fbz+Gwjhv67bydie3YU7wJrOVzzLW+5zk+fpve1biq/inazPTjbPeXm0Y7TEcSgiSmgm47gfanQQp1WaFlKLNj+bJlCVJqWHLkR9LXVCtrRlz6NZ23ZRT+9J4Jz2Cl6fRHobLNOy9BC5KJNfdDSu0AqTe+rFNBC6d4sDWus0/KJGfRH2pBChPoan194FksCGsvkzFvA82dpWUqL+l21huuMjQCHDlfJGf4sxuv6EjnPxtrtYXUT9zn+OBxrE9R3xvsxzlcivMQPjSBI7whNylfcQ3oojj7o0tKKU3k892we7WQgDtx7kJb4FTYpVlxEOuOpUqlUKpVKpVKpVKqeSr94qlQqlUqlUqlUKpWqp7qqUFshFODsAvCDcgPHI0QhTZIbbChMeGsOjriNzjXDhCz1kTPYHdfe6qeLS4Sb0XR63xC5tFZxz3KZNl0nTGh6DNcfGRn10ytF4BWjGcKNArz5LW3O26PfFVYINY6ngIK0aKNuRuWi5FzZdDBVv+U6dE6pCLyA3TjHRlEut9+GTakzLurr+W894qffNHEt8tapmgoxKKMjAGieffZ5P90Pqqsnetv3vMdPjw8DRTx89LCfPnFyyU8vzwF96aqovl7S6YTbGTapbfE5dJhPIse2c8fwnMcee9RP18hR9Dxt1nxsFgjKBTomOwEEi10eWy3cJxAAviScX+HDnHnmAulZGdN9ScTuRW625Xj3+mYmiTbUctLbHieTOhnsB1KTWwPqMzUGrKpJDqhreXKoJufqhqAvx8iBtEWOw/U60C6HHA9tJ2ZtcRYlF88YLSvI5xEPdowhHk4Shl8iNDtPsZeag0QTqLzJcWBCZxaQR3YRZzfZlQLaZqXSG7Sv3iAXzQbKPEyIbJDaZZRcenmZiaFxsEXorCVHVzL1lSot22hSfwiQe2yTULkwdXLbWYriEMLJ3SjADc+grmlVwZZe0SLut1lDnRYrxO8SAiwNnMNuw1sctaNo7xJETDD0fN1UXxAbwTvraFvzeaCwd954jZ+u0bKgSQpHsQSe4fYsrnktuQBXqSLXoxhDqwXcl4sr1AQ2t/Mc3NXj+XYbGBgmtPMwXMYZ3X3kCJaoHF/E2FR3gf8tnANKvJpDjLntdUCad2bhqvuRv/iMn27WgME/8RjeEVZWTvvpm+9D+T32yDGBfkq6JafWG5fV7TQzjXIvbqK84nHUqRdBHzhfBAI6mEZMnuhvt+/0ADmq0lhTIdfq0+extOXMN7Gkq1lCOZtpHK+uIg6P78RLUjxL42yA+jj1tQShmM0aGmQ4gOs0W5dGNF+JUobKkFzBmxQiYoHtz49VcP7mCpZ2tOg9IJFC34zQkowoIc4hcinfJDRaDMXkFuJ/qFN2DdphY4pw9JEhcoGu0+4QHjmXN2kHjxr6fZRw6LFhXIe/G9Xovuc28dxiyXmeBtdUBGOxay/9kqQzniqVSqVSqVQqlUql6qn0i6dKpVKpVCqVSqVSqXqqqwq1PT0HLGCthGlgoqdkZxx4zbvvuslPT41jGvuTT2CT40dOtbEPtwU8IERoaykPLKJaBpqRTgMtYIvCWIw2Oo8hLwl2/CP0acc0ObxuYLp8bABT5EuEu4QivJF6b1Aidp1lC8NiEXhPg1CxYhHPz06bfB3ejL5QRD3u2Tvjp284BLe70UFgHIe//KSfPv4snL9ufgvycGHT8/37D/rHzpye89Oz5AjYfw3y2AuNjQIBnh4HEjOz54CfPn4CqNLRIyd7kg97MSL0ImoRFBcg1sSQW6klvLZFXWBzAVjP8w8/6KdrK0CKc1VgHIfnycGvSshTBw9OpIF27JiZoVyabVJbZem04amKAAAgAElEQVQ5rGVEkHFFyvyWC10auzUvCbXtniplxIV0KrXt8VgUaP4AOcBWS+T4XaGNwMnRNRUHSlStkqM3uVuPD6LPNJo4niaEPhIi9KizkbcNkqtuDShXlDbfduvIy9wp4IE37kM8WC8Btw9a/J7aInx3fR1tjV1g3QbyUCCncUdw3w1y9qv2CLUtbcC9OULuhAOjcEiPE3EaoHEiGEc8tAF6HnJRr5VRvjt3IdaUHNTv5iZibzQKDM6hJRJGGNnt5JPG2BYFFgoNEhHkKxDEB1wHHclji1120m2gzFv5eT+dW8BYLZY/i748OIl2EiC800Tx3N3UJDlCZmhZztObiOmbDZTzzjE4cf/g6i4/HS7imQdP4rPR02jHXgtlOsOOx/TOEQih73v0ntF4FONmXweTbQ2hTDxyL/74jutx8Rj65p4BIKJ7ZHv9zz/+/Yv8BbptD5yAVwtoa8tlwgWrQP7OnDy57fFuisu211M0IzPkhnoW9T6/Buw1nqY2PYzxrz/DCHu7nYTIQTkSgIus2wSm3XJ4XQze3Q7egLZ5zS6k0wmMw/3DyEu1ijbTbCIOlXKIZ14T58cjtJaJAkRv3lZF1ubxjj4yg1jqtfA8IYuxaXUe/Wv+HLBvl/BTXv4Qi+F5uJn001icpyUt1QreNdNpcutOYkwKd2KvQ3GVdwko0xhUofExSrh9wKANlFyc3yS321QC8SlAu3w4Dn0fcWnnBcLpQwb9o1jA+Fsi3Pdi0hlPlUqlUqlUKpVKpVL1VPrFU6VSqVQqlUqlUqlUPdVVhdpuFAmbo43Dg+ROmInhnJEQoWgbmELel4Hb6dJA+7MreaB/jSpQkKdOnPDTAUJTnCRQCOmDE6MEaEPpPkyzp8mdrk7OkbYJPGpmGEhDjbC0AbJhjRECEAi8JAvOl62RESCyHjkVsgNtkZxpMxmURSQCFCCTGPTT584CoZucxPXfev89fnp6ChtX1zbhZhijzctnJoBSpWJUB37VEApKSFgg0hvsajutr6FsiuT4N70XTpsT0zN+2gRxvJtyqSzYudVeBAkN8jmM1G6xuEW6UUQdPffgQ3767HFgUuU6+tIpwlo2NoBYNR302ZldO0VE5M677/GP9fcDHxIDYIdztbUnMGpLGyvTRvPhEGJGkCEgNum8KGt7sX7Xm/5YIZfA7ACQ1xxtPl0jR9mhQbSnyR3oL0XaFDtELtsN2gg8EkEfTyXQv1ZXUHchKqRsH+omShttSycdSSBOV2tAgFyPcMU+PNPCeSwrqBDqe/AA8PXnjp/y0806L7lAXafjeI44YapBwpObLsq1P0WYsOWdx7unyWHghmKoLZITd8xFuxwZgQtxPQ6sqkkuz3HCwIIJIJIJisnZJFDPsSGMg+yIW6dYWaXjy2ttnM6pALUKU/mEyGU9SMtVHAflHAoijy3Bc7RorBRyZSwuzvnpxiZwvjItdUlRfRUJ2c1GgYYKYafdVDIHHD0YQFntnwLmV1pB3Gcr8EmKpYkIjgcJJzX0rkCGtdJgq2AaZ8NUdyHqA2FCsp10u79ZWtbgNnqzNGA7XXfNXj89Tu9X48Qa7t+D94K9Q8D/7t2BcaabCtAYIC/PAP5l6+zycT8dS6GfZgYJs4+iP4z2j9NxhlTbderQmOk4wO1tGM9U3ERcpXAg93wP3suigvfe8TGUOS/jOvEcCmdjE+N2vYi4ZV3kp4/qzqPjkVBv5sFmTwDNH3oj2lCIxqMW9Qs+f3WdviOQS36YluiUaRxKkdN4iN2EyWW6VMD5QbpOQMh9t7OUyQRxzKH3E14Ks2UZBBXh+BjqkUyFZe4Mlie4dM16A32/VEdMbnhodwFymx8eAsIdCRAqvsFRaXvpjKdKpVKpVCqVSqVSqXoq/eKpUqlUKpVKpVKpVKqe6rKitq3Zb/rpgIcpZkMcgyFcM0LOXJYQnCMngbcuLAFZiZGDYoyQsHgS08b95Kb3xClgOm4T5zf6gNoO98c6eQSL4BA+VCUHxwptXN6kqXBDU+FM24UJhbW0mXU4xI6LmP62F3EASxNCxTCf6REfEk8Bea03UP4NQuWyaSBNHrkvhujZYlFc55ZbgSPfeuOMnx5I4knZwTBCeFY8RbheGWnGAkwnDy1DTrcRXDsYR14++S04sO4bApbIjrilGNzARqbQXuqEGFdzwM8MoS/pfnIeI3yjRGZgsSDqrmIvjS68Ei1twl2NkZdgCOUSJWRrIAG0okm4l0PtLEEh5cxTcBg+/o3H/XS5gAc9uQEM6NwayouN0aKEEb7jXe8SEZE73nSXf6xF92xZ5J1Jc7sFsUZ+G4SaHDuKjdHj5OS6b+8+P81uqLxJ/Rbs9mKo7cXZ3+9IEcpriZyB+bhH8XZ1HX12F6G2SdoI22vh/BI1zBhtaB6mWFp0UXeWNpZuCjtDEubVccBOUVyv1XBug8onk2Z3VRxfWwXSmIzjHEPRMUjskeehrpMp9N/lDeBkEXIs37Fnxk+3KIbVqr3Z6HypgOs26H5mGfFo1yDw2pFpoLnHyNnckuN0gjCwviTK+rn5Z/x0agyOh6kokKnZE0f8tJdEfMzuO4TzJ9qIZOUs+k6Q3HMzFhhktYw2Ui1RmYeB3hXrqLt4FijgYJwcHblNcbdj1JSOh+j8ZBDtOkfLZ7qpxiauW6M4Xu3Dc8ar5OZ8FM6lHuXPTTLihrYbJSzQUP/hjds9wqEtYfPbL4oQCY20nX/TeZRhPSaXTbkzx/x0M4Ex10XzlWoOuGIxjPKorvfGZZqdQyV+8fO6oaUVvIvumrzOT/fFyWHfoTG3gLbUT++30imXap3yHkIsadE70ZlTaBf9o4jrN98CRDMuGPscD325XkFbcx3kvUlIfJQw0XgS6SAvVyEUvUcrw2RxAeNEkxzMWyF6X6cBh89fJ3yfNqSQGCHL6yU883A/yi5PrtRLi4zWow6S5HzrNKl+Tfu+XG787lyjd+0SLWkaHEPfydB7sXHRRjJJ+r5A7yoOjZWbDXJxN7hXK0S26lRfMXpP7Cen3ItJZzxVKpVKpVKpVCqVStVT6RdPlUqlUqlUKpVKpVL1VJcVtU0kgXKx41OAEBnG9qJxoH3ryzT1uw7kYvcAeJAGo4qEHxygzYkDdJIbxNz5BfRLRCQUpE1zO26ng/3YHnnPvh1+evbcY3762Amgi5EQIbKEG7ku4TPkvMsb0rKbYIvmsw1NhQfoeIgw3S1l2ewNorlnJ/CPw0fgZlkmtCASJpdFB2WeICRuz35c58CenX7aNoBkFdeAPaTS1FypjMq08b0lB+MAuZZdQA1btNM5FZvE44Qf0CbmTXLVi7ETJrU7dogsV1DvXgj4gdtEGXiE3cYJtai6qNNwA3lYKfbGte9bzzzgp+t1wiljeLa+NPrgeAYOew4VXjKLc1orqLtv/c1X/HRpBa6M6zU8z7EFtJ8KlXXEoKzvufduP/3O73+niIik0sA5ak2UOSPoHm10HyCkhN1rH/jGg3764Yce9tOZDJ7pfe99r5+emaZ22mJ8F/c1jHjTfRltC1zU+fblKxJFfYXCiCmRi5hSeh766eIyYhZvPs1YT71OLqXkNNqiWD04gPJip+BqA5hOtYp69zpufm6LnA9bVHuM5RbQv+vkgpxbB9IZYTw8Sm67NFY0mrh/fhPHw1H0/V074BY5NIL4VMij/SZ75L4YGqLNvMmd2XWQ1503A8PbpPJvkrN5kPpOIIO2kS+iHEu0iXirij7bIBfgPvrsfBllV1kDHr8z215SMXEA+G3+CGJJZQGY8OYK0sUKruGRU2+hhn4R70f5p6eRdquo93oNfT9AMYmPu1Gc3ywAC1ykjeW7qfkcNqB3WxzTsCQjQU7cOcITxxhPrBMmXiQMnVzvZQjXSe6HM2zdRX2VqZ9ECcMO0jKexlonD7T8xWQxNvVa6bOIQxtZtOvcMp5jafW8n94VQT9deB7l3U3VKO71GrW1Ljm9Nqgdl4HfRw3enwNBLGXaWMc5oWQ7ruZKqNsa9V0J4XPzC4gT41N4542lyN2c4m2thhhjG7jO1CTO6aP37uWzGGeSKfpsgFxYCV/tkVm41GlccWmg9tgN1ka2Pb9E/aVJS2satJSuTH3cpXHo/DLeXTdoeVHA0BidoC8t9NlAx5ncs/Q+G0Q8rhAG3qB3ywzF7BBh/qkEjg8OoO42CzQmUuxfruG9bGAPlneYCtrM2gbGJYnjXgNskXwR6YynSqVSqVQqlUqlUql6Kv3iqVKpVCqVSqVSqVSqnuqyorbNOuEvAUz9lqu0OTSlJ6fJQdLF8Z1DmP7eM0Ebl9ZpI+b9N/rpiMVU9GaBNtfOwoFKcpjmnh4DxpHvTGnvvgbuXhnCmjL9cDrdXEMeNwtAF8IRQiQsbQhLzpFEjopHyOEWZ06a6ufjfL4QamcDvdkg+6aDQIayaTzPF74AtHJtBZgBbzR75xtu9dMzU8B6ihvAaAxhDAMZ2rQ4hocuFNCWIvTMsQFyt2TUtoOltQid8GiDeHGAq4xmgLzt3om2MDmO43F6piBt+BxtoK6L5Ea7Vge+UiIXtHwRCF+C7htpolyt1xtk+uzCST/dIgQlQI0rEkY+Zi02Vk4kgFP0k0Po6Yee9NOLhw/7aVPD9U8uA+PYINdNjzZTvuFGoHv/9Cd+xE9PTLaxD8Y8V9aBz2WzwEj6+tgNFW1qdQNt8+sPPOCnDx+Bi2eGcJRQHP3ojbfe5qfHhoGgsON0Xz+c5QYGKMZYdpnuXugNUixwaClBkHDVMDlbBikGGULP2QW7SddpkeNxiHDzBCG+AYvr7N67i85H36iUEXtLnfhYqqD9FzbJIbyC++c30XfEwzWyg4gNhtrO8hLQu2oVx/MlxORAAMfvuxdjhaE6ipPDa3IESONmnhCjLmo8zE6RKM+RPQf89O5xuBA/fw4u39koYqBLCNnIGNpxgDZurxAuHEjjs5tr6Js7R4DWVyO45iah2hubbVw1MI7lJ1PX3u6nF87DrbReQ4wNU6Pd4tbeIqSUXGfXBDHTJVfhQJBdi2Xb43z+2jzKrJEny9QuapmQcoeWnwyNAhe204gd0X7Ez2iRHODJCbNJyGWZ+qlHY094J+ogZFAYySw+65w4hzQhu/VAO52++1r/WDWPOClrvXFyvqAs1VHfTsSnc8uI7fEg2kmYxnw+3k3Zunfpk7qk6fEJPz0yiFgznEU7EYd2PghG6DDaW7HSbjPnV2b9Y8vnUYYbSIrbwBibzqKtLa9jHOwzGEMTEbSNkYn9fnpiEu3XuBgTSgfRNpuEfnu0swAvxUhEe8MzBxOEMW8xnyfs9iLnV2nJAzv5N3nHCNpVwiHL3uUcxq1gmB3mEZsYwW1RTEym2ufUGuwKj35RLhNqS+/LLRofyyXaVaGFcbsvjecrkGvvehlj8cg+xKo7P/A2P/38EYytD/85likVqX7j9M54MemMp0qlUqlUKpVKpVKpeir94qlSqVQqlUqlUqlUqp7qsqK2QwNAKxfXMMU+ex7T/KEwprMjK9gUu76Cc/aNAIG67x4gsKcXMFWcnsRU8dAg0NDVNcbyCIFt4ZoRcsdbXWu7rYVimLZey2O6eWEJU8zhMLCEbAZT8TXCDC0hToaQxlZrewdO3hSbiCQJ0XFr+Do4x/aoeit54FiTwyjnd7/rrX76M1/8Ko6/434/PToERJOvE4vgGVLkpuc0gHiFCTWgfd4lEgF2EoqRexqZZF4gbEPkJDzQj7y89b43+uniBvCHPbuAulxzEAhhipDOFrnUzmeQseMuULHxEHCMWALtMZnBdTzC/MrkOnnMhQNkN9VyUBYNcjiMErZHRmdSIdzNI/yycAwboM8+DtTWEsZxnpCS83kgKC2qyGgabeDg63cjDy0gkt9+8iEREUmlUW7HTwGfC1NbmJxC2+Tnm5sDMryaA04WI9ftQBLXeeo4kOG583DmHKK62zkBLPGuu+70030UY5jOCQS61zety47JhM6GyYU5QZtJk2tzfx9QYIcQzUCQnbJRLw1yORRC//uz6Es3XHeNn84Mog5CBhjW8mI7tj/88Nf9Y3tvRJ17FOyKBcKKCD3aNY44sb6G8WEldwJ5F463iPEeYVPRKMVb3qS9hvuGya0xGunNEoZ7ZoBKug3U48wEjrOTsh1CHClQ3VUI9ZwaQvxyOb4QymxjiKspizE6SGPSaB8h06so6/JCG5tzGrh2chR9YeK6u/x0y0E/Xl1EzKiWEVeEHCIzSbTBkJD7MXUdh1BqS07vLZfw3Qw+wNfJxBh9754CcVqSQTSvRxvER8nOc7OCePjwPJacTNRRLtdQvtnVtrYAN9jmk0Aka+ShbSbh6l/fjzZTpU3lD+1pY5SVAPpUbXEOmQ/3do4iOIX4kR1Cvd+cgov4334TiHs0i7gSSSAmd1MRWpZz0vl7Px2uoc25MaRNGm23Sk7NuRL6YypDY9xuvLvuGcSuCfyOUq+gnYTJWtcQllmmdnV8to1SL+WBVAfo/aSVx/UGLNrU/n5G0xEbmiHE7KCDsZLfSyNxnD86hGcayiBuFSvkLu5gLE6GMP5UevSeE6TlYIkknsej8StIYxOfXySMNRFE32jRuFKj4+tUF7kq4kt/DDEoRuNHk8buPLUTE2z3TTLIl4ag3HJVxABLzrdrC4ixxsX9eclULI72xYhs8gDeZ77vh+/z03tfj6UefSMYH2afet5PLx7Bu27Kw3UuJp3xVKlUKpVKpVKpVCpVT6VfPFUqlUqlUqlUKpVK1VNdVtS2sIFp4LPngLyWaYPbeAzfhZdmgaCMxgihmwR+kZ0A/hguEToTw3T21I1woozRhulxF8iQJ7TpNWFI44k20tH0cG2TxNT6VBJuZOksMJZSDhjp6goQAocQmzptfM/saJLcIps1Qnlpir5VRx6jdH6lCTTAafXG7Y3RO68JBHmYXBPvvOv1dDyx7flbED4H6ECKsB5DGHGAMOUWIYWWsKIg4YWWcBSng3AFCSHM9gGJuv+eN/jpuRNzfnp5FchKfwabt/engQZ5TSAr8Z2jfnqCyoMRuSgh2WlCRj1yWVteA7owlu2N29tDX3/OT1fJVTCRQP5CVJ5OBG1ugHDN7BqhnuRAukkb1p/IA1tkf8QQuczdeBMQzcQQ7vvXX/2sn1483y4Xh7q6oY2VI9QXohF2MCY0uIj6WlhCHDp4/Q1+engP3Ixz67QUgBC+c0uIJX3kSLywiDZTpc3uGQM+sO910i1lEoRKJuGoW6ZNpgcy5KpXxzMMUr65jELkguuSY20zxigk2u7YMKFyfSlK43huCfG/UWzHtWHqR1OjcIpml+UauRM3CbUd6kNf7iNn1hOngSvG0+g7hRraaSyJ566UyX3QxfWNoAzWyKWc+0o39f4feLufdqiduVW01xo5w+9qAqGsNlB3ZXKKDlP/3SziOWO7UF41wtBtFo6aC8tYUnJyFm362n7gu+fWOstbaNN1L4ayTe282U/ftWfGT2/MA7U9/uQTfnp1+bifThpyD26gLdc93Mu0uM3iWRk55/OjhPL2pwmD76IiEUL7+tlGE8fDNPYtreP94A+eAb52gFybfzqGvCZousBW8H6w8RxQ241h9KUzVHZNGisn9uPdZUd/+/wmxcPUPOpfdqOt9UJ9u4Ca1snJdjiB537d9Xv99PQu4Nx1cu/sap5oKY6lgSpJS4Fcj3ZqcFHO5QJiRLBMYz8tuZEaIfs19DsTAkbsubhXNExLkMi9tEDdxBbbuyzEHYwDcYv7RIOox+X84356JoQ+PRW7HvcJENZNCH+hibbRovd600KMySZ5SQ3afqmIPhtJAt0sVXpTjxnagSBFSyY2yBm2j47z+Q3C9w2NSQ1atnBiAQhyvYG4ykv2LC2d6qP3vGAI5zTo3b2Vb5d1msbSSAZtx9KODRubGNdOn8NyQ2minIP07myjtGvGOPJ7/49+n5++7nY4GNsAzt97AO30+z7wZj/9Nx//Fp5j9dJu0DrjqVKpVCqVSqVSqVSqnkq/eKpUKpVKpVKpVCqVqqe6rKhtiDDHahnT84y8ZMl1qkbY3sgE3K8mD2GK9/B5uPmdOIX0HeNADfJ5HB/dg83CAwT9NRvA6bI0jV5cbWMwcXKSGx+ga3uEzxwCNlAj59uHvvQ5P31+HvcJbnFHxJQ+meCKQ78NBAhHZVctdrtlG6xgsDe/Kzz9bbh8vulOIINVB2hlIkYbhPNxmvJ/6Ju4zs5d2Bh9eAh1HSDExSNMqlkDusBYnscOv+Ra5nZ2F/dctv3F9dwW2sLQENpjkpzZGrTZsUd14ZKrWLIPKEJ6APgK+yfS3t7iEcLtEj4zPApkd5TS3dTCKbi7Fmmj8wxhmVPTQE7dGpCSUolwvgbaHG98fy4PhKxExx1CTa6/Bk6m97wFuHOiH32jlkDdRJJttKlA2GAhj7/XCZNfX0Bfq5Az6iY5cDoWqEk2BXR0YgRlXiuh/cbjiE9r5/F8hw7d6qf378ZSgC98AZhwpY48dBe1xTMQgSMBOh4Lo18EaVPwCMWRMD0b4+tFwoSE6i6TBlKZSOCzYeqz6yvA5jbXUGe1Tv1Nj2J5wuQI4esU1Krsskhuryk8hngWbXZ8AqjawjI54jYxDuzZjyUa9TraT4oQSHYezhM2Xic0tZvauxd5ahncu7iOfpfPg6sbHCBHyBpiR7VGDreEk5XKQMgO7EG/qxCSXae4OhynsY0Q51vecIef3ui4K84tYzxvBsg5kvBm6Ue9TBzCsw4f+h4/7W7SZvdHv+2nZw8/5qfXT5NrcQR5D4TQr61Bu3Za9H4xho3vd12P5SDdVMCiLMIh1GMziDadd1HOGzTgu2TZWwwDrVtgx3xG3wmDsxbtskDj2flViu1UN5tEfX5uof2OcoAccPcM4Nxea45c8BMUexq0ZGrntVjq4izCyXanvTTa90rkVWnMstS/6H2rSv0rHKHYSDE2SshlxEWMSwYxTgQbQI1bNYw98TA5hHq0BMnDM4+ncZ2x7O0iIlLzEK8qG7SDxCrKrT8ErLvPIr87RpCXo8tA4gMG77dhg/JoEuZfJ8ffWgr914ugsRXraFclek+2MWKGu6iAIRfXTYrjLvpLkd5R+XyXxkcniLqeIMf85XNYFmUIp983hfhcyvPyDDy/U6P3VVrakW2182biyOOAAXY7PY5r59ZP+ulT6xQPaQwJR9EGm4SE33/LW/z09bccRH5rwJ4jEXw2Sd/V7n0rxoGdU4irn/j9v5JLSWc8VSqVSqVSqVQqlUrVU+kXT5VKpVKpVCqVSqVS9VSXFbX1DLCbTczUS7FIm2I3gAmN92Fa9/X33uunpw7c7qf/6o/+0E+PkdtskJxGF84AFxjbDbem2CBc0pIWU+3VDUydx1ttvKBZw1T5egnp7DCQocGxGT9dKwOpCCApHjmDGsK6ePN2Q+6ShjAS10V1BYRc+5qYOreGNn4PAxnopuwKynbtDNCoqQNwmlsPwAGtnzYoP38cjpN8neQ48h1rMN5ADoYBQmpclBdjgV6I8GWPHAU7LpFei9yJCW8WIQeyKK6RSgCrZnyrZQiRJZA2wHgv4dnC7ry0+XKIEMhAEPflZ+rVr0OGHPmaVbR/k0Q7m5kECllZIQQ9h3pkN7ZVQmDXqS+3yLV5YhII8vd8H/r12DAw5WYT7YdoZEl3HI8ndsGRcWmeMNoATr5+D/DtEmH7p9eAwh09DWfa8jpQnzNPoTxWc0Ad2ZH4LDl9Li4iZtxyPWJMsYD7npwFEtNNjQ4AxyqSQ/gAHW9RTAmQk14qDgQqRGh+o4E4laT+sE7I6fnzi7i+B/TIHAfOdeoUymhiFPEh08nD0BDuHwxyf0Q9ZrNAesPUv4O0AXqZnGZ37UCbPXnqKT9d2gQ+xE7UjFaxc3aDHGQtob+NOtp1N+UQyp8kt2sh9+0gIW5k2Ct9KZxjA4SK0bhy9MgxPz08jPpKJLCcgNHBG2eAXb75VrjT1shx9wKNuG8a7Wslh+dYXIbL4vIs0P5zVJ71BPIez6KNZK+Hy+JNB97opydnn/XTzz78JT+9tjzrp0enMC4fugNuwcO7DuE5wkCPu6lIi5yXA7RZfBBxMkT1Uq2jrU9SvUzR8pOFMuHuNMZECEU19H7QbKHtjg8iroaQBSmuwXnfbrT7z2IO9V8gVP+WWfTpwDpiptTIIdMN0GFyMfdorCacP9GHNr60gTiZoLGy4uL6WcLshw4B7Vt/Fuh1N+WuoV6aAcTDSJycS8O0LKhJqDe9n7SoXkYmbvLTYe+An15bRGdmPNuN45k92gWhRjh9jMr0wuqAviyWyEQy5IA6jDxGyMm1WMfYt1LDEqjUGOo05gG1bdTpXdvDWGzpnWp5A7E3GkYfHxhAHww4uE4rTG28i1o4jyU3TgOxztC7stOw257fpPfFm18/46fvfgvi0R98+BN+OjMBTPq2d2A3ja996Wt+OrdM45BLrvLkvl3vYNXRLOpolByAZ0ZQboUS2siDz+K7TpNQ4kwM7cslt/8dN+zz05EU2l21hPJo0hIw46EfhGhJzd4bMebe9Ga064tJZzxVKpVKpVKpVCqVStVT6RdPlUqlUqlUKpVKpVL1VJcVtbXkvEekogwMYjp5LIFp45tvBU5x8A7gtZurwMmiLrC53VPAdBiFHBsBvuLWcf0qud02CelwarQRtbSntE8vAC187jA23r3jdlxjcAzYRbEE9I7oPBmaAT7cIuTSo81eXUIUC2uYlm+UyCktDbSsbxjISigKDCQY6U317txBjp9VwnzJdbaP3K/4OJ/P18mkgeO2CAE1QZRLq4VyqdfQBiwdT6cJOyEcOdBBMNnpK0hIWp0cL/N5tKlSAeVfoo2SQ1R3iSS7e6KOwoQ6h8NAokKE80UJY4zQ+clUks7B8b6+7uHTkTjuPTSCNsT4VqWOcl6kjc6r5GobJcRqMY/za4SyRMit+j3vfjPiGVsAACAASURBVK+fvuHANX567hRc9jZyQLjFIbQs3E6fehzYamkTf88VqI0cAir4ljvv8dO3pIBP/+qHft1PH30M/Xp8AnhaMAFMxYvhWbN9YOjPzs/56UIJ7SRKbq/Vem9Qogg5Wbueu+1xRkWDZH1rySFTWrRZNvXBVALtwbVo60eOwpFwaQX1npkFqrRRRFl4hDYeHGn3k74xtPMg9c1KGfEtSS7LQXKIjJGbtVB+J6gtZ6nu9kxSvKF6DAfRp1qE+8YzOD4VQh6qfUCeuqn/+B9+3k+PTgKzvOvNd/vpyWGUYX2TNqnnjekproUI29sxgXzHqY9HIyiLTIQGqzSu4xBmV6JxvNZZznD05Jx/bJMc4m/ejbG3PIK8zC4B8zx6FgjwM2dO4T5R1ONQBvm6dhT9+ta74Yj71CN/66df90Yc9wZx/jcJj18vstPkvdItrbmooxFCNPvJKTK0ir7jloA5HrwWiPCOA8DgNp457qfHKa5K2FIS9RgnZDpEyzYS1B9OnJ7z00OV9md3zyA2no+gnldOIb/xEvBpQ9i1YVQwyM67yFezguMb5LyaSCCWlggprRACubGAMSFEOP0ZOt5NHZiG67FH4/d4FmNDjMYAQ/FzbQ3tbIOeORjD8q56He275qCdxOLkEN3E8VoF7ZWdqD1yuPU68Z8dx+MpckdeQ93Vg+hTSxX02VSOxop+fNYpzvnpBMWY/viMnw4Rxuk26H0minfjqTG067CgbzpCCHcX9fTjQIfveRfuPTaDmLY8t7nt+QNDQFrf9QOIwy4hxVUH5T+0A/HutrfBpX+lhuv/9V9+w0+XyTm55aGsyx2n4CJh+2u0/GUkixjjhPCOETS4hvXwvhGJ4v1690F8T5rYiX7kEp4fIzSXkWR2tRfq446gL7/prbfIpaQzniqVSqVSqVQqlUql6qku64xnMIBfjfeO4deGWBzff2d24pfeG+/Er5DjB7Ag+elH/shP75jGdcauu8FPR4axF1EogV8HqjSDUyOjjJVFGB9srmB203PavzLFaSZtaAi/fs0vYgH16Dh+vXGrNCNXoxm8Cn758Mgcw9IvCXGeBRtDuhjFrywZ2ucwGOJ97vBMXp3NOrqn3TfM+GmHzI9o4kAGBvBLYJD2S92xB7+2hOmXlL4B/ELXsHieFs1aejQrHSTToX761TFNv+RbWgi9stL+xXb2DH5VOz+PX91n6ZfwYgkze7xf5zwZLGRS+CVs/z7sf5QkQ5cGzT5t0mzhxhpmw4NkMMX7dfIsJ/2QKh/9yG9It5QdRL8I0uyYR/leobyu0V6Y1SrRC/SrdI1+CZUortlPe7OeOnHGTz/3OPbnMw1cn7aOks08+kmmYzhWITOkfI7afwsfrBylPa1cFGI0g1+saes1KdGek/U1GAo59PPcMMUnl2auv/ng1/10KgLaYu4cnrVKz9dN5Wmf0SbtMcnHQzSbGKFfa8tkuJOm2dkIGQ0F2LCMZi/E0sw4zYK5LTxnkeiNI8cwa3NorP2raJyoDDZbM8IzcmTY5aKu2SgsFGK6AOmxcfyi2zcEw42WQb4ch+IzzajGE/iV2KX9f2OB3gybD34Nexm+6d77/PSf/48/89Pvuu9OP90fQ7nE4oifIUJsarRH6fAg7SscxVi8eZF9SQ21Ad5P2oTRTk6dbY+Vv/kbv+kfW1/FrMobbkd+3/m+H/XTI2OYGUjSnpYT1E+fz6PMW7Rf5eo5xOF9RM3sPgBTr9Qgjp+k84fjiM+jad4Au3t6oED7QyPsyZuIzImvYuyJOaij19G+ehPTmB37/KPP+ekCzXx4IfQ7h/pp3KIc6+dxryDtQb67H3Gw7rVn2UJJ9J1Dd8IcZYOayMYTGBMaRB20QugvNbp/MkmFEEe7q1GQbw3iPa5ORn/LNENXyCMmbx4D8XK20pt9dW+59a1+OkAEUjaG/hWkmaQgxaznj4OeyZ3DuDK7jLoO03tbPEV0EO0paR3UR6WAfuLSnq0R2g++Wm5/9swcTGZSMVzDayF2lcngao3eefY4M356YwHt69zcUeS9ifxmU3i+iRm8UxRc1F2LTHIGwjS7GkXcqhAp0U3Nncb7/Pk5zNzv3DtBx49se/4tN8MoZ98BxM/Pf/FRP91qofwndqF/mSxi5u3fhzhoiU759iO47+oaUQpeu101iGgo45VTBkZQnrkFzJAbpuaIbOonKvJd732bn86OovyZyArwexzHfiJJo2EctzTrHoxf+nuHzniqVCqVSqVSqVQqlaqn0i+eKpVKpVKpVCqVSqXqqS4ralsjXGaEDIXml4B77bkZe3dN3YC0CFAMp4Qp6b40pvaH92OPpEoIU97PPwWcr1EjDKyI+64vALUM0r5TsVi7iCZ3AaM9tB8IjBsEghEOYjo7TAvzQ3UgFdWzQD15bz3yZ5EyYaSJQVx/dILMi1ZwnQLtvRWkKe8QGTh1U9EsUKs4oXetIMqNdtMkcEYkM4ip/Rahcg6ZCDm0WJsfwQSAtYyMANmN0DM3yGCBtryTY0fbaM6nPvlp/1i+ABx6iPY6278PmPauHTuRlzdg76ZEFGXALOzfPfqIn+4nM5M6tSne/zDEZimEv63ngL6cPQ/0o5saGALu1iJkmtFKjzbRNCtAgMou9l0L0v6mXgR1ZKkdVyroAw8/DEwlSzhjjPLA6IYXIzwr1D4ej6F/18Pox9kB9JGdZB4QI3OY/jSOTxMef34Rz8QNL0Ko0uYysKKq69DpiG1HnkZMcolYSfej7XdT6wXUS5zwZj5u7fYGAUZQzmMBlF0fGVKUyISl6TK2hXqvN3CdYoXMucj0p1RCG0h08tmkJQnROLUd+lw4gL7WovzWaV8/10NeCrTnYb6C6zcc9FPGzA0hSdUayszxyEyM4nypwqY03dPd993jpz/4S7/op3/7o7/jp7/4+c/56WsmUV9hwhaTaSw9YOORgT70meEBxCY2IIpEaE9TwrPLhN83CTH/3d9rL3s5cgwoaJRwr09/7i/99NQBLIW5YR+MA+MUSzNkdjVBaBlj7RXan9mSEc3OSezR59DxDBmeJNO0h7Lbm9/dT+WAttYctOnsFMaYG8O0HyttrrlrGih/hkzQGjR+NKpIR8LUHywdJ1QuQvss1zaAPwao3lvBdkxYobxvHgUGmIihfZViqJhSHO9xDdpLls1vEkN4jg0yyynx3sIO+uzSMvpsgGJ/kdDQJJmWHXF6s6/urr14n2SkORTk9y20XROnOHIYz7YwT8ts6kinabmOu4zrJ6I4PjIAvHMwg5hVJpNGNiByOnsMl/MYy+pk7hgg3Ltcx/KyMp1TbNGe3rT/dtggZhw5BZS3bwjnb4Zoz+UknqlM+HBuE/W7a/RWPz27Ajz5zTt+WbqlAM2v1Sp4fs9pbXucz7/1ZrzrJxM4f6QFE7B//WNI02uAfPG/ISayQoI48Kbr7972nMz+dv9NJ/EmPUKmeeOjWDbyuU98xU8//Szq1NL7yd4xjAnTh2b8dC2MsczyHvc0VoZpGYsYGnOp/7ZcXg5DhXAR6YynSqVSqVQqlUqlUql6Kv3iqVKpVCqVSqVSqVSqnuqyorYR2ieGXbz+0fv/kZ++421w88sMYWp/5QwctYKE7OZpz7y1ObgmLpYwDfzAZz7jp1O0d2G9gSn/sVFgDBnag3L2fHvqukn3HJiY8dP7b6A9azxgNRt54JFVcpfdrOE6xqL46zVMc5ctpq1tGRjFQcy0y4mzOG5qOD9lcC92C+6mgrSvnyFEQwzKPMIbtTLGGSTMLxCm4+ScSfuORclt9ewcENVHvwIsY/cYuTXGgL7MxOFyONDfRio/8IF/4h/buQuuZiMjwIFitC9SiDBSok4kSXgY72m53IBrcYL2dR0cpD0fydU1EgDSGCXEZnyMXMsKvUH7hMp/i3UuuXYWyfW0UqF9KMOMblI90m9ZLXI8tNQcajXaP5euOZxFGUVpP8EG2SVXyh18pEzIDF3b1HC93AI52fYjv2fmnvHT6zk8XzTBzsqE/VJbjhF+OJVBffFeso0NXHNoP9C5CGHz3VSJXLMrVaRbBvHIoVhjeX9b7qd5qmtyznQJ1YuEyLW5ifot0P6tGxSTZ9fgIrh/Ep+NdhxTG1XUl0eOpozbJ8jNu0L1fo7iQZMawbFTcDGt0f68ZLwrLj0r76ubJpxtIwcsMUSOu57tjRvq+DSwb3Y5n57AsoKvfvZTfrq0DGQ8QZhyNE57epKDcTSEfppKoO0mCJfkvYRjEdoTLobrrxGO/HwHx7z/fozbN950o5/+2B/Agf6Rv/9rP717jPYNTqBvri8D9Xzm5Ak/HU4iL6MZfNaroZ3EaT/SOu2b51L6xAr2CXUqvdlX9wf3I448NovY/bdz6Bfx3YgFiRTKNk17KzqEpns0tnLfjLEbObkQC41bvF/4BqHnlvY0j3SWQji0t7k9jeVHCYrrTdpz8zkX8WZuHW63MV6q0EI5h2PkYk04Xz2PvlahvYJDKbRZL4zzd/ajDXyr2Jt9PKO0l6HHRp1h2nvRon5j9E7r0L6YKyeBLFtyxx0eu85PnzqOWFajvRgNOfaGJnmJBNJL5+b8dKXaRmyrtISBl18ZS87qMcRAS/uUzi8D1+zvQ36ndyAONRrIY62JezXpnTo9wO/aaBBNwqSjAmR3nfZJ76b6aVkfbYu5ZVkKH+fzb7wN+34maFmfSE56qbe8vb2LRzhOS1to2InSkoiRPRgHqjS2FyqI06kJ9JdAFu20Rt8XwrRArtlEgXhb5ifp+4uDdCzEi+vU1ValUqlUKpVKpVKpVFdY+sVTpVKpVCqVSqVSqVQ91WVFbVvkjBeLAte46RbgqlGa8j/y9FN+enMRU/INQk1Km0A05k8BaShboABhD+enQphmzpBj2nA/ptGXVoD7uE57yrlaAkIwPwsEReR53LOMqe1YiFyeokBBcy6eOx4HrplII79x2oi5VIU7mUtYazyG6fX0AK7ZIvSl0ugNSnRyFhhJH7luhcn5LkFupYx9OHWkCxVM59eqqKPCJpCLAjlqnjyKTaMXjwKn2xgDkh2kzZTnN1Eut77jPSIiUmwCB3rkoW/46akpYLeVIsqwQO2rVERd9KXQXvrTQIPOn5v106NB1JdDZRCLIo+W8MLcOvCNFCE5tkdo39oq4Ybk/OgSQrlB2Ci7EIcJUwmS65lH7mYR+l0rRv266AB9a9I16+RIHKBN6l1yn4sl2uVSLqM8s4RdjQ0Db5598lk/vRlFmy0FcD2PEO8oiCSJjAK9bpFroBAaOk5YZqhF7pKEoEQIf7OBSyMor0SrefSXIYoj64S8xlKIFwFC7yw50Flyg12nek8QcsmIeZHcxddzqNNNillxWrbQn0U/sZ2xYG0N537ui0AxDeE6+/bBWbBQICfyDdxzYhouf0dPAWWXMDm8WrTTBj1rKIR7BWh8CIYRw87NkxN1hFHW7qlYRqxbWUV75ec8v4wYwa6FsSjyxJgyR45oGH02STEoSM8cp6UKsRghakGU0bk1Qhtt+/i73/Me/9Add9zhp+fnseTk05/7vJ9+6hm4hXt14J2bK2izzRyc20MebXTuou+f2QQWmIiifqsNXNNYcsFlN9A8xutu6p+9Ce6601E8w9eOI99/N4e6u2knxp7yaYwfeYqfQYqT+SbayTAtD/DIudohdnCN1jmsE2JdJzfddGcJQbKPXOcJsZcc+mmU3NfP06bzOWIBxyjeJ5K4ZzpJ41oNcXW9Sf0xiOcLbiB9PfXfVAnP9zrLm913T5Yclh1yznXpfbIVoaUNlCdTRj91y+gv/cNwQG1QP6qsoh27tOzFKaPcc3R+MEq4JKHvtVr7/FIV9w/S0hkJIu9Tu3B8ZBxxkl7dtrx7VBy8F++aQRsPeVgiUG3ifTgQQt9vejSGpIDsMuI6kEJM6KZ27ME7gSVmOkSYKR/n8wd3YNz0Ar1ZKrOdrLTjVMNFu2sFUTEBcnEfGKCxncYsN4g+mB3HOQ1qv/RqJS2KExV6TxdBmt8B47T8IkTvgOyWfTHpjKdKpVKpVCqVSqVSqXoq/eKpUqlUKpVKpVKpVKqe6rKitg3CFUb7MPX75c99wU8PjGKqfmQcjpDNKhCcME0np5JABEIB2kSbUI+xEUyd10rAluI0dZ1bW/fTThOoUrrjbNYktO/kU3BUXToG572Gu73rp8f5mqLp+iRtWhslpzpCavsFiMLB64BpBDzctybAh+rkvhugTd27qf/n1//CT/MGt1M7Z/z097/3h/z0Zz8FpPX82Tk/XSLU1msAWeHyr7jkoEvoS4IgssJpYGmZKE3/5x7x0wfvulNERP77x37XP/b4U0/46V0zB/30xBieIxLDbzODg2hHe/YAxUwOAk8ytKH0V/7yy3666eD5SnlyliO8dG2duiMhLlVC6ropx8E9GL90iL9IEJplyDm0zPg0IVZNQkOihF/UCX0LkiNbiFBUl5zRDCGdAQ9ll4i28b/SJvCiSgHluUZ5qZHjJaPOTcKUQgn0L5dceGMJYIZCiOzaInAjIcfBveT4NzwGdK5J2HJlE9hUN9WkfFearW2Ph8ilt0QbeAfp2SpVnJ8gFDNKKKrj4vqra+h3JsSOmjhnjByn00mU+9Ji2+12pYq4e3oO12vWkd8jx4CYRWPIV6UGDC+UBPbsGdrsnpZIGHKQJmJI0mnUdYFwV48xZHIJzRV647741LOH/fQNN95Cx7ERucPuoiE8T9Ohsl1CmdZpWUqEEKgtptSUhzAtVQgTmusRhlUmvHKg4zw/RLGRlySMjY/56Y1N1O9XvvIl5LGM/pvLob4qVOYhcu0NEjrbPzrsp0dGca9VWi6zuYL7euQW7JIbdDc1EMM93rgfm8WvV1CGTyygDR1dwTvJPirbZoScqFsoi1Id+bYNxFJ2jLXU94XScYqrJYu2UdzRrsfB667xjwWpjzz35Qf99DTdf6of5S8U42MhfLjg4JkqOfSvMRpbJobQfiKEhobJ6XQn9eXpLJZXFFZ6s6SoTg7pdUKBPYu062Ipjit4/moB41OAXLlDSTxbfh39ZH2JsFSqF9dDeaWyWE7g1tGBW4ReV2vttl73sKTIUJ8OhdEWhqZwvb378W65nEO8jeD1WkwAx5sVPPdY/w04KYCxz6bwfMePoY2PD2NpVDKK2Ftp9Ma9P0lutJUKxpVq1dn2+JbzG+xW35u4v52czjjuuiiTACHQjQaNz2HEmDA5MQcjaIPpPvS1MuHbZGwtMXIF59VdAXpHiNByBn5fyhXRN/n72cWkM54qlUqlUqlUKpVKpeqp9IunSqVSqVQqlUqlUql6qsuK2vbFgZm2aIPS9XVgMeU1pOMOpoRbArRgoB9YRnYCqIdLSN4CIXFWGCnEIzcJpQoaTF0nyc3vAlkWdHmXelzPa2L6PUBuZMUqOXdGgWakJ8h1Ng6HxhJtQF+vEN6Z2e2nhwgZTgzRBsJFOKI5SXIeDfTG7W3uHDAOh555KY+yveM+HH/sGWAkSwtwhwwHCG0k18ShIaAYu3fB0XJgDK5nVXKldGiT+EKenC4ngF4dOdFGkw+fOOYfC9LG6eyoWifOYO4U3JRvJjTomt1AkjLjcHXLjqK+GgIEd3ER7obDo0BpmnVgZoaQhhZhrb3asJ43oQ4Y2tibNgm2xCQaQmTDLZSFJYQ+QE5q7Ewq1H9SUWD27KgZJIfZOm167dJ11jbb9Vule1YYYyb80qNrN6mNGHLejVO+tpTHKlwBPQexKk8oq0tIaZLwzgAh/FHaDJxRtG5qcBgxsFFr0nG0v1gCiN16DnEnk6TN2ulnyBgheakUzjl9Ys5PBwiTzmTBZBHRKskE2kPAkntpx7l6jnDdRBbI0FgaKB0jn2u0JOLQO98v22nn9LaHX7aefBg4aILKSSKxbc7+znXyzJyf/vBvf9RPnz0FB+8yLUM4tYCyY7TSo77jUBwxND4GqU8Z+v3ZEJ5uDW0WzhmleBRPtq+Zy6G/8ObmxQLG8EYD15ubw5hgXMovDbOWxmGOgJEwrp+MIg5VGZej45Uwxugqub73JqqKGFoqM55FW7ljFxC+Ijllz+UJlaRxcGQaDTkYobGK4le9BJwuRG7GkTA6IW977xJ2nKFdBhod9/gNqoBsP+J0lrDncB35nSSXWnYxN0nEQBPGOYEyLbcK4Zli1MB4iVCVnq+P3G737EC5zp+ipStdFNPKsQj6v9NgZ+QlP73hkBP4IOLXm996l59epPfC+Q16J9iD8mpRWXsOnrlJ7qLJDJDW1Xnkod5s1+++m7AUSOKERBbQT7MjFKjp/bdWRmUMDKPuXIu8D42iVQ0Pkxt9ADE8X0P9DmdxTjSI46uLhO2P0PKWLqpWx/OfPIUyLxSdbY87DepftOSj1aNdIrZTsbOzQiCM+5s6LXeo4X3j7CzqJU8YbYB2KvAoJlUq1H7JHTccRF1vWWJF7r+cXqOYX6ognqVp7L6YdMZTpVKpVCqVSqVSqVQ9lX7xVKlUKpVKpVKpVCpVT3VZUduRQTgcVh1MzQ6myeWSML9mAS5aLcJGq2FMOY+Owo2rRTjdgUNwmXz463/3/7d3Zk1yXIl5zdr3ql7Q3Wg0ADaIhQYobuIMh0NZGtERctghPTgcoSXCP8O/wOE3vfqP2CGNHWEpFEOGBvRwyJkROVwAgtjR6AZ6qa49a8nSQ3fndyrUHUOPOjEv33m6LFZnZd68efMm8tzvaptTqQs5vDbuI82wXpM2dpQEmEkx1U/7fu8pXnM3oa6k9Dp76Zqe79fmkESIhY/3tvX7+QG03zUk8vaYvKXPu1vSQYdIfl0+o9flp0kqJ83lzJn1uNyDpvMPH3yMz6WRnFl7JS63tu/H5fl57WsV9d9EWtb5S/P4vhYw7kG1rJ9Hqi9Usb/74B+DIAiCIf6tJZ1H+hbUwg40zo1nUrbn7mgfv/3qVlxegC44Rermu2+/EZdLP/xhXM7lkXgc6Jymodhwod4xdLnTZAzV/KRyFCEZGfuXghabTxfxfe1rBOUvlZG+kkHKM9N0Ty6rm4oO/adaDimm/B1qITB3mFDK7590rD0kbVI3nuK4x1BAd4dqg1mkei7AX2Ui62nCxNEMF5SPpAZVy1h8HYp5r6O+LB1oOw2oNiOokD1odi9fuRqXv7n/TVwuQrNbmtP+XD2rbbYP+9sIyazFOdXnGNcu/3m0uphMn3Yc+RKTjbUTu83mMd/+17N2Xn1ac0+/sbik+2ZjQVr1X/z5f0hkP47467/+H/oPtLE0xNvmYVv/6c2fxp+9//77cfmLL7+Ky7jsZhKXM5hGQ82QmvAkRLI3kqIfPdA0k0wBbT+UojlB0naI+2OUUL86RT+Sj9Qv3FiAMr4KFRiJ7uO+rsczizrXxar6uybqboQpS2OUwwyua0wDqFOnxz4PW4c6MsY2001NqTmPc57LqA5rfWnMyxldy3vQhws13bejEVLMe2rjLSSaMow/gta6ekPXwaWLqptb60jWPVUw/QSpwsEEqbtF9XVF9HXVrsrtu2qj33tV+3r5VURLpzW9aNjXb/38Q/3t9rbuH6Watt/ra4zUWDj4zuvf17Ske880VglqOo/nLmoq0vy8Em6rFWm8/bHG4O2e2mk01b483lYa9wKmS4Q9tdlGSW1gBJ0/RELyaJDM1LAx7t8bDzXViwnaD/D58pKOoYvk60w+mf7iOPr9g2ssO9T5Yj95/67GpT/+338fl5t9PY/U53APR783M8UA/9HrHj99rYyE/wJXG8D0rDzG0pnsbx7n+I2nMcYYY4wxxphE8YOnMcYYY4wxxphEeaGqbQ8LCWewkG0hL0UjhwS0fBmv6uv6fPO5Xv/31qTULl9QAuqTZ0o/fPX7fxCXO8/1Sv3u7S/icrcj7SOb0X42GgfaZwoaGpNZHz5Aqm1B+1hfQaLXgtTRFFSW1K6+P7+nU7G2rESy83M6vjtf6vX6pUv6/FpRqueTZ1rYNzX6zQu5/lak9Sp9ZW09Lg+R2skg1uVV7SsXoG0h1W2IRL5CGenHU6T5tVXXk640iQ6SClNY+H4M7XJ75+BvM9CBzmK/gpTqar6hVK7vvf29uFzNSjPoIj1spaDfX1uVLrexJ1Xp3ldqM2PUX1DUNufnmeSp72yhLQfBfwxOizwWli4UkKoHVSwFjTYcQVPD+aXeSnV1Am2OKZo5LGTPv83icybcBlOms0WHf8cjOT51LQM9kqrtaHS8djuCLhtBz6NqW8BC9qwz7k+rBc1vIv2tUkWK4Cmyva32MU0jjS7SNVKCxbR0RipRu6W+rrOvviOkcodpEctM1kZiLdNrzyyp364WdR7LqK9e8+D6aUItbHdVb0vQnfpIfq41kC6bMCko4SGSjZNKmZ5b0DHPz6scQHWOoqSyWP8l164qoXuE6QcT7E94qIn+7Y9/HH/26y+/jMuffPqLuJxCvzfBNTtGfVIbn46p7WMqAPYxjXNUnOocDXpSN2emC5CEzuMESlww1j41sjrmty7o/O60dd0Nt5RQOkL6ZL6ivmNAHRlp7OkIbRT309QEdY2/HebYiR7UKhOGJxl0GkhcZ186RT9RnOj8Mul8s6ix1Qj3/wjDk1xFf9vr6W/zaA9LUEOL2fyxn58mIaZfZbNqK6ksVimo67xM+jrOJw+lmH/z6zv6flFp+IMFjef6qK/FksYQ6Uj1uzR/LS4XsEJEiClOjTMHY4gR7l/ttu4Pa+el+qYm2vYH//CzuJwra3vLF6GNI619c0NjzuFE6aa7HWm6C0Wl/TeqGgOPMf2EGmyllkxaeKmK5Na2xhhdJMDnitljv9+Hclr6DgrpaTE4HEvvb2lM8vVnel754ENMZcM+/uhP/igu38YKDnzF2OuxXasdpZBS39zTWDscoA9DYO0c/qPE8dJ0ZnB2LH7jaYwxxhhjjDEmUfzgaYwxxhhjjDEmUV6oarvfklJCra5eLswGzAAAGlpJREFUl76Vh2LY70pnLCFBMhiq/MnNm3H55Vek4D5+LI0hDU2kDBUzA3WgBHWh25FK0e8flMd4lVyFbvfeW9IfikhjHSP5jYsA9x8hba4ttWAZaZRvXXtVn88p7ezTp/fi8g/fUjps+tprcXm0o9fxrU0kAZ4iJSwaff26jn/Qk9p3YVUJdAVoDEWkZT6++yttlIuVZ3SO1i9I9wqxEHDYl4bUGeh8pbCwbh4LvWeDAzXn7KLS2/7qz/9LXP7icyW/hdBol9bX4/LejhLDNvekr7x2Xgt9X76ich9JaX/zP/8mLj+4r6S6MRTuYonKueqs1dPx/ff/9l+D06JY1O9RG51CQQuhT//oDansSfDJ1xvHfj5F9Nr0GD2OSi2dVyqyvR4S6SJq/lK2itBFg4h6I8pIwiyVqM5qm0OoLzSGg1QyqXgLUF5gEQe5rD7vddUXlIs6ntWzWNx9VddLa1/XVzULJ66k7ey1Hsbl61d1vadQ76mpVKHtp1LPc4fa0tMtXUdppHwjqDrIQPnrD9V/KicxGYZIGw2hIFbrySTrUq9n2uAUbbow06CS5dIlJcZPoTxOcA0cXRvdru5xm1u6D6+vaxtttMFen4ux62BP1G7RptKZk1KxVU8RlOwTk7bTyfy7exma9LCpeyL113NzmkLy2r7a9FdN1d3mhq6vVl/3pA6OZ4BjyDEJHfWYnkIvRFvqQYnLHr6DiELUVYhpQahb9p+DLPpGKLhdfgdTUYK0vl/MYXrHRH1mBUnAV1Z0Hufz2mZvp3ns56dJNIaSiCT6wUR91sbWZ3H5608+j8u1jPqIykj96lc/0ZinsK463YGyXL6sfnv9vNrJY2iXk6HqOot72MqhGhtN1e6inv5/Oa06v3dLSeQ3f/Y4Lp+/gRT5GtrXWOP0cUvbXFjS9+/f0woLX2Pqxr9//w/j8tnzum92xzv4XNs/TRqL2r90TvepDlT28xf1OYbxQRdj2v0OJf9ktdtW++B6//SjB/Fn//d/adpCpqR9+dO//FFcngzV1r68rbThYlmfczpSDsp6tao2m4F2m+JqCxOMxTLqK/JpqOiBVVtjjDHGGGOMMb9j/OBpjDHGGGOMMSZRXqhq227p9XC1Cv2jh7RSJDFm8Fy8+1yv5Nt45T0Y6W8zU5VrVYlYW5t65f+4K6UhgmqysqTX/Cmkw+01D/TKAhZFn4PGk8cr6XCI9DwkYHVDJMl1kASJRYmvXFAy27mz2pdHj6Xe7DyX+vF0Q59fgKJw5YJSxW59Jd34NJlAd9t4IEW1D3Vh2pOWuvFMCWjUdLmdG2+/FZdTTJ6Fxnrr1t24zDS0CRbITqNJ95CYeZT4V8JCt0P8/3NnpTT//KZ0kY9/9o9xeXFJiXDvvat63uyrPX702e243EW64WvvKFn56qs6j1zcPAUNLI12Ncklc5mGUAmZ7spkWn4naZhkSz0uk/mX6sZMei3+jp8PkSiZgSqVmaK9ZLgdpOpicWRaZqynGdUWtlcIjTDLJLyEUjTLJelYM79B7Rif96EPpdFGmSpYqWqbK3WlbDd31e8w0bFcQlIxfF8mXY+wGHbuULlfWpLi1N5VW3u+pf6+3NA1myu9uFtWPo/2wAWyE9Jd2eZntNvs8e07aTJImx8jVZaHnztUPUs16YFrF5FcCuWyj/sjk6V53ExWZJud7Q+0A7we2Vdx6sBJ308npNqmsqo3FINBWv1RDnroxVVda/ce63oZhrqfTiJ93sQ9Yxv9VA39VwoLxrPN7MP238T5SB/qdJkTEilZUzn0GVsYK+0H2l4Hv7OGDnQOunFmVynWK0iMfxtjocsX0Mf01W+FUHPLmNpwmrRD6acwyYOtppTajb0P4vL2pvTfszlNl1rE+KSF5NvcppzOPMYQjycaQ7zy716KyzuR/nZvQ+d6aVV1+vr3D85UsaL63N5WSu7z5xoLV6oax16/roT/+nkd7HSie9lkpN/cfKK22d3V50Po2c2OxuNPrks/r9TU5z/dlqq8oiHVqXLxqtJ1Bz215N5Qx3D5usaZRaT67u3z2UT1nD2vcWxrT583mxpzN86oD19d0xSvel31nsupr9yHmvz40cF27j1QG0xjFYIf/Ns34/KVV3R+P/pQacojzFcZh7jWkT4doQ8YD3UtVyvax9mxFtRrpBOPcF33epxGcTx+42mMMcYYY4wxJlH84GmMMcYYY4wxJlFeqGpbqurV7wB6RDYv5YDWRITXt6OJ/na/L42zgoTZQU+v+fsDJY8NsR0my02h3HVa0gvqUMjq9YPF0Pt9/f9tpJtWq1iAFepOaiyVJg/fpoA1cvPQ/9avrGvfe/rbDz/UYtyf3VYqZKGg19//+T+9e+x2Hm9KZTlNBm3pB99+oZS2xUWpqNN56XlhW4rIxn0lqXE7P//kl3F55eLluDy3pmO4g5Su8UDnI1eQqlRAam4Fukl97qCd5JFevLMnBXhnW+c0V1abWlyW9rO0LEXk8jUlvC6vSiNh+mMGC/Vev6GFo9NF7WOIxbJTUDSHU30+CJJJQ6V2RpWNiltSOtpxDKhGQ4HNpZliefQ50i+RsskE3AjHkStAnc2eoJNBCaNCzzrg+WUyNxOC5xoN/S50lOOU4dOAinAfbYgqMMu1MvqsQO0snULaL1VF6MKNhq7rUhr65VRtKZfTdspQDUdo6+PDc1bFwuLzVf0mgvSCEa6FEAuqJ025qP5jgEW0M98hte+3YTTCb6BtpaDasn0nTXUOKZbQ70e4SUeH1wP/f2/IBFy1iwHU9xllGNfFFN9nkm0eyZ287ki5XD72c2rLMyp+Qopm2FG/T3WVybDTGa1N1+OZuvZp97nu9+1NlffRN92E6jqPplFPYUoPjnmE9MkWxiiDQ02WLZtp4XnoyuXZb8WlLFK7y/idiOr9RH9bgprbqCIxdIQE3z1ts1XHeURbaoXJ3B+ftb6OyxOk6Dc7mooTIVG/UUbq7v6duFxZwP0L/V0O44D6SPeM9Ira8fwSxjAN1d3DWxpTpXAOdrcOzlk41vh35aw02kdP1DZ3tjE1Kqd2t4wxKseZvHZC1PnT2zpflZz++NqbSrTuQLvd3uN9eYLPNY3jNMlWcB8s6rezSLEvIVGd92mucNGo6xzlochW9HHwUl7jvD7U/yzup+GIKd5q950epjgdVml5Xr/zyg80vn7pNZV7UGrnkXC/sqRx6dYTpAcv6zvlqq6jCc5FlMEzE6ZZBBgjDNEn9Prahw76v5PwG09jjDHGGGOMMYniB09jjDHGGGOMMYnyQlXb1XVpUq09/PREz7+lonTGSQ7KTii1IF/W33IB1ExGikIIJW44kkYwpfrCRbqRsDqBzXW00HmAVMPmnrTMPpSZxpw0iiwXtsY+9gK9tt7alka6h6Tedldawt//RLrHFt5g70P3fevta3G5Bm3nEVSK0+S995TQurykNNgzS9JSiwXtx8WXlBi2/VypX8+eS614sintlUmYX/xaCXKPHkjTnSDxj+mLVG3rDZ2P8uECurW02s7Tjadx+f79ezqOBf3dm29KnRhBJxtA925UlZoGCyoIu2qz3aZUqY1tHcf2M21nuK/vD8ZSPK7eWA+SIA8FjfoatbkooSTW44hwzaYCJqZBBz2s4PRJ6Z5cRH5Gu8W2cY64GP3MNYvvMMktk4HuNfPD+q0KrsECkuiCVDJKWKultEcqify8h2kIczUl1pVLVGq1zQh9MjVTapbZko6Nau6gDfUK+uMIelZ4uAB6hFqsILE2V9BxNNvSeIqF43XKJEhj30qonBn16DR/L802DwVqwjTUZBcun90hXRu5AlKhoZkd6XcnafsjKJHpCP0KU7RRppoaoR+i/j89oU9i/Z00dYBlXiunyRRJvkzpzWMcMO2P8AcqLlf0nV98rqklOxu6P46RZPscbbSFtNsyph+U0VEVmBqM4z+qO+qUTOSeoP9soT2Ox5y6xAXl9ZsBE4w5hSGL/jnQNpsd3QczUBELaSRtRqqDZi+Z6zGVUZ3nauo/G6jQ8K76o9oS+skzSihN5TQ94dzC78Xlx080Ftr/RmO+G2s34nK1qjq6cF7nd2dD27/7JZKjWwfXaaaswWK+pLHEyjnty+Zj6bhhhLEirpEUpvkcTVcKgiC4dFmrRjy/8yguj0dSbVtIKd98qntCONH5XTwj7fObbx9qH74XnBq1hvZ1CP01j3sW38Hx83JJba6A6yUcajthqPqNAl0z+bzqolxUO2E6LvuseUxjKTQOxrTz81BqhzqntQWNMTiGuXhF6bnT7Nv4vsbIgzGeqzAumOLa5HTCcIBnI9z7CpheNE1z/IjVPU7AbzyNMcYYY4wxxiSKHzyNMcYYY4wxxiTKC1Vt187p1XOjode9nVYfZemXnR5SbQcq1/JK2ytCFxwzRQppktQ+qAyl4NyVq0y91PePFs7OQwOrz+m1+S4WQW5DNakvaB97UGC+ua90qa8/l6KwAr1z5Tx0srS2eaah1/7BBL/b1WvxO7eexOWffKD0tdPkD977UVyeWUwdCyUHU9VttYbUvkXVy9Wr0liH0K12W9I+Nraku5SCN+Ly3s5TfEfKyH5bCsp+U3V9JI9U6tIuavNqa2MsSL3xQHrz5/CxS9AGn21Izd24IyWqgiTMUontXdFnH3/0icoff6p9RDrZ2RUlkr39hhYIPlWQRDoeHa9HnKi0JsAUKuoE+llmRp895u9Qpj6fQR/Aw4hmFqZHkhxSHgtIEs1lCihDuaRSCB14AA08nZaOkk9Ik2xDqWV6LRNuWW6XpXXNL6hNp1JUyXVsixVdM82mrrXqovos9k0jJNwx/a8LPSk6SmKcsSb1m7u70MCga05DnciGzP5EYN82DrUP2YSuCeqhReheebTF4AWm2gZQwkb/H6m2lNC/U6otGDOh+oRU25P+lvVH+P0XkWqbKyKdGacrRZcd9TLp6vpdrel6XMwhKRrpqfVIxzDAGCaN8jiruuiiXvpsPlBmM4d9XwrnLj3mFCXoyilOiRC5FPtMHWsJ+1XFWKySwvHN3H70HyHSZFFNQTldxue/OUXzt2E03NB+4JjzNd3XV189p+/jHjouQPveVz/ZeqZ97TRV7j/V+f3857fj8mId49KcphG9+8c6/vVLmu60sHTQx9aX1QZLi9rfdFqd5vYTpc4+21UKb1SA8jrCPSvSNZgv43qEsVqrqp6iSGPUDqaSjZGGzjT459tScE8TNPMgndbOdtpSSIsYt/H7nDbDKTeDge5lKTwwDKDQ9zFFJMzrt6o1ncdsDonHGDMfjR05BaoQat+zBc4XUjFT1H+s5aXp1vCbRUTNcxrHBP1Kr6fP95o6jxNoupWqvp/BVMQixiAn4TeexhhjjDHGGGMSxQ+exhhjjDHGGGMS5YWqtg297Q0WFvTTna6Ugyb0g72dPMr620yk19PU5qj1UBPi0zUXceYi3X2kOCJILcgdLtA87knhnPS1jxMkvzWhfAyxK7tQie/f0YE0d6SRDLv6g7NwyK6/pMRUbCaol+rHfv/RLSWmcvunyU9vfhCXv0uqLdXD75JqO0rpvJcbUnM///Sf4vJJqbaVmpLBjk21xfbmse8zqbaXpaCsntUxUTe78W+kCb/zupLqmGq7gdRcamDv/KG025df+f24fFKqbWuUzL8PMTWyCD2MGgnTYJMmBTWWigk1mCNOStGkihjNpMfhuof6lcnquCOksUVIFe10sNB2IN+Lals+h0RWLJ6exWLU04TqslZXx0olkRrP/ILSA2dTbVkX2iZTbUtwqRp16D5ItS3loRthUfsc6qUCHad3lGo7Y03qNxcWpJo325pKkM7rmNoPpayPoQCNM9wXql/6pRSSR1stJS5SxayW1Ycx4XWcUKptDtNGZhR36KcnaaZJ0MFUhZljhq55lMpIDY3pjxFOMI9pxKRTnJgU9OYpEh/HJyTcsh8IMdWmMJNYKfh99jGnSSaH3+Y1P6PaIlEcfVM1Jb31j6Bx7vf0+S8fSkPfDqHHox5VE0EQ4XcjXGMTXAPpQ5WU04/S6eO1bvafCKYNSlAOy2kdXy2r46th6tAiqqOMtpFD8n8e+zBFGxxAPZ4mdD1WMFUmG6g85TSuee3HcE/9ak8h9sHeV7qO8h2k7ocai4xz2mY4hco+kVK7tyVds42pHS9fUl8ZHl5Xu4/0m+mOdqYI1/nSJU1dWlnTGGpvoPb7/Lk0y2ioOsjkdb7e+MG6Pp+or44CqMQYz6RQl6k0E3Q5peD0eLqhuojQ5pkum0njvo7r8dw5TTOJ0M52dzRWKyH5NhojGfYp6h19YgFTdyaY3rK7p21OFg72k6ngIZbbKEyQRg/tNl9G6iw07eEUicuYRkMlPg3FfQptPpM9Xp0Nh2izXY2RKsPfPB3EbzyNMcYYY4wxxiSKHzyNMcYYY4wxxiTKC1Vts+WX4/Lckl63zyPlaqGn18DNXb3ibW7r1W+/i2TJMRaBRipUBPVq0NcraqpoGaQFtgdIfuvo+7lD7aGGxYujtNSs0QipUxW9Yi5Ct5nLQ4sIpLy99oZUrldel/awfuVKXH7nXekKjzekA5w/V8X3X4/LrYHUjD9JJiQsuHnzp3F58cyyyotK0Vq/qHN9/+HduLyzI6V2Z1sqQqkq/XTl4uW4/OrvvRaXm9tSjMYD1UsOi8oXyqqXSgVJZeOD73Nx4NVzWmyXi/0+R5Lur36lhNulZR3r778lBWO/I7GJOuUYynBtTtrvK2dVN+tQqFJIHh1Ctxkcm+X6r4epu1T4+PkLzNCc1UShwVEuTMVqH5S86fF67QTRuDOK/Ql/WyggqQ/aI1VAqnqsG+pyXIB7Nu0zGepQbZley8+ZdstEzSGSZjNc+B56bbECLRBtegglLgzVXvNQfEfYfsDFpw/73iHOxXCI5FTUG9vmINR1X4Satd9Gsm9DfWyxpD6Wi4f3Mb2Di3jPKNw4wwPseyaTzG1zZj+gfU9RF1nU1+1vN/Ad9CPoL8ZQWpnCzDbKBchnDN8R9VpeA0hzPrzeuqjPdlt6Xjar9tLuSvfqoZ3yShpTo8VvMuGW+8s6oyJHpY7Xb4Tzm0on9O/uSHJO4XgC9GljKHQRhmHURlcRbv9nb2jKzUpOx3NnS2ORLdTv3hjXDKYmhdidMTS/I7WOdct+cia9FnWL8NygguuiAB23gETYekb1Pw8FtwJVuwg9n4HOI0x/6CERdzRK5v64sqD79LPHTZSRhl/GSgpDjWHST7R/xV1cR0hVDcYaE1SuIM34suo3g20Gz7QPm3e1D5M9XXvLlw6+n8Y5L4Ua5+zuS4nMTZReu7iiaUdnF25o2wOtkvDoiX6zVNX+zi/pmMYD6Mk5tJptHVO4z9UqcA/ZT2a0sXpOSjN/YTA4PtWWbb1chtLKeVT41hDtkuO/pVUp0AVMM8mVdT9l6vsKfqteOjiPvA92Mb0si/TamVTbSNdOu62+oYYk5nLp+FRbJvkPseLBBJr0bKqtxhflqu6zTrU1xhhjjDHGGPM7xw+exhhjjDHGGGMS5YWqtuOelMu9p3rm7XT1OrkJPfTkVFss6hrgNT/VIC4YT4UMz9oZJNKOkVRGO2Z0uMD8/kT6UB+ptrmxthEiaS1TkorQRBztXaTabm7o1fl52WFBvajtf/z/fhGXmWq7gTV+L13Ub936TKm2f/d/PgqSoFiT/nH51Tfjch/JVinodoWaDu7yWSlD3V/8LC5//+239LcZvapnAueVa0qPHQ+lCAxx3tNM8IO2tLl9kLaW6kpJe2deanAxp9/ZuPc4Lu88UwpvAP3g29tacLnU0zldOC9lZQJ14asvpewOoKVNqMhBA0tD65jkkrlM+32qJr/7VFsu6E49jhrMkXL3XVJtp1AO0wHTZaEyQX+jLspU21m9lkmQ+Hc7nKNC7nhlOIfPT5MJ2jnTdfk5tZuwp/ZXKauNVqDjjIdqGxOo7Pv7SvfujvWd+bq+w8TSIeo0PdQ5KM8d9Am7He1Xe1ftbjRWudyAplTS8W09V4Jisah7Qn9Pn5dgqvV76J+gi1IpZApvDyrWBFpvlJCATpUwxZ+AdjsK9R1qkeOhzmk0UZ1Ti+V1wnsc75udrpRlpsSOkKJJDfjoO9S31i8pFfyTT3X/arZUn6kTrscJytMZx/74NN800oaLZbXBAdo49fsZpglNJEBdzSRZM0UTqu0UynqEc/pUhxD87T9JefzlQ12D2yGmFJ2Uaot+cDbVVt85SrRMYXpCOn1834+g7plU2w5OWDnDJFuc64mOLwsFN4PfSmFZgRyScqtI0S4XMXZDYudpsrWr8WpYVt9YW1SnwlTbzp5U1GFFJ2+ANNJ8BymiE/V93Tvqs3a7qq/l9zHoW1YdnZ3XPlw6JtV2GGpM0i8oXX9+USn9a2sX43JviHHp7m3tC7T5+oJ+MwwxzeE5pmvgWGdSbScq1xZUl7ki7puNZBK7M2oqM9Mtqrjf8XNOx+piTMu0bt76pxFWB0AbzeWot6pvYppuhFTfCBdk//Dew8T13pB1KM01iwMcdrWNJ/c1ZWxuQf36uTW1lxzSiTNIjC9DB85gmiHHFAWOy9KcRsDe53j8xtMYY4wxxhhjTKL4wdMYY4wxxhhjTKKkpknpJsYYY4wxxhhjTOA3nsYYY4wxxhhjEsYPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEsUPnsYYY4wxxhhjEuWfAbXsR0AoczWLAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1152x144 with 8 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA54AAACOCAYAAABZsdfhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvXeUZNd93/m7Va9y6uocJnRPxCATiQSYwJzXMq1grSxZkkVbkm2F9bHS0e7qeCV79xzbkijtWitpJS2l1S6pQEWSIikSJEGARCCAwWAGE3tmejp3deX83rv7RxXe99vQDAahagYEft9z5uDi9av37rvhd2/V+9zvNdZaUalUKpVKpVKpVCqValgKXe8MqFQqlUqlUqlUKpXqtS394qlSqVQqlUqlUqlUqqFKv3iqVCqVSqVSqVQqlWqo0i+eKpVKpVKpVCqVSqUaqvSLp0qlUqlUKpVKpVKphir94qlSqVQqlUqlUqlUqqHqdf3F0xhz2BjzhDGmaoz5ieudH9WrQ8YYa4w5cL3zoXpl0nq8/jLG/IEx5pevdz5Urz4ZYx4wxvzIFf62xxhTM8aEr3au6vpK6/HayBhz3hjz7sscf6sx5uRLvJbG5WugK9XZ612v6y+eIvIzIvKAtTZjrf349c6M6sVLO/RrQ1qPKtX106v1i4C19qK1Nm2t9a53Xr4dpPX4+pW19mvW2sPXOx8q1YvV6/2L514ReeZyf3juFzrVt5+MMc71zoPqlUvrUXU5abtQqVSqq0tj5be/Xot1+Lr94mmM+ZKIvENEfrOPgvyxMea/GWM+Y4ypi8g7jDE5Y8wnjDGbxpgLxphfNMaE+p8PG2P+izFmyxizaIz5N3207zXXSF5tMsb8oYjsEZG/7tfdz/TL/l8YYy6KyJeMMfcbYy4973PB27V+/f2CMeZsH7V+3Biz+zL3eosxZskY845r8nCvI2k9vvZkjHmDMeZb/br4pIjE6W8fNsY8aYwpGWMeMsbcSn+bNcb8WT/WLvLSB2PMLxlj/tQY80fGmIqI/OA1fahvAxljfo76wHFjzD/uH/8lY8wf0Xnzz41TxphfEZG3CsbA3+yfc58x5lFjTLn/3/vo8w8YY365X381Y8xfG2PGjDH/jzGm0j9/ns6/4rX62m+MeaT/9780xow+P59XeN4fNsacMMYUjTF/Z4zZO6CivK7Senxt1OMQdXe/XRSNMb9vjIk/f4zsj48/a4w5KiL1fhu5YlxWDV23G2OO9vvGJ40xcRERY8zHjDFnjDHbxpi/MsbMPveBfp/518aY0yJy2vT0q8aYjf51jhpjbu6fGzPG/GdjzEVjzLox5reMMYnr9KwvTtba1+0/EXlARH6kn/4DESmLyJul94U8LiKfEJG/FJGMiMyLyCkR+Rf9839URI6LyC4RyYvIF0XEiohzvZ/r9fBPRM6LyLv76fl+2X9CRFIikhCR+0Xk0gt85t+LyNMiclhEjIjcJiJj/b9ZETkgIu8TkSURued6P+9r9Z/W42vnn4hEReSCiPy0iERE5DtFpCsivywid4jIhoi8UUTCIvLP+/UY68fbx0Xkf+pfY5+InBOR9/Wv+0v963xH/9zE9X7WV9s/EfkuEZntl8/3iEhdRGb6ZfdHdN5zfczp//8D0h8D+/8/KiJFEfl+EXFE5Hv7/z9G558Rkf0ikpPeGHhKRN7dP/8TIvL7L+FayyJyc7+//9lzeX2hfPbbwRkROdK/7i+KyEPXuw60HrUeh9w2zovIMRHZ3a+Tr0svtt4vNEb2z3uyf15CXiAuX+9neq3/69fFI/0+PSoiJ6T33eGdIrIlvXExJiK/ISJfpc9ZEflC/zMJ6c1hHheREenNc46IyEz/3F8Tkb/qn5sRkb8Wkf90vZ/9hf69bt94XkF/aa39urXWl17H/B4R+XlrbdVae15E/ov0gq+IyHeLyK9bay9Za4si8r9elxyrWL9kra1ba5sv4twfEZFftNaetD09Za0t0N+/S0R+W0Q+aK19ZCi5VV1JWo/fnnqT9CY2v2at7Vpr/1REHu3/7WMi8n9aa79prfWstf+3iLT7n7lbRCastf/BWtux1p4Tkd8RkX9K137YWvsX1lr/RbaL15WstX9irV3pl88nReS0iNzzMi71IRE5ba39Q2uta639f0XkWRH5CJ3z+9bas9basoh8VkTOWmu/aK11ReRPROQNL+Faf2itPWatrYvI/ygi322uvszlX0lvYnWif8//KL23Ct/2b8u0Hl8b9ThE/aa1dslauy0ivyK9HwEup4/3z2vKC8dl1fD18X6f3pbel8LbReT7ROT3rLXfsta2ReTnReRepgyk1ze2+3XYld6XyhtExPT7zKoxxkhvbP3p/rlV6fUjHjtfddIvnju1ROlxwS9Fz+mCiMz107PPO5/Tquujl1IHu0Xk7Av8/adE5FPW2qdfWZZUL0Naj9+emhWRZWt7P8P29Vz83Csi/870MNuSMaYkvbqb7f9t9nl/+wURmaLraHx9ARljfsAAYy5J7+3T+Mu41KzsHPNEdo57IiLrlG5e5v/TL+FaS8/7W0Sunu+9IvLr9Kzb0nsLMPfCH3v1S+vxtVGPQ9Tzy3n2RZz3QnFZNXytUbohvX61o09Za2siUpAr9Clr7ZdE5DdF5H8XkXVjzG8bY7IiMiEiSRF5nPrR5/rHX7XSL547xR1zS3q/MvCvb3ukh5SIiKxKD7N9Tv9gXZlqqLJXOVaXXocUkcAsijvjkvQwoyvpu0TkO4wxP/VKMqm6qrQeXztaFZG5/q+wz2lP/79LIvIr1toR+pfsvz1ZEpHF5/0tY639IF3ncu1EJSL9N0S/IyL/Rnr444j0kDwjz+s/IjL9vI8/v1xXZOeYJ7Jz3HspejHX2v28v3WlN/a+kJZE5F89r70krLUPvYw8vmqk9fjaqMch6/nlvHKF87g9vFBcVl0f7ehTxpiUiIzJzj61o09baz9urb1TRG4SkUPSW2a0Jb0fiW6iPpSz1qblVSz94nkF2Z7996dE5FeMMZn+oPA/iMhzC/w/JSI/aYyZM8aMiMjPXqesvl61Lr21YFfSKRGJG2M+ZIyJSG/9SIz+/rsi8r8YYw72F27faowZo7+viMi7ROQnjDE/PujMqwJpPb529LCIuNIra8cY81EBJvg7IvKjxpg39usp1a/TjPTWwFT6hhgJ0zOMutkYc/d1eo5vN6WkN0nZFBExxvyQ9N6UifTWer3N9PZTzEkP6WI9v/99RkQOGWP++34dfo+I3Cgif/My8vVirvXPjDE3GmOSIvIfRORP7dW33vgtEfl5Y8xNIiKmZwL4XS8jf682aT2+NupxmPrXxphdpmfe9Asi8skX8ZkXisuq66M/FpEfMsbcboyJSQ+P/WZ/Sd8/kDHm7v7YGZHej1AtEfFsb1ng74jIrxpjJvvnzhlj3ndNnuJlSr94vrD+rfQq+ZyIPCi9xvJ7/b/9joh8XkSOisgT0gvOrojoflXXRv9JRH6xjxZ85/P/2F+38uPS+2KyLL16ZHfU/yq9Hw8+LyIVEfm/pLeIm69xUXpfWn7WvAr3SHuNSOvxNSJrbUdEPio919mi9NbI/3n/b49Jby3Kb/b/dqZ/3nM/8n1EemtfFqX3K+7vSs/0RHUVWWuPS89/4GHpfQG5RXrGI2Kt/YL0JqdHpWdO8fwvHr8uIt9pei6ZH++vj/6wiPw76aFfPyMiH7bWXu3t1eXy9WKu9YfSM/Zbk56h30/IVWSt/bSI/G8i8v+ZnsvxMRH5wEvN36tNWo+vjXocsv5YemPduf6/X77aB14oLquuj6y1fy+9tdB/Jr030vvlhddlZqX3naMoPUS3ICL/uf+3n5XeePqNfj/6ovTMFl+1Mjuxb9XLlTHmAyLyW9ZaXRivUqlUKpVKpVKpVCR94/ky1UfCPthHF+ZE5H8WkU9f73ypVCqVSqVSqVQq1atN+sbzZaq/luEr0rM3borI34rIT1prK9c1YyqVSqVSqVQqlUr1KpN+8VSpVCqVSqVSqVQq1VClqK1KpVKpVCqVSqVSqYYq/eKpUqlUKpVKpVKpVKqhyrmWN/vVH39PwPXWquXguOdhBxJLe6Yag+/F6XQqSLc6bZxPqHCjgePdjhukY9F4kI44kSAdDiNvnW4zSDsR7LO7Xmj08hLC1oFRh/bhtT7yS3mJR1C0iRTuz/ktbFWD9PLGdpA+t46yWdxq4Tr5iSDdLG4G6YVxXH/fFHYgmJscDdK//tnjvHnwK9IPf+wfBw/RbDaC47VqB+laPUjHYtEg3W7jHK7TdAbnJBLYJ/vSpdUgHY/j/Gx2MkiHwyhrz+sG6T27Dwbpt73l3SIiMprHvrqdLsp2JI9rj49iG8hEFOd/9ZFHg/SjJ44H6cO33BakIz6eI5lE+52ewTUjDupLqD12uyibjSXsC33xzIUg/aM//E8HVo/yDzcd/7ZRcZ3L52SQrlNcKVfQBt/8zncF6ez4OF0JQcAXFO01+EVuYPX4gXsWgnpMp9COa3U8fzSKdhbN8jOjCcSorXNfdiIojXYX1xwbzeM61O/qNcSEWAL5abcRn2PRXjy1Pu7vt2gnKmqZExPo666L+7RcxOwoxdhOB3mvVhBjR7Oo90gEMWZlcx3nTCLGNlq4fruK5+50kYfPfvnowOrxE391PHhqrhePxgzfx3gTCqFewobbLtI83oRpwOO95Pk6vMV8iPIQCtE1Kc/PXYcLge8ZCtE9faRrdfRT36JdpBNog2HqhZ0WzonHsVtSOIF4aynzvqHyM2hXHo3z3TrsGL73I/cOrB4/dP+bgpvn47hs0tA46OF5hOY5TgT5jkbQZzsuyq5M7a/dRXsQF+l33X9vkD5yAEb73WoxSI/nUNZO/14PfP2R4Njpi5hjVJq49vYmdlHptjGGcr17Hs4fieE5PnLv7UF6z61vCNJHy+izIw3UkV3C+O/R8yVSKMtmHZ/9sT/6g4HV47/8jz8QPFAmhTaXTqAuUg7mWOMju4L05BRiY3oc5VIJPxWkm3GMWy2/FKT9Jp4t5mHemcrgOtanNhNGPWbTvcdPRvG5bn0a96zj3EoB7f/MKcwxwjHkXSJ47kvLa0E6k0YeS0XE2I6LuhBBGy+V0d9HcrxLFs+rcc1f+7efGlg9fnnxN4KCsyXEgtZGLUj7RbS5rVNLQTohNG5mMMZkJ1FGU/NzOGeE2sk4njOawfG2xaM12kgXVlFGxx49ISIij37taHBs9RLKf2t9I0g7FD88+t5TpzG82cY51kFeEiMYn32DzyYctPG5uRHcK4P8tuO4ZqyFvulW0U4//bd/f9l61DeeKpVKpVKpVCqVSqUaqvSLp0qlUqlUKpVKpVKphqpritq2a8A8QoQMCWE8jDHFYni1zQhllJChLuFbIRevitOE4zQIOXMJFErE8Ro9EcN38EwGGFa53MPGXB/3z6QyuF4HqEmlhPu08NZc2g2gZ+EQnjtCZTCVQV4cB6/x623gLl16FZ7JoOoOzeH8McJAIj7yNkhVKoTSEUYTdgjPc1CeLtULHydCVkIhXIev73ZRX/kpoHIJwm7LFbSrTBZ1t7W5HKQ7nR5WcuNNdwTHWi3cp0HIMGNo0SjK851vA7701rcj3SAMLMJIVBl1t7x2KUhvEuISy6DuWoRNnTp5BnmIEJo7JL2a3a13oHt9hG5taTE4dvThrwbpLtVjJI2ybVbQIbOjwKMYr7WErAy7NLiNvWJRPExmgffUWuj/bcLUCivoL7kRnD9CcS9EQ4NL5W9DtLShSfgj4bgxwnp917tsuiu9WGEpBjr0HLwkolBCP+Jyc8K453YBSxXKNaBfGRoHWjQONAVlw9cplgpB2qUlIN062tWw+oq5Ai7ri3+503eeT233StirXOE4p80OXHXHzfjT/yDFuC6nm9QGz5wFzndu8XSQjkZxvUwUY6sQTtjsoq0lM8AFQ7SkJUpthkhbafv4bI1QU6+F9vC9H0E8f6W6/fZDQTrSARJXXUO7jNI4ITS3SKfxbIkEymJiEjjfsXMYGzbKGEtCYUKyOzju05KSrQLNwQzKLpLsYY7HTl0MjhW2kV/PIr8xamv5HGLszraG9pJNUDygZVKyjj771IlzQfognT+3DdS27eKaxRKhpt7l+8cr1a4ZIP4RKqt0Cvkbz80G6XwSdTRCcXh1BWV6bhNzoezcVJAem0O9R7KINc06EFxDqLbjAH/0Pcy7tvvYq80C+40lUI/ZBJao2CjqYk8Iz9eqoWwbbTzrgQO4ZpHqJZ1B26jUkBcriJ8UEiQcwXEjvPQKsXqQMoSNhsbwnKOj6F9OE20rNzuDD7epT4VRd4a+L9SzuGY9imereYgvpoo6bXVwDq08k3Ac87zJw722dCMt8csTalvYQFyJE1ZdpD51+kkgw34DfcTSHHxkkmIMtfdLl4DTn1vG3CkcRt5d+i7jUKz2PIptV5C+8VSpVCqVSqVSqVQq1VClXzxVKpVKpVKpVCqVSjVUXVPUtkQuk4wSRQmXScXJ5SqSDdINl9xdycHQI9TVJ4fZRpteZ/t4RV5rAkFJtoAuZJLAzJotoC/dTi/tE87hWGCe4lARktttgl0k6bi5gstgmtz53BDuP5FGeTSpuhIxHM+R49pEFvf1fHKJHKCMIK8hQjTyedQXuy92CFeIxsxlz3ddRsUY68K9KhVyJCREgXFfn/BizwVKcvZcz4X2sW9++TJPNDgd2ndTkL64BCRqvQQEYnrX/iB98x1vDtIOtaXxMbhr+kNCiVgDRT8HLEuoYbfdQ1ZWloDtZZNo/8kRoCMb5LZXWAV2PbV7Dy5Orps7QLHQq7c8nq94Gv2oQkgoH49QjO1skMtnhxxryRXciYQpjf7VImy+S9hcPHwl/A7XaRF22egvP4hQvG+RQyYj7oYQdHasrZURD4RiQJSWaITJtThOKKZLbrpdiskVwnHDVGa8RGBYfYWxY3ZotS/C1Zbd4OUlosA73Egp1FhCqYiw3umO2+8ndVpOsrVFmNYikPiLK+v0OVw7Rc6RTXJHbNEY3rTk1rgJzC9MDpFxwiGJOhWP7tXtoH7jVyfCXpaOTACbXCHnejuFmJ4jd91aEfjj5BiWAUzPzAfpkVF8tksccfmpJ3BjilmxHTM7nN8kXHWthP62erZXNyVyr+1S2baa1NeoDTICzf2b29QKZcaEycHYB44aqtNOAuSKvU6u1B2Lfp1M4LONJq1rGqAurQJnZFfbVgcNp0kOns0RPINr8Az5WTz/wd1IN+PoD1V2ta2wqy3mmjZGMcvD9dnV9jnn7iTh6906xsRKHRhplZb8XNzhaktBIIK56JVcbWtVwi9dHOd2xyvrvC53vMtfZ5AqbWPM6Fg8j0/LuHwUp4Q7NJ9s0I4bhKxH44T4d2j5HLkth+q4l0tLCHlabny6DoZCWV3pLS85c+48DtLahxtvvTVI59Ko32NPYLeFlWXk19Bz1JroU/y9ylLVFWksrhQRMxMc+ynIWg95a3pXX+KnbzxVKpVKpVKpVCqVSjVU6RdPlUqlUqlUKpVKpVINVdcUta3RRr+ZNBCCRGYMGbJAKNbXgGVc3IIzWCIHLDZCCE6lQYgsvc/OxnDcYxyX3QJpY/KwpdfP/TfIlpzDOq0a/Z2c9AjfikRos+MovX6n7/oRcttqM7ZGDNDMBF6jV7u4V4ZQOD6fne0SieG4hLVbKPNkEvhL163Scdw7RGhQPGHofDxz2OA524TsNBso924bOEo6iXNGx9AexKLeY8QurKz0kK8EbRw/DD3yGDbg9sgFLUJOjPE82nsoSQ6rTZwfJpff5cUTA8/nq12Xc7IVEdnc7rmOnj+PeNDehhNpJo6+1qgBJXqWkLTpeaDOI9NwIpQdTp84/GrGkEVEPIpjm1twgJ2YAJ7HMSgRR1/zKO61WuhrSWp/HsUsRmdzhNRGyMGQuqDEKQZ5HsWBftcMEaKbJky63gDe44SRl+woxofyKvD1GPWvOsXSNG1KPj1Grn2rwMY8ctsdIZffDmHF8Sg5rNIG94OUu4OWRZ36O1Dby6ddg/N5qcKOtsvLPHYcJlfbHW665DZPZVHaRBvb3Or1vaVLcO0ukYO0S2UbYuyZ2kWN8GmP0K8aod9talTsrJxPYh4R5etTukPOmRHaPD1Ec4dByjDiX8QUa9tHrB9zyFl6HO1y174DQfr2u98epM+cORukK1W0v+kZxK+pGfT3+XksJ3CiGPOi5KJ+4ixi6KkzvWUhjJrXCHP1ulRWEUJKyS3bIxfoHTh2C/VVqhKaW0ddp/PIl1tBPC+QI7GlOGCoAW+HhoNoHrhxH/0f4uTyIhx4F5eBfc/NgtesW8SyPNW1m302SIfSeM42ze2qJZT1qEN1R/hsNod4l0nAbbbdRzo7LsY+oToqr6ONFM+hbZ567MkgndqN+88dQNuMk5tvpYrr83xQaD68VUCc6BCm2m2j7sbH4ArcprFokPrK544G6So5WbfIUrZZQf7CxJx6LbStzjaeeWHv7iA9MUsYPMWULI1nMVoaFwvh+u0aOW6TS/xasTfXXbyAJUI87nRpSUKeHJRP0G4IxTY5sVPf8aOo92ob5bFRxBIJcWg5Ds2/Ol1mpqnf+ejvHq9zuIL0jadKpVKpVCqVSqVSqYYq/eKpUqlUKpVKpVKpVKqh6pqitoa+5ybI6SxHGwbX+G1vBxjn7YdvC9IPHX8sSJcJB2EXtpjBa+DJMbzm5rf5y+TE1wkhnY7hs3MTPZewXBYOkYY2fM7l8Jo7Qhsrd8hpjd0ikwlGPckpkX4CYMewuRCQikoHZZaN4pV+OonzjWVcYTgbncfI0csntCDioJwrZWCx7GrrUx2NjWETZJdQLr5+gp4tl8X5CdoYPkJOuQ5tTtwhB7/Z2QURESluwkluGLrnrnuC9JVcbVtFIDZ+A8dDFs/tURub2cPIz+tFjL2iTpf7SN/iRaB9S2eAPo0TcrlrHBje6kW49j392KNB+q770aaShKzIq5uu3aEsOS763cxljxP5JmN5lFGBsMld08CqOuQWvllCbHQ7aJdtQhjj5Kznt8mZtAXMrNtGH38Ol2QnWyFX0ngY6VIJiNOeaWy6PkeuvdUGlj+UGhRviAKLJVGpczPAHs8tI4+1BuWRMOD1MpCkOuGIg1TXMtJ8eezb45BOeBMTtYzX8tIOw5a1NN5EyLW4za7BNG6dJbxwdRUurPVaD/9qE1LrkQOwR5nPEvbc6uB4lTZUZ9fidhv3j9KyCV6m0yV32A7VdSKCuYNvCFGjJTPD6uJNB86hU7uBkB59FOVWcvD8uSjmMHfcD4Rvz613BumHnzkdpI8TInvvfW8K0h/88PuC9Mo5oLmjU9NB+pFTXwnSW2305QNHDvc+dwmbzl+8gPbSpnObVOYuYf5xWnYTIQfpSI2WvxC2l0qQa7KgDA7nEbe227hOSVDBxmDcnEkOB5kuNDB+1ym+pDKYw1Vpnnfs5NNBOj17MEiPZdDmXHJYvnQO8wCxuGY+CvzUCuP+QDpHc4iDtTL61bMneufnU6jzTBb9sTuGmFFfxjlr6xgHF3bhnGQan3V95JGXmzm0DKG4TY6phFIbGn8aNLZUqe3nR2m3iAHqC3+B7wsdQsYNOdqHKc3xMEKIarOIdl/YAgIbOUbLNqi+0hk8T24My6vSDvrJGqG0xSrKIj3VOz9KY3WzhnJb3sSYVaclWl16jlie2mmZljC08Rwhao9hWo4zPYN0lzBkj9xxfcOu57RDR5KdjS8vfeOpUqlUKpVKpVKpVKqhSr94qlQqlUqlUqlUKpVqqLqmqG00zk6seIUttOGoSxhHowa8w5SA2jjkamcI3UjSa/F7D8Dp6zvvBKp4aRWIwG/8HV7BLzfxCjnp4NV1q95DKQ7sHQ+OzdIG0V1y63IIzUoSgsvoU52cqwwVfzJDDrTktsUbETsZ4GxOByhrjDY6T8bwer3dHI5LmCXEjp0Vi7TRbJMwrR2oGBlhFUNA6FIp2ihZON+4VzaLMmJnznIF2EGUnj9EyMT+fTeKiMhjQ0Zt3/eRjyJfZWCMy+tAQzdp42YhNMN18azsCBclbGx4IgzvihDai4DT2Jlzx//Q9ak/mCv+9kXumsRLdvusfLWBfndpHdjVOqU9j9wiJ3GfZx+F8/DkNLC4Q3cDk+bQGCI83DDqSFm3jDrys15JZnC/+dVrQHQy6fRlj8djaEOjhCM3qojJjTrOZ/Q9nUDfbDSwFqJZR6yeGQMG1u7geCaOuB0lFN9J9tI2TK66TfSLWAx93SXXxPNnFoP0bQcR17eqiIdhQlZ9wne3tuBA6VD9uoQRlmuIYV3qm9slRsiGg9r6hNq7bLgujK6ScyjhTQ61OSJdJRTmIR6Nt064VXENqNilZcSpagsY4dY24lEoxIhv72YOObSzyzL3kTrF9U4Xz9HsYqxwCa91iM+LkYu7S3h2izZDj8XRrn2aI+ygk7mfynBUSMwH6QlC7PbvRR88vXQqSLdLeOa6iwKrE0p9ZhVYZq2Lc0p1fHaFYt8anb9ZQl3zhvT3vP1+5LM/bP7tCv4eJbfYGOGlaQ/j6u49cNXddwCOvOsbG0H6iW9iaUM9ivlMMo7xboH61xtzmLuVp9HHFw3647kqnnVfBnF+kOqEgEZ7DtruSB6Y/sHDC0F6fQP9pU7zwqPPoC5cKtORceC4YvH8kRg+mx/FvdJJzEGrFbTerXW0Ab+/HCuexfyo0gHu/XQL5dkeJXf9SSxFScaR3yItEVpdofGBlk10qT/W6ojh7g5XcMT+FvXxWh1lFksMZ2nYe77jriA9OFdbtPtX5Gp7K7Dqy7naniUUN0ExbW4Cdcqutgu8XGhQOnwkSP7eJz6H45Qfj3YRaXpX/96hbzxVKpVKpVKpVCqVSjVU6RdPlUqlUqlUKpVKpVINVdcUtc3lgGv4IXItJBfESxWgkBc28WrbL5I7H+U6QxvsTo0AuTiYpY1vaYP5SYfQzTg5Ovm4qKXv49uV3ivkc5dwjalp4BXRKJCHUhH5tYS/MnjnEj7UIlyBKQPHIVfXKPCcBKERzW2gASEqkFqD3Oc6w3F7y2ZR5s0m0BmPbscbS8dihBa0O5c93ycXV774KBqFAAAgAElEQVR+hTYjL5aB+XmE00XjOL9BePGe3fM4JwoHzGHqy08cD9LJJPI4PQN8aGQXobPk/tftomxSCZxz8QwwmOHp6piLvRKcxh/d4cbJuBu5yfHvXTuwW06z8H975udFRCSZQX1WCPlkhPXYEnCvBGOeLZTzMw/B5XFsDk6B+V1AkoxLaD8xtVwejPaFXgQxZAbI+dWbeJ6RUWA/BYpHTYo142PAt+b2wEWzUsb5DjuddtGnorT5dDqJOthYB67pCGLcSA54GOOS0k9HyQGv0SS3TA/nZnN4puVL5KhK9X7k8I1B+mnaRLtDmG6DYlImQTE2gTyECU/uEPqeT1P7scNZwsCYryWcld1uPVqrECar4gg5DBrCntg1uEbPtrIC99LNTYqrhLWHyN1d6Po+oVTP1TWPQTzieYJrNAi94w7gU2yIOjieJJdFQ9dkx2UaKsUP4TpNg7rreDwmAmcblh59+KtBempqf5BmDG8yhzz9xI/9HM5fAMK3QcstlsnFu+XiGU4++0yQjtGSk4Uk2muXnv/9E3AvTZ+BU+7kSK8g39khnDKPMSgcBcaczAHzG5/DuJacQF8/3kFbriQREMdoMpQl5PAt07jmuaeOBenECI7nxpGuUT5z4eG4oTaraLv5acRMG6N6PEC7DvhYxlWjmJwQfLZQQNvNRIFFzu5CvXQF41bZx/n1bfTTeBifrdHwl8n26t2Noi1s1DEX/synaZmURSzdH8U5jPBvrWBM6LRQj2Hqpy0aHyz163SGlpvRuOlk0K5cijd8nUHq7e+/NUh3KHb7XXI/xzRWwp0IHUcbsIRPR2nnhWgKfS2SQpsORWl5AD2bvyOsok1HY2jHi+d7dePvCKsow339eZCISI4c5c8/dUaGqWiEvoeFaS7kIe15V58A6RtPlUqlUqlUKpVKpVINVfrFU6VSqVQqlUqlUqlUQ9U1RW0jIULp+LU6vZntEI5jw/hDmkxfk1H8T4Q2Yx0nNGSzjnfnX1ghDCyK6+ezeLW9P4WiaNJnW31ctd4GOnLiHHDgGw8eQh7zeM3OOJClTWtDZDnoECqVTgHZGMni9TsjqEKufXkqkFCINlkm3KZbB/oxSHm0CXeB8I+QAHllXHhkBMhFoUD5o3IpkGPYzDQ2Np6YAKbSIAc/RsJiSZRLPIxycSJAIL71xNErPs8g9bWngNpWyNV2hlCisRHaTHgEiE02ifI7tAsI07454JDD09V/gzJXICjsFXa49wmz6xK2GGWEbsdFGWPlw+gn+XwP53rL2+4Pjj395LNB+vwisGSPNrU/E0YMiM/DSc47Cdzs6a98PUi/8SPAOxMJtC+PXTGv4JDpXgFbZpR4kIE3Sq6z7PbLxxnR3NhCu1wg1DaVpvOJB6pWcc14grCiMPpgxYWrrKW+2WGH6hDaWKUfp9KCmN0kTL5NBZolR00eNjY3EHtSCZxjCPUNG8ZUCWUnPGltG2hblDjpPfvng7RP6wKaDeKyBqgHH3kgSMfjKJcY9Rfe3LxLhdHu4NlaLdRXg/LapjHJ3YELU7uk8cnS9a+Ehrv9DhGmMvfoGi657bKRc4RQ4jg54jq8qbu5vNO2Q7i3S5hu22J86JBzpuvjOs0W4dzucJaiNBpwAu2E4Jo9nUadTuaA1B64GThumDDE1RNwvp2wyHeKiPUJcli+bR3LgXZ7NPZnqUypbkJbdTre69fvMLHLnutT4OtWcW3vBGJv+OTZIJ2j+HEoh3jbpTnX5BRi7IU1OE7XLfJ1MLsnSFeojU2EUU5zdHyQKm7gGSotuIvaEOrltpsx/7v3fZi3pKKIL90G0qdOoR4rRcThBMVVL4r7XqpcDNJjGfTHWZprZkYx54n2x/E6LQ85ewlj4rkHsXSpU0V9md043tigudhexNXECO4pIVrqFcbxZBLxqUO4cSSE63SIa00maR4bHuD6E9LIKM03LLnLknut08S9mxsc32nJHM3XTQx9I0ZLwySO4xFCcBlLbXVQv7TyTMI0Fs70nXJbdXxu9RLmMMePYj7LjsH5COaTw9BIHvXrhshJ3cdzZ7yr78KgbzxVKpVKpVKpVCqVSjVU6RdPlUqlUqlUKpVKpVINVdcUtbXkytR1yZmOkK2YA0RmbgqvdW84BDRlYc/dQfrsaaAInTZQE88DOlAhTDebBd5xIAt0YR9tsH7yLLCKVn9T9RihT8Ut4KJ1ZEvGp3Ft08T9I/T93ifsxyWnvlgcz8rkiAnjuCWk1hByxeLrJGU4bm+MQ8XIlbVJm+1ms3C8dMhRNMNupBUgHQnKN1+/3ebNkQm9ipDLIb3mn90FZz3TxvXnD9wgIiLPPokN6IehFGFgG7Qp8XoHbbNEGzF3Ke9dwlSPzcJhdXQMqOcN+4HgDlT2CtzojnPYpZacXukURt9Ok2tiswl86oYj2JA4FkNdh67A8/nksuf3Q9Z9b35rcOziIvrr7/7W7yIvhPpc3AQKGiPHx4OjKP+TX3ssSE+Qq+0N990TpBv0fBGPnJVpGcF2A32/TS6ajP4uTMEZ+5UqSrHLiQAfil4BjfY81MXKGsqON7lmVL7VIhfTELt1ow+OjQKnt4RXNtpAyxoNQvRsryxcH3+3PgU+xnLL6C8tciQubKF/RwnXjMXIbZdidbuD+5eKOB6JYYBY2AM0cnwS8bxMG6mnnOH8Xru2SbGJsGBpI9+G+sgOxD2GGCGEvpkQlylh8IRJca/rkuMi93efUG3uppF+nRmOY4TPd/kahF9GaUxMhgmdpWt7lPcQtQdrCOEnvLbtop161Aa7HUKG+WnNcBDNd7z3PUE6lUdbnCSn+1Fyc2630BZlA209fv58kH43jQfhItpD1BI+HkI65pFjP7mzOjFa7pQhV+Rqr4wszU88qrsuxT2P8N4YnU8gpiQoHmYoPjW4bawCHfQ2gJ3yMpNwHbE0VUUbuCGDuU2qAOR+kIqnkddaHc/fpTj+7HH02dVlzEUzhExPTWE5w+Q8uXhfQDkubQJ7TWQork6g/eSzhLeG4HLs0M4H0VAvDrsdzIP8LrV5H23tyC0o2xsWkM4k0afyExTLGyjzTgfPUS1g6ZlHc7RElBBUmttwr6ttY1wOR4cTVy3Vl19CensDczK/iHFo6xQcvxOCeWmU6jQ7iaVTo/P4EhANYSxJE1YezeB4m+ZaDVpTUlhFHWyc7I3Lx78GpJZR2611LA9xaA3D/R/8oAxT+w9imVibsOJYi77XVNXVVqVSqVQqlUqlUqlU11n6xVOlUqlUKpVKpVKpVEPVtUVtCRWNEroaNrSpM7kjThGmctttwPPGJ2gD4wngYeeeBVIzkgIytUSOaZkxuD45JeAd+RxQpdmZ24L08qlHREQkGSeXsBXaVNYFGlNv4Xv8Mr0WzybhapYm51Lfx3PXPELOdjh9ooqsRy5ZXdo1mDCYep02/O0O53eFWgXPnCV0NhYHRNEhvImdFdlJkN1uYxGUKV9fCHVok1uj28U5iQjKt1kBEpcJ4/V/LouyG6a+/JXPBOlKgfLCrsVUZgnCAj3C1Zo1YCDhGGHV//yjA8sryyesipsf43yWnD13ULGEVS0RbvTXn/mbIF2pACO5bwuYyDve/s4gzagn54e9Ld0+QpbOoM4//I8+HKTPnITj4Bc/+wXcnxyUn11G38wbIDBx6r/f+Nzng7RDqHNoCvGjXiKcnjDC1QowqHIV57Db6MIH/6UMSmwG2G3jHmHCE9kNNUzOfoZih6GS7tB1fEKVnATiUZIQupDFdfYdWKDzUb71GmJCtdwrl2odfaRMuFO9jvuXiohp4lGMp3oxFnlcW0W8b9AG4CWqi1AIx9/1DsR7Q4hmgvpdahLoWrFEaOQA5RGi7ZeAexWWz+EkSzGdXF/H5oCGh0aA9hnalFzo2YRQ23CE+zjqt0Njj2FUjnHffrAIEXrPG9DHCf1uE26fzyP2ZyKo0wuLJ4N0Io3YmM4D8TK0ZMdrot3RYWnSzGZtGyhgp4OTEingq4NUg/DuGNVXm5b8FNbQ7usNxPrKM4hfrRKQ2hyh7+lxzH+cEdrIPk6wq4O+Eb2A/MSpjKojaAPtvsMpo9y8zIcdlHmzeGo60hT02TZtNO/TgOIWaJkJoY6JBGJJzeC5/TLSIVrasE04X9sdjhtqKkUxU9BeV1ZQFhWLfFeKQJ2dOOaWhTrSuQzqLk47E2THgBcnYqjTqfwMHWdIFXnoEh/f7fbK1FL5V4pYMkAroOT+96BPxQRj8sw0tR2656mn0V62i7TzQ4XmrjRW5MbJDZ6OR2mpQrmG6xhnOOh7h8ad419+Okg3LqJfuAXEXkP9bjaL8T5OSzKi02jr4uF4Zg85G9OYGCEnW4/q6+Kz54P0A59/KEgXL/XyFqqhnU93MW5Pk1O0uLw+Yri65034HpaanwzS9fNoPw9/9htXvY6+8VSpVCqVSqVSqVQq1VClXzxVKpVKpVKpVCqVSjVUXVvUlm7HeIShzW5HskDoIhFgqZdWgabYCDm5xfGae88CcKjJCThN7TkC9MsP4bV7qTRL5wO72SoAxcvFRkVEZP/caHCs9XfYNPn0yvkgXffIBbEK/KCwjdfyBxf2BulZQoa9LpCDDpWHMYTC0cbCjD0yDpmg8kjTxuuDVLUKZCpKeGQuP0ln4ZxmE2WRIPQulUL+ykW8qufNv6enUUcZQq9Cgs/GCF/ZO4br33oDNp/Oz/Tq5o0/84vBsTOngbA9/fQJ5IVQusL6SpA+T+6pC7fdG6SXyM03kQEWWC4CJTp3Hq51bptwQUKMRjJAUyxhkuksl+uwRLgGuaQV6RnKRfRB3hB5bZMwi8ceCdKPP/NUkK6Qe127i/570y03B+nJcfTfMKFllSr6RqnUu878HNCk2TmUzw9+7J8F6aVllPk3n4I7XLsOpOc0IfHJaRwvHDsWpBt/HiRl/5vvCNLFGrCxRgP13jZ41k6XXJn9q7u9vRxZcgj3CCk25KKZIBdNdiDN54BbdaleQmF2FEVdt8lVll1P8yPom7fcdEOQzo4hJjoGbX1tpdevHnroy8GxA7cBF/UI7ayUEUva1HcWZtBftjaBs60XgCuG6LdVQ0s9PKqLGDt9dsl1k9DQCMWqWPTyjuKvWE20pwqNK+0iUNFaDe0pnUbsrVCfHYnRJuIGeQ35aBshwiJdwt0itGzBhNBmXPKi7Bgct6levY/R+Jmk2BUndJOdaSPkLG3rQKMPTOI6HrGzYXLqjRICHaf63dhAHHIdYIETIxQ/adP4RBjPOkgtP4OxxJ9EnBonhDFTwjOXnsB8okXOzyMHDwTp1BzmM+FJYnbJqdk+g/EsNAbE2h1D36wSRtjeBF4Y7vbq16XxvE0uyIkZXCNOGG/j1BnkZQtxz6E+Es4iv1UaNwoe2rKTRRwqF+FSu7mG+MyO2oUUYskSYfmDVJeWCIUE6UQM+Q5RTMvk0e882qmh2UFsaqwjfi7M3RSkcwnESeEYVEZZ5ykGCc2BGy3EKXF69/XJKfrcGdRFfgr1e8edKPOEHMQ9PbSRVh315XYJWadYFQvjmokU0mHenYHQfhpOJJ/D3K3ZoiV3A1R7C+VTPoqlQO01Gr/LGL9HIuSiT7h5g8aDaVp6EWmgvhoXUNetVXKeJQx+g5aOnHwM85JkCc8/k++1hzpZ07damEeHeVkM5YuXJQ1DY1NAzg/cjph0pom8lwnDvpL0jadKpVKpVCqVSqVSqYYq/eKpUqlUKpVKpVKpVKqh6pqitl3Cp+JRvHqOOEjv2XsoSGfG4BCVI9QjFQNulaT39vkRclILA0UYJSwjLMBrsrsOB+nxabw23nros8hb39VqZBa47twcXnlfIMfc1UWkPXKRbNLmqhcuwq0wFQZWlCIULkROgOk0bQ6fBnbhOkA5ajWgJpYcs9hRc5BqNVG2xSK5y6aBmvjkoFhv0IbTcaAYHUL71teA1yTonGoFWEKE0MHpKdTjm+65NUhnyRX5mW88HKTfPHtjL19ElNZrqMepSbS1o0efCdInjgOFaBLyN3/LffjsLDCVW2+6HXkvALW4cBYbTW+uAVnZOwEkamYCSNKxE0A9T51GuxqsCAOlDeL556hyBfXytYceDNIXVuDculVBmRfraA+hFNpxvI3n3CjwNb8WpOfn4cbJDrfLl4CvdDu9NtNs4J61KtLUROTI3UA3nzwDN7sOOSJeKqF9JaO4564c+t3iY98K0uEYCic0C/y+7AIv2eHNR2hfuz0clChL2GI6hTzVaKP3UdrM2mvh+cdoaQM7DzoUv1xy5+vEkc6m0N+naaPzEXIIH8nhOG+Q3a70cK4JcnjeRRgPY8nNHBCzDqG24zmUdC6Dcj51Fm0zQRt3l5uIk/EUnrteQxsIu7i+IUftzTLyzi7dg1QqTM7e5FophKaZUOiyxx06n69TKCEGNcnFPRvHOZ0u4Z0ZYKlZiudbBnXaSQOPy872cNDlCtpa6TRiZriOZQsLh24M0jWqx9oacM391HbKhFA2yJnSEAJ6+zTG0O0y4sAixV47jeUtMUIgE53huBMnwoSo0pKTBcLazQrakOfjOaduxZwkf8/dQXqbNoxvLKJ9x0J4HmkTcrlBzvCHgb4nb8Xyk+LfPxCk451euzIxlH/qAM4NjaMPdsm1uLyC5SfhFcS3DLnax+JoL7FZjLOxLIK1Sw6v9QLGxy1yvo1QDCtSkF0jPHmQWj6D/jI5izpNJRBvy020uYyD5x+dQgY3Nwk/9cjptY1zWoS7xwyeMxRGH9zewjlOivp4Ffdt1vpl4eBzS8so55ldiGPxNOKeQ0somk3UtW3jOrvmcE6OsN+1C2h3qTR9NoTzifgXy6HNR9yu14YTV9MG5R+vo78U17F0iB2xk2mUfzSKeBujnREcWp5R3MZ8Rgzqxfcxv3TCPA/AObviaA+T4xiTav3lWzWPlkR0UD4NRp2d4bg6X05bW+iPd+YOX/Z4JJSSq0nfeKpUKpVKpVKpVCqVaqjSL54qlUqlUqlUKpVKpRqqrilqGya3qE4br6End8PR654P/ndBOjGCzXO7PrCYkTDQgsY2XpeHyEktOwmsxSPHv2iMNtMlZ9TCMtxL0w5wtaNnerhAKAUMbO7WtyPvq18M0vWLtCEyuTyWdiBDQB2c8CylBTJAFLoNfLbJKFwHr9oj9HzM+Tnh4WzIm0wCD8iNjNFfkL9KBfXVbjfoOLCAiQnUBbvXdjvkkFVBee0/MB+kb7kVGOXUGDCGY38HLPLkUSBfd7yzlwdLv7UcOoTNcM+dPR+kFy8CH7q4gefIO2hftgNko06oSzeDthMmTMaJ4Pj4BNrd9BTc0XbPADea3w+M4eQpoFWD1DMn4Drr0LM9h7OKiBRLQIlK1HYvrqKMctTXRnN4hrFx1O/mWeDCJ44Be/3CF7+A65D7YZg6RLsDzKnT7vWHz/0d+gXtlS2zu4AKJsfxTLfdDtzsiQexSX2DfOBOEeKV8ICL5F0gMGe+8XiQLk0Ax90m175IB8ddcpndgWj+mAxMWcL0KcRKiI7HI+QQatD+otQfIuSwbIVcZclNT2gz+GwG5ZJM4rMRQn+21lGmxU2gXc0+Qr97Cujd3CRiJrvaNigGdtooT6K6xLOErM+i3S2vkSMutev9h7B0otVCvaRD7MKK4bFUQbxtDQmZ7pRRVi65JLfIMTAUCl/2uBvD+XydlSXggs0i0vUEyjdMDrB+ilxSp4Comjlg8LOHsbSh0i/S5U2MfdLENXZn0S4KZcSAto+yrVP7KhHiHp4BounT2O4R+l2MIu9774BL6NK30GacacTYkMUY1a2SQ/MAFU2iDd1xJ8pwPI6+VtpFaPpuxPoO0bKXHn0ySPuE2nY3UL/tKcIC47ivU8azdVaoTkfng3TijXBmP3r2uIiIJB1yr6X5hn0czrsJYrwbyxgHEuSw61F9faOOdreyhHq5eR6Ov3lC7j2K94kIns9PIN7s3o2lUSFymx+kom2K41U8jx9D2+20EBe2yD3VUrxN0TNMTNJOCmNolzucl7vo4xFCNLthlG+FyvTSOpbxrF3qtY1tNBFx2+ivmRF8bm3reJDO0ZiQjAKJn5zF0rfZOXK8dlE21SMYtzsutQFDaCjN95MxGuctjUXkljxIbVIMXLoIl2SXkNcItdd4HHniN3N5WupWIhf1Rh3tL0M7ScRTaCeRGKHqRMcLuX7z0ph6q3cSLzliB+Wqi3M7FD+H/YXOUExI0NIkPl4tluRq0jeeKpVKpVKpVCqVSqUaqvSLp0qlUqlUKpVKpVKphqpritp65MBmo0AuQqAsZPYgsJNEGjhlYe18kN7aPB2kc3mgLNEEXmc7tDFtdpQwBgNcoEXumsvn4K6Z7ODVdb7/+v+pR+C899Ef+uEgfXcHr5hLn4Yb7uo2UJeNNtARIsjEM7TxMbnR+l1UCyNv4Rg5UBLOZAl9CQnK2DPDQW3ZAHWO0BF+HnagrZAzbTaLyo7Sq/psErjmxQtAR+bmcP33vvv+IL17F208XERZxMlRcH4WeFg63r/vjh12ca5vkQ5FCe+JIb9pcjuzHi7kEUbSdZH2LFA4xtk8cj7bIvywUgC+sfsA0NzZ3fMyDD30yENBukmulKk4nvPDH/5HQdq1qK/HnwZ6lctQXZBD4yxtBt9dR7mU64TgnAb2mifH2FQOeUjngU7GU71yzI2Qoym1qWwWOEyCWMz73/lG3H8LyPCxY9h03euiH10scVsGhuOsoX9Vi0i75J4aSgChWl4CXlihMh6kpkaBdFdq6AujdNwnFCZEGHM6gXw7YZR/u43nT8Xw/FuEnF66hOUJvoc6MicvBOkzZ7Bh9+wU0LpsPw/j44RdhdmdD/1lZARxL0JIeNiiTdUIY17YA3z39JkngjQjQB7F+JAhR3GHyoA2NLcUuNut4SCam0toiy5vXk/1ssN8mo7z+Xyddok2HfdhJxkyhIFRubeauE7bAu3adeObgvRaCHFgu9ir35CHtp0fRX8cHUHZXtjGkoH8BBxTkxHE/rk9iBm+gzx2i7REhRzx2cV93x7MBer70Q822qgvh7j8dpysNgeoN9x8IEhPEN7dKqGMcgv7g/TyEvpL/SSVES3hMISDd6i+0h718SqNQx7u61IMWp1EP43OoNzX+mW9cgrznCjNK/ZR+Y9uIh0TcpQNoTyLHtrOJ7ewHOpkAfHgZ8nJdnIWMTNKLqTRPNpSg+p9bBbtpLpMrqIDVDZKqC3N80wMsTFJDvyFDfQvMiOVI/swD5kbA+Lv0G4OrTrKLiIoF0N1XSME+eQiynG1hHSo3x/8Eq43SnHyUJ5jBi1hcAhx76I82UU7miD35XE4+Y9n0Zcr5GLd7qL9phy0tboLLD8WoTlgCvOIQWrxFHaS2NjC+BUP0RIVQvBrdZRXmsYbh97T1ct4zmqZYixdJyR4No9sc00Yx3lZk+tiPtHt7/jAq+hmplGGSQpd588h3mef/CauR0tCtquYCz26hv673EBd3LEPc+0o9eWVbZzzHR97G25MzHCExqXIi3ifqW88VSqVSqVSqVQqlUo1VOkXT5VKpVKpVCqVSqVSDVXXFLXNz+N2IR8oXYTcCStVuLcl0sBC1teeCdIPfeORIH3LzXcF6QMHgNT6DXIeIxe8qINX234Lx6cJQSldAJqyb1fvePEZIDAVwlyP3AcEqbB6Nkg//tipIO028Gp9qwBkxjaBZjhpQoB85Mv3gZyFGdPt8oba+CyRYjuuP0jdeReeudYklKgNVLTdBH4wkgEO5Hl4NodQqngMmMWddwGjueu2+SA9mgIiWFgGXhANowEl0sAnGjVC6MK9exm6v0+uazaKa4cTyEs0BRQ2PQb81SfHyxC5EHsCvKHVxfWj1NNi5DaaIVQnROhHlVCdeHgHHzwwnTuPMiyTe+/BBWA0iQT66coK+uYFQn3SKaBBbXpmU0EbaJbIyo2cQw/sB06/fwLO0RlCrDY2gIbkR3sNfGY38lWtUDlTUcV91GmWrv2e978jSG8XUb/rl/B8W21cKFnGOZOE9Trk8DpHbsYpcmpdPn8+SHcatKn7ABUN00bs1L75OKOi7C5uKTbyLtrGQztOE9fjWmCvx08QwreOmJhdRBzYJldqL4I6ODLZ67O5adRjmPBP3kw8RRvHh8nVNe5zQER+ZycRb0aSaJv75xBXsuQwys6RPqFtiSyO73KQh0ZuOEiYI+gvluJFt0HuooQ/+i7VaRYf4Otk42jHVXJxbdRw3ORRp5Oz6I+zN70Vn6XN7lfXzwfp9lZvWUSsDYx0iso/LLjn7nGUW4o2YzdRtIv91Hcc4swyhBmm2FU9QnVEzpG734B5wfkVxCqHHEljcXx2kNpLuN32o4/hfi2Mxw12hm2RG+o2+kuRUG9GLpO09KC2BSwynCAn1Xm4vpbbOL5IdXfyW19H3vp9qbWFcWCDlk2M0f1jIdpVwJC7O/XfErmkVmhZimtxze0m6rRES1QsTXSiDuHhhIyWPeTTsSizQSo1gnbmUnxxyVG0VkacCteQ75iDuCNN4iKbQIqNg3r0XNwrFkG66+H5ie4UW4Ejf6KLsSdhe/eKhVH/ayW0wXkHc+Rd8ZtxnxDu02wglpc7iPE+tU3jY0wcoTmSTxh+tYJ+ECWMtlrHkofIBOJK07A78btlUFohFHurhnpk0j5OCP5WFeP0RB54a4mWyqyuYIzjYJ0i59suxayuwX0TKZQRz4GbNGeuVnp1MDaNus3S3Na46F9Zmn8ZcsntUvwstvFMDYP7+A4xsrTSJU7L4PJJjA/1AhphhdJ8nM+/kvSNp0qlUqlUKpVKpVKphir94qlSqVQqlUqlUqlUqqHq2rraVoCrhg1eD1tyEXXpNX9hHejq0grcCcdpY/huA6/tv/X1B3B94nfz5KY4RU6niSjhK2PAsKKJe4J0Z6yXhz30mv3CWWy8e+NbfyhI3/YWuDwWinid7V7Abr4jEbyRt5MAACAASURBVCAH4RCe2yOHM9cF9tBpAmNwyJmySShgNIXX+w67PoaG4774tnuxIfHJs3DkO3Ycm/PWqLyiETimdbrkCEcuxPsPATs5vB/uhLaN+q1sAplIZ6jpEo5cqwMpsOT6GOrjhR7ZQvrkAEwEnyTI6dP4uEaHXLziCdw/MYrnSxKKWasDr/AcoAtuB2XgkStjgjCNhou2GWkP5/ehehnoTIM2IY4l8TzlKs65sHQ+SI/kaMPvOp7HEB62ugaHxNUVdsrDOd/9Tz4apP0a3A+/9OADuO9RIDhjuV4/WTuN8pmbhateuUs7Z0eAzrbcy+MfH3zPey97/KXqsUcfDdKzVDbRBPrjOCGIg1SJ0KAOOdnxcYcwOHarrJHTa4bqPcp9h9yxQyFm+fFsdcL6XR99v0KOosefhYPxrdN3iohIghh0Qwi2EVw7FgWy5LrUd6jfs0NkhPDL6Rmgm7nxmSDtEx7fJfdFQ4EgkaQ+6+Je8dBwhs0QLbFoeciH8REjHHIL97qXPz9G18lnMMZE07R5/TTc4w/fcWeQHt0Nt9VNF5hXeQV9ML1BmH2j12f3TKBt19bhIjmzgL45Nz2PZ+Jypk3SJ/O4zsrS40F6147uS+sQCK1fLqzI5eSweS2d/10f+v7Lnv9KtfnEU0G6dBJzhRRhdQ1yUU+SC3CjQzgjOUsnCQe3ZOJaLdED0Zjkt9BmLpIrdb1BS2M2MF63R/voewYFnbnzviDNvrEv1UP2/iscb3z+wSDtdJGv8TAtF6CY0GZXZkt93x/OkqJGBeXfqCGmRSgeZQkpjlGbjrqo01QY85lwG/3Lb2LOmYjQ2OAhxhqar8xkcJ3pESx3anqI8/Xt3ji+uIF5Wd7BMrWcRX73TCIvJ9Yw1w4ZzFEjhuaibeSl1aS5aBpOql4Uc6dKC+NJtQRk18aBZWZodwufENBBqkVLbqq0nKRDfbBNjrI1ip8uOcBfWkPL3y7Trhzkih5NUmyiz4ZoTu9ZGlvDKKN6HW2s3Z8jZrP4u0NLrtI0Vo+Re32xjO9PJVpKuNZEXx/dD9za1BEzNreJ5U7gXqM0p+1WgZb7ne5lj4+mFbVVqVQqlUqlUqlUKtV11jV949k5hwW5/EvCOO17lyGjAQnjm/zC3tuCtEN7hp0/hV921hbxq2yLvu179Gvh+Bx+gc1mcf0Ymajc9Ia7g3Sub7yQPHYiOFau8n58+HVo1y3vD9L7l/GLweNH/48gvXsG92zSvqa1Kv2C1KB9gcisZXQW1VWkXxi8FhlikJlGOETmGwNUt4Y317cfwRuFkQx+uf6bv/l8kN5cxy9FE+Oo37e8EQYQ87QvZ4X2ezP0S9Qo7dEYjeOZy2W8sYjS2474KH7dM8+98aSfvH369ctz6e0wlflUFr/e7NuLNyZzMzieoGcK095ksTbaVMXi+pstvMWu0lupUgVv/JJ032gH5TpIdWj/0QaZg5xZxJvKT//FnwXpB7/ylSBtLMpuvYJf2TYv4G1HhH6M79Jbm+g0yuvrX8X+ue0K2snx0zDnqq+jDZQ2e9cZGcMvfpu0t2alTPsJjqAu9t6CtzrD0CbdN7yJN60jY9Q2wsMJt1tltKEEmafwcWvplT6ZIhlBvUyHYKSQo3bM5k0dF/3LIxqgRb+GV+p4a1MlU44qOWYl+/nsEOESS6CdW/pcJIS69im/rTb9Mk1vCco1tOtSHddv0z6tXC+GzJYaTZRZ16O9I8NkZkX70A5SI9N4i7F1Fu0/FEXbCtF+wJbMI7pk1jcyfShIL9yMsWz0CPaydfJ42xKh/YxbVZSXQ+Pcvgzq4IYj2Kdyvv/mepQMqB7/JvYHvmlhPkjvXcAehnGiSlIp5F3ITIXfeA5DczOTVz/pZejcMqiLLr2JTsXorTyZd2xvo82FaN4SpfGp20F/aNNbMBumdBXnN5u0V+CN80F6n6TpfMydCn6vz8zuB3mEUXg4cqt4wxLfIIPEJiiMElFemz767GwTY/uKS3PGAaqzjfKM0l6zoQ71QZo3+C5i7+Ts7UE64oEu2FxBu4/QG3A3QfNVqusmmTfGE+iDDF3kRjAviWZ77W17gok4MnpqoczXm8eCdHoa5R/3MBdrt9Bewh6ICTY5W9sGjRiLYN4yOgoyLtTFdfwI4nOlhTjXprmQYFr5ihUmM7IG0TsdMqfrMApBc+huGOevFTBvC0eIVqSxh9+EslFdisxDm216Q09mTDV6q/7cG1if4mGtin5hfDSAXAbPVybzpC0iyCYPol+/5fs+EKSfOY430Q/9EeZ3FRfjQIL2Wo0Y1C/vaxohoqrTRB6uJH3jqVKpVCqVSqVSqVSqoUq/eKpUKpVKpVKpVCqVaqi6pqhtpUP7x5ExwjihopUSXtOO0kL3XXuAyp16Gnje4gXCRVaB+S3swrv6Cr2iPvnN00E6HsMr6hqZY8RoL8bZTC/PJcLHnBw+V7iIRdlT+7C30n0feFeQbjUBrZyhvbMYqTCE4UWSVE4tHG83/yGqJiJi4jg/QviupT39Bqk4bZZYL2HR8twEXud/x0dg2vIXf/tFHP8Q9meaGseiZb5OnEyf0rSXVpeMSiKEKDBRHI0CZ3LiQEyeIw2JXtphSDJK+0a+9133BunKNsp8/wLQrBuOABtLjwCf9sksaCmLjJ10gV/O0B5f8STaaYrQb4/ME2qt4Sy6z40CUerST1CVGpCS408+GaTXFxeDdIhCR5LcO6K0F6ntdOh8FPyuGewxNpoB1lMkxHzfPPCkC7RnW2m7ICIiXgxltU7mRg3a87C0DeRt7y0yVBkyVUjTPo9hMunxaX+/QapKaEu9gbRvCLejvcYsm2oZQoxKiJN1MjZxO7wPMsp9/k5gnC9Vf3+xXxYXa3QU6e++G+WZjKHt1GtoUxfPw0ymQ3sFPnsGY0KT2iBva+zSszq0+XEmiz6xXQCq5JDJiWeHs4Th9n/yE0F6bvFokD760GeC9OYa+uDULsSgW+/7YJCeWADi1qS9Uy9RO2mexNKROO0vO0p42O5xxPPZXUhPjWG5Sr4/ViVpJvHGg98TpHMZMl8hJCyZpn1Rqd9vbmIcGLZKBcTk1NTcC5z50vQkmcrlxmFmOD0Bs6Y4IeNt2gMv1cHxThPoXbFaCNIJGrdChKTHPNTd6P75IL3vHW8O0pUHEc9jNG/Yl+/lLVwazp6Yl1O8i3tFNtA5Wz767FM0Lj1YRBz6UB3nP1DG8Z8bYP5uPvz2IB0nwzhDk4jNTRhtbdeRj3AcOHqrRcuryFwxnsDzd8hwsEkoPxvOsDEi74eepXlyor9EYnkTsatF+5yv1rHcLV2gfYDzhJdWzgfpJI3n+cR8kHZojua2cU4qBiR51zT2A48I+ldXsCRuqYp03bs6ovlyFKYlYBVaupUME/5L7+CadHyL9uIsNDDG5OO0B30U858Omd+VaNmCoTrgFTdt+q5RaKBN274B0eYymY2RwV2U8Nd4AuXPiGzqMNrd+78X30cO3I25VW4Sc5XFJ2BCtXIcsTHt4TrTEzCkipFRFR9/3CODxytI33iqVCqVSqVSqVQqlWqo0i+eKpVKpVKpVCqVSqUaqq4paluPAjVpEja6EAPCuLaJ17TnV+l1L+0Ns7QIvKFBr7PdLl5VbxfgkDkzBZzRJ9c4n15dtz3kp7yG63vrPQyoVAS6kIkDJzjzLF5PF2jfp30LuOftdwITvvAMnPq6tEenJfc2Q85bltxAfXJGFTrOr+BdizIwZjhIWJvK2QkDM/A6wNcmxoErvOWtd9Px5GXPD/F+XXT9dIT2tDI4J0R4nE/YtiWHxjDteWf7CFmX9mgKk2PZSA7t6933w/3x/KnzQXqN9q/LZ29COkN7WnaAjSX2wjlylsrDkptajPabzWQItaW6WyOX1EEqTaitQ/v9dQrAe7ZOAV/fnSYnUEJwqtSXW4R6G3Lhi1Gb3lxHX3r8m9jzboqQoQLtL1luokxr/abe3ELfEcJ4HXKwTkSG0/4vp3qdnPoqSOfHaH825rwHqA3C48YzQKa2aA/WeBpIDe/FaQk9suQGu7WN8k8SRpyKoU7hRT54/f7v/1WQPngQ2Fq5jHxt0b5js7vh7HjizDlcKEJ906JttOlZHYfjCtppmHCmi0vAeh0qj0HqQRrXbiT08663vSdIP/HwF4L0G+7FcW8M5/N1tsiROEN7It+WQb0f2QscdIH2PU2nUXbbRSB62QjK+tB8776JMPpahvaujRC6KxSPDSHocRpDErTP9bD195//XJD+gRvfMLDrHqc9KbO0H+3TS8CIZ6YJ8yPcPZNCGY2S83BjC9dhF+BYDZh0pom6zhFGe+EE9hLtFJAHn+pM+v2h6ZKz6JCVcdAGm+Te+UwX/fQLtHficgVYYqeC8jtTGA6efdud7wzSYdprNkx1+szJx4J04SLmrotrqIuIg/Exkaa9PrvkOk7PXC/THvCWHPsJ6WxQvZ87j+Ve6XjvOh65ntaoPW4Ssr2/Ox+kt5dR/hfPA8OP0PK4kTSeb3Yec4Gyi/HcH8F8ZjRCWG8MY3u9jeNFiuebJVxnkMrSrgNtmisYmoe1aWnTqWW0uRZhxNEQL4FDHeVorham8aNNS1T8Er6nZGipXjSLscSSu/p2sddmzl6kMiEMP0zzXxujpTMzyO+7vx+7bNz0phtxPs3RDhzGKP7+7wNa/rlPfgPPsYH7hmK053Cle9nj7RcxPOobT5VKpVKpVCqVSqVSDVX6xVOlUqlUKpVKpVKpVEPVNUVtQ3vwyjZVBy7CG44//AAQKyGH0Bg5gdYYURW87s3EyZmLXksXNvHqPEyv1DttYAzTk8BnU3EgDU4fU0in8Co5QghWvQh0oUsbmh97CE6u2+twIsyQ42WR8hWhquDX6C5tUCy0ya1Pxx3CNT1CdjsyHFfbagEYyYVFoJhvfgtwpQZhJMk48AbLx+k5v/4gNjPeuwD0a2KcNm4mJM4jZLZDKKZP+IRHaK7fd6J02RmOEGWxtBm9j+cbH0c7TWXh5thu454eocGuxTOlcsAYMqPAyemuYsjo1CNnTpfQo4mpKRmG/CjhGuSIGCUMLkJ9cE8WqLxL2EmVyj+cBUYSigLLbK6jj7dLKN9qAe1hy8d9S22cM38HXDrXNnv9rVTE9dLkkNlqABPuRnD/Ycv3UXcbm8CHqDvucAIcpHgj7Dptbs7HHXLUrRaB/YQJca83cH4yhhgYo37X5T4zRB1/dpXSQLxiFJvrhBY6KeBUHmHdVXYWDGF8oOqSDDmvlmu4pscYskHbLJSH4/w5QQjUxkU48x7cg/6/7zCQqfQYjp+m8ycSwLunMqjTm3Zj7Jl1cPzmN8Dy2RBK/Td/8odB+pvfeDBIj08iDvzMv//pXh737gqONcllOkLRznqXbztRwhgb7fplzxmGPvfZvw3SP/BTPz+w63oR9JcaOX67NCfxluB0H6fnL9DygNMlzC06TZRdi9xu5yNA6+5KYo6y/i2Mpw2qA0Ob17uE4pdTvXo38Ws3JUym8dynWnjuL5ynHQRoGc13HgKu+d69iPmb2zg+SI1M4R5hD/3CJBBfGsdQp8tLqK/tFtKZNMZEdw3ln6RdFSZpfjBGzto1Gs/Y+bbbwvyvVsJ8uNWfF4XIGbjWwhytRvOmio+x19DWABGDuHL8DDDe3DjOLzoo80gKz1Sj+V2BxpmFqbuC9OI68OSKi7zVW8NxtQ0Z9BGX3rV1wyhbdu1eu4ilTYac4Q/uwly0SnMYEbSNbpPmnIRkj/i0g0WCdu4waAO7Z3D9wlZv940zW8hjyNDOFzG0wY6Lc95NePjNd2KXjWoTc5JoFJ9N0RKrd7z3viC9d9ehIP2p3/7zIP3Q09iVY+Tm++j4Q0F6Yj/a75WkbzxVKpVKpVKpVCqVSjVU6RdPlUqlUqlUKpVKpVINVdcWtZ2mjZKbwIHK6+eDdJQskbJRvIb2CB3YQYPQxugRQjcpKS4hU4xPhUK8qTpQg40NuG4tzPZc/m64Bc60JcJeHjgHpEKE0yRn12UPh45gg13eWp7T0eNAj31yxYySe1bcIYSJ0Dkbxmv0QSrVInevddr8+xyQuF2H8cxbISAX+TjQhUsngRvxdVIz9GxtRiNw3ISolAg75nr0CHOS51BSl/5OvJ0RxiDp+Qg5TCeBmDF6x66ELmFNIdpo3u3Q7vXszksOow4hWiFyeuRnGqRKJWAx7QbKMEVOdhPTs0G6cAEIypnzQPs2aVPs0VGUUYjQ97qPvu91afPpBrCTVht16pJT3CY5G9b7KKTtMhYKVLJDDruGELZhK0POxokk4laX8sl1PUiNTQATajc7dBw4VDwJHGirAOwmm8I5HDPjhFyyo/hZcnmeuREI9KC15yA2uY6QG+omLU8oVFHXG9toy0Q0SigGzKpDm4G7hAxb2rA9QhhSk1DeJJWTRIeDcE9nUAF1Qt9tB31k7xxw/y4dzxLGnaLrOC7SWYqfiRjKlDHxp48+GaQ//Vd/gmsmcP7GNjZ9/2+/9esiIvKTP/ZDwbEolW2adkv3aTlDp4N2yuPz5vbVNx8flC6srl79pJehN5B78oltoIQ823KozEeSiF/sBF4lFLDZIoSRxhKHlo7ckMSYwZijjCE2hXK4fp3x+z5qG48ik4j8w1FRMD958jy5RtP6hI/egH73nrvg3Lx3BPn84ST6xCAVJrdQ3yIWxMmZtlvHXHH9NNyDLS3/mJiGA/6Zk3jOpsH4aOroy84c4pShsX/14vkgXW8Ar200aOlEvz0YS8h6HPHeRtBGltbQNvM55Hf3Hszd2jQfb3Zwn06bXFpHcc1Wm5Z6VIBMxwTI7tY2jvsxxK04jeOD1PIl1FGH+t0dd88H6be9894g/bu/9qkgnZ0FdnzPh+4J0l/6zJeCdGEN5eu5aK+eh3bS8hCHY+T8O5XC8of5ScwbytVe2/vKUZRbx6V4T3Mrl2L/nlvwnSKaRh9pVNGOOrSMy9BuHuzufuA22pXj7RiLn3oMuxBM70c7cdOI57ffdZtcTfrGU6VSqVQqlUqlUqlUQ5V+8VSpVCqVSqVSqVQq1VB1TVHbSBQISpxQn+VlYAObZ/HaOmbx+n+E0IqFGbyeTseBjoRD7AYLTKJGToUxdr4lFNIh19zDt8KBy+07dsYo740CXGrZ0WoYiseQrxA5bPFxS+61qQRe46cyw9niPZtBPvaS42KTHNgsuaflyDmLj/P5fB2+vu+RG1gYCI5P6HWrCezD0vFMBnUT6pdRKEQoLLmEhUOo3xZhYKUSsJAqbXZcJVzEoXbHSF6SEKoIuQ9GCHdxiG+MEdYbpfNThO1MTsL97hWrSSgyGcq55AJXBy0iq+QWuko4XY2cVKWAcglH0O8ahLhYQsab7BxK6Bc//zLhlW7fGZPR6M0iMF7GmK3H0PpwZSg+WEa4ySGY29sg5ZFTdzhJqI8PfD2dRP7SFCMaNaA2IcLEc4wLE/rWaLGb3/Dk0uby/PNoeow23y6ifzdayKNJIH5EHDy3T/00TFh7hDbuZoQtRuUk9NntEuLAIHXq5KNBeiqLpSheE8+WICfqVhPt3qX0qfUzQbpbpyUMB+FUOLcP2GK9CoT+4a9+NkiP0sbrH/sRoLRPPQnc6hvf+KqIiHzta18Jjk0k0AYNOVW2OshLhza1bxDSXCNsMH5geCi3iEg0lb76SS9D+108p5OgfjSK8uzQONil5R9rBWCBbpgwxDTG+ykX48EuQskbZbTdpQTayS1vxTIhIv5k9Zlnca9KbyzOx66dE/iJdbTZGi3XuHMXyuneQ5jDjMZRTu1Og44Pxy282UKe/CjasV8lVLGG5VVuDZh4fmIhSLc3cby+AbzVpXGwW0PdFej8cIzR/yqlcX61gTyEn1s+Fkbedy1g3JmcAXadpJUolubC9e5akF6YB8bseIgZjc4zQTrkYMlUx0M7TaWBYvq00mg0vRf38vHZqIN+M0g9+RgcnkfH0R8/8tG3BWmX5hMNcvIf34NlLPd84I1Bep3i7Wf/5GtButYgPJvKokbLiCpUGJtt1NPkCNp91+n1wzDh2NZD+bAT+L4jKOfZvUBkXXLSjdP3HkPzOKElTRKmmCRoa29+L+LH3W/FzhUj02hLP/LTP4i8Ja7+PlPfeKpUKpVKpVKpVCqVaqjSL54qlUqlUqlUKpVKpRqqrilqa1aBiBhywpoYBZpTSwMhqJDTaaUILDNOCNmh/fO4Jr1CbjRwTr2GV86JNDDdIzfCbSw9AQ+3ho/X2HO7e3jBxhLcpapF4DAiu2WY6rSR9yRhmQ06Ho6jXP0wbcbdHQ66MDUP/CWexf26hEqGQQDJ6CheyYcFiMwecsWKUN3lqD20LeEucnl34jC5NeZzuFeGNpu3ffeu9XU4GS6egzvjpSXgJYuLF4N0pQqMxesCkVhahKtrljaIPnQQm/amRoHLtT3kt7hFG01vAnMLkyPx1BTQ4xjhVB//jf8qg5JjUD6Mndf+//bO7MeS67DPdevuS+/7zHA4GykOGYqUqIWmRMlKYMeGYQeKkNgADRt5SR6SfyRPCZCHIDDyEOchgBMjcQQliOwQoSFyLEqyGErch8PZp3t679t3rVtVeeie+n0X7MY4ma7Jy+970VHx9r1Vp845VTX1nd/pqW1t7ak/biFFc4Sk0XSE9Damyg6k00XQQcMQCxjjfPE8Fkv4fvzz2AMlaOyzKIchE4ODxwdTjhOk88IYpf5/kmxsSEVOQ7XFQiJtsY7+uDCv/tve0xixv7uVlQfQzFJocIuLWuQ6T6jr9Poa+yemoHEi2TuG7V2FXhtQgeY5gvpVLFK71ecLaKcD9P04zSdlOt2TYtcd6dxd25ae16Baif5VSKGYc9F5JFdvT6q/Xw10rot39Z3vvf/XWfm7v/PbWfm1134/K58/fyEr/9WVA83sL16XbnZ2QWNXGKuu+tC0qTHGULk7A9Xzqzmrts3Jhy90/v/CFNJoW0iTbCDBMqlIfevjLqxc1ueTFtpuhKkKu6qjBdRdGfU7jcEvuqVr3g6mJRTXdB1qHCZmVouP75ZwD/v71ILuI145r/OyMi31t4BpNIWyPl9A2udJ0t/V2Dgaqb+MAmjiu+pfIRJaS03V486GrqEb96Cl4t5mFKsuWtMr2o4VBBLoxd2e7kH7se4hCpWDPl5CO5o/o++79LQU4NVNjTcVXYaDQqjtw46Oe3nmeX0o1P1yinv2jz5U+1rBONBEYm1noOPYXpfuff4J9Y+T5PqnqvOXvqyE1qe+oGlL3//B21k5STROnjqvlP7CtPrUy7/xzaycYprej68o2fj+usaBUqz2OsC52YftP7uoOtq8czBlqcApV5gSMrOge8vf/t5vZuXpJV0fu3213xBTyfi+sYAVGaplbec0pWIdielI3e4M1Pabs9qeBLi4HoPfeBpjjDHGGGOMyRU/eBpjjDHGGGOMyZXHqtqGq3odO8T6tg2keX7xotTV0jntXr+t1/n9nlIF792TIlligiECLet1qahFqKj7WOD2PpIzk0SvwttnDlKirn0k1Xa6hsXEc67BZk06zNKCNIFoU9t78Mw6qNc0zCfVM0K6bBUqTB16T1KUjoLs1AB5WsHknOqRi5jz+yMkjMGmCwqhzuPiopTdChSBARzNwuHb/w8/+CTb9qf/6T9n5Z1dqW3zc1IRn37qYlY+f1ZpbMnXteBwg0mASKr7n29fycozp6Wd9GPVze3bWDgbal+tqja7gXN9kuy3dcx7e2o4nX1o6h2oswgPnJyWm1NFiigpoD3U0TfLFX2emmwZ+i5V25iJuJnmmGKbfpPJ1kxse/1tJDiOJenqMxG2x/j+IpTOEhVg/G0DCkqVGjIGomr16Hp6VGanpN1EsHnLJW3vdqS/NGrap5VltbNwRcewtwtdsIT9rj9cozkJIthzRXj7vSEWvK4iPbikc1FB4ncNOuXutsZ7tp8ECaMVTGfoYTrDAEpjazKfNNSr772TlYtQZ2eWlKy4uKTUwvtruvZtr0m9i9HuR9DmNz9RuuMc0oFrSM3ewjSS5RX9VhvK/fycdOsH5+bDTzT14PYtTB+A3hyhcUZDtSOKyxjug3f++b/SPmKMXVnRWBrgGnd3Vfv+6nf+dlb+3j/8+1n5j//9v9Mx7eeT0LxyTufr5++rLtqrUhgRVhpUoCGehkIaqqkHg7ba7gxUwBRJwSGuuQ1c+6IPPsrKLajMra7aRuUw4rQy0n8v/5HqKsE0mhRJ8ynO3k9GGveWX/5GVv7H/+h7Wfm9H0nJvl/Xcby0ouN+dlbfU0EyZzp2HYBanuRzn1MJ1C+GO9KVtyLdfzbmNMZ++9dfzcoTRW0PLqn4ystKln4U/s2f/HFWfupF6aBB/eB8bO7qnmF6UeNhgOk1vX3V4eyCjnWUSpedX9L4ubCAhPZQ90g7PbXfBeio1aK237+rdkqldGFaY0mtkk+icoj3a1/5sk5Gs6H2dO1jTbsqhUiNflJ1u91TndZmdO/6d7/7q1n58kvPZuU79/RMUUrVpieaOgeLi2onK0tSov/rn/wwCIIgeOddTbNI0TcvIVH2iS+ey8o9rCQwlq4fYToJp6IUNCaPcI1LkLQ9NkUIai5Xlkj4CjN9eH/0G09jjDHGGGOMMbniB09jjDHGGGOMMbnyWFXbJtJik32kKeH5twntNkj1ind2Qa+2h7Fehe8gVWyE5Lf5WSVkpaG+c+2+XpdvQq+daOo1/9KiXq/f+PAg5W/tjvSK3TLS8C7q9XgePHdZ+zV1Rq/IpwrafvWWXq+vresV+TDOR+0La3hVX8bC7QlfyWt7Ba/n+Ro+LGpfCzhHIVNKofIwce/GdSmqb//wp1n5wrKSypKa6uhc/UCBmJ3RIsivvfYHWfnJ80ppW8T5r5X0HSVoYyH8sCY0sG5bSsnqQMpKY1lKydycNIkbH1/LypVQSky1KhVuZRkL2Z8gVHijoc5LYatiHAAAGgVJREFUvy+FYjiEMo2U4DLaQK9H3Ys6DnwylKlJjZD2G0JzrGN1ayq7D7xa6rekAB+4EBy9sHi3q/4SQ80uUZFFOi5/n9+fjqWb4rewmQs356XaNupoH9ynY/a115ViHUJ+b06oHTdb+s6lSfWHnS3pgnmyzsTNKdVbua5zVKupXAiwQDe00Ahp2QNougVMT+A5ZcJtpQL1+xg9/CRha6owzRBjQbcTH7m9U5aK2R1RhRQ9LFa+torU4pLGKaZv7+0i0RpjxSaneRyqvO2Oxol292iNtpAeramnmERBxT1Fn2Kq8P4daY8F9NMEfflHb2qaw601XSs2t7Tv1OZPkoWzUpR7V6VDD0tIqG/o/EaYOhSqawblVdXpNBIqA7SNQVH1UsA1NxmbqoA6Hek704Bj/sE5SCIodkgiT3GtLuLil+C63cf1sdLUuBdh/s8QqfLPzekzX3pCY8xUSW2wQ82vpN9K8FtxkE9/DGLVYbmm/l+b1r1oq6Ny+5q0yCCAapsDE1OYnvZVTQH67P6hVj2hfnEK7XFmRverrabueXojjettKNhJqt+5vSFVf3Zaqu2gq/vhqbruu6Mepj319Z1RH/exde3PoHv0Nf1RmZnTteyFrz2VlRsT2u9KWc8UE5Nqr/MLum/j9TvG9SPEPfATT+nzTz6jKQGtGq9hOn6EfgdVJLAvXjyoxy5Wctjt6FmndUrtK5xW++/hWlbGJLchpjbEY+8b9f0FTIWolThBDvdrGId5reT2Uunh/dFvPI0xxhhjjDHG5IofPI0xxhhjjDHG5MrjVW1nkBIGta+CBVgjJDcxrYyv4ff39Sq8jMTFyUksLh7q0Np7UuuokqRQU4Z4VXwfC7IXDlcaL+F1+m5PughfSOdBpap9n5xBItm6ts8s4tV2U1rBxlo+Cyu/8+71rFxvQAVuUsXUPjWgTXah+UV9lXeRutnrSgNjEuUuzuMnSKe9+4ESFbeWpTcUK9qfW9sHeutXfuu72ba9oRIHr7yptL0zZ6SgdPZUh7vbWkyZKY9TLSkbMxNqg7dvfpaVl4pIdEQd1KCWpyOpC5sbUsJaSJ08SaJI2lWAFMQSNAvaodU60/FQxChCDRGGVxBDr6XeWoSCW6xAscZixhXszwNFj98xrrwKhh2G0GWnp5ECi34/gFbMZNDj9FqmvY0wbgUxk1/1ee7zSVKEgk7tuY7zxfJEQ+2pgMXQQxwztaIi6n9qSkpcZ/0X+gwU+jLPB4pRl/V7cK6H2PdyoN+8dElp0lGqvxskGhtmzqqvbe+oP9aR8htCdS5DA+91NJYMkV7L61KrJZW134didIzC/agMkDgaDFVxt25I4StWdcwxFvCOoVAOMEUlGVOKdR6pUca4JlL9eu/9D7LyN77xSlZ+8603s/LO4TiYQn1MWD2cnlDguCKtjGNGWIYqhgXTmXJdYvJrQd9TgnZbggbWR72ePnNW+5nmpPZVVImXsaD71Q21sz6GAi4o32ecM87jDjS4dYwjYUnHUBvheEIkd2Mg5puGEOdmdDg+c1JHE/04wb6UcIIjjKv7mP6w11N/XL17U7+zI6Xz+dP6tVNIT4376psltIHKtPpjkXp8LZ+pKIWSxtKJSY0jMdToOzfVRz755dWs/KUXNKUnD6bmmWQOzb19cO96+oySlQuxxsw3Xv9xVi4j1XXxLO7HsfLD6l0lRQ9j3ZNs7eseabamY51qaRrRCFNnRmgbnNJRqp3Lyrt9tZmT5OxFTHM6KxU4Rnrtymkdz72bmnrQmkSCN+72IyS69oOjU/JrUOJTfGaA85WgrsNY9TV7OFUwxFg3Kur3p1d0HAOcX8w2HBvfOrjXDgKVuZpEHTowVwgJqe1jjBkOVS4Web/28MdKv/E0xhhjjDHGGJMrfvA0xhhjjDHGGJMrj1W1vbUhRXYCWmwDr5ATqCZU2XahUiVQZ1aW9Gq/jOfoEVLgUqYZjllg0jWGWLm6BA2oerhv5bJ+swJF92jJ7+SoTUIlQoojt8+2oEliwfByPR+V6F/+i/+QlYtQ8rgw7pknz2Xlv/e938vKf/anUlpv37ieldtQbWOob0xb7Yz0/QWoPw2chd1PpYZMYt9Kmwcph5df/Wa27d/+0b/Oyj/9+c+y8vlzl7PyqWUdR6Wmep7DIuoXL0o/bM5B946llPzwP/6PrDxEG2/vqE9Qd13fQNc8RiV9VHgMITSSGL5dBLWC+mkf/asAra1A5Q4axxDKXzE5OvVsXNPVeec+HJVUC+sqSBKqsEh5xDEVkbrGMSZiGQnNTFk+TrstHpPETL02OSaJ91HZ25M6U0EyHrd3oa9PQwdv1KnU6jsTaD/UTCPoyKU61Ul9T7+tsbrZkAYXDXT8g+FB/SY4n00k1pYxBu+01V9qVX1fONT2qQlpeDDWg35P/YvVXykjjZOpzFAHQ+xbHZUTI/n1JBn1dY74C0yHrqU6F32kMyfJMRo32ugAwwhVRfZZlt+88pb+FtfQ659pCsHk9MH1d1zN1v6WoZvxM62mzhfTpNmXeS6ocoVFXBMrag9MtQ2h3QbYHxj/4wujnyCNnvrdd85J4TtbVDv72S0pp3dGqtsSxpEdHMNHaLyquSCYGuozS1g1IEJi7XGjDsfzB+PtMuu8hLEO+9LH3Io2xvUd1GcMFXE4UB+slbT97IrOXQw9OUq1nVNRCpiSFaRM881nwlMUa8rV3bV3s/KHP9UUg4mizkYz0vU+b5agxiYp1MnuQd9ohGoLn32kaUlv/fh2Vj7zrM5jMqHzXh7pvmC0h/vMBX3++mefZuUPdzUF6de/82pWXj6j89UZbWK7vn9tU33irb9Svf6TXwlOjCbSazsDTEnY0jSuBlKYe0ONq1v4TGkWuizaN1PlOSWAQ3IELXU00vfjNAWDAa6n5YPU4DKmihQrmloxMaV2t7+v620fQ1qtonPKW0iOjRVcZ0e4d9rEvUOZui9c3rGpTBifIkzfOQ6/8TTGGGOMMcYYkyt+8DTGGGOMMcYYkyuPVbX9X2/o9fwzzz+RlWsXFrNyFYpde1vpYUMsaj87K1WsAj0rGkCP0xvkoAEtNd1nAhUXiddr5hLeIT9Iq2xW9Lq+WNCr9XxyuECRr9TLR25vNaXqlOs6jmY1H/VjfQOv9qllIu3u3o7Oyyt/R9t/8r+lety7o4W9y6j/GtTN+Xml1F44fykrzy5r0eTuvnQNqoC7O0onO3XqYBHl9z/+ONv2y48/zMpF6BJ9KK99OArXr6r9frmh+n/mwjNZeXJFCW/TSxey8iBQm717905WXljSuRv2pSSNL4yej6I5OSlNPYnhoOH4B1h4eA9pvEwbLKI8ltyKYjk8OuGOehz1WqZhFujHJZ/XjhOmL6KuUvy7GhPehj21EabaJhTnUf/8ReqyKf5Lo6a+VoHaF8IdzGvB+ja0GKbXMuGW5XZD+tDMrNp9oaCxsd/Tcc41laC3syP9rDWn9jM/pfYdQfevQLXsQI1NHiz6PnY69ZtbW/qdAZPOB9BfC2hHGLM7UFB3cdxMOp+Z0jFNQD2m3slpGSNM7ygVPq97nwQLi0qipN49pv9yXECdJ8f0qQLVSfSp8WNQucH0Y0ydWF2TGrqwpDG5eZiiTh2YWizV3TKUWqpf3F+6tkXsO3Vc9usC0hf5pSPWAfsdh7mcpjDsdZDAv4epFJH66VRNfe3eLtL1URWcNtCA9loKjh6r93Dd7HIaEeoiHhu/xIPRq1LS1jbOXSdVW9hDX9vGuNpGm22vKjG+vz6fleeRDttq4r5siIXscfPG4OHBvvp1HVMEBj1tp4b8qFy9JvXz7vYbWXljVfely+XnsvJc4egpJHnwxa+qXGvq2rOxcZDavL4u/bXZ0jhx+fKZrDx5RvWWxhon40j9ZfWO2m9nS9uHA9237OzrenLnss51c0L39fc2pCovyT4P7qCd3Pz0epAHvT6nCOF+Y4ApFkjR38VUkZu3tIJFc0XPLL1Ix9/voe1i/KpVMA0NymyIFTIKfYwVPbWfG58d3LvuQKMNK1gZAFOBOh2doyGmyJSLSu1t4H61MDbNQuX1TenQ7Y6Ob2JCCcpUbUu4to+wQkKp/PD3mX7jaYwxxhhjjDEmV/zgaYwxxhhjjDEmVx6raruzqVfC1z/WotiTiV75z0LF6EKhYLpT3FW6Uw96XlyE7hUyKQ+Lm5a5GCoULognI+iFDzYXQr1KLpakjE2u/SArzyisK5iZ0+exXn3QauL3mXQV6bV/vyet5fZNvSKvwusd7EivmFjQa/daXfs+dZLeCZhBGurqtvSh+flzWbmLV/Kvv/E2tks/mD/9hay8t3Fd3z8DjXhCOt8OlMIz52fweS0K3kVi7OQZqNeHGuGfv/GjbNsQ/+4SQosIoCjvQyG4e381K09f1T5++sFHWXl2Xam6aU3n8eWXXsjK9V9RZFu5Ak0Vbmp4TDrsSVKgsgavkKm7fegoEeqCKmAJSlwK1XU44qLJ1P+ouHOBeW2nXpiMtG8PShQFWTtpgToKtDJomSG0tXLx6ETE9Bglj4m/Y9YvnDCeO24fRcckjz4iE5Nqi0y1pQ49M6tBaDzVlimW+k6m2tYL6htTk0iKRqptvaI/7icag6hXNtHHug9SbcesVS6gLWVrpy1lPkR/YartaKTtTSTfFpGeym6U4uS127qeUD1qNaQqBVBJRzml2r7w4otZmSnJ1NcHSPweS5I95vNj6jATmdFP+RmerzrrGn12gOkMD9T2sTEK57TIRnWMMoxZFmPHUUbyMMcM9mt+J5X7Unr0rQ3V+kJOyvTOvjS1jS21rZ0O0kcxajEFOxlxqoI+U8H4Rf20h7+NaBRjXCuh/6a850k5TebgS1djpH7i97s4R10YpQOozqVA/2HzrtTs67/Q9zzzjBLgi5zZgAG3WNN3FtBP2WZjbj9i+sVJsLGjaTkJUtynGvq97u7VrNycfXyq7daafmsw0rSEpeUDlfbWHd07b2I1ibSs87uImVjV6tH65QBJ5Pc+Rlp5WX/89Ivns/I+tNuNbdVTuRpj+xo+r+88v6LvOUk4taeDaSDFCrTRKjRi9MF7a9JPLwzkCPfxjNDeU11zOk2C++8SVPUSkqiL6DPXr+n+8gf/7S+CIAiCnZ6ufZPT+sIIqzqMtX72087RU9AaSJqv4l43RFp1pcKUcu07rz+9faTpM7289PCUab/xNMYYY4wxxhiTK37wNMYYY4wxxhiTK49Vtf2135RiyCSm6XmpX7W6XtnGDb3WHUDFqJf1mnmqIRWznei1cVzQ99Rb0FJretaO8Up9iFfRtTpeP5cOPw/VpYhFkOtN7cv0vL5velavuZsNPN8nqvKddSk53U0s9J1IM4vwyrtU5vav4POqpxDaRW0qH5VoSGUw1Gv1pdPn9Bks1MvwwMUVpapx8dq9LSW98vur0N0SLCzdb0vpiKEw7Y9UFwUsPj061Ec2NvV3xaL07WXsVwA1aWZKiuJXXlKdt0pqIx3oIktV/f7pFSnAd7eV3vbZB0rzHaH+AiSjzszod6mcnSTU4wYDJr1i8W8sHM9zOozU7pkYW4BnR22uBi0wROprPGLKNJQz7FsBmuOD76ceWCkerTj1+9p3LhZfHFuYHovL4/eplHS7SEZlah3OF79zBAWU2m2tBp37BJmEasv0Wm5n2m0Qc9yDegRlrYA+UGtS21OfHQZQqdE2KlB8I3x/AEW1etgGhqj/IZIto9HRSYH9gbSmGjSl3TaSfdFna3WNH0NqQh0s4o1zR02VKcd97PuYPnqCUNM6LgGZmhQZ01JRX7yuUWsvIHmWx8m+n0APK471MWj2h/26zPRa6t4lTnnh9ehoPTKGTlmu6PcjqG1xAbooo4eP0bZTXOep2uZFZ1967aCHVFvsRxkp4k2qjRjrUtRzxFRuaHY9KnfsM8coxeOp35hC8OAz+J0E9zwx2ksZdVuE9ltLodhjFYI5pHiuTCCdOFYfLNbgJWLBeo4Z1MBDtPdyPqZtMLWI83JN/W5iQccZzSs9tlCWRnxM8z4xentIkW6oHiv1g/F/6ZT2ZfU2EsITtEdc7wpQvyenVf/nL2pK0/pVTY8bRbr27W1pXF29p3uhQaz037l5jcmffHozK2/f1+/ubeVzfay31F57GA/rUEJj9CO+jruKpN2n15VqW8VzSrmka+v4/YTKfVwfd9dUXx+++15WfuMvMSXtcD+//WvfyrZ9jFUYuI/dLlVfXe84xu9s67530Md0PEwDnMb/qY94/w4Nu6HjLur2KqhBj6841dYYY4wxxhhjzP9v/OBpjDHGGGOMMSZXHqtq++f//UpWXj6lV8KtQCmpIVJtB8ek2vZKelUcJlKsmGqbINW2q2CoYDiCPjKWaouFXENGWh78DxewjgO9Ko9DqQuPkmpbKUmRSyIpAMOhFJRi9LWsXB7+NCtXQ6RLoW5GXeh1Jwl0oHpTesTly09n5X5X+/TEihYSrtZ0/DUsanv72jv6fixkz3S+c09cyMoDLApMnWkf6XMFLMhbqRyoIaVASsDy3EpW/r1/8FpWfu8XSqkdQKNdOHcuK29vqlGtbktlef6MdIyLl7Dg8E3ptd//L9/PyjeuQ19BG6xBjaRi9M/+6R8GJwW1Myp21FKp44zpf2P6qyges/B9ij5FjZLfSc2xAFepiDbwYKH6sYTOlAmH6FNQ/rgvxym41BWPOw7uI/+2Ao22gVRV1k1eKZoxdLSx1G5sb2NR7AHUnGYDabAT0qdGUKxjHM/urtSyzkifmZnUZyJo2NSzQ6T5NQ4HxS30rzaUrQjKfGNKdVuu6/jW1tUHazVdT3rb2l6fCrS9q3GiAOVwPEkV1w20kxjaY5KTRzfWjtOjf2Ms7Zmpzcf87VgfYF+DolpiW0e5gmtoucQ++HlVnb/JxFoeRnLMPibJ0eV4gGvZ6GgVjgnD3C+qudRB+Rnq9CdJyv6IyN4y2k2joHqex8LwhRGmECQoI9mbqbI91NcIw0s81nxQvwHPAZN1D76UGnk4lryrcpXXqVDHOl/V/n7p7EJW/tZzSgNtFTANgcnGnE7CMu/FoHNnU6CCICim+bw/SRuqixUcQ4SpQCMcc7KraV+9z3LZpYyz5zXeTi5qfKzPHYzhYbicbdu4o7TY+1tK4U2qUl4D3HMGCbRJqJWYfRFMtKBkJ1LL9/fVHkZoGzXcv69vSMHt7yMFNtG14iSZmtNvdHBfuot9bUOPf+LJs9iua8D9+0q4PXNW7SFF+2PbSPGcMEj0Wz+7ciMr//DP/jorF+v6zG/97reDIAiCeKhr8vsf/zIr1xrazvuKMlKmWy08O0C75SofCQaKFAMlxy1Onxoh9brMVRiwvVZ8+HOH33gaY4wxxhhjjMkVP3gaY4wxxhhjjMmVx6raTs9Jhzr3tDTEJy9AxUS61J6srqCPlLTipJTa+rTSu6KBXv3WsTrxaMQFZI9O0eSa7xWk6aWHfxsyzTHSq+fekOqdHJheR9rF3o7UttYkyk0kXqLcmkFqb12v+qsKGAsGk9DAakiXhKqDsNcT5dm/9XxWfvtnP8/Kd29IUe0htTiF63z3/npWrjfVHmKofc++9KWsXGDyLDTWjz66lpW5QHAMhSlE8+72Dr4/xOL2dSQGD3v6/VPLS1n5J299mpXf/vGPsvLcglSiV16WdrGK9nXlXS1A3YGq9/zXvpGVn3pO7SFGkliBi8AX8/n3IaqlyTGLcA8j6m4Rtv/fpdpSpWKq7ejYVFuWmUR58PmjdL/P7fvwaH2Yeu9xqbZj6aZoX/zdscWisb9MmXscqbZdTEnAaQlgjY6pyw0o7rUGxrpEx9zZV/9d6yPFGp9p70trr5a56DvaAxalrkIpDA7TbtfXlfYcpvqOxSX1rzSEgl1C0mwfKbk1ncfJSY29tQkda4JUUbqIScIxHG18yHFV2lSakzJNZeo4LZtt8W/yt2Optkw9xZgSB5hywj4IvTYa62Of1+/Dv0GqbchjQrFY1P+J8R94HcYlN4gwHoTh0bcwJSyMzlTbPlKf80oL5xQGDHVBrYJE1wj1X0L7S9HWkSZZpSKMPtVBO8alP4ioO2PfxsY1Tm14MKUIUyiquJbW0Adn0AeXkHh9aUn97qvPns7KKwhiTnGPluA4YrS1Eq7LAe6pxvpdhecdg94JUipp7KjMqN0Mt3X/2dXwFWx/IBWzsv96Vg4HqoCta7oXGqzoeBa/o/lYuxtQHtFOzp/XqgaDSOWtW/rdcP9gh2ot/t0LWXnptO6ntvuq5/V1qaYJ9M4ixuwXvn5O22MdRxLo+tMbYapTAB10TN3k9V+fn1jQ/eBJcvYptcXtXayGgHF/blE31y9/S6sX7OysZuX2vo5zZ1fXx0k8j5RxHdzc1APMYF8345/duJ2VQ4xxX//mi1n50hcOdN8rf/lBti2KdA0aDai54j6OzyOYbtBqah95faCCWy4drQwz1b839lABBXfIqRvBQ/EbT2OMMcYYY4wxueIHT2OMMcYYY4wxuVI4Lj3PGGOMMcYYY4w5CfzG0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvjB0xhjjDHGGGNMrvwfVrivm/HEEIkAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wvhwlXlDtEwQ",
        "colab_type": "text"
      },
      "source": [
        "Now let's see how fast it is:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S0YBtX02q8T6",
        "colab_type": "code",
        "outputId": "d4bedf88-ee09-483e-f53c-7ee3a7f9aa59",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "t = Timer(synch=torch.cuda.synchronize)\n",
        "batches = train_batches(batch_size=512, transforms=(Crop(32, 32), FlipLR(), Cutout(8, 8)))\n",
        "\n",
        "for epoch in range(24):\n",
        "    for batch in batches:\n",
        "        pass\n",
        "print(f'{t():.3f}s')"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.362s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x-mg89jmtg1A",
        "colab_type": "text"
      },
      "source": [
        "This is great! We're able to iterate through 24 epochs of training data, applying data augmentation and shuffling in less than the time taken to transfer the dataset once to the CPU! Moreover, since we're no longer racing CPU preprocessing queues against the GPU, we can stop worrying about dataloading altogether, even if training gets faster. Note: we are relying on the fact that the dataset is small enough to store and manipulate as a whole in GPU memory, but a more sophisticated implementation could work around this - or one could switch to an industrial strength solution such as [Nvidia DALI](https://github.com/NVIDIA/DALI).\n",
        "\n",
        "Let's see if this gives us a speedup on our baseline. The code we use here is essentially equivalent to the DAWNBench submission, apart from the GPU data processing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hNFoYZEjH_w8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        },
        "outputId": "e47debaf-d181-4838-ea70-754e85371f15"
      },
      "source": [
        "baseline_net=network()\n",
        "show(baseline_net)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<__main__.DotGraph at 0x7f41409c0518>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n -->\n<!-- Title: G Pages: 1 -->\n<svg width=\"1080pt\" height=\"59pt\"\n viewBox=\"0.00 0.00 1080.00 59.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(0.310167 0.310167) rotate(0) translate(4 187)\">\n<title>G</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-187 3478,-187 3478,4 -4,4\"/>\n<g id=\"clust1\" class=\"cluster\"><title>cluster_prep</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M94,-48C94,-48 320,-48 320,-48 326,-48 332,-54 332,-60 332,-60 332,-111 332,-111 332,-117 326,-123 320,-123 320,-123 94,-123 94,-123 88,-123 82,-117 82,-111 82,-111 82,-60 82,-60 82,-54 88,-48 94,-48\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-107.8\" font-family=\"Times,serif\" font-size=\"14.00\">prep</text>\n</g>\n<g id=\"clust2\" class=\"cluster\"><title>cluster_layer1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M364,-12C364,-12 1498,-12 1498,-12 1504,-12 1510,-18 1510,-24 1510,-24 1510,-163 1510,-163 1510,-169 1504,-175 1498,-175 1498,-175 364,-175 364,-175 358,-175 352,-169 352,-163 352,-163 352,-24 352,-24 352,-18 358,-12 364,-12\"/>\n<text text-anchor=\"middle\" x=\"931\" y=\"-159.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer1</text>\n</g>\n<g id=\"clust3\" class=\"cluster\"><title>cluster_layer1_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M724,-20C724,-20 1490,-20 1490,-20 1496,-20 1502,-26 1502,-32 1502,-32 1502,-132 1502,-132 1502,-138 1496,-144 1490,-144 1490,-144 724,-144 724,-144 718,-144 712,-138 712,-132 712,-132 712,-32 712,-32 712,-26 718,-20 724,-20\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-128.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust4\" class=\"cluster\"><title>cluster_layer1_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M814,-28C814,-28 1040,-28 1040,-28 1046,-28 1052,-34 1052,-40 1052,-40 1052,-91 1052,-91 1052,-97 1046,-103 1040,-103 1040,-103 814,-103 814,-103 808,-103 802,-97 802,-91 802,-91 802,-40 802,-40 802,-34 808,-28 814,-28\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust5\" class=\"cluster\"><title>cluster_layer1_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1084,-28C1084,-28 1310,-28 1310,-28 1316,-28 1322,-34 1322,-40 1322,-40 1322,-91 1322,-91 1322,-97 1316,-103 1310,-103 1310,-103 1084,-103 1084,-103 1078,-103 1072,-97 1072,-91 1072,-91 1072,-40 1072,-40 1072,-34 1078,-28 1084,-28\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust6\" class=\"cluster\"><title>cluster_layer2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1534,-53C1534,-53 1850,-53 1850,-53 1856,-53 1862,-59 1862,-65 1862,-65 1862,-116 1862,-116 1862,-122 1856,-128 1850,-128 1850,-128 1534,-128 1534,-128 1528,-128 1522,-122 1522,-116 1522,-116 1522,-65 1522,-65 1522,-59 1528,-53 1534,-53\"/>\n<text text-anchor=\"middle\" x=\"1692\" y=\"-112.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer2</text>\n</g>\n<g id=\"clust7\" class=\"cluster\"><title>cluster_layer3</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1894,-8C1894,-8 3028,-8 3028,-8 3034,-8 3040,-14 3040,-20 3040,-20 3040,-159 3040,-159 3040,-165 3034,-171 3028,-171 3028,-171 1894,-171 1894,-171 1888,-171 1882,-165 1882,-159 1882,-159 1882,-20 1882,-20 1882,-14 1888,-8 1894,-8\"/>\n<text text-anchor=\"middle\" x=\"2461\" y=\"-155.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer3</text>\n</g>\n<g id=\"clust8\" class=\"cluster\"><title>cluster_layer3_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2254,-16C2254,-16 3020,-16 3020,-16 3026,-16 3032,-22 3032,-28 3032,-28 3032,-128 3032,-128 3032,-134 3026,-140 3020,-140 3020,-140 2254,-140 2254,-140 2248,-140 2242,-134 2242,-128 2242,-128 2242,-28 2242,-28 2242,-22 2248,-16 2254,-16\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust9\" class=\"cluster\"><title>cluster_layer3_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2344,-24C2344,-24 2570,-24 2570,-24 2576,-24 2582,-30 2582,-36 2582,-36 2582,-87 2582,-87 2582,-93 2576,-99 2570,-99 2570,-99 2344,-99 2344,-99 2338,-99 2332,-93 2332,-87 2332,-87 2332,-36 2332,-36 2332,-30 2338,-24 2344,-24\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust10\" class=\"cluster\"><title>cluster_layer3_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2614,-24C2614,-24 2840,-24 2840,-24 2846,-24 2852,-30 2852,-36 2852,-36 2852,-87 2852,-87 2852,-93 2846,-99 2840,-99 2840,-99 2614,-99 2614,-99 2608,-99 2602,-93 2602,-87 2602,-87 2602,-36 2602,-36 2602,-30 2608,-24 2614,-24\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust11\" class=\"cluster\"><title>cluster_classifier</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M3154,-49C3154,-49 3380,-49 3380,-49 3386,-49 3392,-55 3392,-61 3392,-61 3392,-112 3392,-112 3392,-118 3386,-124 3380,-124 3380,-124 3154,-124 3154,-124 3148,-124 3142,-118 3142,-112 3142,-112 3142,-61 3142,-61 3142,-55 3148,-49 3154,-49\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-108.8\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n</g>\n<!-- prep_conv -->\n<g id=\"node1\" class=\"node\"><title>prep_conv</title>\n<g id=\"a_node1\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 3, &#39;out_channels&#39;: 64, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M132,-92C132,-92 102,-92 102,-92 96,-92 90,-86 90,-80 90,-80 90,-68 90,-68 90,-62 96,-56 102,-56 102,-56 132,-56 132,-56 138,-56 144,-62 144,-68 144,-68 144,-80 144,-80 144,-86 138,-92 132,-92\"/>\n<text text-anchor=\"middle\" x=\"117\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_norm -->\n<g id=\"node2\" class=\"node\"><title>prep_norm</title>\n<g id=\"a_node2\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 64}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M222,-92C222,-92 192,-92 192,-92 186,-92 180,-86 180,-80 180,-80 180,-68 180,-68 180,-62 186,-56 192,-56 192,-56 222,-56 222,-56 228,-56 234,-62 234,-68 234,-68 234,-80 234,-80 234,-86 228,-92 222,-92\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- prep_conv&#45;&gt;prep_norm -->\n<g id=\"edge2\" class=\"edge\"><title>prep_conv&#45;&gt;prep_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M144.403,-74C152.393,-74 161.311,-74 169.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"169.919,-77.5001 179.919,-74 169.919,-70.5001 169.919,-77.5001\"/>\n</g>\n<!-- prep_act -->\n<g id=\"node3\" class=\"node\"><title>prep_act</title>\n<g id=\"a_node3\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M312,-92C312,-92 282,-92 282,-92 276,-92 270,-86 270,-80 270,-80 270,-68 270,-68 270,-62 276,-56 282,-56 282,-56 312,-56 312,-56 318,-56 324,-62 324,-68 324,-68 324,-80 324,-80 324,-86 318,-92 312,-92\"/>\n<text text-anchor=\"middle\" x=\"297\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- prep_norm&#45;&gt;prep_act -->\n<g id=\"edge3\" class=\"edge\"><title>prep_norm&#45;&gt;prep_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M234.403,-74C242.393,-74 251.311,-74 259.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"259.919,-77.5001 269.919,-74 259.919,-70.5001 259.919,-77.5001\"/>\n</g>\n<!-- layer1_conv -->\n<g id=\"node4\" class=\"node\"><title>layer1_conv</title>\n<g id=\"a_node4\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 64, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M402,-92C402,-92 372,-92 372,-92 366,-92 360,-86 360,-80 360,-80 360,-68 360,-68 360,-62 366,-56 372,-56 372,-56 402,-56 402,-56 408,-56 414,-62 414,-68 414,-68 414,-80 414,-80 414,-86 408,-92 402,-92\"/>\n<text text-anchor=\"middle\" x=\"387\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_act&#45;&gt;layer1_conv -->\n<g id=\"edge4\" class=\"edge\"><title>prep_act&#45;&gt;layer1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M324.403,-74C332.393,-74 341.311,-74 349.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"349.919,-77.5001 359.919,-74 349.919,-70.5001 349.919,-77.5001\"/>\n</g>\n<!-- layer1_norm -->\n<g id=\"node5\" class=\"node\"><title>layer1_norm</title>\n<g id=\"a_node5\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M492,-92C492,-92 462,-92 462,-92 456,-92 450,-86 450,-80 450,-80 450,-68 450,-68 450,-62 456,-56 462,-56 462,-56 492,-56 492,-56 498,-56 504,-62 504,-68 504,-68 504,-80 504,-80 504,-86 498,-92 492,-92\"/>\n<text text-anchor=\"middle\" x=\"477\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_conv&#45;&gt;layer1_norm -->\n<g id=\"edge5\" class=\"edge\"><title>layer1_conv&#45;&gt;layer1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M414.403,-74C422.393,-74 431.311,-74 439.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"439.919,-77.5001 449.919,-74 439.919,-70.5001 439.919,-77.5001\"/>\n</g>\n<!-- layer1_act -->\n<g id=\"node6\" class=\"node\"><title>layer1_act</title>\n<g id=\"a_node6\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M582,-92C582,-92 552,-92 552,-92 546,-92 540,-86 540,-80 540,-80 540,-68 540,-68 540,-62 546,-56 552,-56 552,-56 582,-56 582,-56 588,-56 594,-62 594,-68 594,-68 594,-80 594,-80 594,-86 588,-92 582,-92\"/>\n<text text-anchor=\"middle\" x=\"567\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_norm&#45;&gt;layer1_act -->\n<g id=\"edge6\" class=\"edge\"><title>layer1_norm&#45;&gt;layer1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M504.403,-74C512.393,-74 521.311,-74 529.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"529.919,-77.5001 539.919,-74 529.919,-70.5001 529.919,-77.5001\"/>\n</g>\n<!-- layer1_pool -->\n<g id=\"node7\" class=\"node\"><title>layer1_pool</title>\n<g id=\"a_node7\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M672,-92C672,-92 642,-92 642,-92 636,-92 630,-86 630,-80 630,-80 630,-68 630,-68 630,-62 636,-56 642,-56 642,-56 672,-56 672,-56 678,-56 684,-62 684,-68 684,-68 684,-80 684,-80 684,-86 678,-92 672,-92\"/>\n<text text-anchor=\"middle\" x=\"657\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer1_act&#45;&gt;layer1_pool -->\n<g id=\"edge7\" class=\"edge\"><title>layer1_act&#45;&gt;layer1_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M594.403,-74C602.393,-74 611.311,-74 619.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"619.919,-77.5001 629.919,-74 619.919,-70.5001 619.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_in -->\n<g id=\"node8\" class=\"node\"><title>layer1_residual_in</title>\n<g id=\"a_node8\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M762,-92C762,-92 732,-92 732,-92 726,-92 720,-86 720,-80 720,-80 720,-68 720,-68 720,-62 726,-56 732,-56 732,-56 762,-56 762,-56 768,-56 774,-62 774,-68 774,-68 774,-80 774,-80 774,-86 768,-92 762,-92\"/>\n<text text-anchor=\"middle\" x=\"747\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer1_pool&#45;&gt;layer1_residual_in -->\n<g id=\"edge8\" class=\"edge\"><title>layer1_pool&#45;&gt;layer1_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M684.403,-74C692.393,-74 701.311,-74 709.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"709.919,-77.5001 719.919,-74 709.919,-70.5001 709.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_res1_conv -->\n<g id=\"node9\" class=\"node\"><title>layer1_residual_res1_conv</title>\n<g id=\"a_node9\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M852,-72C852,-72 822,-72 822,-72 816,-72 810,-66 810,-60 810,-60 810,-48 810,-48 810,-42 816,-36 822,-36 822,-36 852,-36 852,-36 858,-36 864,-42 864,-48 864,-48 864,-60 864,-60 864,-66 858,-72 852,-72\"/>\n<text text-anchor=\"middle\" x=\"837\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_res1_conv -->\n<g id=\"edge9\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M774.403,-67.9993C782.481,-66.1634 791.507,-64.1121 800.105,-62.158\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"800.943,-65.5568 809.919,-59.9275 799.392,-58.7308 800.943,-65.5568\"/>\n</g>\n<!-- layer1_residual_add -->\n<g id=\"node16\" class=\"node\"><title>layer1_residual_add</title>\n<g id=\"a_node16\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M1482,-97C1482,-97 1452,-97 1452,-97 1446,-97 1440,-91 1440,-85 1440,-85 1440,-73 1440,-73 1440,-67 1446,-61 1452,-61 1452,-61 1482,-61 1482,-61 1488,-61 1494,-67 1494,-73 1494,-73 1494,-85 1494,-85 1494,-91 1488,-97 1482,-97\"/>\n<text text-anchor=\"middle\" x=\"1467\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_add -->\n<g id=\"edge16\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M771.737,-92.1018C780.735,-97.9639 791.381,-103.78 802,-107 854.782,-123.006 870.844,-112 926,-112 926,-112 926,-112 1288,-112 1337.77,-112 1394.08,-99.3711 1429.8,-89.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1431.04,-93.0265 1439.74,-86.9884 1429.17,-86.2796 1431.04,-93.0265\"/>\n</g>\n<!-- layer1_residual_res1_norm -->\n<g id=\"node10\" class=\"node\"><title>layer1_residual_res1_norm</title>\n<g id=\"a_node10\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M942,-72C942,-72 912,-72 912,-72 906,-72 900,-66 900,-60 900,-60 900,-48 900,-48 900,-42 906,-36 912,-36 912,-36 942,-36 942,-36 948,-36 954,-42 954,-48 954,-48 954,-60 954,-60 954,-66 948,-72 942,-72\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm -->\n<g id=\"edge10\" class=\"edge\"><title>layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M864.403,-54C872.393,-54 881.311,-54 889.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"889.919,-57.5001 899.919,-54 889.919,-50.5001 889.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res1_act -->\n<g id=\"node11\" class=\"node\"><title>layer1_residual_res1_act</title>\n<g id=\"a_node11\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1032,-72C1032,-72 1002,-72 1002,-72 996,-72 990,-66 990,-60 990,-60 990,-48 990,-48 990,-42 996,-36 1002,-36 1002,-36 1032,-36 1032,-36 1038,-36 1044,-42 1044,-48 1044,-48 1044,-60 1044,-60 1044,-66 1038,-72 1032,-72\"/>\n<text text-anchor=\"middle\" x=\"1017\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act -->\n<g id=\"edge11\" class=\"edge\"><title>layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M954.403,-54C962.393,-54 971.311,-54 979.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"979.919,-57.5001 989.919,-54 979.919,-50.5001 979.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_conv -->\n<g id=\"node12\" class=\"node\"><title>layer1_residual_res2_conv</title>\n<g id=\"a_node12\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1122,-72C1122,-72 1092,-72 1092,-72 1086,-72 1080,-66 1080,-60 1080,-60 1080,-48 1080,-48 1080,-42 1086,-36 1092,-36 1092,-36 1122,-36 1122,-36 1128,-36 1134,-42 1134,-48 1134,-48 1134,-60 1134,-60 1134,-66 1128,-72 1122,-72\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv -->\n<g id=\"edge12\" class=\"edge\"><title>layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1044.4,-54C1052.39,-54 1061.31,-54 1069.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1069.92,-57.5001 1079.92,-54 1069.92,-50.5001 1069.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_norm -->\n<g id=\"node13\" class=\"node\"><title>layer1_residual_res2_norm</title>\n<g id=\"a_node13\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M1212,-72C1212,-72 1182,-72 1182,-72 1176,-72 1170,-66 1170,-60 1170,-60 1170,-48 1170,-48 1170,-42 1176,-36 1182,-36 1182,-36 1212,-36 1212,-36 1218,-36 1224,-42 1224,-48 1224,-48 1224,-60 1224,-60 1224,-66 1218,-72 1212,-72\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm -->\n<g id=\"edge13\" class=\"edge\"><title>layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1134.4,-54C1142.39,-54 1151.31,-54 1159.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1159.92,-57.5001 1169.92,-54 1159.92,-50.5001 1159.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_act -->\n<g id=\"node14\" class=\"node\"><title>layer1_residual_res2_act</title>\n<g id=\"a_node14\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1302,-72C1302,-72 1272,-72 1272,-72 1266,-72 1260,-66 1260,-60 1260,-60 1260,-48 1260,-48 1260,-42 1266,-36 1272,-36 1272,-36 1302,-36 1302,-36 1308,-36 1314,-42 1314,-48 1314,-48 1314,-60 1314,-60 1314,-66 1308,-72 1302,-72\"/>\n<text text-anchor=\"middle\" x=\"1287\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act -->\n<g id=\"edge14\" class=\"edge\"><title>layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1224.4,-54C1232.39,-54 1241.31,-54 1249.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1249.92,-57.5001 1259.92,-54 1249.92,-50.5001 1249.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_out -->\n<g id=\"node15\" class=\"node\"><title>layer1_residual_out</title>\n<g id=\"a_node15\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M1392,-82C1392,-82 1362,-82 1362,-82 1356,-82 1350,-76 1350,-70 1350,-70 1350,-58 1350,-58 1350,-52 1356,-46 1362,-46 1362,-46 1392,-46 1392,-46 1398,-46 1404,-52 1404,-58 1404,-58 1404,-70 1404,-70 1404,-76 1398,-82 1392,-82\"/>\n<text text-anchor=\"middle\" x=\"1377\" y=\"-60.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_act&#45;&gt;layer1_residual_out -->\n<g id=\"edge15\" class=\"edge\"><title>layer1_residual_res2_act&#45;&gt;layer1_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1314.4,-57.0003C1322.39,-57.9083 1331.31,-58.9217 1339.82,-59.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1339.59,-63.3847 1349.92,-61.0362 1340.38,-56.4294 1339.59,-63.3847\"/>\n</g>\n<!-- layer1_residual_out&#45;&gt;layer1_residual_add -->\n<g id=\"edge17\" class=\"edge\"><title>layer1_residual_out&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1404.4,-68.5005C1412.39,-69.8625 1421.31,-71.3825 1429.82,-72.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1429.47,-76.3242 1439.92,-74.5544 1430.65,-69.4237 1429.47,-76.3242\"/>\n</g>\n<!-- layer2_conv -->\n<g id=\"node17\" class=\"node\"><title>layer2_conv</title>\n<g id=\"a_node17\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 256, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1572,-97C1572,-97 1542,-97 1542,-97 1536,-97 1530,-91 1530,-85 1530,-85 1530,-73 1530,-73 1530,-67 1536,-61 1542,-61 1542,-61 1572,-61 1572,-61 1578,-61 1584,-67 1584,-73 1584,-73 1584,-85 1584,-85 1584,-91 1578,-97 1572,-97\"/>\n<text text-anchor=\"middle\" x=\"1557\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_add&#45;&gt;layer2_conv -->\n<g id=\"edge18\" class=\"edge\"><title>layer1_residual_add&#45;&gt;layer2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1494.4,-79C1502.39,-79 1511.31,-79 1519.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1519.92,-82.5001 1529.92,-79 1519.92,-75.5001 1519.92,-82.5001\"/>\n</g>\n<!-- layer2_norm -->\n<g id=\"node18\" class=\"node\"><title>layer2_norm</title>\n<g id=\"a_node18\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 256}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M1662,-97C1662,-97 1632,-97 1632,-97 1626,-97 1620,-91 1620,-85 1620,-85 1620,-73 1620,-73 1620,-67 1626,-61 1632,-61 1632,-61 1662,-61 1662,-61 1668,-61 1674,-67 1674,-73 1674,-73 1674,-85 1674,-85 1674,-91 1668,-97 1662,-97\"/>\n<text text-anchor=\"middle\" x=\"1647\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer2_conv&#45;&gt;layer2_norm -->\n<g id=\"edge19\" class=\"edge\"><title>layer2_conv&#45;&gt;layer2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1584.4,-79C1592.39,-79 1601.31,-79 1609.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1609.92,-82.5001 1619.92,-79 1609.92,-75.5001 1609.92,-82.5001\"/>\n</g>\n<!-- layer2_act -->\n<g id=\"node19\" class=\"node\"><title>layer2_act</title>\n<g id=\"a_node19\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1752,-97C1752,-97 1722,-97 1722,-97 1716,-97 1710,-91 1710,-85 1710,-85 1710,-73 1710,-73 1710,-67 1716,-61 1722,-61 1722,-61 1752,-61 1752,-61 1758,-61 1764,-67 1764,-73 1764,-73 1764,-85 1764,-85 1764,-91 1758,-97 1752,-97\"/>\n<text text-anchor=\"middle\" x=\"1737\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer2_norm&#45;&gt;layer2_act -->\n<g id=\"edge20\" class=\"edge\"><title>layer2_norm&#45;&gt;layer2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1674.4,-79C1682.39,-79 1691.31,-79 1699.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1699.92,-82.5001 1709.92,-79 1699.92,-75.5001 1699.92,-82.5001\"/>\n</g>\n<!-- layer2_pool -->\n<g id=\"node20\" class=\"node\"><title>layer2_pool</title>\n<g id=\"a_node20\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M1842,-97C1842,-97 1812,-97 1812,-97 1806,-97 1800,-91 1800,-85 1800,-85 1800,-73 1800,-73 1800,-67 1806,-61 1812,-61 1812,-61 1842,-61 1842,-61 1848,-61 1854,-67 1854,-73 1854,-73 1854,-85 1854,-85 1854,-91 1848,-97 1842,-97\"/>\n<text text-anchor=\"middle\" x=\"1827\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer2_act&#45;&gt;layer2_pool -->\n<g id=\"edge21\" class=\"edge\"><title>layer2_act&#45;&gt;layer2_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1764.4,-79C1772.39,-79 1781.31,-79 1789.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1789.92,-82.5001 1799.92,-79 1789.92,-75.5001 1789.92,-82.5001\"/>\n</g>\n<!-- layer3_conv -->\n<g id=\"node21\" class=\"node\"><title>layer3_conv</title>\n<g id=\"a_node21\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 256, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1932,-97C1932,-97 1902,-97 1902,-97 1896,-97 1890,-91 1890,-85 1890,-85 1890,-73 1890,-73 1890,-67 1896,-61 1902,-61 1902,-61 1932,-61 1932,-61 1938,-61 1944,-67 1944,-73 1944,-73 1944,-85 1944,-85 1944,-91 1938,-97 1932,-97\"/>\n<text text-anchor=\"middle\" x=\"1917\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer2_pool&#45;&gt;layer3_conv -->\n<g id=\"edge22\" class=\"edge\"><title>layer2_pool&#45;&gt;layer3_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1854.4,-79C1862.39,-79 1871.31,-79 1879.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1879.92,-82.5001 1889.92,-79 1879.92,-75.5001 1879.92,-82.5001\"/>\n</g>\n<!-- layer3_norm -->\n<g id=\"node22\" class=\"node\"><title>layer3_norm</title>\n<g id=\"a_node22\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2022,-97C2022,-97 1992,-97 1992,-97 1986,-97 1980,-91 1980,-85 1980,-85 1980,-73 1980,-73 1980,-67 1986,-61 1992,-61 1992,-61 2022,-61 2022,-61 2028,-61 2034,-67 2034,-73 2034,-73 2034,-85 2034,-85 2034,-91 2028,-97 2022,-97\"/>\n<text text-anchor=\"middle\" x=\"2007\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_conv&#45;&gt;layer3_norm -->\n<g id=\"edge23\" class=\"edge\"><title>layer3_conv&#45;&gt;layer3_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1944.4,-79C1952.39,-79 1961.31,-79 1969.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1969.92,-82.5001 1979.92,-79 1969.92,-75.5001 1969.92,-82.5001\"/>\n</g>\n<!-- layer3_act -->\n<g id=\"node23\" class=\"node\"><title>layer3_act</title>\n<g id=\"a_node23\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2112,-97C2112,-97 2082,-97 2082,-97 2076,-97 2070,-91 2070,-85 2070,-85 2070,-73 2070,-73 2070,-67 2076,-61 2082,-61 2082,-61 2112,-61 2112,-61 2118,-61 2124,-67 2124,-73 2124,-73 2124,-85 2124,-85 2124,-91 2118,-97 2112,-97\"/>\n<text text-anchor=\"middle\" x=\"2097\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_norm&#45;&gt;layer3_act -->\n<g id=\"edge24\" class=\"edge\"><title>layer3_norm&#45;&gt;layer3_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2034.4,-79C2042.39,-79 2051.31,-79 2059.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2059.92,-82.5001 2069.92,-79 2059.92,-75.5001 2059.92,-82.5001\"/>\n</g>\n<!-- layer3_pool -->\n<g id=\"node24\" class=\"node\"><title>layer3_pool</title>\n<g id=\"a_node24\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M2202,-97C2202,-97 2172,-97 2172,-97 2166,-97 2160,-91 2160,-85 2160,-85 2160,-73 2160,-73 2160,-67 2166,-61 2172,-61 2172,-61 2202,-61 2202,-61 2208,-61 2214,-67 2214,-73 2214,-73 2214,-85 2214,-85 2214,-91 2208,-97 2202,-97\"/>\n<text text-anchor=\"middle\" x=\"2187\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_act&#45;&gt;layer3_pool -->\n<g id=\"edge25\" class=\"edge\"><title>layer3_act&#45;&gt;layer3_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2124.4,-79C2132.39,-79 2141.31,-79 2149.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2149.92,-82.5001 2159.92,-79 2149.92,-75.5001 2149.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_in -->\n<g id=\"node25\" class=\"node\"><title>layer3_residual_in</title>\n<g id=\"a_node25\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2292,-97C2292,-97 2262,-97 2262,-97 2256,-97 2250,-91 2250,-85 2250,-85 2250,-73 2250,-73 2250,-67 2256,-61 2262,-61 2262,-61 2292,-61 2292,-61 2298,-61 2304,-67 2304,-73 2304,-73 2304,-85 2304,-85 2304,-91 2298,-97 2292,-97\"/>\n<text text-anchor=\"middle\" x=\"2277\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer3_pool&#45;&gt;layer3_residual_in -->\n<g id=\"edge26\" class=\"edge\"><title>layer3_pool&#45;&gt;layer3_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2214.4,-79C2222.39,-79 2231.31,-79 2239.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2239.92,-82.5001 2249.92,-79 2239.92,-75.5001 2239.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_res1_conv -->\n<g id=\"node26\" class=\"node\"><title>layer3_residual_res1_conv</title>\n<g id=\"a_node26\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2382,-68C2382,-68 2352,-68 2352,-68 2346,-68 2340,-62 2340,-56 2340,-56 2340,-44 2340,-44 2340,-38 2346,-32 2352,-32 2352,-32 2382,-32 2382,-32 2388,-32 2394,-38 2394,-44 2394,-44 2394,-56 2394,-56 2394,-62 2388,-68 2382,-68\"/>\n<text text-anchor=\"middle\" x=\"2367\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_res1_conv -->\n<g id=\"edge27\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.4,-70.299C2312.57,-67.608 2321.7,-64.5979 2330.38,-61.7368\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2331.52,-65.049 2339.92,-58.5949 2329.33,-58.4007 2331.52,-65.049\"/>\n</g>\n<!-- layer3_residual_add -->\n<g id=\"node33\" class=\"node\"><title>layer3_residual_add</title>\n<g id=\"a_node33\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M3012,-93C3012,-93 2982,-93 2982,-93 2976,-93 2970,-87 2970,-81 2970,-81 2970,-69 2970,-69 2970,-63 2976,-57 2982,-57 2982,-57 3012,-57 3012,-57 3018,-57 3024,-63 3024,-69 3024,-69 3024,-81 3024,-81 3024,-87 3018,-93 3012,-93\"/>\n<text text-anchor=\"middle\" x=\"2997\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_add -->\n<g id=\"edge34\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.22,-93.0359C2312.83,-96.9904 2322.61,-100.808 2332,-103 2385.71,-115.535 2400.84,-108 2456,-108 2456,-108 2456,-108 2818,-108 2867.77,-108 2924.08,-95.3711 2959.8,-85.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2961.04,-89.0265 2969.74,-82.9884 2959.17,-82.2796 2961.04,-89.0265\"/>\n</g>\n<!-- layer3_residual_res1_norm -->\n<g id=\"node27\" class=\"node\"><title>layer3_residual_res1_norm</title>\n<g id=\"a_node27\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2472,-68C2472,-68 2442,-68 2442,-68 2436,-68 2430,-62 2430,-56 2430,-56 2430,-44 2430,-44 2430,-38 2436,-32 2442,-32 2442,-32 2472,-32 2472,-32 2478,-32 2484,-38 2484,-44 2484,-44 2484,-56 2484,-56 2484,-62 2478,-68 2472,-68\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm -->\n<g id=\"edge28\" class=\"edge\"><title>layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2394.4,-50C2402.39,-50 2411.31,-50 2419.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2419.92,-53.5001 2429.92,-50 2419.92,-46.5001 2419.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res1_act -->\n<g id=\"node28\" class=\"node\"><title>layer3_residual_res1_act</title>\n<g id=\"a_node28\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2562,-68C2562,-68 2532,-68 2532,-68 2526,-68 2520,-62 2520,-56 2520,-56 2520,-44 2520,-44 2520,-38 2526,-32 2532,-32 2532,-32 2562,-32 2562,-32 2568,-32 2574,-38 2574,-44 2574,-44 2574,-56 2574,-56 2574,-62 2568,-68 2562,-68\"/>\n<text text-anchor=\"middle\" x=\"2547\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act -->\n<g id=\"edge29\" class=\"edge\"><title>layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2484.4,-50C2492.39,-50 2501.31,-50 2509.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2509.92,-53.5001 2519.92,-50 2509.92,-46.5001 2509.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_conv -->\n<g id=\"node29\" class=\"node\"><title>layer3_residual_res2_conv</title>\n<g id=\"a_node29\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2652,-68C2652,-68 2622,-68 2622,-68 2616,-68 2610,-62 2610,-56 2610,-56 2610,-44 2610,-44 2610,-38 2616,-32 2622,-32 2622,-32 2652,-32 2652,-32 2658,-32 2664,-38 2664,-44 2664,-44 2664,-56 2664,-56 2664,-62 2658,-68 2652,-68\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv -->\n<g id=\"edge30\" class=\"edge\"><title>layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2574.4,-50C2582.39,-50 2591.31,-50 2599.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2599.92,-53.5001 2609.92,-50 2599.92,-46.5001 2599.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_norm -->\n<g id=\"node30\" class=\"node\"><title>layer3_residual_res2_norm</title>\n<g id=\"a_node30\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2742,-68C2742,-68 2712,-68 2712,-68 2706,-68 2700,-62 2700,-56 2700,-56 2700,-44 2700,-44 2700,-38 2706,-32 2712,-32 2712,-32 2742,-32 2742,-32 2748,-32 2754,-38 2754,-44 2754,-44 2754,-56 2754,-56 2754,-62 2748,-68 2742,-68\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm -->\n<g id=\"edge31\" class=\"edge\"><title>layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2664.4,-50C2672.39,-50 2681.31,-50 2689.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2689.92,-53.5001 2699.92,-50 2689.92,-46.5001 2689.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_act -->\n<g id=\"node31\" class=\"node\"><title>layer3_residual_res2_act</title>\n<g id=\"a_node31\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2832,-68C2832,-68 2802,-68 2802,-68 2796,-68 2790,-62 2790,-56 2790,-56 2790,-44 2790,-44 2790,-38 2796,-32 2802,-32 2802,-32 2832,-32 2832,-32 2838,-32 2844,-38 2844,-44 2844,-44 2844,-56 2844,-56 2844,-62 2838,-68 2832,-68\"/>\n<text text-anchor=\"middle\" x=\"2817\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act -->\n<g id=\"edge32\" class=\"edge\"><title>layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2754.4,-50C2762.39,-50 2771.31,-50 2779.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2779.92,-53.5001 2789.92,-50 2779.92,-46.5001 2779.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_out -->\n<g id=\"node32\" class=\"node\"><title>layer3_residual_out</title>\n<g id=\"a_node32\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2922,-78C2922,-78 2892,-78 2892,-78 2886,-78 2880,-72 2880,-66 2880,-66 2880,-54 2880,-54 2880,-48 2886,-42 2892,-42 2892,-42 2922,-42 2922,-42 2928,-42 2934,-48 2934,-54 2934,-54 2934,-66 2934,-66 2934,-72 2928,-78 2922,-78\"/>\n<text text-anchor=\"middle\" x=\"2907\" y=\"-56.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_act&#45;&gt;layer3_residual_out -->\n<g id=\"edge33\" class=\"edge\"><title>layer3_residual_res2_act&#45;&gt;layer3_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2844.4,-53.0003C2852.39,-53.9083 2861.31,-54.9217 2869.82,-55.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2869.59,-59.3847 2879.92,-57.0362 2870.38,-52.4294 2869.59,-59.3847\"/>\n</g>\n<!-- layer3_residual_out&#45;&gt;layer3_residual_add -->\n<g id=\"edge35\" class=\"edge\"><title>layer3_residual_out&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2934.4,-64.5005C2942.39,-65.8625 2951.31,-67.3825 2959.82,-68.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2959.47,-72.3242 2969.92,-70.5544 2960.65,-65.4237 2959.47,-72.3242\"/>\n</g>\n<!-- pool -->\n<g id=\"node34\" class=\"node\"><title>pool</title>\n<g id=\"a_node34\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 4}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M3102,-93C3102,-93 3072,-93 3072,-93 3066,-93 3060,-87 3060,-81 3060,-81 3060,-69 3060,-69 3060,-63 3066,-57 3072,-57 3072,-57 3102,-57 3102,-57 3108,-57 3114,-63 3114,-69 3114,-69 3114,-81 3114,-81 3114,-87 3108,-93 3102,-93\"/>\n<text text-anchor=\"middle\" x=\"3087\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_add&#45;&gt;pool -->\n<g id=\"edge36\" class=\"edge\"><title>layer3_residual_add&#45;&gt;pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3024.4,-75C3032.39,-75 3041.31,-75 3049.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3049.92,-78.5001 3059.92,-75 3049.92,-71.5001 3049.92,-78.5001\"/>\n</g>\n<!-- classifier_flatten -->\n<g id=\"node35\" class=\"node\"><title>classifier_flatten</title>\n<g id=\"a_node35\"><a xlink:title=\"&lt;class &#39;__main__.Flatten&#39;&gt; {}\">\n<path fill=\"#b3de69\" stroke=\"black\" d=\"M3192,-93C3192,-93 3162,-93 3162,-93 3156,-93 3150,-87 3150,-81 3150,-81 3150,-69 3150,-69 3150,-63 3156,-57 3162,-57 3162,-57 3192,-57 3192,-57 3198,-57 3204,-63 3204,-69 3204,-69 3204,-81 3204,-81 3204,-87 3198,-93 3192,-93\"/>\n<text text-anchor=\"middle\" x=\"3177\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">flatten</text>\n</a>\n</g>\n</g>\n<!-- pool&#45;&gt;classifier_flatten -->\n<g id=\"edge37\" class=\"edge\"><title>pool&#45;&gt;classifier_flatten</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3114.4,-75C3122.39,-75 3131.31,-75 3139.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3139.92,-78.5001 3149.92,-75 3139.92,-71.5001 3139.92,-78.5001\"/>\n</g>\n<!-- classifier_conv -->\n<g id=\"node36\" class=\"node\"><title>classifier_conv</title>\n<g id=\"a_node36\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.linear.Linear&#39;&gt; {&#39;in_features&#39;: 512, &#39;out_features&#39;: 10, &#39;bias&#39;: False}\">\n<path fill=\"#fccde5\" stroke=\"black\" d=\"M3282,-93C3282,-93 3252,-93 3252,-93 3246,-93 3240,-87 3240,-81 3240,-81 3240,-69 3240,-69 3240,-63 3246,-57 3252,-57 3252,-57 3282,-57 3282,-57 3288,-57 3294,-63 3294,-69 3294,-69 3294,-81 3294,-81 3294,-87 3288,-93 3282,-93\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- classifier_flatten&#45;&gt;classifier_conv -->\n<g id=\"edge38\" class=\"edge\"><title>classifier_flatten&#45;&gt;classifier_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3204.4,-75C3212.39,-75 3221.31,-75 3229.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3229.92,-78.5001 3239.92,-75 3229.92,-71.5001 3229.92,-78.5001\"/>\n</g>\n<!-- classifier_scale -->\n<g id=\"node37\" class=\"node\"><title>classifier_scale</title>\n<g id=\"a_node37\"><a xlink:title=\"&lt;class &#39;__main__.Mul&#39;&gt; {&#39;weight&#39;: 0.125}\">\n<path fill=\"#bc80bd\" stroke=\"black\" d=\"M3372,-93C3372,-93 3342,-93 3342,-93 3336,-93 3330,-87 3330,-81 3330,-81 3330,-69 3330,-69 3330,-63 3336,-57 3342,-57 3342,-57 3372,-57 3372,-57 3378,-57 3384,-63 3384,-69 3384,-69 3384,-81 3384,-81 3384,-87 3378,-93 3372,-93\"/>\n<text text-anchor=\"middle\" x=\"3357\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">scale</text>\n</a>\n</g>\n</g>\n<!-- classifier_conv&#45;&gt;classifier_scale -->\n<g id=\"edge39\" class=\"edge\"><title>classifier_conv&#45;&gt;classifier_scale</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3294.4,-75C3302.39,-75 3311.31,-75 3319.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3319.92,-78.5001 3329.92,-75 3319.92,-71.5001 3319.92,-78.5001\"/>\n</g>\n<!-- logits -->\n<g id=\"node38\" class=\"node\"><title>logits</title>\n<g id=\"a_node38\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M3462,-93C3462,-93 3432,-93 3432,-93 3426,-93 3420,-87 3420,-81 3420,-81 3420,-69 3420,-69 3420,-63 3426,-57 3432,-57 3432,-57 3462,-57 3462,-57 3468,-57 3474,-63 3474,-69 3474,-69 3474,-81 3474,-81 3474,-87 3468,-93 3462,-93\"/>\n<text text-anchor=\"middle\" x=\"3447\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">logits</text>\n</a>\n</g>\n</g>\n<!-- classifier_scale&#45;&gt;logits -->\n<g id=\"edge40\" class=\"edge\"><title>classifier_scale&#45;&gt;logits</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3384.4,-75C3392.39,-75 3401.31,-75 3409.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3409.92,-78.5001 3419.92,-75 3409.92,-71.5001 3409.92,-78.5001\"/>\n</g>\n<!-- input -->\n<g id=\"node39\" class=\"node\"><title>input</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-92C42,-92 12,-92 12,-92 6,-92 0,-86 0,-80 0,-80 0,-68 0,-68 0,-62 6,-56 12,-56 12,-56 42,-56 42,-56 48,-56 54,-62 54,-68 54,-68 54,-80 54,-80 54,-86 48,-92 42,-92\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">input</text>\n</g>\n<!-- input&#45;&gt;prep_conv -->\n<g id=\"edge1\" class=\"edge\"><title>input&#45;&gt;prep_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4029,-74C62.3932,-74 71.3106,-74 79.8241,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"79.919,-77.5001 89.919,-74 79.919,-70.5001 79.919,-77.5001\"/>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3v3IsSpij-PL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 24, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, 5, epochs], [0.0, 0.4, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "model = build_model(baseline_net, x_ent_loss)  \n",
        "warmup_cudnn(model, next(iter(train_batches(batch_size, transforms))))\n",
        "logs, state, timer = Table(), {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize)\n",
        "for epoch in range(epochs):\n",
        "    logs.append(union({'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "10lxhHspp79n",
        "colab_type": "text"
      },
      "source": [
        "Total training time (including the negligible time spent on preprocessing) is under 70s, moving us up two places to fourth on the leaderboard!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZiaIqUQKWHgT",
        "colab_type": "text"
      },
      "source": [
        "#### Aside: mixed precision training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8FoPeZCcWmYR",
        "colab_type": "text"
      },
      "source": [
        "In our original DAWNBench submission and in the code above, we simply converted the model to float16 without all the niceties of so-called [mixed precision training](https://arxiv.org/abs/1710.03740) although we include a basic sort of 'loss scaling' by summing rather than averaging losses in a batch. It is straightforward to implement proper mixed precision training but this adds about a second to overall training time and we found it to have no effect on final accuracy, so we continue to do without it below. For completeness, here is a simple implementation which might be useful elsewhere: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qvk0e9AfWMjp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "FP32_PARAMS = 'fp32_params'\n",
        "\n",
        "def copy_grads_to_master(batch, state):\n",
        "    if not batch: return\n",
        "    for float_param, param in zip(state[FP32_PARAMS], trainable_params(state[MODEL]).values()):\n",
        "        if float_param.grad is None: \n",
        "            float_param.grad = float_param.data.new(*float_param.data.size())\n",
        "        float_param.grad.data.copy_(param.grad.data)    \n",
        "    \n",
        "def update_params_from_master(batch, state):\n",
        "    if not batch: return\n",
        "    for float_param, param in zip(state[FP32_PARAMS], trainable_params(state[MODEL]).values()):\n",
        "        param.data.copy_(float_param)\n",
        "    \n",
        "train_steps_mixed_precision = (forward(training_mode=True),  log_activations(('loss', 'acc')), backward(torch.float32), copy_grads_to_master, opt_steps, update_params_from_master)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Q9-IXXGYJFL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size =24, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, 5, epochs], [0.0, 0.4, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "model = build_model(network(), x_ent_loss)\n",
        "fp32_params = [v.clone().to(torch.float32) for v in trainable_params(model).values()]\n",
        "logs, state, timer = Table(), {MODEL: model, OPTS: [SGD(fp32_params, opt_params)], FP32_PARAMS: fp32_params}, Timer(torch.cuda.synchronize)\n",
        "for epoch in range(epochs):\n",
        "    logs.append(union({'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size), train_steps=train_steps_mixed_precision)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-RpWD1rHwNoz",
        "colab_type": "text"
      },
      "source": [
        "### Moving max-pool layers (64s)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTwu1qXPwVCI",
        "colab_type": "text"
      },
      "source": [
        "Max-pooling commutes with a monotonic-increasing activation function such as ReLU. It should be more efficient to apply pooling first. This is the sort of thing a friendly compiler might do for you, but for now let's switch the order by hand:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vHSpmXRd-Acj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        },
        "outputId": "f5e673a2-23e9-411f-bff5-e454588428a4"
      },
      "source": [
        "conv_pool_block_opt = lambda c_in, c_out: reorder(conv_pool_block(c_in, c_out), ('conv', 'norm', 'pool', 'act'))\n",
        "show(network(conv_pool_block=conv_pool_block_opt))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<__main__.DotGraph at 0x7f4140b19ac8>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n -->\n<!-- Title: G Pages: 1 -->\n<svg width=\"1080pt\" height=\"59pt\"\n viewBox=\"0.00 0.00 1080.00 59.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(0.310167 0.310167) rotate(0) translate(4 187)\">\n<title>G</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-187 3478,-187 3478,4 -4,4\"/>\n<g id=\"clust1\" class=\"cluster\"><title>cluster_prep</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M94,-48C94,-48 320,-48 320,-48 326,-48 332,-54 332,-60 332,-60 332,-111 332,-111 332,-117 326,-123 320,-123 320,-123 94,-123 94,-123 88,-123 82,-117 82,-111 82,-111 82,-60 82,-60 82,-54 88,-48 94,-48\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-107.8\" font-family=\"Times,serif\" font-size=\"14.00\">prep</text>\n</g>\n<g id=\"clust2\" class=\"cluster\"><title>cluster_layer1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M364,-12C364,-12 1498,-12 1498,-12 1504,-12 1510,-18 1510,-24 1510,-24 1510,-163 1510,-163 1510,-169 1504,-175 1498,-175 1498,-175 364,-175 364,-175 358,-175 352,-169 352,-163 352,-163 352,-24 352,-24 352,-18 358,-12 364,-12\"/>\n<text text-anchor=\"middle\" x=\"931\" y=\"-159.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer1</text>\n</g>\n<g id=\"clust3\" class=\"cluster\"><title>cluster_layer1_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M724,-20C724,-20 1490,-20 1490,-20 1496,-20 1502,-26 1502,-32 1502,-32 1502,-132 1502,-132 1502,-138 1496,-144 1490,-144 1490,-144 724,-144 724,-144 718,-144 712,-138 712,-132 712,-132 712,-32 712,-32 712,-26 718,-20 724,-20\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-128.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust4\" class=\"cluster\"><title>cluster_layer1_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M814,-28C814,-28 1040,-28 1040,-28 1046,-28 1052,-34 1052,-40 1052,-40 1052,-91 1052,-91 1052,-97 1046,-103 1040,-103 1040,-103 814,-103 814,-103 808,-103 802,-97 802,-91 802,-91 802,-40 802,-40 802,-34 808,-28 814,-28\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust5\" class=\"cluster\"><title>cluster_layer1_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1084,-28C1084,-28 1310,-28 1310,-28 1316,-28 1322,-34 1322,-40 1322,-40 1322,-91 1322,-91 1322,-97 1316,-103 1310,-103 1310,-103 1084,-103 1084,-103 1078,-103 1072,-97 1072,-91 1072,-91 1072,-40 1072,-40 1072,-34 1078,-28 1084,-28\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust6\" class=\"cluster\"><title>cluster_layer2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1534,-53C1534,-53 1850,-53 1850,-53 1856,-53 1862,-59 1862,-65 1862,-65 1862,-116 1862,-116 1862,-122 1856,-128 1850,-128 1850,-128 1534,-128 1534,-128 1528,-128 1522,-122 1522,-116 1522,-116 1522,-65 1522,-65 1522,-59 1528,-53 1534,-53\"/>\n<text text-anchor=\"middle\" x=\"1692\" y=\"-112.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer2</text>\n</g>\n<g id=\"clust7\" class=\"cluster\"><title>cluster_layer3</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1894,-8C1894,-8 3028,-8 3028,-8 3034,-8 3040,-14 3040,-20 3040,-20 3040,-159 3040,-159 3040,-165 3034,-171 3028,-171 3028,-171 1894,-171 1894,-171 1888,-171 1882,-165 1882,-159 1882,-159 1882,-20 1882,-20 1882,-14 1888,-8 1894,-8\"/>\n<text text-anchor=\"middle\" x=\"2461\" y=\"-155.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer3</text>\n</g>\n<g id=\"clust8\" class=\"cluster\"><title>cluster_layer3_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2254,-16C2254,-16 3020,-16 3020,-16 3026,-16 3032,-22 3032,-28 3032,-28 3032,-128 3032,-128 3032,-134 3026,-140 3020,-140 3020,-140 2254,-140 2254,-140 2248,-140 2242,-134 2242,-128 2242,-128 2242,-28 2242,-28 2242,-22 2248,-16 2254,-16\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust9\" class=\"cluster\"><title>cluster_layer3_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2344,-24C2344,-24 2570,-24 2570,-24 2576,-24 2582,-30 2582,-36 2582,-36 2582,-87 2582,-87 2582,-93 2576,-99 2570,-99 2570,-99 2344,-99 2344,-99 2338,-99 2332,-93 2332,-87 2332,-87 2332,-36 2332,-36 2332,-30 2338,-24 2344,-24\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust10\" class=\"cluster\"><title>cluster_layer3_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2614,-24C2614,-24 2840,-24 2840,-24 2846,-24 2852,-30 2852,-36 2852,-36 2852,-87 2852,-87 2852,-93 2846,-99 2840,-99 2840,-99 2614,-99 2614,-99 2608,-99 2602,-93 2602,-87 2602,-87 2602,-36 2602,-36 2602,-30 2608,-24 2614,-24\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust11\" class=\"cluster\"><title>cluster_classifier</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M3154,-49C3154,-49 3380,-49 3380,-49 3386,-49 3392,-55 3392,-61 3392,-61 3392,-112 3392,-112 3392,-118 3386,-124 3380,-124 3380,-124 3154,-124 3154,-124 3148,-124 3142,-118 3142,-112 3142,-112 3142,-61 3142,-61 3142,-55 3148,-49 3154,-49\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-108.8\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n</g>\n<!-- prep_conv -->\n<g id=\"node1\" class=\"node\"><title>prep_conv</title>\n<g id=\"a_node1\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 3, &#39;out_channels&#39;: 64, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M132,-92C132,-92 102,-92 102,-92 96,-92 90,-86 90,-80 90,-80 90,-68 90,-68 90,-62 96,-56 102,-56 102,-56 132,-56 132,-56 138,-56 144,-62 144,-68 144,-68 144,-80 144,-80 144,-86 138,-92 132,-92\"/>\n<text text-anchor=\"middle\" x=\"117\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_norm -->\n<g id=\"node2\" class=\"node\"><title>prep_norm</title>\n<g id=\"a_node2\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 64}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M222,-92C222,-92 192,-92 192,-92 186,-92 180,-86 180,-80 180,-80 180,-68 180,-68 180,-62 186,-56 192,-56 192,-56 222,-56 222,-56 228,-56 234,-62 234,-68 234,-68 234,-80 234,-80 234,-86 228,-92 222,-92\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- prep_conv&#45;&gt;prep_norm -->\n<g id=\"edge2\" class=\"edge\"><title>prep_conv&#45;&gt;prep_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M144.403,-74C152.393,-74 161.311,-74 169.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"169.919,-77.5001 179.919,-74 169.919,-70.5001 169.919,-77.5001\"/>\n</g>\n<!-- prep_act -->\n<g id=\"node3\" class=\"node\"><title>prep_act</title>\n<g id=\"a_node3\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M312,-92C312,-92 282,-92 282,-92 276,-92 270,-86 270,-80 270,-80 270,-68 270,-68 270,-62 276,-56 282,-56 282,-56 312,-56 312,-56 318,-56 324,-62 324,-68 324,-68 324,-80 324,-80 324,-86 318,-92 312,-92\"/>\n<text text-anchor=\"middle\" x=\"297\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- prep_norm&#45;&gt;prep_act -->\n<g id=\"edge3\" class=\"edge\"><title>prep_norm&#45;&gt;prep_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M234.403,-74C242.393,-74 251.311,-74 259.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"259.919,-77.5001 269.919,-74 259.919,-70.5001 259.919,-77.5001\"/>\n</g>\n<!-- layer1_conv -->\n<g id=\"node4\" class=\"node\"><title>layer1_conv</title>\n<g id=\"a_node4\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 64, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M402,-92C402,-92 372,-92 372,-92 366,-92 360,-86 360,-80 360,-80 360,-68 360,-68 360,-62 366,-56 372,-56 372,-56 402,-56 402,-56 408,-56 414,-62 414,-68 414,-68 414,-80 414,-80 414,-86 408,-92 402,-92\"/>\n<text text-anchor=\"middle\" x=\"387\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_act&#45;&gt;layer1_conv -->\n<g id=\"edge4\" class=\"edge\"><title>prep_act&#45;&gt;layer1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M324.403,-74C332.393,-74 341.311,-74 349.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"349.919,-77.5001 359.919,-74 349.919,-70.5001 349.919,-77.5001\"/>\n</g>\n<!-- layer1_norm -->\n<g id=\"node5\" class=\"node\"><title>layer1_norm</title>\n<g id=\"a_node5\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M492,-92C492,-92 462,-92 462,-92 456,-92 450,-86 450,-80 450,-80 450,-68 450,-68 450,-62 456,-56 462,-56 462,-56 492,-56 492,-56 498,-56 504,-62 504,-68 504,-68 504,-80 504,-80 504,-86 498,-92 492,-92\"/>\n<text text-anchor=\"middle\" x=\"477\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_conv&#45;&gt;layer1_norm -->\n<g id=\"edge5\" class=\"edge\"><title>layer1_conv&#45;&gt;layer1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M414.403,-74C422.393,-74 431.311,-74 439.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"439.919,-77.5001 449.919,-74 439.919,-70.5001 439.919,-77.5001\"/>\n</g>\n<!-- layer1_pool -->\n<g id=\"node6\" class=\"node\"><title>layer1_pool</title>\n<g id=\"a_node6\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M582,-92C582,-92 552,-92 552,-92 546,-92 540,-86 540,-80 540,-80 540,-68 540,-68 540,-62 546,-56 552,-56 552,-56 582,-56 582,-56 588,-56 594,-62 594,-68 594,-68 594,-80 594,-80 594,-86 588,-92 582,-92\"/>\n<text text-anchor=\"middle\" x=\"567\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer1_norm&#45;&gt;layer1_pool -->\n<g id=\"edge6\" class=\"edge\"><title>layer1_norm&#45;&gt;layer1_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M504.403,-74C512.393,-74 521.311,-74 529.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"529.919,-77.5001 539.919,-74 529.919,-70.5001 529.919,-77.5001\"/>\n</g>\n<!-- layer1_act -->\n<g id=\"node7\" class=\"node\"><title>layer1_act</title>\n<g id=\"a_node7\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M672,-92C672,-92 642,-92 642,-92 636,-92 630,-86 630,-80 630,-80 630,-68 630,-68 630,-62 636,-56 642,-56 642,-56 672,-56 672,-56 678,-56 684,-62 684,-68 684,-68 684,-80 684,-80 684,-86 678,-92 672,-92\"/>\n<text text-anchor=\"middle\" x=\"657\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_pool&#45;&gt;layer1_act -->\n<g id=\"edge7\" class=\"edge\"><title>layer1_pool&#45;&gt;layer1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M594.403,-74C602.393,-74 611.311,-74 619.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"619.919,-77.5001 629.919,-74 619.919,-70.5001 619.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_in -->\n<g id=\"node8\" class=\"node\"><title>layer1_residual_in</title>\n<g id=\"a_node8\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M762,-92C762,-92 732,-92 732,-92 726,-92 720,-86 720,-80 720,-80 720,-68 720,-68 720,-62 726,-56 732,-56 732,-56 762,-56 762,-56 768,-56 774,-62 774,-68 774,-68 774,-80 774,-80 774,-86 768,-92 762,-92\"/>\n<text text-anchor=\"middle\" x=\"747\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer1_act&#45;&gt;layer1_residual_in -->\n<g id=\"edge8\" class=\"edge\"><title>layer1_act&#45;&gt;layer1_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M684.403,-74C692.393,-74 701.311,-74 709.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"709.919,-77.5001 719.919,-74 709.919,-70.5001 709.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_res1_conv -->\n<g id=\"node9\" class=\"node\"><title>layer1_residual_res1_conv</title>\n<g id=\"a_node9\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M852,-72C852,-72 822,-72 822,-72 816,-72 810,-66 810,-60 810,-60 810,-48 810,-48 810,-42 816,-36 822,-36 822,-36 852,-36 852,-36 858,-36 864,-42 864,-48 864,-48 864,-60 864,-60 864,-66 858,-72 852,-72\"/>\n<text text-anchor=\"middle\" x=\"837\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_res1_conv -->\n<g id=\"edge9\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M774.403,-67.9993C782.481,-66.1634 791.507,-64.1121 800.105,-62.158\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"800.943,-65.5568 809.919,-59.9275 799.392,-58.7308 800.943,-65.5568\"/>\n</g>\n<!-- layer1_residual_add -->\n<g id=\"node16\" class=\"node\"><title>layer1_residual_add</title>\n<g id=\"a_node16\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M1482,-97C1482,-97 1452,-97 1452,-97 1446,-97 1440,-91 1440,-85 1440,-85 1440,-73 1440,-73 1440,-67 1446,-61 1452,-61 1452,-61 1482,-61 1482,-61 1488,-61 1494,-67 1494,-73 1494,-73 1494,-85 1494,-85 1494,-91 1488,-97 1482,-97\"/>\n<text text-anchor=\"middle\" x=\"1467\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_add -->\n<g id=\"edge16\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M771.737,-92.1018C780.735,-97.9639 791.381,-103.78 802,-107 854.782,-123.006 870.844,-112 926,-112 926,-112 926,-112 1288,-112 1337.77,-112 1394.08,-99.3711 1429.8,-89.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1431.04,-93.0265 1439.74,-86.9884 1429.17,-86.2796 1431.04,-93.0265\"/>\n</g>\n<!-- layer1_residual_res1_norm -->\n<g id=\"node10\" class=\"node\"><title>layer1_residual_res1_norm</title>\n<g id=\"a_node10\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M942,-72C942,-72 912,-72 912,-72 906,-72 900,-66 900,-60 900,-60 900,-48 900,-48 900,-42 906,-36 912,-36 912,-36 942,-36 942,-36 948,-36 954,-42 954,-48 954,-48 954,-60 954,-60 954,-66 948,-72 942,-72\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm -->\n<g id=\"edge10\" class=\"edge\"><title>layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M864.403,-54C872.393,-54 881.311,-54 889.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"889.919,-57.5001 899.919,-54 889.919,-50.5001 889.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res1_act -->\n<g id=\"node11\" class=\"node\"><title>layer1_residual_res1_act</title>\n<g id=\"a_node11\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1032,-72C1032,-72 1002,-72 1002,-72 996,-72 990,-66 990,-60 990,-60 990,-48 990,-48 990,-42 996,-36 1002,-36 1002,-36 1032,-36 1032,-36 1038,-36 1044,-42 1044,-48 1044,-48 1044,-60 1044,-60 1044,-66 1038,-72 1032,-72\"/>\n<text text-anchor=\"middle\" x=\"1017\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act -->\n<g id=\"edge11\" class=\"edge\"><title>layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M954.403,-54C962.393,-54 971.311,-54 979.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"979.919,-57.5001 989.919,-54 979.919,-50.5001 979.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_conv -->\n<g id=\"node12\" class=\"node\"><title>layer1_residual_res2_conv</title>\n<g id=\"a_node12\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1122,-72C1122,-72 1092,-72 1092,-72 1086,-72 1080,-66 1080,-60 1080,-60 1080,-48 1080,-48 1080,-42 1086,-36 1092,-36 1092,-36 1122,-36 1122,-36 1128,-36 1134,-42 1134,-48 1134,-48 1134,-60 1134,-60 1134,-66 1128,-72 1122,-72\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv -->\n<g id=\"edge12\" class=\"edge\"><title>layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1044.4,-54C1052.39,-54 1061.31,-54 1069.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1069.92,-57.5001 1079.92,-54 1069.92,-50.5001 1069.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_norm -->\n<g id=\"node13\" class=\"node\"><title>layer1_residual_res2_norm</title>\n<g id=\"a_node13\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M1212,-72C1212,-72 1182,-72 1182,-72 1176,-72 1170,-66 1170,-60 1170,-60 1170,-48 1170,-48 1170,-42 1176,-36 1182,-36 1182,-36 1212,-36 1212,-36 1218,-36 1224,-42 1224,-48 1224,-48 1224,-60 1224,-60 1224,-66 1218,-72 1212,-72\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm -->\n<g id=\"edge13\" class=\"edge\"><title>layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1134.4,-54C1142.39,-54 1151.31,-54 1159.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1159.92,-57.5001 1169.92,-54 1159.92,-50.5001 1159.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_act -->\n<g id=\"node14\" class=\"node\"><title>layer1_residual_res2_act</title>\n<g id=\"a_node14\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1302,-72C1302,-72 1272,-72 1272,-72 1266,-72 1260,-66 1260,-60 1260,-60 1260,-48 1260,-48 1260,-42 1266,-36 1272,-36 1272,-36 1302,-36 1302,-36 1308,-36 1314,-42 1314,-48 1314,-48 1314,-60 1314,-60 1314,-66 1308,-72 1302,-72\"/>\n<text text-anchor=\"middle\" x=\"1287\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act -->\n<g id=\"edge14\" class=\"edge\"><title>layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1224.4,-54C1232.39,-54 1241.31,-54 1249.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1249.92,-57.5001 1259.92,-54 1249.92,-50.5001 1249.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_out -->\n<g id=\"node15\" class=\"node\"><title>layer1_residual_out</title>\n<g id=\"a_node15\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M1392,-82C1392,-82 1362,-82 1362,-82 1356,-82 1350,-76 1350,-70 1350,-70 1350,-58 1350,-58 1350,-52 1356,-46 1362,-46 1362,-46 1392,-46 1392,-46 1398,-46 1404,-52 1404,-58 1404,-58 1404,-70 1404,-70 1404,-76 1398,-82 1392,-82\"/>\n<text text-anchor=\"middle\" x=\"1377\" y=\"-60.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_act&#45;&gt;layer1_residual_out -->\n<g id=\"edge15\" class=\"edge\"><title>layer1_residual_res2_act&#45;&gt;layer1_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1314.4,-57.0003C1322.39,-57.9083 1331.31,-58.9217 1339.82,-59.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1339.59,-63.3847 1349.92,-61.0362 1340.38,-56.4294 1339.59,-63.3847\"/>\n</g>\n<!-- layer1_residual_out&#45;&gt;layer1_residual_add -->\n<g id=\"edge17\" class=\"edge\"><title>layer1_residual_out&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1404.4,-68.5005C1412.39,-69.8625 1421.31,-71.3825 1429.82,-72.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1429.47,-76.3242 1439.92,-74.5544 1430.65,-69.4237 1429.47,-76.3242\"/>\n</g>\n<!-- layer2_conv -->\n<g id=\"node17\" class=\"node\"><title>layer2_conv</title>\n<g id=\"a_node17\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 256, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1572,-97C1572,-97 1542,-97 1542,-97 1536,-97 1530,-91 1530,-85 1530,-85 1530,-73 1530,-73 1530,-67 1536,-61 1542,-61 1542,-61 1572,-61 1572,-61 1578,-61 1584,-67 1584,-73 1584,-73 1584,-85 1584,-85 1584,-91 1578,-97 1572,-97\"/>\n<text text-anchor=\"middle\" x=\"1557\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_add&#45;&gt;layer2_conv -->\n<g id=\"edge18\" class=\"edge\"><title>layer1_residual_add&#45;&gt;layer2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1494.4,-79C1502.39,-79 1511.31,-79 1519.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1519.92,-82.5001 1529.92,-79 1519.92,-75.5001 1519.92,-82.5001\"/>\n</g>\n<!-- layer2_norm -->\n<g id=\"node18\" class=\"node\"><title>layer2_norm</title>\n<g id=\"a_node18\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 256}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M1662,-97C1662,-97 1632,-97 1632,-97 1626,-97 1620,-91 1620,-85 1620,-85 1620,-73 1620,-73 1620,-67 1626,-61 1632,-61 1632,-61 1662,-61 1662,-61 1668,-61 1674,-67 1674,-73 1674,-73 1674,-85 1674,-85 1674,-91 1668,-97 1662,-97\"/>\n<text text-anchor=\"middle\" x=\"1647\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer2_conv&#45;&gt;layer2_norm -->\n<g id=\"edge19\" class=\"edge\"><title>layer2_conv&#45;&gt;layer2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1584.4,-79C1592.39,-79 1601.31,-79 1609.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1609.92,-82.5001 1619.92,-79 1609.92,-75.5001 1609.92,-82.5001\"/>\n</g>\n<!-- layer2_pool -->\n<g id=\"node19\" class=\"node\"><title>layer2_pool</title>\n<g id=\"a_node19\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M1752,-97C1752,-97 1722,-97 1722,-97 1716,-97 1710,-91 1710,-85 1710,-85 1710,-73 1710,-73 1710,-67 1716,-61 1722,-61 1722,-61 1752,-61 1752,-61 1758,-61 1764,-67 1764,-73 1764,-73 1764,-85 1764,-85 1764,-91 1758,-97 1752,-97\"/>\n<text text-anchor=\"middle\" x=\"1737\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer2_norm&#45;&gt;layer2_pool -->\n<g id=\"edge20\" class=\"edge\"><title>layer2_norm&#45;&gt;layer2_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1674.4,-79C1682.39,-79 1691.31,-79 1699.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1699.92,-82.5001 1709.92,-79 1699.92,-75.5001 1699.92,-82.5001\"/>\n</g>\n<!-- layer2_act -->\n<g id=\"node20\" class=\"node\"><title>layer2_act</title>\n<g id=\"a_node20\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1842,-97C1842,-97 1812,-97 1812,-97 1806,-97 1800,-91 1800,-85 1800,-85 1800,-73 1800,-73 1800,-67 1806,-61 1812,-61 1812,-61 1842,-61 1842,-61 1848,-61 1854,-67 1854,-73 1854,-73 1854,-85 1854,-85 1854,-91 1848,-97 1842,-97\"/>\n<text text-anchor=\"middle\" x=\"1827\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer2_pool&#45;&gt;layer2_act -->\n<g id=\"edge21\" class=\"edge\"><title>layer2_pool&#45;&gt;layer2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1764.4,-79C1772.39,-79 1781.31,-79 1789.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1789.92,-82.5001 1799.92,-79 1789.92,-75.5001 1789.92,-82.5001\"/>\n</g>\n<!-- layer3_conv -->\n<g id=\"node21\" class=\"node\"><title>layer3_conv</title>\n<g id=\"a_node21\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 256, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1932,-97C1932,-97 1902,-97 1902,-97 1896,-97 1890,-91 1890,-85 1890,-85 1890,-73 1890,-73 1890,-67 1896,-61 1902,-61 1902,-61 1932,-61 1932,-61 1938,-61 1944,-67 1944,-73 1944,-73 1944,-85 1944,-85 1944,-91 1938,-97 1932,-97\"/>\n<text text-anchor=\"middle\" x=\"1917\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer2_act&#45;&gt;layer3_conv -->\n<g id=\"edge22\" class=\"edge\"><title>layer2_act&#45;&gt;layer3_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1854.4,-79C1862.39,-79 1871.31,-79 1879.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1879.92,-82.5001 1889.92,-79 1879.92,-75.5001 1879.92,-82.5001\"/>\n</g>\n<!-- layer3_norm -->\n<g id=\"node22\" class=\"node\"><title>layer3_norm</title>\n<g id=\"a_node22\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2022,-97C2022,-97 1992,-97 1992,-97 1986,-97 1980,-91 1980,-85 1980,-85 1980,-73 1980,-73 1980,-67 1986,-61 1992,-61 1992,-61 2022,-61 2022,-61 2028,-61 2034,-67 2034,-73 2034,-73 2034,-85 2034,-85 2034,-91 2028,-97 2022,-97\"/>\n<text text-anchor=\"middle\" x=\"2007\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_conv&#45;&gt;layer3_norm -->\n<g id=\"edge23\" class=\"edge\"><title>layer3_conv&#45;&gt;layer3_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1944.4,-79C1952.39,-79 1961.31,-79 1969.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1969.92,-82.5001 1979.92,-79 1969.92,-75.5001 1969.92,-82.5001\"/>\n</g>\n<!-- layer3_pool -->\n<g id=\"node23\" class=\"node\"><title>layer3_pool</title>\n<g id=\"a_node23\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M2112,-97C2112,-97 2082,-97 2082,-97 2076,-97 2070,-91 2070,-85 2070,-85 2070,-73 2070,-73 2070,-67 2076,-61 2082,-61 2082,-61 2112,-61 2112,-61 2118,-61 2124,-67 2124,-73 2124,-73 2124,-85 2124,-85 2124,-91 2118,-97 2112,-97\"/>\n<text text-anchor=\"middle\" x=\"2097\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_norm&#45;&gt;layer3_pool -->\n<g id=\"edge24\" class=\"edge\"><title>layer3_norm&#45;&gt;layer3_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2034.4,-79C2042.39,-79 2051.31,-79 2059.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2059.92,-82.5001 2069.92,-79 2059.92,-75.5001 2059.92,-82.5001\"/>\n</g>\n<!-- layer3_act -->\n<g id=\"node24\" class=\"node\"><title>layer3_act</title>\n<g id=\"a_node24\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2202,-97C2202,-97 2172,-97 2172,-97 2166,-97 2160,-91 2160,-85 2160,-85 2160,-73 2160,-73 2160,-67 2166,-61 2172,-61 2172,-61 2202,-61 2202,-61 2208,-61 2214,-67 2214,-73 2214,-73 2214,-85 2214,-85 2214,-91 2208,-97 2202,-97\"/>\n<text text-anchor=\"middle\" x=\"2187\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_pool&#45;&gt;layer3_act -->\n<g id=\"edge25\" class=\"edge\"><title>layer3_pool&#45;&gt;layer3_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2124.4,-79C2132.39,-79 2141.31,-79 2149.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2149.92,-82.5001 2159.92,-79 2149.92,-75.5001 2149.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_in -->\n<g id=\"node25\" class=\"node\"><title>layer3_residual_in</title>\n<g id=\"a_node25\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2292,-97C2292,-97 2262,-97 2262,-97 2256,-97 2250,-91 2250,-85 2250,-85 2250,-73 2250,-73 2250,-67 2256,-61 2262,-61 2262,-61 2292,-61 2292,-61 2298,-61 2304,-67 2304,-73 2304,-73 2304,-85 2304,-85 2304,-91 2298,-97 2292,-97\"/>\n<text text-anchor=\"middle\" x=\"2277\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer3_act&#45;&gt;layer3_residual_in -->\n<g id=\"edge26\" class=\"edge\"><title>layer3_act&#45;&gt;layer3_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2214.4,-79C2222.39,-79 2231.31,-79 2239.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2239.92,-82.5001 2249.92,-79 2239.92,-75.5001 2239.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_res1_conv -->\n<g id=\"node26\" class=\"node\"><title>layer3_residual_res1_conv</title>\n<g id=\"a_node26\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2382,-68C2382,-68 2352,-68 2352,-68 2346,-68 2340,-62 2340,-56 2340,-56 2340,-44 2340,-44 2340,-38 2346,-32 2352,-32 2352,-32 2382,-32 2382,-32 2388,-32 2394,-38 2394,-44 2394,-44 2394,-56 2394,-56 2394,-62 2388,-68 2382,-68\"/>\n<text text-anchor=\"middle\" x=\"2367\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_res1_conv -->\n<g id=\"edge27\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.4,-70.299C2312.57,-67.608 2321.7,-64.5979 2330.38,-61.7368\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2331.52,-65.049 2339.92,-58.5949 2329.33,-58.4007 2331.52,-65.049\"/>\n</g>\n<!-- layer3_residual_add -->\n<g id=\"node33\" class=\"node\"><title>layer3_residual_add</title>\n<g id=\"a_node33\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M3012,-93C3012,-93 2982,-93 2982,-93 2976,-93 2970,-87 2970,-81 2970,-81 2970,-69 2970,-69 2970,-63 2976,-57 2982,-57 2982,-57 3012,-57 3012,-57 3018,-57 3024,-63 3024,-69 3024,-69 3024,-81 3024,-81 3024,-87 3018,-93 3012,-93\"/>\n<text text-anchor=\"middle\" x=\"2997\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_add -->\n<g id=\"edge34\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.22,-93.0359C2312.83,-96.9904 2322.61,-100.808 2332,-103 2385.71,-115.535 2400.84,-108 2456,-108 2456,-108 2456,-108 2818,-108 2867.77,-108 2924.08,-95.3711 2959.8,-85.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2961.04,-89.0265 2969.74,-82.9884 2959.17,-82.2796 2961.04,-89.0265\"/>\n</g>\n<!-- layer3_residual_res1_norm -->\n<g id=\"node27\" class=\"node\"><title>layer3_residual_res1_norm</title>\n<g id=\"a_node27\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2472,-68C2472,-68 2442,-68 2442,-68 2436,-68 2430,-62 2430,-56 2430,-56 2430,-44 2430,-44 2430,-38 2436,-32 2442,-32 2442,-32 2472,-32 2472,-32 2478,-32 2484,-38 2484,-44 2484,-44 2484,-56 2484,-56 2484,-62 2478,-68 2472,-68\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm -->\n<g id=\"edge28\" class=\"edge\"><title>layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2394.4,-50C2402.39,-50 2411.31,-50 2419.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2419.92,-53.5001 2429.92,-50 2419.92,-46.5001 2419.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res1_act -->\n<g id=\"node28\" class=\"node\"><title>layer3_residual_res1_act</title>\n<g id=\"a_node28\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2562,-68C2562,-68 2532,-68 2532,-68 2526,-68 2520,-62 2520,-56 2520,-56 2520,-44 2520,-44 2520,-38 2526,-32 2532,-32 2532,-32 2562,-32 2562,-32 2568,-32 2574,-38 2574,-44 2574,-44 2574,-56 2574,-56 2574,-62 2568,-68 2562,-68\"/>\n<text text-anchor=\"middle\" x=\"2547\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act -->\n<g id=\"edge29\" class=\"edge\"><title>layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2484.4,-50C2492.39,-50 2501.31,-50 2509.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2509.92,-53.5001 2519.92,-50 2509.92,-46.5001 2509.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_conv -->\n<g id=\"node29\" class=\"node\"><title>layer3_residual_res2_conv</title>\n<g id=\"a_node29\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2652,-68C2652,-68 2622,-68 2622,-68 2616,-68 2610,-62 2610,-56 2610,-56 2610,-44 2610,-44 2610,-38 2616,-32 2622,-32 2622,-32 2652,-32 2652,-32 2658,-32 2664,-38 2664,-44 2664,-44 2664,-56 2664,-56 2664,-62 2658,-68 2652,-68\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv -->\n<g id=\"edge30\" class=\"edge\"><title>layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2574.4,-50C2582.39,-50 2591.31,-50 2599.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2599.92,-53.5001 2609.92,-50 2599.92,-46.5001 2599.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_norm -->\n<g id=\"node30\" class=\"node\"><title>layer3_residual_res2_norm</title>\n<g id=\"a_node30\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2742,-68C2742,-68 2712,-68 2712,-68 2706,-68 2700,-62 2700,-56 2700,-56 2700,-44 2700,-44 2700,-38 2706,-32 2712,-32 2712,-32 2742,-32 2742,-32 2748,-32 2754,-38 2754,-44 2754,-44 2754,-56 2754,-56 2754,-62 2748,-68 2742,-68\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm -->\n<g id=\"edge31\" class=\"edge\"><title>layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2664.4,-50C2672.39,-50 2681.31,-50 2689.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2689.92,-53.5001 2699.92,-50 2689.92,-46.5001 2689.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_act -->\n<g id=\"node31\" class=\"node\"><title>layer3_residual_res2_act</title>\n<g id=\"a_node31\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2832,-68C2832,-68 2802,-68 2802,-68 2796,-68 2790,-62 2790,-56 2790,-56 2790,-44 2790,-44 2790,-38 2796,-32 2802,-32 2802,-32 2832,-32 2832,-32 2838,-32 2844,-38 2844,-44 2844,-44 2844,-56 2844,-56 2844,-62 2838,-68 2832,-68\"/>\n<text text-anchor=\"middle\" x=\"2817\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act -->\n<g id=\"edge32\" class=\"edge\"><title>layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2754.4,-50C2762.39,-50 2771.31,-50 2779.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2779.92,-53.5001 2789.92,-50 2779.92,-46.5001 2779.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_out -->\n<g id=\"node32\" class=\"node\"><title>layer3_residual_out</title>\n<g id=\"a_node32\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2922,-78C2922,-78 2892,-78 2892,-78 2886,-78 2880,-72 2880,-66 2880,-66 2880,-54 2880,-54 2880,-48 2886,-42 2892,-42 2892,-42 2922,-42 2922,-42 2928,-42 2934,-48 2934,-54 2934,-54 2934,-66 2934,-66 2934,-72 2928,-78 2922,-78\"/>\n<text text-anchor=\"middle\" x=\"2907\" y=\"-56.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_act&#45;&gt;layer3_residual_out -->\n<g id=\"edge33\" class=\"edge\"><title>layer3_residual_res2_act&#45;&gt;layer3_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2844.4,-53.0003C2852.39,-53.9083 2861.31,-54.9217 2869.82,-55.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2869.59,-59.3847 2879.92,-57.0362 2870.38,-52.4294 2869.59,-59.3847\"/>\n</g>\n<!-- layer3_residual_out&#45;&gt;layer3_residual_add -->\n<g id=\"edge35\" class=\"edge\"><title>layer3_residual_out&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2934.4,-64.5005C2942.39,-65.8625 2951.31,-67.3825 2959.82,-68.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2959.47,-72.3242 2969.92,-70.5544 2960.65,-65.4237 2959.47,-72.3242\"/>\n</g>\n<!-- pool -->\n<g id=\"node34\" class=\"node\"><title>pool</title>\n<g id=\"a_node34\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 4}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M3102,-93C3102,-93 3072,-93 3072,-93 3066,-93 3060,-87 3060,-81 3060,-81 3060,-69 3060,-69 3060,-63 3066,-57 3072,-57 3072,-57 3102,-57 3102,-57 3108,-57 3114,-63 3114,-69 3114,-69 3114,-81 3114,-81 3114,-87 3108,-93 3102,-93\"/>\n<text text-anchor=\"middle\" x=\"3087\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_add&#45;&gt;pool -->\n<g id=\"edge36\" class=\"edge\"><title>layer3_residual_add&#45;&gt;pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3024.4,-75C3032.39,-75 3041.31,-75 3049.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3049.92,-78.5001 3059.92,-75 3049.92,-71.5001 3049.92,-78.5001\"/>\n</g>\n<!-- classifier_flatten -->\n<g id=\"node35\" class=\"node\"><title>classifier_flatten</title>\n<g id=\"a_node35\"><a xlink:title=\"&lt;class &#39;__main__.Flatten&#39;&gt; {}\">\n<path fill=\"#b3de69\" stroke=\"black\" d=\"M3192,-93C3192,-93 3162,-93 3162,-93 3156,-93 3150,-87 3150,-81 3150,-81 3150,-69 3150,-69 3150,-63 3156,-57 3162,-57 3162,-57 3192,-57 3192,-57 3198,-57 3204,-63 3204,-69 3204,-69 3204,-81 3204,-81 3204,-87 3198,-93 3192,-93\"/>\n<text text-anchor=\"middle\" x=\"3177\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">flatten</text>\n</a>\n</g>\n</g>\n<!-- pool&#45;&gt;classifier_flatten -->\n<g id=\"edge37\" class=\"edge\"><title>pool&#45;&gt;classifier_flatten</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3114.4,-75C3122.39,-75 3131.31,-75 3139.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3139.92,-78.5001 3149.92,-75 3139.92,-71.5001 3139.92,-78.5001\"/>\n</g>\n<!-- classifier_conv -->\n<g id=\"node36\" class=\"node\"><title>classifier_conv</title>\n<g id=\"a_node36\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.linear.Linear&#39;&gt; {&#39;in_features&#39;: 512, &#39;out_features&#39;: 10, &#39;bias&#39;: False}\">\n<path fill=\"#fccde5\" stroke=\"black\" d=\"M3282,-93C3282,-93 3252,-93 3252,-93 3246,-93 3240,-87 3240,-81 3240,-81 3240,-69 3240,-69 3240,-63 3246,-57 3252,-57 3252,-57 3282,-57 3282,-57 3288,-57 3294,-63 3294,-69 3294,-69 3294,-81 3294,-81 3294,-87 3288,-93 3282,-93\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- classifier_flatten&#45;&gt;classifier_conv -->\n<g id=\"edge38\" class=\"edge\"><title>classifier_flatten&#45;&gt;classifier_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3204.4,-75C3212.39,-75 3221.31,-75 3229.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3229.92,-78.5001 3239.92,-75 3229.92,-71.5001 3229.92,-78.5001\"/>\n</g>\n<!-- classifier_scale -->\n<g id=\"node37\" class=\"node\"><title>classifier_scale</title>\n<g id=\"a_node37\"><a xlink:title=\"&lt;class &#39;__main__.Mul&#39;&gt; {&#39;weight&#39;: 0.125}\">\n<path fill=\"#bc80bd\" stroke=\"black\" d=\"M3372,-93C3372,-93 3342,-93 3342,-93 3336,-93 3330,-87 3330,-81 3330,-81 3330,-69 3330,-69 3330,-63 3336,-57 3342,-57 3342,-57 3372,-57 3372,-57 3378,-57 3384,-63 3384,-69 3384,-69 3384,-81 3384,-81 3384,-87 3378,-93 3372,-93\"/>\n<text text-anchor=\"middle\" x=\"3357\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">scale</text>\n</a>\n</g>\n</g>\n<!-- classifier_conv&#45;&gt;classifier_scale -->\n<g id=\"edge39\" class=\"edge\"><title>classifier_conv&#45;&gt;classifier_scale</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3294.4,-75C3302.39,-75 3311.31,-75 3319.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3319.92,-78.5001 3329.92,-75 3319.92,-71.5001 3319.92,-78.5001\"/>\n</g>\n<!-- logits -->\n<g id=\"node38\" class=\"node\"><title>logits</title>\n<g id=\"a_node38\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M3462,-93C3462,-93 3432,-93 3432,-93 3426,-93 3420,-87 3420,-81 3420,-81 3420,-69 3420,-69 3420,-63 3426,-57 3432,-57 3432,-57 3462,-57 3462,-57 3468,-57 3474,-63 3474,-69 3474,-69 3474,-81 3474,-81 3474,-87 3468,-93 3462,-93\"/>\n<text text-anchor=\"middle\" x=\"3447\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">logits</text>\n</a>\n</g>\n</g>\n<!-- classifier_scale&#45;&gt;logits -->\n<g id=\"edge40\" class=\"edge\"><title>classifier_scale&#45;&gt;logits</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3384.4,-75C3392.39,-75 3401.31,-75 3409.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3409.92,-78.5001 3419.92,-75 3409.92,-71.5001 3409.92,-78.5001\"/>\n</g>\n<!-- input -->\n<g id=\"node39\" class=\"node\"><title>input</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-92C42,-92 12,-92 12,-92 6,-92 0,-86 0,-80 0,-80 0,-68 0,-68 0,-62 6,-56 12,-56 12,-56 42,-56 42,-56 48,-56 54,-62 54,-68 54,-68 54,-80 54,-80 54,-86 48,-92 42,-92\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">input</text>\n</g>\n<!-- input&#45;&gt;prep_conv -->\n<g id=\"edge1\" class=\"edge\"><title>input&#45;&gt;prep_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4029,-74C62.3932,-74 71.3106,-74 79.8241,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"79.919,-77.5001 89.919,-74 79.919,-70.5001 79.919,-77.5001\"/>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uCx6niO29qtZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 24, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, 5, epochs], [0.0, 0.4, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "model = build_model(network(conv_pool_block=conv_pool_block_opt), x_ent_loss)\n",
        "logs, state, timer = Table(), {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize)\n",
        "for epoch in range(epochs):\n",
        "    logs.append(union({'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KpB6j6de7keL",
        "colab_type": "text"
      },
      "source": [
        "A further 3s off! \n",
        "\n",
        "Perhaps we should try something more radical and move max-pooling before batch norm. This will achieve a further efficiency gain but will change the network \n",
        "so we need to test the effect on training.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YfDw9V35-1Ng",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        },
        "outputId": "7e314bfa-b3c0-4a01-c738-f38fa1a6ddd5"
      },
      "source": [
        "conv_pool_block_pre = lambda c_in, c_out: reorder(conv_pool_block(c_in, c_out), ('conv', 'pool', 'norm', 'act'))\n",
        "\n",
        "maxpool_pre_net = network(conv_pool_block=conv_pool_block_pre)\n",
        "show(maxpool_pre_net)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<__main__.DotGraph at 0x7f4142311e80>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n -->\n<!-- Title: G Pages: 1 -->\n<svg width=\"1080pt\" height=\"59pt\"\n viewBox=\"0.00 0.00 1080.00 59.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(0.310167 0.310167) rotate(0) translate(4 187)\">\n<title>G</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-187 3478,-187 3478,4 -4,4\"/>\n<g id=\"clust1\" class=\"cluster\"><title>cluster_prep</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M94,-48C94,-48 320,-48 320,-48 326,-48 332,-54 332,-60 332,-60 332,-111 332,-111 332,-117 326,-123 320,-123 320,-123 94,-123 94,-123 88,-123 82,-117 82,-111 82,-111 82,-60 82,-60 82,-54 88,-48 94,-48\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-107.8\" font-family=\"Times,serif\" font-size=\"14.00\">prep</text>\n</g>\n<g id=\"clust2\" class=\"cluster\"><title>cluster_layer1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M364,-12C364,-12 1498,-12 1498,-12 1504,-12 1510,-18 1510,-24 1510,-24 1510,-163 1510,-163 1510,-169 1504,-175 1498,-175 1498,-175 364,-175 364,-175 358,-175 352,-169 352,-163 352,-163 352,-24 352,-24 352,-18 358,-12 364,-12\"/>\n<text text-anchor=\"middle\" x=\"931\" y=\"-159.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer1</text>\n</g>\n<g id=\"clust3\" class=\"cluster\"><title>cluster_layer1_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M724,-20C724,-20 1490,-20 1490,-20 1496,-20 1502,-26 1502,-32 1502,-32 1502,-132 1502,-132 1502,-138 1496,-144 1490,-144 1490,-144 724,-144 724,-144 718,-144 712,-138 712,-132 712,-132 712,-32 712,-32 712,-26 718,-20 724,-20\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-128.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust4\" class=\"cluster\"><title>cluster_layer1_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M814,-28C814,-28 1040,-28 1040,-28 1046,-28 1052,-34 1052,-40 1052,-40 1052,-91 1052,-91 1052,-97 1046,-103 1040,-103 1040,-103 814,-103 814,-103 808,-103 802,-97 802,-91 802,-91 802,-40 802,-40 802,-34 808,-28 814,-28\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust5\" class=\"cluster\"><title>cluster_layer1_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1084,-28C1084,-28 1310,-28 1310,-28 1316,-28 1322,-34 1322,-40 1322,-40 1322,-91 1322,-91 1322,-97 1316,-103 1310,-103 1310,-103 1084,-103 1084,-103 1078,-103 1072,-97 1072,-91 1072,-91 1072,-40 1072,-40 1072,-34 1078,-28 1084,-28\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust6\" class=\"cluster\"><title>cluster_layer2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1534,-53C1534,-53 1850,-53 1850,-53 1856,-53 1862,-59 1862,-65 1862,-65 1862,-116 1862,-116 1862,-122 1856,-128 1850,-128 1850,-128 1534,-128 1534,-128 1528,-128 1522,-122 1522,-116 1522,-116 1522,-65 1522,-65 1522,-59 1528,-53 1534,-53\"/>\n<text text-anchor=\"middle\" x=\"1692\" y=\"-112.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer2</text>\n</g>\n<g id=\"clust7\" class=\"cluster\"><title>cluster_layer3</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1894,-8C1894,-8 3028,-8 3028,-8 3034,-8 3040,-14 3040,-20 3040,-20 3040,-159 3040,-159 3040,-165 3034,-171 3028,-171 3028,-171 1894,-171 1894,-171 1888,-171 1882,-165 1882,-159 1882,-159 1882,-20 1882,-20 1882,-14 1888,-8 1894,-8\"/>\n<text text-anchor=\"middle\" x=\"2461\" y=\"-155.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer3</text>\n</g>\n<g id=\"clust8\" class=\"cluster\"><title>cluster_layer3_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2254,-16C2254,-16 3020,-16 3020,-16 3026,-16 3032,-22 3032,-28 3032,-28 3032,-128 3032,-128 3032,-134 3026,-140 3020,-140 3020,-140 2254,-140 2254,-140 2248,-140 2242,-134 2242,-128 2242,-128 2242,-28 2242,-28 2242,-22 2248,-16 2254,-16\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust9\" class=\"cluster\"><title>cluster_layer3_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2344,-24C2344,-24 2570,-24 2570,-24 2576,-24 2582,-30 2582,-36 2582,-36 2582,-87 2582,-87 2582,-93 2576,-99 2570,-99 2570,-99 2344,-99 2344,-99 2338,-99 2332,-93 2332,-87 2332,-87 2332,-36 2332,-36 2332,-30 2338,-24 2344,-24\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust10\" class=\"cluster\"><title>cluster_layer3_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2614,-24C2614,-24 2840,-24 2840,-24 2846,-24 2852,-30 2852,-36 2852,-36 2852,-87 2852,-87 2852,-93 2846,-99 2840,-99 2840,-99 2614,-99 2614,-99 2608,-99 2602,-93 2602,-87 2602,-87 2602,-36 2602,-36 2602,-30 2608,-24 2614,-24\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust11\" class=\"cluster\"><title>cluster_classifier</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M3154,-49C3154,-49 3380,-49 3380,-49 3386,-49 3392,-55 3392,-61 3392,-61 3392,-112 3392,-112 3392,-118 3386,-124 3380,-124 3380,-124 3154,-124 3154,-124 3148,-124 3142,-118 3142,-112 3142,-112 3142,-61 3142,-61 3142,-55 3148,-49 3154,-49\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-108.8\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n</g>\n<!-- prep_conv -->\n<g id=\"node1\" class=\"node\"><title>prep_conv</title>\n<g id=\"a_node1\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 3, &#39;out_channels&#39;: 64, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M132,-92C132,-92 102,-92 102,-92 96,-92 90,-86 90,-80 90,-80 90,-68 90,-68 90,-62 96,-56 102,-56 102,-56 132,-56 132,-56 138,-56 144,-62 144,-68 144,-68 144,-80 144,-80 144,-86 138,-92 132,-92\"/>\n<text text-anchor=\"middle\" x=\"117\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_norm -->\n<g id=\"node2\" class=\"node\"><title>prep_norm</title>\n<g id=\"a_node2\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 64}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M222,-92C222,-92 192,-92 192,-92 186,-92 180,-86 180,-80 180,-80 180,-68 180,-68 180,-62 186,-56 192,-56 192,-56 222,-56 222,-56 228,-56 234,-62 234,-68 234,-68 234,-80 234,-80 234,-86 228,-92 222,-92\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- prep_conv&#45;&gt;prep_norm -->\n<g id=\"edge2\" class=\"edge\"><title>prep_conv&#45;&gt;prep_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M144.403,-74C152.393,-74 161.311,-74 169.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"169.919,-77.5001 179.919,-74 169.919,-70.5001 169.919,-77.5001\"/>\n</g>\n<!-- prep_act -->\n<g id=\"node3\" class=\"node\"><title>prep_act</title>\n<g id=\"a_node3\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M312,-92C312,-92 282,-92 282,-92 276,-92 270,-86 270,-80 270,-80 270,-68 270,-68 270,-62 276,-56 282,-56 282,-56 312,-56 312,-56 318,-56 324,-62 324,-68 324,-68 324,-80 324,-80 324,-86 318,-92 312,-92\"/>\n<text text-anchor=\"middle\" x=\"297\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- prep_norm&#45;&gt;prep_act -->\n<g id=\"edge3\" class=\"edge\"><title>prep_norm&#45;&gt;prep_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M234.403,-74C242.393,-74 251.311,-74 259.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"259.919,-77.5001 269.919,-74 259.919,-70.5001 259.919,-77.5001\"/>\n</g>\n<!-- layer1_conv -->\n<g id=\"node4\" class=\"node\"><title>layer1_conv</title>\n<g id=\"a_node4\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 64, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M402,-92C402,-92 372,-92 372,-92 366,-92 360,-86 360,-80 360,-80 360,-68 360,-68 360,-62 366,-56 372,-56 372,-56 402,-56 402,-56 408,-56 414,-62 414,-68 414,-68 414,-80 414,-80 414,-86 408,-92 402,-92\"/>\n<text text-anchor=\"middle\" x=\"387\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_act&#45;&gt;layer1_conv -->\n<g id=\"edge4\" class=\"edge\"><title>prep_act&#45;&gt;layer1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M324.403,-74C332.393,-74 341.311,-74 349.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"349.919,-77.5001 359.919,-74 349.919,-70.5001 349.919,-77.5001\"/>\n</g>\n<!-- layer1_pool -->\n<g id=\"node5\" class=\"node\"><title>layer1_pool</title>\n<g id=\"a_node5\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M492,-92C492,-92 462,-92 462,-92 456,-92 450,-86 450,-80 450,-80 450,-68 450,-68 450,-62 456,-56 462,-56 462,-56 492,-56 492,-56 498,-56 504,-62 504,-68 504,-68 504,-80 504,-80 504,-86 498,-92 492,-92\"/>\n<text text-anchor=\"middle\" x=\"477\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer1_conv&#45;&gt;layer1_pool -->\n<g id=\"edge5\" class=\"edge\"><title>layer1_conv&#45;&gt;layer1_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M414.403,-74C422.393,-74 431.311,-74 439.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"439.919,-77.5001 449.919,-74 439.919,-70.5001 439.919,-77.5001\"/>\n</g>\n<!-- layer1_norm -->\n<g id=\"node6\" class=\"node\"><title>layer1_norm</title>\n<g id=\"a_node6\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M582,-92C582,-92 552,-92 552,-92 546,-92 540,-86 540,-80 540,-80 540,-68 540,-68 540,-62 546,-56 552,-56 552,-56 582,-56 582,-56 588,-56 594,-62 594,-68 594,-68 594,-80 594,-80 594,-86 588,-92 582,-92\"/>\n<text text-anchor=\"middle\" x=\"567\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_pool&#45;&gt;layer1_norm -->\n<g id=\"edge6\" class=\"edge\"><title>layer1_pool&#45;&gt;layer1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M504.403,-74C512.393,-74 521.311,-74 529.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"529.919,-77.5001 539.919,-74 529.919,-70.5001 529.919,-77.5001\"/>\n</g>\n<!-- layer1_act -->\n<g id=\"node7\" class=\"node\"><title>layer1_act</title>\n<g id=\"a_node7\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M672,-92C672,-92 642,-92 642,-92 636,-92 630,-86 630,-80 630,-80 630,-68 630,-68 630,-62 636,-56 642,-56 642,-56 672,-56 672,-56 678,-56 684,-62 684,-68 684,-68 684,-80 684,-80 684,-86 678,-92 672,-92\"/>\n<text text-anchor=\"middle\" x=\"657\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_norm&#45;&gt;layer1_act -->\n<g id=\"edge7\" class=\"edge\"><title>layer1_norm&#45;&gt;layer1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M594.403,-74C602.393,-74 611.311,-74 619.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"619.919,-77.5001 629.919,-74 619.919,-70.5001 619.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_in -->\n<g id=\"node8\" class=\"node\"><title>layer1_residual_in</title>\n<g id=\"a_node8\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M762,-92C762,-92 732,-92 732,-92 726,-92 720,-86 720,-80 720,-80 720,-68 720,-68 720,-62 726,-56 732,-56 732,-56 762,-56 762,-56 768,-56 774,-62 774,-68 774,-68 774,-80 774,-80 774,-86 768,-92 762,-92\"/>\n<text text-anchor=\"middle\" x=\"747\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer1_act&#45;&gt;layer1_residual_in -->\n<g id=\"edge8\" class=\"edge\"><title>layer1_act&#45;&gt;layer1_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M684.403,-74C692.393,-74 701.311,-74 709.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"709.919,-77.5001 719.919,-74 709.919,-70.5001 709.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_res1_conv -->\n<g id=\"node9\" class=\"node\"><title>layer1_residual_res1_conv</title>\n<g id=\"a_node9\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M852,-72C852,-72 822,-72 822,-72 816,-72 810,-66 810,-60 810,-60 810,-48 810,-48 810,-42 816,-36 822,-36 822,-36 852,-36 852,-36 858,-36 864,-42 864,-48 864,-48 864,-60 864,-60 864,-66 858,-72 852,-72\"/>\n<text text-anchor=\"middle\" x=\"837\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_res1_conv -->\n<g id=\"edge9\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M774.403,-67.9993C782.481,-66.1634 791.507,-64.1121 800.105,-62.158\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"800.943,-65.5568 809.919,-59.9275 799.392,-58.7308 800.943,-65.5568\"/>\n</g>\n<!-- layer1_residual_add -->\n<g id=\"node16\" class=\"node\"><title>layer1_residual_add</title>\n<g id=\"a_node16\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M1482,-97C1482,-97 1452,-97 1452,-97 1446,-97 1440,-91 1440,-85 1440,-85 1440,-73 1440,-73 1440,-67 1446,-61 1452,-61 1452,-61 1482,-61 1482,-61 1488,-61 1494,-67 1494,-73 1494,-73 1494,-85 1494,-85 1494,-91 1488,-97 1482,-97\"/>\n<text text-anchor=\"middle\" x=\"1467\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_add -->\n<g id=\"edge16\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M771.737,-92.1018C780.735,-97.9639 791.381,-103.78 802,-107 854.782,-123.006 870.844,-112 926,-112 926,-112 926,-112 1288,-112 1337.77,-112 1394.08,-99.3711 1429.8,-89.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1431.04,-93.0265 1439.74,-86.9884 1429.17,-86.2796 1431.04,-93.0265\"/>\n</g>\n<!-- layer1_residual_res1_norm -->\n<g id=\"node10\" class=\"node\"><title>layer1_residual_res1_norm</title>\n<g id=\"a_node10\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M942,-72C942,-72 912,-72 912,-72 906,-72 900,-66 900,-60 900,-60 900,-48 900,-48 900,-42 906,-36 912,-36 912,-36 942,-36 942,-36 948,-36 954,-42 954,-48 954,-48 954,-60 954,-60 954,-66 948,-72 942,-72\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm -->\n<g id=\"edge10\" class=\"edge\"><title>layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M864.403,-54C872.393,-54 881.311,-54 889.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"889.919,-57.5001 899.919,-54 889.919,-50.5001 889.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res1_act -->\n<g id=\"node11\" class=\"node\"><title>layer1_residual_res1_act</title>\n<g id=\"a_node11\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1032,-72C1032,-72 1002,-72 1002,-72 996,-72 990,-66 990,-60 990,-60 990,-48 990,-48 990,-42 996,-36 1002,-36 1002,-36 1032,-36 1032,-36 1038,-36 1044,-42 1044,-48 1044,-48 1044,-60 1044,-60 1044,-66 1038,-72 1032,-72\"/>\n<text text-anchor=\"middle\" x=\"1017\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act -->\n<g id=\"edge11\" class=\"edge\"><title>layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M954.403,-54C962.393,-54 971.311,-54 979.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"979.919,-57.5001 989.919,-54 979.919,-50.5001 979.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_conv -->\n<g id=\"node12\" class=\"node\"><title>layer1_residual_res2_conv</title>\n<g id=\"a_node12\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1122,-72C1122,-72 1092,-72 1092,-72 1086,-72 1080,-66 1080,-60 1080,-60 1080,-48 1080,-48 1080,-42 1086,-36 1092,-36 1092,-36 1122,-36 1122,-36 1128,-36 1134,-42 1134,-48 1134,-48 1134,-60 1134,-60 1134,-66 1128,-72 1122,-72\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv -->\n<g id=\"edge12\" class=\"edge\"><title>layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1044.4,-54C1052.39,-54 1061.31,-54 1069.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1069.92,-57.5001 1079.92,-54 1069.92,-50.5001 1069.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_norm -->\n<g id=\"node13\" class=\"node\"><title>layer1_residual_res2_norm</title>\n<g id=\"a_node13\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M1212,-72C1212,-72 1182,-72 1182,-72 1176,-72 1170,-66 1170,-60 1170,-60 1170,-48 1170,-48 1170,-42 1176,-36 1182,-36 1182,-36 1212,-36 1212,-36 1218,-36 1224,-42 1224,-48 1224,-48 1224,-60 1224,-60 1224,-66 1218,-72 1212,-72\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm -->\n<g id=\"edge13\" class=\"edge\"><title>layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1134.4,-54C1142.39,-54 1151.31,-54 1159.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1159.92,-57.5001 1169.92,-54 1159.92,-50.5001 1159.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_act -->\n<g id=\"node14\" class=\"node\"><title>layer1_residual_res2_act</title>\n<g id=\"a_node14\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1302,-72C1302,-72 1272,-72 1272,-72 1266,-72 1260,-66 1260,-60 1260,-60 1260,-48 1260,-48 1260,-42 1266,-36 1272,-36 1272,-36 1302,-36 1302,-36 1308,-36 1314,-42 1314,-48 1314,-48 1314,-60 1314,-60 1314,-66 1308,-72 1302,-72\"/>\n<text text-anchor=\"middle\" x=\"1287\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act -->\n<g id=\"edge14\" class=\"edge\"><title>layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1224.4,-54C1232.39,-54 1241.31,-54 1249.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1249.92,-57.5001 1259.92,-54 1249.92,-50.5001 1249.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_out -->\n<g id=\"node15\" class=\"node\"><title>layer1_residual_out</title>\n<g id=\"a_node15\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M1392,-82C1392,-82 1362,-82 1362,-82 1356,-82 1350,-76 1350,-70 1350,-70 1350,-58 1350,-58 1350,-52 1356,-46 1362,-46 1362,-46 1392,-46 1392,-46 1398,-46 1404,-52 1404,-58 1404,-58 1404,-70 1404,-70 1404,-76 1398,-82 1392,-82\"/>\n<text text-anchor=\"middle\" x=\"1377\" y=\"-60.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_act&#45;&gt;layer1_residual_out -->\n<g id=\"edge15\" class=\"edge\"><title>layer1_residual_res2_act&#45;&gt;layer1_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1314.4,-57.0003C1322.39,-57.9083 1331.31,-58.9217 1339.82,-59.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1339.59,-63.3847 1349.92,-61.0362 1340.38,-56.4294 1339.59,-63.3847\"/>\n</g>\n<!-- layer1_residual_out&#45;&gt;layer1_residual_add -->\n<g id=\"edge17\" class=\"edge\"><title>layer1_residual_out&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1404.4,-68.5005C1412.39,-69.8625 1421.31,-71.3825 1429.82,-72.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1429.47,-76.3242 1439.92,-74.5544 1430.65,-69.4237 1429.47,-76.3242\"/>\n</g>\n<!-- layer2_conv -->\n<g id=\"node17\" class=\"node\"><title>layer2_conv</title>\n<g id=\"a_node17\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 256, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1572,-97C1572,-97 1542,-97 1542,-97 1536,-97 1530,-91 1530,-85 1530,-85 1530,-73 1530,-73 1530,-67 1536,-61 1542,-61 1542,-61 1572,-61 1572,-61 1578,-61 1584,-67 1584,-73 1584,-73 1584,-85 1584,-85 1584,-91 1578,-97 1572,-97\"/>\n<text text-anchor=\"middle\" x=\"1557\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_add&#45;&gt;layer2_conv -->\n<g id=\"edge18\" class=\"edge\"><title>layer1_residual_add&#45;&gt;layer2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1494.4,-79C1502.39,-79 1511.31,-79 1519.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1519.92,-82.5001 1529.92,-79 1519.92,-75.5001 1519.92,-82.5001\"/>\n</g>\n<!-- layer2_pool -->\n<g id=\"node18\" class=\"node\"><title>layer2_pool</title>\n<g id=\"a_node18\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M1662,-97C1662,-97 1632,-97 1632,-97 1626,-97 1620,-91 1620,-85 1620,-85 1620,-73 1620,-73 1620,-67 1626,-61 1632,-61 1632,-61 1662,-61 1662,-61 1668,-61 1674,-67 1674,-73 1674,-73 1674,-85 1674,-85 1674,-91 1668,-97 1662,-97\"/>\n<text text-anchor=\"middle\" x=\"1647\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer2_conv&#45;&gt;layer2_pool -->\n<g id=\"edge19\" class=\"edge\"><title>layer2_conv&#45;&gt;layer2_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1584.4,-79C1592.39,-79 1601.31,-79 1609.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1609.92,-82.5001 1619.92,-79 1609.92,-75.5001 1609.92,-82.5001\"/>\n</g>\n<!-- layer2_norm -->\n<g id=\"node19\" class=\"node\"><title>layer2_norm</title>\n<g id=\"a_node19\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 256}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M1752,-97C1752,-97 1722,-97 1722,-97 1716,-97 1710,-91 1710,-85 1710,-85 1710,-73 1710,-73 1710,-67 1716,-61 1722,-61 1722,-61 1752,-61 1752,-61 1758,-61 1764,-67 1764,-73 1764,-73 1764,-85 1764,-85 1764,-91 1758,-97 1752,-97\"/>\n<text text-anchor=\"middle\" x=\"1737\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer2_pool&#45;&gt;layer2_norm -->\n<g id=\"edge20\" class=\"edge\"><title>layer2_pool&#45;&gt;layer2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1674.4,-79C1682.39,-79 1691.31,-79 1699.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1699.92,-82.5001 1709.92,-79 1699.92,-75.5001 1699.92,-82.5001\"/>\n</g>\n<!-- layer2_act -->\n<g id=\"node20\" class=\"node\"><title>layer2_act</title>\n<g id=\"a_node20\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M1842,-97C1842,-97 1812,-97 1812,-97 1806,-97 1800,-91 1800,-85 1800,-85 1800,-73 1800,-73 1800,-67 1806,-61 1812,-61 1812,-61 1842,-61 1842,-61 1848,-61 1854,-67 1854,-73 1854,-73 1854,-85 1854,-85 1854,-91 1848,-97 1842,-97\"/>\n<text text-anchor=\"middle\" x=\"1827\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer2_norm&#45;&gt;layer2_act -->\n<g id=\"edge21\" class=\"edge\"><title>layer2_norm&#45;&gt;layer2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1764.4,-79C1772.39,-79 1781.31,-79 1789.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1789.92,-82.5001 1799.92,-79 1789.92,-75.5001 1789.92,-82.5001\"/>\n</g>\n<!-- layer3_conv -->\n<g id=\"node21\" class=\"node\"><title>layer3_conv</title>\n<g id=\"a_node21\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 256, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1932,-97C1932,-97 1902,-97 1902,-97 1896,-97 1890,-91 1890,-85 1890,-85 1890,-73 1890,-73 1890,-67 1896,-61 1902,-61 1902,-61 1932,-61 1932,-61 1938,-61 1944,-67 1944,-73 1944,-73 1944,-85 1944,-85 1944,-91 1938,-97 1932,-97\"/>\n<text text-anchor=\"middle\" x=\"1917\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer2_act&#45;&gt;layer3_conv -->\n<g id=\"edge22\" class=\"edge\"><title>layer2_act&#45;&gt;layer3_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1854.4,-79C1862.39,-79 1871.31,-79 1879.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1879.92,-82.5001 1889.92,-79 1879.92,-75.5001 1879.92,-82.5001\"/>\n</g>\n<!-- layer3_pool -->\n<g id=\"node22\" class=\"node\"><title>layer3_pool</title>\n<g id=\"a_node22\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M2022,-97C2022,-97 1992,-97 1992,-97 1986,-97 1980,-91 1980,-85 1980,-85 1980,-73 1980,-73 1980,-67 1986,-61 1992,-61 1992,-61 2022,-61 2022,-61 2028,-61 2034,-67 2034,-73 2034,-73 2034,-85 2034,-85 2034,-91 2028,-97 2022,-97\"/>\n<text text-anchor=\"middle\" x=\"2007\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_conv&#45;&gt;layer3_pool -->\n<g id=\"edge23\" class=\"edge\"><title>layer3_conv&#45;&gt;layer3_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1944.4,-79C1952.39,-79 1961.31,-79 1969.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1969.92,-82.5001 1979.92,-79 1969.92,-75.5001 1969.92,-82.5001\"/>\n</g>\n<!-- layer3_norm -->\n<g id=\"node23\" class=\"node\"><title>layer3_norm</title>\n<g id=\"a_node23\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2112,-97C2112,-97 2082,-97 2082,-97 2076,-97 2070,-91 2070,-85 2070,-85 2070,-73 2070,-73 2070,-67 2076,-61 2082,-61 2082,-61 2112,-61 2112,-61 2118,-61 2124,-67 2124,-73 2124,-73 2124,-85 2124,-85 2124,-91 2118,-97 2112,-97\"/>\n<text text-anchor=\"middle\" x=\"2097\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_pool&#45;&gt;layer3_norm -->\n<g id=\"edge24\" class=\"edge\"><title>layer3_pool&#45;&gt;layer3_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2034.4,-79C2042.39,-79 2051.31,-79 2059.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2059.92,-82.5001 2069.92,-79 2059.92,-75.5001 2059.92,-82.5001\"/>\n</g>\n<!-- layer3_act -->\n<g id=\"node24\" class=\"node\"><title>layer3_act</title>\n<g id=\"a_node24\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2202,-97C2202,-97 2172,-97 2172,-97 2166,-97 2160,-91 2160,-85 2160,-85 2160,-73 2160,-73 2160,-67 2166,-61 2172,-61 2172,-61 2202,-61 2202,-61 2208,-61 2214,-67 2214,-73 2214,-73 2214,-85 2214,-85 2214,-91 2208,-97 2202,-97\"/>\n<text text-anchor=\"middle\" x=\"2187\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_norm&#45;&gt;layer3_act -->\n<g id=\"edge25\" class=\"edge\"><title>layer3_norm&#45;&gt;layer3_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2124.4,-79C2132.39,-79 2141.31,-79 2149.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2149.92,-82.5001 2159.92,-79 2149.92,-75.5001 2149.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_in -->\n<g id=\"node25\" class=\"node\"><title>layer3_residual_in</title>\n<g id=\"a_node25\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2292,-97C2292,-97 2262,-97 2262,-97 2256,-97 2250,-91 2250,-85 2250,-85 2250,-73 2250,-73 2250,-67 2256,-61 2262,-61 2262,-61 2292,-61 2292,-61 2298,-61 2304,-67 2304,-73 2304,-73 2304,-85 2304,-85 2304,-91 2298,-97 2292,-97\"/>\n<text text-anchor=\"middle\" x=\"2277\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer3_act&#45;&gt;layer3_residual_in -->\n<g id=\"edge26\" class=\"edge\"><title>layer3_act&#45;&gt;layer3_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2214.4,-79C2222.39,-79 2231.31,-79 2239.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2239.92,-82.5001 2249.92,-79 2239.92,-75.5001 2239.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_res1_conv -->\n<g id=\"node26\" class=\"node\"><title>layer3_residual_res1_conv</title>\n<g id=\"a_node26\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2382,-68C2382,-68 2352,-68 2352,-68 2346,-68 2340,-62 2340,-56 2340,-56 2340,-44 2340,-44 2340,-38 2346,-32 2352,-32 2352,-32 2382,-32 2382,-32 2388,-32 2394,-38 2394,-44 2394,-44 2394,-56 2394,-56 2394,-62 2388,-68 2382,-68\"/>\n<text text-anchor=\"middle\" x=\"2367\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_res1_conv -->\n<g id=\"edge27\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.4,-70.299C2312.57,-67.608 2321.7,-64.5979 2330.38,-61.7368\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2331.52,-65.049 2339.92,-58.5949 2329.33,-58.4007 2331.52,-65.049\"/>\n</g>\n<!-- layer3_residual_add -->\n<g id=\"node33\" class=\"node\"><title>layer3_residual_add</title>\n<g id=\"a_node33\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M3012,-93C3012,-93 2982,-93 2982,-93 2976,-93 2970,-87 2970,-81 2970,-81 2970,-69 2970,-69 2970,-63 2976,-57 2982,-57 2982,-57 3012,-57 3012,-57 3018,-57 3024,-63 3024,-69 3024,-69 3024,-81 3024,-81 3024,-87 3018,-93 3012,-93\"/>\n<text text-anchor=\"middle\" x=\"2997\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_add -->\n<g id=\"edge34\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.22,-93.0359C2312.83,-96.9904 2322.61,-100.808 2332,-103 2385.71,-115.535 2400.84,-108 2456,-108 2456,-108 2456,-108 2818,-108 2867.77,-108 2924.08,-95.3711 2959.8,-85.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2961.04,-89.0265 2969.74,-82.9884 2959.17,-82.2796 2961.04,-89.0265\"/>\n</g>\n<!-- layer3_residual_res1_norm -->\n<g id=\"node27\" class=\"node\"><title>layer3_residual_res1_norm</title>\n<g id=\"a_node27\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2472,-68C2472,-68 2442,-68 2442,-68 2436,-68 2430,-62 2430,-56 2430,-56 2430,-44 2430,-44 2430,-38 2436,-32 2442,-32 2442,-32 2472,-32 2472,-32 2478,-32 2484,-38 2484,-44 2484,-44 2484,-56 2484,-56 2484,-62 2478,-68 2472,-68\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm -->\n<g id=\"edge28\" class=\"edge\"><title>layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2394.4,-50C2402.39,-50 2411.31,-50 2419.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2419.92,-53.5001 2429.92,-50 2419.92,-46.5001 2419.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res1_act -->\n<g id=\"node28\" class=\"node\"><title>layer3_residual_res1_act</title>\n<g id=\"a_node28\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2562,-68C2562,-68 2532,-68 2532,-68 2526,-68 2520,-62 2520,-56 2520,-56 2520,-44 2520,-44 2520,-38 2526,-32 2532,-32 2532,-32 2562,-32 2562,-32 2568,-32 2574,-38 2574,-44 2574,-44 2574,-56 2574,-56 2574,-62 2568,-68 2562,-68\"/>\n<text text-anchor=\"middle\" x=\"2547\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act -->\n<g id=\"edge29\" class=\"edge\"><title>layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2484.4,-50C2492.39,-50 2501.31,-50 2509.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2509.92,-53.5001 2519.92,-50 2509.92,-46.5001 2509.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_conv -->\n<g id=\"node29\" class=\"node\"><title>layer3_residual_res2_conv</title>\n<g id=\"a_node29\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2652,-68C2652,-68 2622,-68 2622,-68 2616,-68 2610,-62 2610,-56 2610,-56 2610,-44 2610,-44 2610,-38 2616,-32 2622,-32 2622,-32 2652,-32 2652,-32 2658,-32 2664,-38 2664,-44 2664,-44 2664,-56 2664,-56 2664,-62 2658,-68 2652,-68\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv -->\n<g id=\"edge30\" class=\"edge\"><title>layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2574.4,-50C2582.39,-50 2591.31,-50 2599.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2599.92,-53.5001 2609.92,-50 2599.92,-46.5001 2599.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_norm -->\n<g id=\"node30\" class=\"node\"><title>layer3_residual_res2_norm</title>\n<g id=\"a_node30\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2742,-68C2742,-68 2712,-68 2712,-68 2706,-68 2700,-62 2700,-56 2700,-56 2700,-44 2700,-44 2700,-38 2706,-32 2712,-32 2712,-32 2742,-32 2742,-32 2748,-32 2754,-38 2754,-44 2754,-44 2754,-56 2754,-56 2754,-62 2748,-68 2742,-68\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm -->\n<g id=\"edge31\" class=\"edge\"><title>layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2664.4,-50C2672.39,-50 2681.31,-50 2689.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2689.92,-53.5001 2699.92,-50 2689.92,-46.5001 2689.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_act -->\n<g id=\"node31\" class=\"node\"><title>layer3_residual_res2_act</title>\n<g id=\"a_node31\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.activation.ReLU&#39;&gt; {}\">\n<path fill=\"#fb8072\" stroke=\"black\" d=\"M2832,-68C2832,-68 2802,-68 2802,-68 2796,-68 2790,-62 2790,-56 2790,-56 2790,-44 2790,-44 2790,-38 2796,-32 2802,-32 2802,-32 2832,-32 2832,-32 2838,-32 2844,-38 2844,-44 2844,-44 2844,-56 2844,-56 2844,-62 2838,-68 2832,-68\"/>\n<text text-anchor=\"middle\" x=\"2817\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act -->\n<g id=\"edge32\" class=\"edge\"><title>layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2754.4,-50C2762.39,-50 2771.31,-50 2779.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2779.92,-53.5001 2789.92,-50 2779.92,-46.5001 2779.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_out -->\n<g id=\"node32\" class=\"node\"><title>layer3_residual_out</title>\n<g id=\"a_node32\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2922,-78C2922,-78 2892,-78 2892,-78 2886,-78 2880,-72 2880,-66 2880,-66 2880,-54 2880,-54 2880,-48 2886,-42 2892,-42 2892,-42 2922,-42 2922,-42 2928,-42 2934,-48 2934,-54 2934,-54 2934,-66 2934,-66 2934,-72 2928,-78 2922,-78\"/>\n<text text-anchor=\"middle\" x=\"2907\" y=\"-56.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_act&#45;&gt;layer3_residual_out -->\n<g id=\"edge33\" class=\"edge\"><title>layer3_residual_res2_act&#45;&gt;layer3_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2844.4,-53.0003C2852.39,-53.9083 2861.31,-54.9217 2869.82,-55.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2869.59,-59.3847 2879.92,-57.0362 2870.38,-52.4294 2869.59,-59.3847\"/>\n</g>\n<!-- layer3_residual_out&#45;&gt;layer3_residual_add -->\n<g id=\"edge35\" class=\"edge\"><title>layer3_residual_out&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2934.4,-64.5005C2942.39,-65.8625 2951.31,-67.3825 2959.82,-68.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2959.47,-72.3242 2969.92,-70.5544 2960.65,-65.4237 2959.47,-72.3242\"/>\n</g>\n<!-- pool -->\n<g id=\"node34\" class=\"node\"><title>pool</title>\n<g id=\"a_node34\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 4}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M3102,-93C3102,-93 3072,-93 3072,-93 3066,-93 3060,-87 3060,-81 3060,-81 3060,-69 3060,-69 3060,-63 3066,-57 3072,-57 3072,-57 3102,-57 3102,-57 3108,-57 3114,-63 3114,-69 3114,-69 3114,-81 3114,-81 3114,-87 3108,-93 3102,-93\"/>\n<text text-anchor=\"middle\" x=\"3087\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_add&#45;&gt;pool -->\n<g id=\"edge36\" class=\"edge\"><title>layer3_residual_add&#45;&gt;pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3024.4,-75C3032.39,-75 3041.31,-75 3049.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3049.92,-78.5001 3059.92,-75 3049.92,-71.5001 3049.92,-78.5001\"/>\n</g>\n<!-- classifier_flatten -->\n<g id=\"node35\" class=\"node\"><title>classifier_flatten</title>\n<g id=\"a_node35\"><a xlink:title=\"&lt;class &#39;__main__.Flatten&#39;&gt; {}\">\n<path fill=\"#b3de69\" stroke=\"black\" d=\"M3192,-93C3192,-93 3162,-93 3162,-93 3156,-93 3150,-87 3150,-81 3150,-81 3150,-69 3150,-69 3150,-63 3156,-57 3162,-57 3162,-57 3192,-57 3192,-57 3198,-57 3204,-63 3204,-69 3204,-69 3204,-81 3204,-81 3204,-87 3198,-93 3192,-93\"/>\n<text text-anchor=\"middle\" x=\"3177\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">flatten</text>\n</a>\n</g>\n</g>\n<!-- pool&#45;&gt;classifier_flatten -->\n<g id=\"edge37\" class=\"edge\"><title>pool&#45;&gt;classifier_flatten</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3114.4,-75C3122.39,-75 3131.31,-75 3139.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3139.92,-78.5001 3149.92,-75 3139.92,-71.5001 3139.92,-78.5001\"/>\n</g>\n<!-- classifier_conv -->\n<g id=\"node36\" class=\"node\"><title>classifier_conv</title>\n<g id=\"a_node36\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.linear.Linear&#39;&gt; {&#39;in_features&#39;: 512, &#39;out_features&#39;: 10, &#39;bias&#39;: False}\">\n<path fill=\"#fccde5\" stroke=\"black\" d=\"M3282,-93C3282,-93 3252,-93 3252,-93 3246,-93 3240,-87 3240,-81 3240,-81 3240,-69 3240,-69 3240,-63 3246,-57 3252,-57 3252,-57 3282,-57 3282,-57 3288,-57 3294,-63 3294,-69 3294,-69 3294,-81 3294,-81 3294,-87 3288,-93 3282,-93\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- classifier_flatten&#45;&gt;classifier_conv -->\n<g id=\"edge38\" class=\"edge\"><title>classifier_flatten&#45;&gt;classifier_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3204.4,-75C3212.39,-75 3221.31,-75 3229.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3229.92,-78.5001 3239.92,-75 3229.92,-71.5001 3229.92,-78.5001\"/>\n</g>\n<!-- classifier_scale -->\n<g id=\"node37\" class=\"node\"><title>classifier_scale</title>\n<g id=\"a_node37\"><a xlink:title=\"&lt;class &#39;__main__.Mul&#39;&gt; {&#39;weight&#39;: 0.125}\">\n<path fill=\"#bc80bd\" stroke=\"black\" d=\"M3372,-93C3372,-93 3342,-93 3342,-93 3336,-93 3330,-87 3330,-81 3330,-81 3330,-69 3330,-69 3330,-63 3336,-57 3342,-57 3342,-57 3372,-57 3372,-57 3378,-57 3384,-63 3384,-69 3384,-69 3384,-81 3384,-81 3384,-87 3378,-93 3372,-93\"/>\n<text text-anchor=\"middle\" x=\"3357\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">scale</text>\n</a>\n</g>\n</g>\n<!-- classifier_conv&#45;&gt;classifier_scale -->\n<g id=\"edge39\" class=\"edge\"><title>classifier_conv&#45;&gt;classifier_scale</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3294.4,-75C3302.39,-75 3311.31,-75 3319.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3319.92,-78.5001 3329.92,-75 3319.92,-71.5001 3319.92,-78.5001\"/>\n</g>\n<!-- logits -->\n<g id=\"node38\" class=\"node\"><title>logits</title>\n<g id=\"a_node38\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M3462,-93C3462,-93 3432,-93 3432,-93 3426,-93 3420,-87 3420,-81 3420,-81 3420,-69 3420,-69 3420,-63 3426,-57 3432,-57 3432,-57 3462,-57 3462,-57 3468,-57 3474,-63 3474,-69 3474,-69 3474,-81 3474,-81 3474,-87 3468,-93 3462,-93\"/>\n<text text-anchor=\"middle\" x=\"3447\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">logits</text>\n</a>\n</g>\n</g>\n<!-- classifier_scale&#45;&gt;logits -->\n<g id=\"edge40\" class=\"edge\"><title>classifier_scale&#45;&gt;logits</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3384.4,-75C3392.39,-75 3401.31,-75 3409.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3409.92,-78.5001 3419.92,-75 3409.92,-71.5001 3409.92,-78.5001\"/>\n</g>\n<!-- input -->\n<g id=\"node39\" class=\"node\"><title>input</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-92C42,-92 12,-92 12,-92 6,-92 0,-86 0,-80 0,-80 0,-68 0,-68 0,-62 6,-56 12,-56 12,-56 42,-56 42,-56 48,-56 54,-62 54,-68 54,-68 54,-80 54,-80 54,-86 48,-92 42,-92\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">input</text>\n</g>\n<!-- input&#45;&gt;prep_conv -->\n<g id=\"edge1\" class=\"edge\"><title>input&#45;&gt;prep_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4029,-74C62.3932,-74 71.3106,-74 79.8241,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"79.919,-77.5001 89.919,-74 79.919,-70.5001 79.919,-77.5001\"/>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0eblbuB2_vCh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 24, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, 5, epochs], [0.0, 0.4, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(maxpool_pre_net, x_ent_loss)\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))       \n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5elIHYXi7AVz",
        "colab_type": "text"
      },
      "source": [
        "The results show a small negative effect on test accuracy to 94.0% (mean of 50 runs) compared to our baseline of 94.1%. They also show a substantial 5s reduction in training time! We can restore the previous accuracy by adding an extra epoch to training. This is the only time in the post that we select an 'improvement' that leads to worse accuracy! The 5s gain from a more efficient network more than compensates the 2.5s loss from the extra training epoch. The net effect brings our time to 64s, up to third place on the leaderboard."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b7ecJCjI1SJV",
        "colab_type": "text"
      },
      "source": [
        "### Label smoothing (59s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-NqB3aXtWoel",
        "colab_type": "text"
      },
      "source": [
        "[Label smoothing](https://arxiv.org/abs/1512.00567) is a well-established trick for improving the training speed and generalization of neural nets in classification problems. It involves blending the one-hot target probabilities with a uniform distribution over class labels inside the cross entropy loss. This helps to stabilise the output distribution and prevents the network from making overconfident predictions which might inhibit further training. Let's give it a try - the label smoothing parameter of 0.2 has been very roughly hand-optimised but the result is not too sensitive to a range of choices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s8UjTsDLul0D",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "outputId": "2ffce5c6-e64a-439f-d59c-2b74838e5312"
      },
      "source": [
        "show(label_smoothing_loss(alpha=0.2), size=2)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<__main__.DotGraph at 0x7f41407b8e48>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n -->\n<!-- Title: G Pages: 1 -->\n<svg width=\"144pt\" height=\"51pt\"\n viewBox=\"0.00 0.00 144.00 50.95\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(0.421053 0.421053) rotate(0) translate(4 117)\">\n<title>G</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-117 338,-117 338,4 -4,4\"/>\n<!-- logprobs -->\n<g id=\"node1\" class=\"node\"><title>logprobs</title>\n<g id=\"a_node1\"><a xlink:title=\"&lt;class &#39;__main__.LogSoftmax&#39;&gt; LogSoftmax(dim=1)\">\n<path fill=\"#ccebc5\" stroke=\"black\" d=\"M142,-36C142,-36 102,-36 102,-36 96,-36 90,-30 90,-24 90,-24 90,-12 90,-12 90,-6 96,-0 102,-0 102,-0 142,-0 142,-0 148,-0 154,-6 154,-12 154,-12 154,-24 154,-24 154,-30 148,-36 142,-36\"/>\n<text text-anchor=\"middle\" x=\"122\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">logprobs</text>\n</a>\n</g>\n</g>\n<!-- KL -->\n<g id=\"node2\" class=\"node\"><title>KL</title>\n<g id=\"a_node2\"><a xlink:title=\"&lt;class &#39;__main__.KLLoss&#39;&gt; KLLoss()\">\n<path fill=\"#ffed6f\" stroke=\"black\" d=\"M232,-36C232,-36 202,-36 202,-36 196,-36 190,-30 190,-24 190,-24 190,-12 190,-12 190,-6 196,-0 202,-0 202,-0 232,-0 232,-0 238,-0 244,-6 244,-12 244,-12 244,-24 244,-24 244,-30 238,-36 232,-36\"/>\n<text text-anchor=\"middle\" x=\"217\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">KL</text>\n</a>\n</g>\n</g>\n<!-- logprobs&#45;&gt;KL -->\n<g id=\"edge2\" class=\"edge\"><title>logprobs&#45;&gt;KL</title>\n<path fill=\"none\" stroke=\"black\" d=\"M154.226,-18C162.48,-18 171.455,-18 179.941,-18\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"179.969,-21.5001 189.969,-18 179.969,-14.5001 179.969,-21.5001\"/>\n</g>\n<!-- xent -->\n<g id=\"node3\" class=\"node\"><title>xent</title>\n<g id=\"a_node3\"><a xlink:title=\"&lt;class &#39;__main__.CrossEntropyLoss&#39;&gt; CrossEntropyLoss()\">\n<path fill=\"#1f78b4\" stroke=\"black\" d=\"M232,-90C232,-90 202,-90 202,-90 196,-90 190,-84 190,-78 190,-78 190,-66 190,-66 190,-60 196,-54 202,-54 202,-54 232,-54 232,-54 238,-54 244,-60 244,-66 244,-66 244,-78 244,-78 244,-84 238,-90 232,-90\"/>\n<text text-anchor=\"middle\" x=\"217\" y=\"-68.3\" font-family=\"Times,serif\" font-size=\"14.00\">xent</text>\n</a>\n</g>\n</g>\n<!-- logprobs&#45;&gt;xent -->\n<g id=\"edge3\" class=\"edge\"><title>logprobs&#45;&gt;xent</title>\n<path fill=\"none\" stroke=\"black\" d=\"M154.226,-36.1314C162.842,-41.1343 172.246,-46.5942 181.058,-51.7108\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"179.563,-54.8904 189.969,-56.8851 183.078,-48.8369 179.563,-54.8904\"/>\n</g>\n<!-- loss -->\n<g id=\"node4\" class=\"node\"><title>loss</title>\n<g id=\"a_node4\"><a xlink:title=\"&lt;class &#39;__main__.AddWeighted&#39;&gt; AddWeighted(wx=0.8, wy=0.2)\">\n<path fill=\"#33a02c\" stroke=\"black\" d=\"M322,-63C322,-63 292,-63 292,-63 286,-63 280,-57 280,-51 280,-51 280,-39 280,-39 280,-33 286,-27 292,-27 292,-27 322,-27 322,-27 328,-27 334,-33 334,-39 334,-39 334,-51 334,-51 334,-57 328,-63 322,-63\"/>\n<text text-anchor=\"middle\" x=\"307\" y=\"-41.3\" font-family=\"Times,serif\" font-size=\"14.00\">loss</text>\n</a>\n</g>\n</g>\n<!-- KL&#45;&gt;loss -->\n<g id=\"edge6\" class=\"edge\"><title>KL&#45;&gt;loss</title>\n<path fill=\"none\" stroke=\"black\" d=\"M244.403,-26.1009C252.481,-28.5794 261.507,-31.3486 270.105,-33.9866\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"269.332,-37.4106 279.919,-36.9979 271.386,-30.7185 269.332,-37.4106\"/>\n</g>\n<!-- xent&#45;&gt;loss -->\n<g id=\"edge5\" class=\"edge\"><title>xent&#45;&gt;loss</title>\n<path fill=\"none\" stroke=\"black\" d=\"M244.403,-63.8991C252.481,-61.4206 261.507,-58.6514 270.105,-56.0134\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"271.386,-59.2815 279.919,-53.0021 269.332,-52.5894 271.386,-59.2815\"/>\n</g>\n<!-- acc -->\n<g id=\"node5\" class=\"node\"><title>acc</title>\n<g id=\"a_node5\"><a xlink:title=\"&lt;class &#39;__main__.Correct&#39;&gt; Correct()\">\n<path fill=\"#e31a1c\" stroke=\"black\" d=\"M137,-90C137,-90 107,-90 107,-90 101,-90 95,-84 95,-78 95,-78 95,-66 95,-66 95,-60 101,-54 107,-54 107,-54 137,-54 137,-54 143,-54 149,-60 149,-66 149,-66 149,-78 149,-78 149,-84 143,-90 137,-90\"/>\n<text text-anchor=\"middle\" x=\"122\" y=\"-68.3\" font-family=\"Times,serif\" font-size=\"14.00\">acc</text>\n</a>\n</g>\n</g>\n<!-- logits -->\n<g id=\"node6\" class=\"node\"><title>logits</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-59C42,-59 12,-59 12,-59 6,-59 0,-53 0,-47 0,-47 0,-35 0,-35 0,-29 6,-23 12,-23 12,-23 42,-23 42,-23 48,-23 54,-29 54,-35 54,-35 54,-47 54,-47 54,-53 48,-59 42,-59\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-37.3\" font-family=\"Times,serif\" font-size=\"14.00\">logits</text>\n</g>\n<!-- logits&#45;&gt;logprobs -->\n<g id=\"edge1\" class=\"edge\"><title>logits&#45;&gt;logprobs</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4013,-34.4706C62.3524,-32.5042 71.2743,-30.2977 79.9406,-28.1545\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"81.056,-31.4841 89.9233,-25.6856 79.3754,-24.6889 81.056,-31.4841\"/>\n</g>\n<!-- logits&#45;&gt;acc -->\n<g id=\"edge7\" class=\"edge\"><title>logits&#45;&gt;acc</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4013,-49.8004C64.0054,-53.0018 75.0259,-56.6753 85.3015,-60.1005\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"84.2609,-63.4429 94.8546,-63.2849 86.4746,-56.8021 84.2609,-63.4429\"/>\n</g>\n<!-- target -->\n<g id=\"node7\" class=\"node\"><title>target</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-113C42,-113 12,-113 12,-113 6,-113 0,-107 0,-101 0,-101 0,-89 0,-89 0,-83 6,-77 12,-77 12,-77 42,-77 42,-77 48,-77 54,-83 54,-89 54,-89 54,-101 54,-101 54,-107 48,-113 42,-113\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-91.3\" font-family=\"Times,serif\" font-size=\"14.00\">target</text>\n</g>\n<!-- target&#45;&gt;xent -->\n<g id=\"edge4\" class=\"edge\"><title>target&#45;&gt;xent</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.374,-99.4098C80.0851,-102.875 120.147,-106.051 154,-99 162.935,-97.1388 172.161,-93.918 180.678,-90.3351\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"182.241,-93.47 189.938,-86.1891 179.381,-87.0811 182.241,-93.47\"/>\n</g>\n<!-- target&#45;&gt;acc -->\n<g id=\"edge8\" class=\"edge\"><title>target&#45;&gt;acc</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4013,-88.4706C63.9064,-86.1199 74.7988,-83.4261 84.9835,-80.9073\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"85.9873,-84.2646 94.8546,-78.4661 84.3067,-77.4693 85.9873,-84.2646\"/>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ofLeJMTpbJVc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 25, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.4, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(maxpool_pre_net, label_smoothing_loss(0.2))\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nc5w9r2HSof1",
        "colab_type": "text"
      },
      "source": [
        "Test accuracy improves to 94.2% (mean of 50 runs.) We can trade this for training time by reducing the number of epochs. As a rule of thumb, we reduce training by one epoch for each 0.1% of test accuracy improvement, since this roughly tracks the gain from an extra epoch of training. We reduce the warmup period - during which learning rates increase linearly - in proportion to the overall number of epochs. \n",
        "\n",
        "Accuracy for 23 epochs of training is 94.1% and training time has dipped under a minute!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ug1Qk7-YyJW",
        "colab_type": "text"
      },
      "source": [
        "### CELU activations (52s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EyAlk-aa9l8E",
        "colab_type": "text"
      },
      "source": [
        "We might hope to help the optimisation process by using a smooth activation function, rather than ReLU with its delta-function of curvature at the origin. This may also help generalisation since smoothed functions lead to a less expressive function class - in the large smoothing limit we recover a linear network. \n",
        "\n",
        "We are otherwise happy with ReLU so we're going to pick a simple smoothed-out alternative. Our choice is the Continuously Differentiable Exponential Linear Unit or [CELU](https://arxiv.org/abs/1704.07483) activation since it is smooth (unlike ELU) and the PyTorch implementation is faster than that of the otherwise perfectly adequate Softplus activation. In addition to smoothing, CELU applies an x- and y-shift to ReLU as shown below, but these are largely irrelevant given our use of batch norm."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K7g9dIFu1jB_",
        "colab_type": "code",
        "outputId": "c9238162-2a23-47f6-88ec-7aad73f62283",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 368
        }
      },
      "source": [
        "alpha = 0.075\n",
        "\n",
        "funcs = {\n",
        "    'CELU(x; α)': nn.CELU(alpha),\n",
        "    'ReLU(x+α)-α': lambda x: torch.nn.functional.relu(x+alpha) - alpha,\n",
        "}\n",
        "\n",
        "xs = torch.tensor(np.arange(-1, 1, 0.01))\n",
        "data = pd.DataFrame([{'func': k, 'x': float(to_numpy(x)), 'y': float(to_numpy(f(x)))} for x in xs for k,f in funcs.items()])\n",
        "alt.Chart(data).mark_line().encode(x='x', y='y', color='func')"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<!DOCTYPE html>\n",
              "<html>\n",
              "<head>\n",
              "  <style>\n",
              "    .vega-actions a {\n",
              "        margin-right: 12px;\n",
              "        color: #757575;\n",
              "        font-weight: normal;\n",
              "        font-size: 13px;\n",
              "    }\n",
              "    .error {\n",
              "        color: red;\n",
              "    }\n",
              "  </style>\n",
              "  <script type=\"text/javascript\" src=\"https://cdn.jsdelivr.net/npm//vega@5\"></script>\n",
              "  <script type=\"text/javascript\" src=\"https://cdn.jsdelivr.net/npm//vega-lite@3.4.0\"></script>\n",
              "  <script type=\"text/javascript\" src=\"https://cdn.jsdelivr.net/npm//vega-embed@4\"></script>\n",
              "</head>\n",
              "<body>\n",
              "  <div id=\"altair-viz\"></div>\n",
              "  <script>\n",
              "    (function(vegaEmbed) {\n",
              "      var spec = {\"config\": {\"view\": {\"width\": 400, \"height\": 300}, \"mark\": {\"tooltip\": null}}, \"data\": {\"name\": \"data-e340a0773542cdd52e6083e2ac95ca3d\"}, \"mark\": \"line\", \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"func\"}, \"x\": {\"type\": \"quantitative\", \"field\": \"x\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"y\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v3.4.0.json\", \"datasets\": {\"data-e340a0773542cdd52e6083e2ac95ca3d\": [{\"func\": \"CELU(x; \\u03b1)\", \"x\": -1.0, \"y\": -0.07499987853024057}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -1.0, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.99, \"y\": -0.07499986120491017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.99, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.98, \"y\": -0.07499984140845384}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.98, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.97, \"y\": -0.07499981878841287}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.97, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.96, \"y\": -0.0749997929420571}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.96, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.95, \"y\": -0.0749997634092146}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.95, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.94, \"y\": -0.07499972966407882}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.94, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.9299999999999999, \"y\": -0.07499969110584694}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.9299999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.9199999999999999, \"y\": -0.07499964704802312}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.9199999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.9099999999999999, \"y\": -0.07499959670619613}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.9099999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8999999999999999, \"y\": -0.0749995391840735}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8999999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8899999999999999, \"y\": -0.07499947345752381}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8899999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8799999999999999, \"y\": -0.07499939835634299}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8799999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8699999999999999, \"y\": -0.07499931254341978}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8699999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8599999999999999, \"y\": -0.07499921449092967}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8599999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8499999999999999, \"y\": -0.07499910245313329}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8499999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8399999999999999, \"y\": -0.07499897443529507}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8399999999999999, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8299999999999998, \"y\": -0.07499882815816866}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8299999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8199999999999998, \"y\": -0.07499866101741696}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8199999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.8099999999999998, \"y\": -0.07499847003724416}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.8099999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7999999999999998, \"y\": -0.07499825181741428}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7999999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7899999999999998, \"y\": -0.0749980024727129}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7899999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7799999999999998, \"y\": -0.07499771756377437}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7799999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7699999999999998, \"y\": -0.07499739201804263}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7699999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7599999999999998, \"y\": -0.0749970200394589}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7599999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7499999999999998, \"y\": -0.07499659500526781}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7499999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7399999999999998, \"y\": -0.074996109348105}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7399999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7299999999999998, \"y\": -0.0749955544212668}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7299999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7199999999999998, \"y\": -0.07499492034476318}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7199999999999998, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.7099999999999997, \"y\": -0.07499419582941311}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.7099999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6999999999999997, \"y\": -0.0749933679758505}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6999999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6899999999999997, \"y\": -0.07499242204486221}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6899999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6799999999999997, \"y\": -0.07499134119496915}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6799999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6699999999999997, \"y\": -0.07499010618257844}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6699999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6599999999999997, \"y\": -0.07498869501936783}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6599999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6499999999999997, \"y\": -0.07498708258080293}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6499999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6399999999999997, \"y\": -0.07498524015881655}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6399999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6299999999999997, \"y\": -0.07498313495068659}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6299999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6199999999999997, \"y\": -0.07498072947501204}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6199999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.6099999999999997, \"y\": -0.07497798090438927}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.6099999999999997, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5999999999999996, \"y\": -0.07497484030290731}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5999999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5899999999999996, \"y\": -0.07497125175488645}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5899999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5799999999999996, \"y\": -0.0749671513693482}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5799999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5699999999999996, \"y\": -0.07496246614249195}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5699999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5599999999999996, \"y\": -0.07495711265792575}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5599999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5499999999999996, \"y\": -0.07495099560150995}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5499999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5399999999999996, \"y\": -0.07494400606437174}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5399999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5299999999999996, \"y\": -0.07493601960387745}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5299999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5199999999999996, \"y\": -0.07492689402803948}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5199999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.5099999999999996, \"y\": -0.07491646686391164}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.5099999999999996, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.49999999999999956, \"y\": -0.07490455246489951}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.49999999999999956, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.48999999999999955, \"y\": -0.07489093870548423}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.48999999999999955, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.47999999999999954, \"y\": -0.07487538320451195}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.47999999999999954, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.46999999999999953, \"y\": -0.0748576090098081}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.46999999999999953, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.4599999999999995, \"y\": -0.07483729966728464}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.4599999999999995, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.4499999999999995, \"y\": -0.07481409358675002}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.4499999999999995, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.4399999999999995, \"y\": -0.07478757760411014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.4399999999999995, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.4299999999999995, \"y\": -0.07475727962534078}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.4299999999999995, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.4199999999999995, \"y\": -0.07472266022126378}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.4199999999999995, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.4099999999999995, \"y\": -0.07468310302347939}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.4099999999999995, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.39999999999999947, \"y\": -0.07463790375046263}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.39999999999999947, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.38999999999999946, \"y\": -0.07458625766844294}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.38999999999999946, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.37999999999999945, \"y\": -0.0745272452638187}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.37999999999999945, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.36999999999999944, \"y\": -0.07445981587201689}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.36999999999999944, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.35999999999999943, \"y\": -0.07438276897132348}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.35999999999999943, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.3499999999999994, \"y\": -0.07429473280863785}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.3499999999999994, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.3399999999999994, \"y\": -0.07419413997660099}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.3399999999999994, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.3299999999999994, \"y\": -0.07407919950726985}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.3299999999999994, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.3199999999999994, \"y\": -0.07394786498548986}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.3199999999999994, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.3099999999999994, \"y\": -0.07379779811425158}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.3099999999999994, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.2999999999999994, \"y\": -0.07362632708334492}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.2999999999999994, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.28999999999999937, \"y\": -0.07343039900010061}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.28999999999999937, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.27999999999999936, \"y\": -0.07320652553528958}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.27999999999999936, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.26999999999999935, \"y\": -0.07295072081645304}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.26999999999999935, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.25999999999999934, \"y\": -0.07265843046290767}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.25999999999999934, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.24999999999999933, \"y\": -0.07232445049895604}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.24999999999999933, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.23999999999999932, \"y\": -0.07194283470162251}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.23999999999999932, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.22999999999999932, \"y\": -0.07150678873332121}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.22999999999999932, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.2199999999999993, \"y\": -0.07100854917458085}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.2199999999999993, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.2099999999999993, \"y\": -0.0704392453031086}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.2099999999999993, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.1999999999999993, \"y\": -0.06978874115828983}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.1999999999999993, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.18999999999999928, \"y\": -0.06904545507921908}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.18999999999999928, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.17999999999999927, \"y\": -0.068196153503294}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.17999999999999927, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.16999999999999926, \"y\": -0.06722571535413534}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.16999999999999926, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.15999999999999925, \"y\": -0.06611686282396463}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.15999999999999925, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.14999999999999925, \"y\": -0.06484985375725394}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.14999999999999925, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.13999999999999924, \"y\": -0.06340213015880578}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.13999999999999924, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.12999999999999923, \"y\": -0.06174791656825511}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.12999999999999923, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.11999999999999922, \"y\": -0.05985776115040069}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.11999999999999922, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.10999999999999921, \"y\": -0.057698011330877604}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.10999999999999921, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.0999999999999992, \"y\": -0.05523021464132028}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.0999999999999992, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.08999999999999919, \"y\": -0.0524104341065846}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.08999999999999919, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.07999999999999918, \"y\": -0.04918846598509379}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.07999999999999918, \"y\": -0.075}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.06999999999999917, \"y\": -0.045506945934854805}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.06999999999999917, \"y\": -0.06999999999999917}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.059999999999999165, \"y\": -0.041300327691208005}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.059999999999999165, \"y\": -0.059999999999999165}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.049999999999999156, \"y\": -0.036493716072555166}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.049999999999999156, \"y\": -0.049999999999999156}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.03999999999999915, \"y\": -0.031001533536747113}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.03999999999999915, \"y\": -0.03999999999999915}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.02999999999999914, \"y\": -0.024725996547326476}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.02999999999999914, \"y\": -0.02999999999999914}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.01999999999999913, \"y\": -0.017555374622650683}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.01999999999999913, \"y\": -0.01999999999999913}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": -0.00999999999999912, \"y\": -0.009362001071778172}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": -0.00999999999999912, \"y\": -0.00999999999999912}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 8.881784197001252e-16, \"y\": 8.881784197001252e-16}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 8.881784197001252e-16, \"y\": 8.881784197001252e-16}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.010000000000000897, \"y\": 0.010000000000000897}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.010000000000000897, \"y\": 0.010000000000000897}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.020000000000000906, \"y\": 0.020000000000000906}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.020000000000000906, \"y\": 0.020000000000000906}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.030000000000000915, \"y\": 0.030000000000000915}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.030000000000000915, \"y\": 0.030000000000000915}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.040000000000000924, \"y\": 0.040000000000000924}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.040000000000000924, \"y\": 0.040000000000000924}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.05000000000000093, \"y\": 0.05000000000000093}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.05000000000000093, \"y\": 0.050000000000000946}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.06000000000000094, \"y\": 0.06000000000000094}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.06000000000000094, \"y\": 0.060000000000000955}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.07000000000000095, \"y\": 0.07000000000000095}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.07000000000000095, \"y\": 0.07000000000000096}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.08000000000000096, \"y\": 0.08000000000000096}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.08000000000000096, \"y\": 0.08000000000000097}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.09000000000000097, \"y\": 0.09000000000000097}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.09000000000000097, \"y\": 0.09000000000000098}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.10000000000000098, \"y\": 0.10000000000000098}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.10000000000000098, \"y\": 0.10000000000000099}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.11000000000000099, \"y\": 0.11000000000000099}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.11000000000000099, \"y\": 0.110000000000001}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.120000000000001, \"y\": 0.120000000000001}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.120000000000001, \"y\": 0.12000000000000101}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.130000000000001, \"y\": 0.130000000000001}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.130000000000001, \"y\": 0.130000000000001}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.140000000000001, \"y\": 0.140000000000001}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.140000000000001, \"y\": 0.140000000000001}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.15000000000000102, \"y\": 0.15000000000000102}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.15000000000000102, \"y\": 0.15000000000000102}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.16000000000000103, \"y\": 0.16000000000000103}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.16000000000000103, \"y\": 0.16000000000000103}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.17000000000000104, \"y\": 0.17000000000000104}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.17000000000000104, \"y\": 0.17000000000000104}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.18000000000000105, \"y\": 0.18000000000000105}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.18000000000000105, \"y\": 0.18000000000000105}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.19000000000000106, \"y\": 0.19000000000000106}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.19000000000000106, \"y\": 0.19000000000000106}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.20000000000000107, \"y\": 0.20000000000000107}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.20000000000000107, \"y\": 0.20000000000000107}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.21000000000000107, \"y\": 0.21000000000000107}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.21000000000000107, \"y\": 0.21000000000000107}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.22000000000000108, \"y\": 0.22000000000000108}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.22000000000000108, \"y\": 0.22000000000000108}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.2300000000000011, \"y\": 0.2300000000000011}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.2300000000000011, \"y\": 0.2300000000000011}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.2400000000000011, \"y\": 0.2400000000000011}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.2400000000000011, \"y\": 0.2400000000000011}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.2500000000000011, \"y\": 0.2500000000000011}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.2500000000000011, \"y\": 0.2500000000000011}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.2600000000000011, \"y\": 0.2600000000000011}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.2600000000000011, \"y\": 0.2600000000000011}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.27000000000000113, \"y\": 0.27000000000000113}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.27000000000000113, \"y\": 0.27000000000000113}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.28000000000000114, \"y\": 0.28000000000000114}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.28000000000000114, \"y\": 0.28000000000000114}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.29000000000000115, \"y\": 0.29000000000000115}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.29000000000000115, \"y\": 0.29000000000000115}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.30000000000000115, \"y\": 0.30000000000000115}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.30000000000000115, \"y\": 0.30000000000000115}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.31000000000000116, \"y\": 0.31000000000000116}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.31000000000000116, \"y\": 0.31000000000000116}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.3200000000000012, \"y\": 0.3200000000000012}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.3200000000000012, \"y\": 0.3200000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.3300000000000012, \"y\": 0.3300000000000012}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.3300000000000012, \"y\": 0.3300000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.3400000000000012, \"y\": 0.3400000000000012}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.3400000000000012, \"y\": 0.3400000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.3500000000000012, \"y\": 0.3500000000000012}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.3500000000000012, \"y\": 0.3500000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.3600000000000012, \"y\": 0.3600000000000012}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.3600000000000012, \"y\": 0.3600000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.3700000000000012, \"y\": 0.3700000000000012}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.3700000000000012, \"y\": 0.3700000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.3800000000000012, \"y\": 0.3800000000000012}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.3800000000000012, \"y\": 0.3800000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.39000000000000123, \"y\": 0.39000000000000123}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.39000000000000123, \"y\": 0.39000000000000123}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.40000000000000124, \"y\": 0.40000000000000124}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.40000000000000124, \"y\": 0.40000000000000124}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.41000000000000125, \"y\": 0.41000000000000125}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.41000000000000125, \"y\": 0.41000000000000125}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.42000000000000126, \"y\": 0.42000000000000126}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.42000000000000126, \"y\": 0.42000000000000126}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.43000000000000127, \"y\": 0.43000000000000127}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.43000000000000127, \"y\": 0.4300000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.4400000000000013, \"y\": 0.4400000000000013}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.4400000000000013, \"y\": 0.4400000000000012}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.4500000000000013, \"y\": 0.4500000000000013}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.4500000000000013, \"y\": 0.45000000000000123}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.4600000000000013, \"y\": 0.4600000000000013}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.4600000000000013, \"y\": 0.46000000000000124}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.4700000000000013, \"y\": 0.4700000000000013}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.4700000000000013, \"y\": 0.47000000000000125}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.4800000000000013, \"y\": 0.4800000000000013}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.4800000000000013, \"y\": 0.48000000000000126}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.4900000000000013, \"y\": 0.4900000000000013}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.4900000000000013, \"y\": 0.49000000000000127}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5000000000000013, \"y\": 0.5000000000000013}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5000000000000013, \"y\": 0.5000000000000013}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5100000000000013, \"y\": 0.5100000000000013}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5100000000000013, \"y\": 0.5100000000000013}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5200000000000014, \"y\": 0.5200000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5200000000000014, \"y\": 0.5200000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5300000000000014, \"y\": 0.5300000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5300000000000014, \"y\": 0.5300000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5400000000000014, \"y\": 0.5400000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5400000000000014, \"y\": 0.5400000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5500000000000014, \"y\": 0.5500000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5500000000000014, \"y\": 0.5500000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5600000000000014, \"y\": 0.5600000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5600000000000014, \"y\": 0.5600000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5700000000000014, \"y\": 0.5700000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5700000000000014, \"y\": 0.5700000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5800000000000014, \"y\": 0.5800000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5800000000000014, \"y\": 0.5800000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.5900000000000014, \"y\": 0.5900000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.5900000000000014, \"y\": 0.5900000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6000000000000014, \"y\": 0.6000000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6000000000000014, \"y\": 0.6000000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6100000000000014, \"y\": 0.6100000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6100000000000014, \"y\": 0.6100000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6200000000000014, \"y\": 0.6200000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6200000000000014, \"y\": 0.6200000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6300000000000014, \"y\": 0.6300000000000014}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6300000000000014, \"y\": 0.6300000000000014}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6400000000000015, \"y\": 0.6400000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6400000000000015, \"y\": 0.6400000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6500000000000015, \"y\": 0.6500000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6500000000000015, \"y\": 0.6500000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6600000000000015, \"y\": 0.6600000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6600000000000015, \"y\": 0.6600000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6700000000000015, \"y\": 0.6700000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6700000000000015, \"y\": 0.6700000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6800000000000015, \"y\": 0.6800000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6800000000000015, \"y\": 0.6800000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.6900000000000015, \"y\": 0.6900000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.6900000000000015, \"y\": 0.6900000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7000000000000015, \"y\": 0.7000000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7000000000000015, \"y\": 0.7000000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7100000000000015, \"y\": 0.7100000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7100000000000015, \"y\": 0.7100000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7200000000000015, \"y\": 0.7200000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7200000000000015, \"y\": 0.7200000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7300000000000015, \"y\": 0.7300000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7300000000000015, \"y\": 0.7300000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7400000000000015, \"y\": 0.7400000000000015}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7400000000000015, \"y\": 0.7400000000000015}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7500000000000016, \"y\": 0.7500000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7500000000000016, \"y\": 0.7500000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7600000000000016, \"y\": 0.7600000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7600000000000016, \"y\": 0.7600000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7700000000000016, \"y\": 0.7700000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7700000000000016, \"y\": 0.7700000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7800000000000016, \"y\": 0.7800000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7800000000000016, \"y\": 0.7800000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.7900000000000016, \"y\": 0.7900000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.7900000000000016, \"y\": 0.7900000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8000000000000016, \"y\": 0.8000000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8000000000000016, \"y\": 0.8000000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8100000000000016, \"y\": 0.8100000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8100000000000016, \"y\": 0.8100000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8200000000000016, \"y\": 0.8200000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8200000000000016, \"y\": 0.8200000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8300000000000016, \"y\": 0.8300000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8300000000000016, \"y\": 0.8300000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8400000000000016, \"y\": 0.8400000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8400000000000016, \"y\": 0.8400000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8500000000000016, \"y\": 0.8500000000000016}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8500000000000016, \"y\": 0.8500000000000016}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8600000000000017, \"y\": 0.8600000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8600000000000017, \"y\": 0.8600000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8700000000000017, \"y\": 0.8700000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8700000000000017, \"y\": 0.8700000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8800000000000017, \"y\": 0.8800000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8800000000000017, \"y\": 0.8800000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.8900000000000017, \"y\": 0.8900000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.8900000000000017, \"y\": 0.8900000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9000000000000017, \"y\": 0.9000000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9000000000000017, \"y\": 0.9000000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9100000000000017, \"y\": 0.9100000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9100000000000017, \"y\": 0.9100000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9200000000000017, \"y\": 0.9200000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9200000000000017, \"y\": 0.9200000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9300000000000017, \"y\": 0.9300000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9300000000000017, \"y\": 0.9300000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9400000000000017, \"y\": 0.9400000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9400000000000017, \"y\": 0.9400000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9500000000000017, \"y\": 0.9500000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9500000000000017, \"y\": 0.9500000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9600000000000017, \"y\": 0.9600000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9600000000000017, \"y\": 0.9600000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9700000000000017, \"y\": 0.9700000000000017}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9700000000000017, \"y\": 0.9700000000000017}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9800000000000018, \"y\": 0.9800000000000018}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9800000000000018, \"y\": 0.9800000000000018}, {\"func\": \"CELU(x; \\u03b1)\", \"x\": 0.9900000000000018, \"y\": 0.9900000000000018}, {\"func\": \"ReLU(x+\\u03b1)-\\u03b1\", \"x\": 0.9900000000000018, \"y\": 0.9900000000000018}]}};\n",
              "      var embedOpt = {\"mode\": \"vega-lite\"};\n",
              "\n",
              "      function showError(el, error){\n",
              "          el.innerHTML = ('<div class=\"error\" style=\"color:red;\">'\n",
              "                          + '<p>JavaScript Error: ' + error.message + '</p>'\n",
              "                          + \"<p>This usually means there's a typo in your chart specification. \"\n",
              "                          + \"See the javascript console for the full traceback.</p>\"\n",
              "                          + '</div>');\n",
              "          throw error;\n",
              "      }\n",
              "      const el = document.getElementById('altair-viz');\n",
              "      vegaEmbed(\"#altair-viz\", spec, embedOpt)\n",
              "        .catch(error => showError(el, error));\n",
              "    })(vegaEmbed);\n",
              "\n",
              "  </script>\n",
              "</body>\n",
              "</html>"
            ],
            "text/plain": [
              "alt.Chart(...)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-28Zj2A3chKd",
        "colab_type": "text"
      },
      "source": [
        "Let's see how it works  - as usual the smoothing parameter $\\alpha$ has been roughly manually tuned."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x8KwIQjrGQlR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        },
        "outputId": "ce717079-d602-40c3-c2f3-b43f34685f24"
      },
      "source": [
        "celu_net = network(conv_pool_block=conv_pool_block_pre, types={nn.ReLU: partial(nn.CELU, alpha=0.075)})\n",
        "show(celu_net)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<__main__.DotGraph at 0x7f41407e1438>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n -->\n<!-- Title: G Pages: 1 -->\n<svg width=\"1080pt\" height=\"59pt\"\n viewBox=\"0.00 0.00 1080.00 59.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(0.310167 0.310167) rotate(0) translate(4 187)\">\n<title>G</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-187 3478,-187 3478,4 -4,4\"/>\n<g id=\"clust1\" class=\"cluster\"><title>cluster_prep</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M94,-48C94,-48 320,-48 320,-48 326,-48 332,-54 332,-60 332,-60 332,-111 332,-111 332,-117 326,-123 320,-123 320,-123 94,-123 94,-123 88,-123 82,-117 82,-111 82,-111 82,-60 82,-60 82,-54 88,-48 94,-48\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-107.8\" font-family=\"Times,serif\" font-size=\"14.00\">prep</text>\n</g>\n<g id=\"clust2\" class=\"cluster\"><title>cluster_layer1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M364,-12C364,-12 1498,-12 1498,-12 1504,-12 1510,-18 1510,-24 1510,-24 1510,-163 1510,-163 1510,-169 1504,-175 1498,-175 1498,-175 364,-175 364,-175 358,-175 352,-169 352,-163 352,-163 352,-24 352,-24 352,-18 358,-12 364,-12\"/>\n<text text-anchor=\"middle\" x=\"931\" y=\"-159.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer1</text>\n</g>\n<g id=\"clust3\" class=\"cluster\"><title>cluster_layer1_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M724,-20C724,-20 1490,-20 1490,-20 1496,-20 1502,-26 1502,-32 1502,-32 1502,-132 1502,-132 1502,-138 1496,-144 1490,-144 1490,-144 724,-144 724,-144 718,-144 712,-138 712,-132 712,-132 712,-32 712,-32 712,-26 718,-20 724,-20\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-128.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust4\" class=\"cluster\"><title>cluster_layer1_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M814,-28C814,-28 1040,-28 1040,-28 1046,-28 1052,-34 1052,-40 1052,-40 1052,-91 1052,-91 1052,-97 1046,-103 1040,-103 1040,-103 814,-103 814,-103 808,-103 802,-97 802,-91 802,-91 802,-40 802,-40 802,-34 808,-28 814,-28\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust5\" class=\"cluster\"><title>cluster_layer1_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1084,-28C1084,-28 1310,-28 1310,-28 1316,-28 1322,-34 1322,-40 1322,-40 1322,-91 1322,-91 1322,-97 1316,-103 1310,-103 1310,-103 1084,-103 1084,-103 1078,-103 1072,-97 1072,-91 1072,-91 1072,-40 1072,-40 1072,-34 1078,-28 1084,-28\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust6\" class=\"cluster\"><title>cluster_layer2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1534,-53C1534,-53 1850,-53 1850,-53 1856,-53 1862,-59 1862,-65 1862,-65 1862,-116 1862,-116 1862,-122 1856,-128 1850,-128 1850,-128 1534,-128 1534,-128 1528,-128 1522,-122 1522,-116 1522,-116 1522,-65 1522,-65 1522,-59 1528,-53 1534,-53\"/>\n<text text-anchor=\"middle\" x=\"1692\" y=\"-112.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer2</text>\n</g>\n<g id=\"clust7\" class=\"cluster\"><title>cluster_layer3</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1894,-8C1894,-8 3028,-8 3028,-8 3034,-8 3040,-14 3040,-20 3040,-20 3040,-159 3040,-159 3040,-165 3034,-171 3028,-171 3028,-171 1894,-171 1894,-171 1888,-171 1882,-165 1882,-159 1882,-159 1882,-20 1882,-20 1882,-14 1888,-8 1894,-8\"/>\n<text text-anchor=\"middle\" x=\"2461\" y=\"-155.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer3</text>\n</g>\n<g id=\"clust8\" class=\"cluster\"><title>cluster_layer3_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2254,-16C2254,-16 3020,-16 3020,-16 3026,-16 3032,-22 3032,-28 3032,-28 3032,-128 3032,-128 3032,-134 3026,-140 3020,-140 3020,-140 2254,-140 2254,-140 2248,-140 2242,-134 2242,-128 2242,-128 2242,-28 2242,-28 2242,-22 2248,-16 2254,-16\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust9\" class=\"cluster\"><title>cluster_layer3_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2344,-24C2344,-24 2570,-24 2570,-24 2576,-24 2582,-30 2582,-36 2582,-36 2582,-87 2582,-87 2582,-93 2576,-99 2570,-99 2570,-99 2344,-99 2344,-99 2338,-99 2332,-93 2332,-87 2332,-87 2332,-36 2332,-36 2332,-30 2338,-24 2344,-24\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust10\" class=\"cluster\"><title>cluster_layer3_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2614,-24C2614,-24 2840,-24 2840,-24 2846,-24 2852,-30 2852,-36 2852,-36 2852,-87 2852,-87 2852,-93 2846,-99 2840,-99 2840,-99 2614,-99 2614,-99 2608,-99 2602,-93 2602,-87 2602,-87 2602,-36 2602,-36 2602,-30 2608,-24 2614,-24\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust11\" class=\"cluster\"><title>cluster_classifier</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M3154,-49C3154,-49 3380,-49 3380,-49 3386,-49 3392,-55 3392,-61 3392,-61 3392,-112 3392,-112 3392,-118 3386,-124 3380,-124 3380,-124 3154,-124 3154,-124 3148,-124 3142,-118 3142,-112 3142,-112 3142,-61 3142,-61 3142,-55 3148,-49 3154,-49\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-108.8\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n</g>\n<!-- prep_conv -->\n<g id=\"node1\" class=\"node\"><title>prep_conv</title>\n<g id=\"a_node1\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 3, &#39;out_channels&#39;: 64, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M132,-92C132,-92 102,-92 102,-92 96,-92 90,-86 90,-80 90,-80 90,-68 90,-68 90,-62 96,-56 102,-56 102,-56 132,-56 132,-56 138,-56 144,-62 144,-68 144,-68 144,-80 144,-80 144,-86 138,-92 132,-92\"/>\n<text text-anchor=\"middle\" x=\"117\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_norm -->\n<g id=\"node2\" class=\"node\"><title>prep_norm</title>\n<g id=\"a_node2\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 64}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M222,-92C222,-92 192,-92 192,-92 186,-92 180,-86 180,-80 180,-80 180,-68 180,-68 180,-62 186,-56 192,-56 192,-56 222,-56 222,-56 228,-56 234,-62 234,-68 234,-68 234,-80 234,-80 234,-86 228,-92 222,-92\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- prep_conv&#45;&gt;prep_norm -->\n<g id=\"edge2\" class=\"edge\"><title>prep_conv&#45;&gt;prep_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M144.403,-74C152.393,-74 161.311,-74 169.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"169.919,-77.5001 179.919,-74 169.919,-70.5001 169.919,-77.5001\"/>\n</g>\n<!-- prep_act -->\n<g id=\"node3\" class=\"node\"><title>prep_act</title>\n<g id=\"a_node3\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, alpha=0.075) {}\">\n<path fill=\"#ff7f00\" stroke=\"black\" d=\"M312,-92C312,-92 282,-92 282,-92 276,-92 270,-86 270,-80 270,-80 270,-68 270,-68 270,-62 276,-56 282,-56 282,-56 312,-56 312,-56 318,-56 324,-62 324,-68 324,-68 324,-80 324,-80 324,-86 318,-92 312,-92\"/>\n<text text-anchor=\"middle\" x=\"297\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- prep_norm&#45;&gt;prep_act -->\n<g id=\"edge3\" class=\"edge\"><title>prep_norm&#45;&gt;prep_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M234.403,-74C242.393,-74 251.311,-74 259.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"259.919,-77.5001 269.919,-74 259.919,-70.5001 259.919,-77.5001\"/>\n</g>\n<!-- layer1_conv -->\n<g id=\"node4\" class=\"node\"><title>layer1_conv</title>\n<g id=\"a_node4\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 64, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M402,-92C402,-92 372,-92 372,-92 366,-92 360,-86 360,-80 360,-80 360,-68 360,-68 360,-62 366,-56 372,-56 372,-56 402,-56 402,-56 408,-56 414,-62 414,-68 414,-68 414,-80 414,-80 414,-86 408,-92 402,-92\"/>\n<text text-anchor=\"middle\" x=\"387\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_act&#45;&gt;layer1_conv -->\n<g id=\"edge4\" class=\"edge\"><title>prep_act&#45;&gt;layer1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M324.403,-74C332.393,-74 341.311,-74 349.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"349.919,-77.5001 359.919,-74 349.919,-70.5001 349.919,-77.5001\"/>\n</g>\n<!-- layer1_pool -->\n<g id=\"node5\" class=\"node\"><title>layer1_pool</title>\n<g id=\"a_node5\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M492,-92C492,-92 462,-92 462,-92 456,-92 450,-86 450,-80 450,-80 450,-68 450,-68 450,-62 456,-56 462,-56 462,-56 492,-56 492,-56 498,-56 504,-62 504,-68 504,-68 504,-80 504,-80 504,-86 498,-92 492,-92\"/>\n<text text-anchor=\"middle\" x=\"477\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer1_conv&#45;&gt;layer1_pool -->\n<g id=\"edge5\" class=\"edge\"><title>layer1_conv&#45;&gt;layer1_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M414.403,-74C422.393,-74 431.311,-74 439.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"439.919,-77.5001 449.919,-74 439.919,-70.5001 439.919,-77.5001\"/>\n</g>\n<!-- layer1_norm -->\n<g id=\"node6\" class=\"node\"><title>layer1_norm</title>\n<g id=\"a_node6\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M582,-92C582,-92 552,-92 552,-92 546,-92 540,-86 540,-80 540,-80 540,-68 540,-68 540,-62 546,-56 552,-56 552,-56 582,-56 582,-56 588,-56 594,-62 594,-68 594,-68 594,-80 594,-80 594,-86 588,-92 582,-92\"/>\n<text text-anchor=\"middle\" x=\"567\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_pool&#45;&gt;layer1_norm -->\n<g id=\"edge6\" class=\"edge\"><title>layer1_pool&#45;&gt;layer1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M504.403,-74C512.393,-74 521.311,-74 529.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"529.919,-77.5001 539.919,-74 529.919,-70.5001 529.919,-77.5001\"/>\n</g>\n<!-- layer1_act -->\n<g id=\"node7\" class=\"node\"><title>layer1_act</title>\n<g id=\"a_node7\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, alpha=0.075) {}\">\n<path fill=\"#ff7f00\" stroke=\"black\" d=\"M672,-92C672,-92 642,-92 642,-92 636,-92 630,-86 630,-80 630,-80 630,-68 630,-68 630,-62 636,-56 642,-56 642,-56 672,-56 672,-56 678,-56 684,-62 684,-68 684,-68 684,-80 684,-80 684,-86 678,-92 672,-92\"/>\n<text text-anchor=\"middle\" x=\"657\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_norm&#45;&gt;layer1_act -->\n<g id=\"edge7\" class=\"edge\"><title>layer1_norm&#45;&gt;layer1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M594.403,-74C602.393,-74 611.311,-74 619.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"619.919,-77.5001 629.919,-74 619.919,-70.5001 619.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_in -->\n<g id=\"node8\" class=\"node\"><title>layer1_residual_in</title>\n<g id=\"a_node8\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M762,-92C762,-92 732,-92 732,-92 726,-92 720,-86 720,-80 720,-80 720,-68 720,-68 720,-62 726,-56 732,-56 732,-56 762,-56 762,-56 768,-56 774,-62 774,-68 774,-68 774,-80 774,-80 774,-86 768,-92 762,-92\"/>\n<text text-anchor=\"middle\" x=\"747\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer1_act&#45;&gt;layer1_residual_in -->\n<g id=\"edge8\" class=\"edge\"><title>layer1_act&#45;&gt;layer1_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M684.403,-74C692.393,-74 701.311,-74 709.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"709.919,-77.5001 719.919,-74 709.919,-70.5001 709.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_res1_conv -->\n<g id=\"node9\" class=\"node\"><title>layer1_residual_res1_conv</title>\n<g id=\"a_node9\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M852,-72C852,-72 822,-72 822,-72 816,-72 810,-66 810,-60 810,-60 810,-48 810,-48 810,-42 816,-36 822,-36 822,-36 852,-36 852,-36 858,-36 864,-42 864,-48 864,-48 864,-60 864,-60 864,-66 858,-72 852,-72\"/>\n<text text-anchor=\"middle\" x=\"837\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_res1_conv -->\n<g id=\"edge9\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M774.403,-67.9993C782.481,-66.1634 791.507,-64.1121 800.105,-62.158\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"800.943,-65.5568 809.919,-59.9275 799.392,-58.7308 800.943,-65.5568\"/>\n</g>\n<!-- layer1_residual_add -->\n<g id=\"node16\" class=\"node\"><title>layer1_residual_add</title>\n<g id=\"a_node16\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M1482,-97C1482,-97 1452,-97 1452,-97 1446,-97 1440,-91 1440,-85 1440,-85 1440,-73 1440,-73 1440,-67 1446,-61 1452,-61 1452,-61 1482,-61 1482,-61 1488,-61 1494,-67 1494,-73 1494,-73 1494,-85 1494,-85 1494,-91 1488,-97 1482,-97\"/>\n<text text-anchor=\"middle\" x=\"1467\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_add -->\n<g id=\"edge16\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M771.737,-92.1018C780.735,-97.9639 791.381,-103.78 802,-107 854.782,-123.006 870.844,-112 926,-112 926,-112 926,-112 1288,-112 1337.77,-112 1394.08,-99.3711 1429.8,-89.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1431.04,-93.0265 1439.74,-86.9884 1429.17,-86.2796 1431.04,-93.0265\"/>\n</g>\n<!-- layer1_residual_res1_norm -->\n<g id=\"node10\" class=\"node\"><title>layer1_residual_res1_norm</title>\n<g id=\"a_node10\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M942,-72C942,-72 912,-72 912,-72 906,-72 900,-66 900,-60 900,-60 900,-48 900,-48 900,-42 906,-36 912,-36 912,-36 942,-36 942,-36 948,-36 954,-42 954,-48 954,-48 954,-60 954,-60 954,-66 948,-72 942,-72\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm -->\n<g id=\"edge10\" class=\"edge\"><title>layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M864.403,-54C872.393,-54 881.311,-54 889.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"889.919,-57.5001 899.919,-54 889.919,-50.5001 889.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res1_act -->\n<g id=\"node11\" class=\"node\"><title>layer1_residual_res1_act</title>\n<g id=\"a_node11\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, alpha=0.075) {}\">\n<path fill=\"#ff7f00\" stroke=\"black\" d=\"M1032,-72C1032,-72 1002,-72 1002,-72 996,-72 990,-66 990,-60 990,-60 990,-48 990,-48 990,-42 996,-36 1002,-36 1002,-36 1032,-36 1032,-36 1038,-36 1044,-42 1044,-48 1044,-48 1044,-60 1044,-60 1044,-66 1038,-72 1032,-72\"/>\n<text text-anchor=\"middle\" x=\"1017\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act -->\n<g id=\"edge11\" class=\"edge\"><title>layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M954.403,-54C962.393,-54 971.311,-54 979.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"979.919,-57.5001 989.919,-54 979.919,-50.5001 979.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_conv -->\n<g id=\"node12\" class=\"node\"><title>layer1_residual_res2_conv</title>\n<g id=\"a_node12\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1122,-72C1122,-72 1092,-72 1092,-72 1086,-72 1080,-66 1080,-60 1080,-60 1080,-48 1080,-48 1080,-42 1086,-36 1092,-36 1092,-36 1122,-36 1122,-36 1128,-36 1134,-42 1134,-48 1134,-48 1134,-60 1134,-60 1134,-66 1128,-72 1122,-72\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv -->\n<g id=\"edge12\" class=\"edge\"><title>layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1044.4,-54C1052.39,-54 1061.31,-54 1069.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1069.92,-57.5001 1079.92,-54 1069.92,-50.5001 1069.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_norm -->\n<g id=\"node13\" class=\"node\"><title>layer1_residual_res2_norm</title>\n<g id=\"a_node13\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 128}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M1212,-72C1212,-72 1182,-72 1182,-72 1176,-72 1170,-66 1170,-60 1170,-60 1170,-48 1170,-48 1170,-42 1176,-36 1182,-36 1182,-36 1212,-36 1212,-36 1218,-36 1224,-42 1224,-48 1224,-48 1224,-60 1224,-60 1224,-66 1218,-72 1212,-72\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm -->\n<g id=\"edge13\" class=\"edge\"><title>layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1134.4,-54C1142.39,-54 1151.31,-54 1159.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1159.92,-57.5001 1169.92,-54 1159.92,-50.5001 1159.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_act -->\n<g id=\"node14\" class=\"node\"><title>layer1_residual_res2_act</title>\n<g id=\"a_node14\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, alpha=0.075) {}\">\n<path fill=\"#ff7f00\" stroke=\"black\" d=\"M1302,-72C1302,-72 1272,-72 1272,-72 1266,-72 1260,-66 1260,-60 1260,-60 1260,-48 1260,-48 1260,-42 1266,-36 1272,-36 1272,-36 1302,-36 1302,-36 1308,-36 1314,-42 1314,-48 1314,-48 1314,-60 1314,-60 1314,-66 1308,-72 1302,-72\"/>\n<text text-anchor=\"middle\" x=\"1287\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act -->\n<g id=\"edge14\" class=\"edge\"><title>layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1224.4,-54C1232.39,-54 1241.31,-54 1249.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1249.92,-57.5001 1259.92,-54 1249.92,-50.5001 1249.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_out -->\n<g id=\"node15\" class=\"node\"><title>layer1_residual_out</title>\n<g id=\"a_node15\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M1392,-82C1392,-82 1362,-82 1362,-82 1356,-82 1350,-76 1350,-70 1350,-70 1350,-58 1350,-58 1350,-52 1356,-46 1362,-46 1362,-46 1392,-46 1392,-46 1398,-46 1404,-52 1404,-58 1404,-58 1404,-70 1404,-70 1404,-76 1398,-82 1392,-82\"/>\n<text text-anchor=\"middle\" x=\"1377\" y=\"-60.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_act&#45;&gt;layer1_residual_out -->\n<g id=\"edge15\" class=\"edge\"><title>layer1_residual_res2_act&#45;&gt;layer1_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1314.4,-57.0003C1322.39,-57.9083 1331.31,-58.9217 1339.82,-59.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1339.59,-63.3847 1349.92,-61.0362 1340.38,-56.4294 1339.59,-63.3847\"/>\n</g>\n<!-- layer1_residual_out&#45;&gt;layer1_residual_add -->\n<g id=\"edge17\" class=\"edge\"><title>layer1_residual_out&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1404.4,-68.5005C1412.39,-69.8625 1421.31,-71.3825 1429.82,-72.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1429.47,-76.3242 1439.92,-74.5544 1430.65,-69.4237 1429.47,-76.3242\"/>\n</g>\n<!-- layer2_conv -->\n<g id=\"node17\" class=\"node\"><title>layer2_conv</title>\n<g id=\"a_node17\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 256, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1572,-97C1572,-97 1542,-97 1542,-97 1536,-97 1530,-91 1530,-85 1530,-85 1530,-73 1530,-73 1530,-67 1536,-61 1542,-61 1542,-61 1572,-61 1572,-61 1578,-61 1584,-67 1584,-73 1584,-73 1584,-85 1584,-85 1584,-91 1578,-97 1572,-97\"/>\n<text text-anchor=\"middle\" x=\"1557\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_add&#45;&gt;layer2_conv -->\n<g id=\"edge18\" class=\"edge\"><title>layer1_residual_add&#45;&gt;layer2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1494.4,-79C1502.39,-79 1511.31,-79 1519.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1519.92,-82.5001 1529.92,-79 1519.92,-75.5001 1519.92,-82.5001\"/>\n</g>\n<!-- layer2_pool -->\n<g id=\"node18\" class=\"node\"><title>layer2_pool</title>\n<g id=\"a_node18\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M1662,-97C1662,-97 1632,-97 1632,-97 1626,-97 1620,-91 1620,-85 1620,-85 1620,-73 1620,-73 1620,-67 1626,-61 1632,-61 1632,-61 1662,-61 1662,-61 1668,-61 1674,-67 1674,-73 1674,-73 1674,-85 1674,-85 1674,-91 1668,-97 1662,-97\"/>\n<text text-anchor=\"middle\" x=\"1647\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer2_conv&#45;&gt;layer2_pool -->\n<g id=\"edge19\" class=\"edge\"><title>layer2_conv&#45;&gt;layer2_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1584.4,-79C1592.39,-79 1601.31,-79 1609.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1609.92,-82.5001 1619.92,-79 1609.92,-75.5001 1609.92,-82.5001\"/>\n</g>\n<!-- layer2_norm -->\n<g id=\"node19\" class=\"node\"><title>layer2_norm</title>\n<g id=\"a_node19\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 256}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M1752,-97C1752,-97 1722,-97 1722,-97 1716,-97 1710,-91 1710,-85 1710,-85 1710,-73 1710,-73 1710,-67 1716,-61 1722,-61 1722,-61 1752,-61 1752,-61 1758,-61 1764,-67 1764,-73 1764,-73 1764,-85 1764,-85 1764,-91 1758,-97 1752,-97\"/>\n<text text-anchor=\"middle\" x=\"1737\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer2_pool&#45;&gt;layer2_norm -->\n<g id=\"edge20\" class=\"edge\"><title>layer2_pool&#45;&gt;layer2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1674.4,-79C1682.39,-79 1691.31,-79 1699.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1699.92,-82.5001 1709.92,-79 1699.92,-75.5001 1699.92,-82.5001\"/>\n</g>\n<!-- layer2_act -->\n<g id=\"node20\" class=\"node\"><title>layer2_act</title>\n<g id=\"a_node20\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, alpha=0.075) {}\">\n<path fill=\"#ff7f00\" stroke=\"black\" d=\"M1842,-97C1842,-97 1812,-97 1812,-97 1806,-97 1800,-91 1800,-85 1800,-85 1800,-73 1800,-73 1800,-67 1806,-61 1812,-61 1812,-61 1842,-61 1842,-61 1848,-61 1854,-67 1854,-73 1854,-73 1854,-85 1854,-85 1854,-91 1848,-97 1842,-97\"/>\n<text text-anchor=\"middle\" x=\"1827\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer2_norm&#45;&gt;layer2_act -->\n<g id=\"edge21\" class=\"edge\"><title>layer2_norm&#45;&gt;layer2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1764.4,-79C1772.39,-79 1781.31,-79 1789.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1789.92,-82.5001 1799.92,-79 1789.92,-75.5001 1789.92,-82.5001\"/>\n</g>\n<!-- layer3_conv -->\n<g id=\"node21\" class=\"node\"><title>layer3_conv</title>\n<g id=\"a_node21\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 256, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1932,-97C1932,-97 1902,-97 1902,-97 1896,-97 1890,-91 1890,-85 1890,-85 1890,-73 1890,-73 1890,-67 1896,-61 1902,-61 1902,-61 1932,-61 1932,-61 1938,-61 1944,-67 1944,-73 1944,-73 1944,-85 1944,-85 1944,-91 1938,-97 1932,-97\"/>\n<text text-anchor=\"middle\" x=\"1917\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer2_act&#45;&gt;layer3_conv -->\n<g id=\"edge22\" class=\"edge\"><title>layer2_act&#45;&gt;layer3_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1854.4,-79C1862.39,-79 1871.31,-79 1879.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1879.92,-82.5001 1889.92,-79 1879.92,-75.5001 1879.92,-82.5001\"/>\n</g>\n<!-- layer3_pool -->\n<g id=\"node22\" class=\"node\"><title>layer3_pool</title>\n<g id=\"a_node22\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M2022,-97C2022,-97 1992,-97 1992,-97 1986,-97 1980,-91 1980,-85 1980,-85 1980,-73 1980,-73 1980,-67 1986,-61 1992,-61 1992,-61 2022,-61 2022,-61 2028,-61 2034,-67 2034,-73 2034,-73 2034,-85 2034,-85 2034,-91 2028,-97 2022,-97\"/>\n<text text-anchor=\"middle\" x=\"2007\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_conv&#45;&gt;layer3_pool -->\n<g id=\"edge23\" class=\"edge\"><title>layer3_conv&#45;&gt;layer3_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1944.4,-79C1952.39,-79 1961.31,-79 1969.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1969.92,-82.5001 1979.92,-79 1969.92,-75.5001 1969.92,-82.5001\"/>\n</g>\n<!-- layer3_norm -->\n<g id=\"node23\" class=\"node\"><title>layer3_norm</title>\n<g id=\"a_node23\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2112,-97C2112,-97 2082,-97 2082,-97 2076,-97 2070,-91 2070,-85 2070,-85 2070,-73 2070,-73 2070,-67 2076,-61 2082,-61 2082,-61 2112,-61 2112,-61 2118,-61 2124,-67 2124,-73 2124,-73 2124,-85 2124,-85 2124,-91 2118,-97 2112,-97\"/>\n<text text-anchor=\"middle\" x=\"2097\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_pool&#45;&gt;layer3_norm -->\n<g id=\"edge24\" class=\"edge\"><title>layer3_pool&#45;&gt;layer3_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2034.4,-79C2042.39,-79 2051.31,-79 2059.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2059.92,-82.5001 2069.92,-79 2059.92,-75.5001 2059.92,-82.5001\"/>\n</g>\n<!-- layer3_act -->\n<g id=\"node24\" class=\"node\"><title>layer3_act</title>\n<g id=\"a_node24\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, alpha=0.075) {}\">\n<path fill=\"#ff7f00\" stroke=\"black\" d=\"M2202,-97C2202,-97 2172,-97 2172,-97 2166,-97 2160,-91 2160,-85 2160,-85 2160,-73 2160,-73 2160,-67 2166,-61 2172,-61 2172,-61 2202,-61 2202,-61 2208,-61 2214,-67 2214,-73 2214,-73 2214,-85 2214,-85 2214,-91 2208,-97 2202,-97\"/>\n<text text-anchor=\"middle\" x=\"2187\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_norm&#45;&gt;layer3_act -->\n<g id=\"edge25\" class=\"edge\"><title>layer3_norm&#45;&gt;layer3_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2124.4,-79C2132.39,-79 2141.31,-79 2149.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2149.92,-82.5001 2159.92,-79 2149.92,-75.5001 2149.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_in -->\n<g id=\"node25\" class=\"node\"><title>layer3_residual_in</title>\n<g id=\"a_node25\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2292,-97C2292,-97 2262,-97 2262,-97 2256,-97 2250,-91 2250,-85 2250,-85 2250,-73 2250,-73 2250,-67 2256,-61 2262,-61 2262,-61 2292,-61 2292,-61 2298,-61 2304,-67 2304,-73 2304,-73 2304,-85 2304,-85 2304,-91 2298,-97 2292,-97\"/>\n<text text-anchor=\"middle\" x=\"2277\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer3_act&#45;&gt;layer3_residual_in -->\n<g id=\"edge26\" class=\"edge\"><title>layer3_act&#45;&gt;layer3_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2214.4,-79C2222.39,-79 2231.31,-79 2239.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2239.92,-82.5001 2249.92,-79 2239.92,-75.5001 2239.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_res1_conv -->\n<g id=\"node26\" class=\"node\"><title>layer3_residual_res1_conv</title>\n<g id=\"a_node26\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2382,-68C2382,-68 2352,-68 2352,-68 2346,-68 2340,-62 2340,-56 2340,-56 2340,-44 2340,-44 2340,-38 2346,-32 2352,-32 2352,-32 2382,-32 2382,-32 2388,-32 2394,-38 2394,-44 2394,-44 2394,-56 2394,-56 2394,-62 2388,-68 2382,-68\"/>\n<text text-anchor=\"middle\" x=\"2367\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_res1_conv -->\n<g id=\"edge27\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.4,-70.299C2312.57,-67.608 2321.7,-64.5979 2330.38,-61.7368\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2331.52,-65.049 2339.92,-58.5949 2329.33,-58.4007 2331.52,-65.049\"/>\n</g>\n<!-- layer3_residual_add -->\n<g id=\"node33\" class=\"node\"><title>layer3_residual_add</title>\n<g id=\"a_node33\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M3012,-93C3012,-93 2982,-93 2982,-93 2976,-93 2970,-87 2970,-81 2970,-81 2970,-69 2970,-69 2970,-63 2976,-57 2982,-57 2982,-57 3012,-57 3012,-57 3018,-57 3024,-63 3024,-69 3024,-69 3024,-81 3024,-81 3024,-87 3018,-93 3012,-93\"/>\n<text text-anchor=\"middle\" x=\"2997\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_add -->\n<g id=\"edge34\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.22,-93.0359C2312.83,-96.9904 2322.61,-100.808 2332,-103 2385.71,-115.535 2400.84,-108 2456,-108 2456,-108 2456,-108 2818,-108 2867.77,-108 2924.08,-95.3711 2959.8,-85.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2961.04,-89.0265 2969.74,-82.9884 2959.17,-82.2796 2961.04,-89.0265\"/>\n</g>\n<!-- layer3_residual_res1_norm -->\n<g id=\"node27\" class=\"node\"><title>layer3_residual_res1_norm</title>\n<g id=\"a_node27\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2472,-68C2472,-68 2442,-68 2442,-68 2436,-68 2430,-62 2430,-56 2430,-56 2430,-44 2430,-44 2430,-38 2436,-32 2442,-32 2442,-32 2472,-32 2472,-32 2478,-32 2484,-38 2484,-44 2484,-44 2484,-56 2484,-56 2484,-62 2478,-68 2472,-68\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm -->\n<g id=\"edge28\" class=\"edge\"><title>layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2394.4,-50C2402.39,-50 2411.31,-50 2419.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2419.92,-53.5001 2429.92,-50 2419.92,-46.5001 2419.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res1_act -->\n<g id=\"node28\" class=\"node\"><title>layer3_residual_res1_act</title>\n<g id=\"a_node28\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, alpha=0.075) {}\">\n<path fill=\"#ff7f00\" stroke=\"black\" d=\"M2562,-68C2562,-68 2532,-68 2532,-68 2526,-68 2520,-62 2520,-56 2520,-56 2520,-44 2520,-44 2520,-38 2526,-32 2532,-32 2532,-32 2562,-32 2562,-32 2568,-32 2574,-38 2574,-44 2574,-44 2574,-56 2574,-56 2574,-62 2568,-68 2562,-68\"/>\n<text text-anchor=\"middle\" x=\"2547\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act -->\n<g id=\"edge29\" class=\"edge\"><title>layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2484.4,-50C2492.39,-50 2501.31,-50 2509.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2509.92,-53.5001 2519.92,-50 2509.92,-46.5001 2509.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_conv -->\n<g id=\"node29\" class=\"node\"><title>layer3_residual_res2_conv</title>\n<g id=\"a_node29\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2652,-68C2652,-68 2622,-68 2622,-68 2616,-68 2610,-62 2610,-56 2610,-56 2610,-44 2610,-44 2610,-38 2616,-32 2622,-32 2622,-32 2652,-32 2652,-32 2658,-32 2664,-38 2664,-44 2664,-44 2664,-56 2664,-56 2664,-62 2658,-68 2652,-68\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv -->\n<g id=\"edge30\" class=\"edge\"><title>layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2574.4,-50C2582.39,-50 2591.31,-50 2599.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2599.92,-53.5001 2609.92,-50 2599.92,-46.5001 2599.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_norm -->\n<g id=\"node30\" class=\"node\"><title>layer3_residual_res2_norm</title>\n<g id=\"a_node30\"><a xlink:title=\"&lt;class &#39;__main__.BatchNorm&#39;&gt; {&#39;num_features&#39;: 512}\">\n<path fill=\"#ffffb3\" stroke=\"black\" d=\"M2742,-68C2742,-68 2712,-68 2712,-68 2706,-68 2700,-62 2700,-56 2700,-56 2700,-44 2700,-44 2700,-38 2706,-32 2712,-32 2712,-32 2742,-32 2742,-32 2748,-32 2754,-38 2754,-44 2754,-44 2754,-56 2754,-56 2754,-62 2748,-68 2742,-68\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm -->\n<g id=\"edge31\" class=\"edge\"><title>layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2664.4,-50C2672.39,-50 2681.31,-50 2689.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2689.92,-53.5001 2699.92,-50 2689.92,-46.5001 2689.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_act -->\n<g id=\"node31\" class=\"node\"><title>layer3_residual_res2_act</title>\n<g id=\"a_node31\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, alpha=0.075) {}\">\n<path fill=\"#ff7f00\" stroke=\"black\" d=\"M2832,-68C2832,-68 2802,-68 2802,-68 2796,-68 2790,-62 2790,-56 2790,-56 2790,-44 2790,-44 2790,-38 2796,-32 2802,-32 2802,-32 2832,-32 2832,-32 2838,-32 2844,-38 2844,-44 2844,-44 2844,-56 2844,-56 2844,-62 2838,-68 2832,-68\"/>\n<text text-anchor=\"middle\" x=\"2817\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act -->\n<g id=\"edge32\" class=\"edge\"><title>layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2754.4,-50C2762.39,-50 2771.31,-50 2779.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2779.92,-53.5001 2789.92,-50 2779.92,-46.5001 2779.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_out -->\n<g id=\"node32\" class=\"node\"><title>layer3_residual_out</title>\n<g id=\"a_node32\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2922,-78C2922,-78 2892,-78 2892,-78 2886,-78 2880,-72 2880,-66 2880,-66 2880,-54 2880,-54 2880,-48 2886,-42 2892,-42 2892,-42 2922,-42 2922,-42 2928,-42 2934,-48 2934,-54 2934,-54 2934,-66 2934,-66 2934,-72 2928,-78 2922,-78\"/>\n<text text-anchor=\"middle\" x=\"2907\" y=\"-56.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_act&#45;&gt;layer3_residual_out -->\n<g id=\"edge33\" class=\"edge\"><title>layer3_residual_res2_act&#45;&gt;layer3_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2844.4,-53.0003C2852.39,-53.9083 2861.31,-54.9217 2869.82,-55.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2869.59,-59.3847 2879.92,-57.0362 2870.38,-52.4294 2869.59,-59.3847\"/>\n</g>\n<!-- layer3_residual_out&#45;&gt;layer3_residual_add -->\n<g id=\"edge35\" class=\"edge\"><title>layer3_residual_out&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2934.4,-64.5005C2942.39,-65.8625 2951.31,-67.3825 2959.82,-68.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2959.47,-72.3242 2969.92,-70.5544 2960.65,-65.4237 2959.47,-72.3242\"/>\n</g>\n<!-- pool -->\n<g id=\"node34\" class=\"node\"><title>pool</title>\n<g id=\"a_node34\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 4}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M3102,-93C3102,-93 3072,-93 3072,-93 3066,-93 3060,-87 3060,-81 3060,-81 3060,-69 3060,-69 3060,-63 3066,-57 3072,-57 3072,-57 3102,-57 3102,-57 3108,-57 3114,-63 3114,-69 3114,-69 3114,-81 3114,-81 3114,-87 3108,-93 3102,-93\"/>\n<text text-anchor=\"middle\" x=\"3087\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_add&#45;&gt;pool -->\n<g id=\"edge36\" class=\"edge\"><title>layer3_residual_add&#45;&gt;pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3024.4,-75C3032.39,-75 3041.31,-75 3049.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3049.92,-78.5001 3059.92,-75 3049.92,-71.5001 3049.92,-78.5001\"/>\n</g>\n<!-- classifier_flatten -->\n<g id=\"node35\" class=\"node\"><title>classifier_flatten</title>\n<g id=\"a_node35\"><a xlink:title=\"&lt;class &#39;__main__.Flatten&#39;&gt; {}\">\n<path fill=\"#b3de69\" stroke=\"black\" d=\"M3192,-93C3192,-93 3162,-93 3162,-93 3156,-93 3150,-87 3150,-81 3150,-81 3150,-69 3150,-69 3150,-63 3156,-57 3162,-57 3162,-57 3192,-57 3192,-57 3198,-57 3204,-63 3204,-69 3204,-69 3204,-81 3204,-81 3204,-87 3198,-93 3192,-93\"/>\n<text text-anchor=\"middle\" x=\"3177\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">flatten</text>\n</a>\n</g>\n</g>\n<!-- pool&#45;&gt;classifier_flatten -->\n<g id=\"edge37\" class=\"edge\"><title>pool&#45;&gt;classifier_flatten</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3114.4,-75C3122.39,-75 3131.31,-75 3139.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3139.92,-78.5001 3149.92,-75 3139.92,-71.5001 3139.92,-78.5001\"/>\n</g>\n<!-- classifier_conv -->\n<g id=\"node36\" class=\"node\"><title>classifier_conv</title>\n<g id=\"a_node36\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.linear.Linear&#39;&gt; {&#39;in_features&#39;: 512, &#39;out_features&#39;: 10, &#39;bias&#39;: False}\">\n<path fill=\"#fccde5\" stroke=\"black\" d=\"M3282,-93C3282,-93 3252,-93 3252,-93 3246,-93 3240,-87 3240,-81 3240,-81 3240,-69 3240,-69 3240,-63 3246,-57 3252,-57 3252,-57 3282,-57 3282,-57 3288,-57 3294,-63 3294,-69 3294,-69 3294,-81 3294,-81 3294,-87 3288,-93 3282,-93\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- classifier_flatten&#45;&gt;classifier_conv -->\n<g id=\"edge38\" class=\"edge\"><title>classifier_flatten&#45;&gt;classifier_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3204.4,-75C3212.39,-75 3221.31,-75 3229.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3229.92,-78.5001 3239.92,-75 3229.92,-71.5001 3229.92,-78.5001\"/>\n</g>\n<!-- classifier_scale -->\n<g id=\"node37\" class=\"node\"><title>classifier_scale</title>\n<g id=\"a_node37\"><a xlink:title=\"&lt;class &#39;__main__.Mul&#39;&gt; {&#39;weight&#39;: 0.125}\">\n<path fill=\"#bc80bd\" stroke=\"black\" d=\"M3372,-93C3372,-93 3342,-93 3342,-93 3336,-93 3330,-87 3330,-81 3330,-81 3330,-69 3330,-69 3330,-63 3336,-57 3342,-57 3342,-57 3372,-57 3372,-57 3378,-57 3384,-63 3384,-69 3384,-69 3384,-81 3384,-81 3384,-87 3378,-93 3372,-93\"/>\n<text text-anchor=\"middle\" x=\"3357\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">scale</text>\n</a>\n</g>\n</g>\n<!-- classifier_conv&#45;&gt;classifier_scale -->\n<g id=\"edge39\" class=\"edge\"><title>classifier_conv&#45;&gt;classifier_scale</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3294.4,-75C3302.39,-75 3311.31,-75 3319.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3319.92,-78.5001 3329.92,-75 3319.92,-71.5001 3319.92,-78.5001\"/>\n</g>\n<!-- logits -->\n<g id=\"node38\" class=\"node\"><title>logits</title>\n<g id=\"a_node38\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M3462,-93C3462,-93 3432,-93 3432,-93 3426,-93 3420,-87 3420,-81 3420,-81 3420,-69 3420,-69 3420,-63 3426,-57 3432,-57 3432,-57 3462,-57 3462,-57 3468,-57 3474,-63 3474,-69 3474,-69 3474,-81 3474,-81 3474,-87 3468,-93 3462,-93\"/>\n<text text-anchor=\"middle\" x=\"3447\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">logits</text>\n</a>\n</g>\n</g>\n<!-- classifier_scale&#45;&gt;logits -->\n<g id=\"edge40\" class=\"edge\"><title>classifier_scale&#45;&gt;logits</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3384.4,-75C3392.39,-75 3401.31,-75 3409.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3409.92,-78.5001 3419.92,-75 3409.92,-71.5001 3409.92,-78.5001\"/>\n</g>\n<!-- input -->\n<g id=\"node39\" class=\"node\"><title>input</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-92C42,-92 12,-92 12,-92 6,-92 0,-86 0,-80 0,-80 0,-68 0,-68 0,-62 6,-56 12,-56 12,-56 42,-56 42,-56 48,-56 54,-62 54,-68 54,-68 54,-80 54,-80 54,-86 48,-92 42,-92\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">input</text>\n</g>\n<!-- input&#45;&gt;prep_conv -->\n<g id=\"edge1\" class=\"edge\"><title>input&#45;&gt;prep_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4029,-74C62.3932,-74 71.3106,-74 79.8241,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"79.919,-77.5001 89.919,-74 79.919,-70.5001 79.919,-77.5001\"/>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cH7GrPu5dWna",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 23, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.4, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(celu_net, label_smoothing_loss(0.2))\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_uu5xtfMc1gK",
        "colab_type": "text"
      },
      "source": [
        "This gives an impressive improvement to 94.3% test accuracy (mean of 50 runs) allowing a further 3 epoch reduction in training and a 20 epoch time of 52s for 94.1% accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7HoNA_JwSkC_",
        "colab_type": "text"
      },
      "source": [
        "### Ghost batch norm (46s)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J1F8M7LfV8dE",
        "colab_type": "text"
      },
      "source": [
        "Batch norm seems to work best with batch size of around 32. The reasons presumably have to do with noise in the batch statistics and specifically a balance between a beneficial regularising effect at intermediate batch sizes and an excess of noise at small batches.\n",
        "\n",
        "Our batches are of size 512 and we can't afford to reduce them without taking a serious hit on training times, but we can apply batch norm separately to subsets of a training batch. This technique, known as 'ghost' batch norm, is usually used in a distributed setting but is just as useful when using large batches on a single node. It isn't supported directly in PyTorch but we can roll our own easily enough."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HwmZ3fJZT6sP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        },
        "outputId": "faff87b8-5481-4e62-e0df-f989b799ff7d"
      },
      "source": [
        "ghost_bn_net = network(conv_pool_block=conv_pool_block_pre, types={\n",
        "    nn.ReLU: partial(nn.CELU, 0.075), \n",
        "    BatchNorm: partial(GhostBatchNorm, num_splits=16)\n",
        "})\n",
        "show(ghost_bn_net)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<__main__.DotGraph at 0x7f4140a9ba58>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n -->\n<!-- Title: G Pages: 1 -->\n<svg width=\"1080pt\" height=\"59pt\"\n viewBox=\"0.00 0.00 1080.00 59.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(0.310167 0.310167) rotate(0) translate(4 187)\">\n<title>G</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-187 3478,-187 3478,4 -4,4\"/>\n<g id=\"clust1\" class=\"cluster\"><title>cluster_prep</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M94,-48C94,-48 320,-48 320,-48 326,-48 332,-54 332,-60 332,-60 332,-111 332,-111 332,-117 326,-123 320,-123 320,-123 94,-123 94,-123 88,-123 82,-117 82,-111 82,-111 82,-60 82,-60 82,-54 88,-48 94,-48\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-107.8\" font-family=\"Times,serif\" font-size=\"14.00\">prep</text>\n</g>\n<g id=\"clust2\" class=\"cluster\"><title>cluster_layer1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M364,-12C364,-12 1498,-12 1498,-12 1504,-12 1510,-18 1510,-24 1510,-24 1510,-163 1510,-163 1510,-169 1504,-175 1498,-175 1498,-175 364,-175 364,-175 358,-175 352,-169 352,-163 352,-163 352,-24 352,-24 352,-18 358,-12 364,-12\"/>\n<text text-anchor=\"middle\" x=\"931\" y=\"-159.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer1</text>\n</g>\n<g id=\"clust3\" class=\"cluster\"><title>cluster_layer1_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M724,-20C724,-20 1490,-20 1490,-20 1496,-20 1502,-26 1502,-32 1502,-32 1502,-132 1502,-132 1502,-138 1496,-144 1490,-144 1490,-144 724,-144 724,-144 718,-144 712,-138 712,-132 712,-132 712,-32 712,-32 712,-26 718,-20 724,-20\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-128.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust4\" class=\"cluster\"><title>cluster_layer1_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M814,-28C814,-28 1040,-28 1040,-28 1046,-28 1052,-34 1052,-40 1052,-40 1052,-91 1052,-91 1052,-97 1046,-103 1040,-103 1040,-103 814,-103 814,-103 808,-103 802,-97 802,-91 802,-91 802,-40 802,-40 802,-34 808,-28 814,-28\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust5\" class=\"cluster\"><title>cluster_layer1_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1084,-28C1084,-28 1310,-28 1310,-28 1316,-28 1322,-34 1322,-40 1322,-40 1322,-91 1322,-91 1322,-97 1316,-103 1310,-103 1310,-103 1084,-103 1084,-103 1078,-103 1072,-97 1072,-91 1072,-91 1072,-40 1072,-40 1072,-34 1078,-28 1084,-28\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust6\" class=\"cluster\"><title>cluster_layer2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1534,-53C1534,-53 1850,-53 1850,-53 1856,-53 1862,-59 1862,-65 1862,-65 1862,-116 1862,-116 1862,-122 1856,-128 1850,-128 1850,-128 1534,-128 1534,-128 1528,-128 1522,-122 1522,-116 1522,-116 1522,-65 1522,-65 1522,-59 1528,-53 1534,-53\"/>\n<text text-anchor=\"middle\" x=\"1692\" y=\"-112.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer2</text>\n</g>\n<g id=\"clust7\" class=\"cluster\"><title>cluster_layer3</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1894,-8C1894,-8 3028,-8 3028,-8 3034,-8 3040,-14 3040,-20 3040,-20 3040,-159 3040,-159 3040,-165 3034,-171 3028,-171 3028,-171 1894,-171 1894,-171 1888,-171 1882,-165 1882,-159 1882,-159 1882,-20 1882,-20 1882,-14 1888,-8 1894,-8\"/>\n<text text-anchor=\"middle\" x=\"2461\" y=\"-155.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer3</text>\n</g>\n<g id=\"clust8\" class=\"cluster\"><title>cluster_layer3_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2254,-16C2254,-16 3020,-16 3020,-16 3026,-16 3032,-22 3032,-28 3032,-28 3032,-128 3032,-128 3032,-134 3026,-140 3020,-140 3020,-140 2254,-140 2254,-140 2248,-140 2242,-134 2242,-128 2242,-128 2242,-28 2242,-28 2242,-22 2248,-16 2254,-16\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust9\" class=\"cluster\"><title>cluster_layer3_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2344,-24C2344,-24 2570,-24 2570,-24 2576,-24 2582,-30 2582,-36 2582,-36 2582,-87 2582,-87 2582,-93 2576,-99 2570,-99 2570,-99 2344,-99 2344,-99 2338,-99 2332,-93 2332,-87 2332,-87 2332,-36 2332,-36 2332,-30 2338,-24 2344,-24\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust10\" class=\"cluster\"><title>cluster_layer3_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2614,-24C2614,-24 2840,-24 2840,-24 2846,-24 2852,-30 2852,-36 2852,-36 2852,-87 2852,-87 2852,-93 2846,-99 2840,-99 2840,-99 2614,-99 2614,-99 2608,-99 2602,-93 2602,-87 2602,-87 2602,-36 2602,-36 2602,-30 2608,-24 2614,-24\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust11\" class=\"cluster\"><title>cluster_classifier</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M3154,-49C3154,-49 3380,-49 3380,-49 3386,-49 3392,-55 3392,-61 3392,-61 3392,-112 3392,-112 3392,-118 3386,-124 3380,-124 3380,-124 3154,-124 3154,-124 3148,-124 3142,-118 3142,-112 3142,-112 3142,-61 3142,-61 3142,-55 3148,-49 3154,-49\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-108.8\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n</g>\n<!-- prep_conv -->\n<g id=\"node1\" class=\"node\"><title>prep_conv</title>\n<g id=\"a_node1\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 3, &#39;out_channels&#39;: 64, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M132,-92C132,-92 102,-92 102,-92 96,-92 90,-86 90,-80 90,-80 90,-68 90,-68 90,-62 96,-56 102,-56 102,-56 132,-56 132,-56 138,-56 144,-62 144,-68 144,-68 144,-80 144,-80 144,-86 138,-92 132,-92\"/>\n<text text-anchor=\"middle\" x=\"117\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_norm -->\n<g id=\"node2\" class=\"node\"><title>prep_norm</title>\n<g id=\"a_node2\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16) {&#39;num_features&#39;: 64}\">\n<path fill=\"#4dddf8\" stroke=\"black\" d=\"M222,-92C222,-92 192,-92 192,-92 186,-92 180,-86 180,-80 180,-80 180,-68 180,-68 180,-62 186,-56 192,-56 192,-56 222,-56 222,-56 228,-56 234,-62 234,-68 234,-68 234,-80 234,-80 234,-86 228,-92 222,-92\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- prep_conv&#45;&gt;prep_norm -->\n<g id=\"edge2\" class=\"edge\"><title>prep_conv&#45;&gt;prep_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M144.403,-74C152.393,-74 161.311,-74 169.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"169.919,-77.5001 179.919,-74 169.919,-70.5001 169.919,-77.5001\"/>\n</g>\n<!-- prep_act -->\n<g id=\"node3\" class=\"node\"><title>prep_act</title>\n<g id=\"a_node3\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.075) {}\">\n<path fill=\"#e66493\" stroke=\"black\" d=\"M312,-92C312,-92 282,-92 282,-92 276,-92 270,-86 270,-80 270,-80 270,-68 270,-68 270,-62 276,-56 282,-56 282,-56 312,-56 312,-56 318,-56 324,-62 324,-68 324,-68 324,-80 324,-80 324,-86 318,-92 312,-92\"/>\n<text text-anchor=\"middle\" x=\"297\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- prep_norm&#45;&gt;prep_act -->\n<g id=\"edge3\" class=\"edge\"><title>prep_norm&#45;&gt;prep_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M234.403,-74C242.393,-74 251.311,-74 259.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"259.919,-77.5001 269.919,-74 259.919,-70.5001 259.919,-77.5001\"/>\n</g>\n<!-- layer1_conv -->\n<g id=\"node4\" class=\"node\"><title>layer1_conv</title>\n<g id=\"a_node4\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 64, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M402,-92C402,-92 372,-92 372,-92 366,-92 360,-86 360,-80 360,-80 360,-68 360,-68 360,-62 366,-56 372,-56 372,-56 402,-56 402,-56 408,-56 414,-62 414,-68 414,-68 414,-80 414,-80 414,-86 408,-92 402,-92\"/>\n<text text-anchor=\"middle\" x=\"387\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_act&#45;&gt;layer1_conv -->\n<g id=\"edge4\" class=\"edge\"><title>prep_act&#45;&gt;layer1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M324.403,-74C332.393,-74 341.311,-74 349.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"349.919,-77.5001 359.919,-74 349.919,-70.5001 349.919,-77.5001\"/>\n</g>\n<!-- layer1_pool -->\n<g id=\"node5\" class=\"node\"><title>layer1_pool</title>\n<g id=\"a_node5\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M492,-92C492,-92 462,-92 462,-92 456,-92 450,-86 450,-80 450,-80 450,-68 450,-68 450,-62 456,-56 462,-56 462,-56 492,-56 492,-56 498,-56 504,-62 504,-68 504,-68 504,-80 504,-80 504,-86 498,-92 492,-92\"/>\n<text text-anchor=\"middle\" x=\"477\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer1_conv&#45;&gt;layer1_pool -->\n<g id=\"edge5\" class=\"edge\"><title>layer1_conv&#45;&gt;layer1_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M414.403,-74C422.393,-74 431.311,-74 439.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"439.919,-77.5001 449.919,-74 439.919,-70.5001 439.919,-77.5001\"/>\n</g>\n<!-- layer1_norm -->\n<g id=\"node6\" class=\"node\"><title>layer1_norm</title>\n<g id=\"a_node6\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16) {&#39;num_features&#39;: 128}\">\n<path fill=\"#4dddf8\" stroke=\"black\" d=\"M582,-92C582,-92 552,-92 552,-92 546,-92 540,-86 540,-80 540,-80 540,-68 540,-68 540,-62 546,-56 552,-56 552,-56 582,-56 582,-56 588,-56 594,-62 594,-68 594,-68 594,-80 594,-80 594,-86 588,-92 582,-92\"/>\n<text text-anchor=\"middle\" x=\"567\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_pool&#45;&gt;layer1_norm -->\n<g id=\"edge6\" class=\"edge\"><title>layer1_pool&#45;&gt;layer1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M504.403,-74C512.393,-74 521.311,-74 529.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"529.919,-77.5001 539.919,-74 529.919,-70.5001 529.919,-77.5001\"/>\n</g>\n<!-- layer1_act -->\n<g id=\"node7\" class=\"node\"><title>layer1_act</title>\n<g id=\"a_node7\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.075) {}\">\n<path fill=\"#e66493\" stroke=\"black\" d=\"M672,-92C672,-92 642,-92 642,-92 636,-92 630,-86 630,-80 630,-80 630,-68 630,-68 630,-62 636,-56 642,-56 642,-56 672,-56 672,-56 678,-56 684,-62 684,-68 684,-68 684,-80 684,-80 684,-86 678,-92 672,-92\"/>\n<text text-anchor=\"middle\" x=\"657\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_norm&#45;&gt;layer1_act -->\n<g id=\"edge7\" class=\"edge\"><title>layer1_norm&#45;&gt;layer1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M594.403,-74C602.393,-74 611.311,-74 619.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"619.919,-77.5001 629.919,-74 619.919,-70.5001 619.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_in -->\n<g id=\"node8\" class=\"node\"><title>layer1_residual_in</title>\n<g id=\"a_node8\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M762,-92C762,-92 732,-92 732,-92 726,-92 720,-86 720,-80 720,-80 720,-68 720,-68 720,-62 726,-56 732,-56 732,-56 762,-56 762,-56 768,-56 774,-62 774,-68 774,-68 774,-80 774,-80 774,-86 768,-92 762,-92\"/>\n<text text-anchor=\"middle\" x=\"747\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer1_act&#45;&gt;layer1_residual_in -->\n<g id=\"edge8\" class=\"edge\"><title>layer1_act&#45;&gt;layer1_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M684.403,-74C692.393,-74 701.311,-74 709.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"709.919,-77.5001 719.919,-74 709.919,-70.5001 709.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_res1_conv -->\n<g id=\"node9\" class=\"node\"><title>layer1_residual_res1_conv</title>\n<g id=\"a_node9\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M852,-72C852,-72 822,-72 822,-72 816,-72 810,-66 810,-60 810,-60 810,-48 810,-48 810,-42 816,-36 822,-36 822,-36 852,-36 852,-36 858,-36 864,-42 864,-48 864,-48 864,-60 864,-60 864,-66 858,-72 852,-72\"/>\n<text text-anchor=\"middle\" x=\"837\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_res1_conv -->\n<g id=\"edge9\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M774.403,-67.9993C782.481,-66.1634 791.507,-64.1121 800.105,-62.158\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"800.943,-65.5568 809.919,-59.9275 799.392,-58.7308 800.943,-65.5568\"/>\n</g>\n<!-- layer1_residual_add -->\n<g id=\"node16\" class=\"node\"><title>layer1_residual_add</title>\n<g id=\"a_node16\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M1482,-97C1482,-97 1452,-97 1452,-97 1446,-97 1440,-91 1440,-85 1440,-85 1440,-73 1440,-73 1440,-67 1446,-61 1452,-61 1452,-61 1482,-61 1482,-61 1488,-61 1494,-67 1494,-73 1494,-73 1494,-85 1494,-85 1494,-91 1488,-97 1482,-97\"/>\n<text text-anchor=\"middle\" x=\"1467\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_add -->\n<g id=\"edge16\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M771.737,-92.1018C780.735,-97.9639 791.381,-103.78 802,-107 854.782,-123.006 870.844,-112 926,-112 926,-112 926,-112 1288,-112 1337.77,-112 1394.08,-99.3711 1429.8,-89.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1431.04,-93.0265 1439.74,-86.9884 1429.17,-86.2796 1431.04,-93.0265\"/>\n</g>\n<!-- layer1_residual_res1_norm -->\n<g id=\"node10\" class=\"node\"><title>layer1_residual_res1_norm</title>\n<g id=\"a_node10\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16) {&#39;num_features&#39;: 128}\">\n<path fill=\"#4dddf8\" stroke=\"black\" d=\"M942,-72C942,-72 912,-72 912,-72 906,-72 900,-66 900,-60 900,-60 900,-48 900,-48 900,-42 906,-36 912,-36 912,-36 942,-36 942,-36 948,-36 954,-42 954,-48 954,-48 954,-60 954,-60 954,-66 948,-72 942,-72\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm -->\n<g id=\"edge10\" class=\"edge\"><title>layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M864.403,-54C872.393,-54 881.311,-54 889.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"889.919,-57.5001 899.919,-54 889.919,-50.5001 889.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res1_act -->\n<g id=\"node11\" class=\"node\"><title>layer1_residual_res1_act</title>\n<g id=\"a_node11\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.075) {}\">\n<path fill=\"#e66493\" stroke=\"black\" d=\"M1032,-72C1032,-72 1002,-72 1002,-72 996,-72 990,-66 990,-60 990,-60 990,-48 990,-48 990,-42 996,-36 1002,-36 1002,-36 1032,-36 1032,-36 1038,-36 1044,-42 1044,-48 1044,-48 1044,-60 1044,-60 1044,-66 1038,-72 1032,-72\"/>\n<text text-anchor=\"middle\" x=\"1017\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act -->\n<g id=\"edge11\" class=\"edge\"><title>layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M954.403,-54C962.393,-54 971.311,-54 979.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"979.919,-57.5001 989.919,-54 979.919,-50.5001 979.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_conv -->\n<g id=\"node12\" class=\"node\"><title>layer1_residual_res2_conv</title>\n<g id=\"a_node12\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1122,-72C1122,-72 1092,-72 1092,-72 1086,-72 1080,-66 1080,-60 1080,-60 1080,-48 1080,-48 1080,-42 1086,-36 1092,-36 1092,-36 1122,-36 1122,-36 1128,-36 1134,-42 1134,-48 1134,-48 1134,-60 1134,-60 1134,-66 1128,-72 1122,-72\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv -->\n<g id=\"edge12\" class=\"edge\"><title>layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1044.4,-54C1052.39,-54 1061.31,-54 1069.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1069.92,-57.5001 1079.92,-54 1069.92,-50.5001 1069.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_norm -->\n<g id=\"node13\" class=\"node\"><title>layer1_residual_res2_norm</title>\n<g id=\"a_node13\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16) {&#39;num_features&#39;: 128}\">\n<path fill=\"#4dddf8\" stroke=\"black\" d=\"M1212,-72C1212,-72 1182,-72 1182,-72 1176,-72 1170,-66 1170,-60 1170,-60 1170,-48 1170,-48 1170,-42 1176,-36 1182,-36 1182,-36 1212,-36 1212,-36 1218,-36 1224,-42 1224,-48 1224,-48 1224,-60 1224,-60 1224,-66 1218,-72 1212,-72\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm -->\n<g id=\"edge13\" class=\"edge\"><title>layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1134.4,-54C1142.39,-54 1151.31,-54 1159.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1159.92,-57.5001 1169.92,-54 1159.92,-50.5001 1159.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_act -->\n<g id=\"node14\" class=\"node\"><title>layer1_residual_res2_act</title>\n<g id=\"a_node14\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.075) {}\">\n<path fill=\"#e66493\" stroke=\"black\" d=\"M1302,-72C1302,-72 1272,-72 1272,-72 1266,-72 1260,-66 1260,-60 1260,-60 1260,-48 1260,-48 1260,-42 1266,-36 1272,-36 1272,-36 1302,-36 1302,-36 1308,-36 1314,-42 1314,-48 1314,-48 1314,-60 1314,-60 1314,-66 1308,-72 1302,-72\"/>\n<text text-anchor=\"middle\" x=\"1287\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act -->\n<g id=\"edge14\" class=\"edge\"><title>layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1224.4,-54C1232.39,-54 1241.31,-54 1249.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1249.92,-57.5001 1259.92,-54 1249.92,-50.5001 1249.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_out -->\n<g id=\"node15\" class=\"node\"><title>layer1_residual_out</title>\n<g id=\"a_node15\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M1392,-82C1392,-82 1362,-82 1362,-82 1356,-82 1350,-76 1350,-70 1350,-70 1350,-58 1350,-58 1350,-52 1356,-46 1362,-46 1362,-46 1392,-46 1392,-46 1398,-46 1404,-52 1404,-58 1404,-58 1404,-70 1404,-70 1404,-76 1398,-82 1392,-82\"/>\n<text text-anchor=\"middle\" x=\"1377\" y=\"-60.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_act&#45;&gt;layer1_residual_out -->\n<g id=\"edge15\" class=\"edge\"><title>layer1_residual_res2_act&#45;&gt;layer1_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1314.4,-57.0003C1322.39,-57.9083 1331.31,-58.9217 1339.82,-59.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1339.59,-63.3847 1349.92,-61.0362 1340.38,-56.4294 1339.59,-63.3847\"/>\n</g>\n<!-- layer1_residual_out&#45;&gt;layer1_residual_add -->\n<g id=\"edge17\" class=\"edge\"><title>layer1_residual_out&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1404.4,-68.5005C1412.39,-69.8625 1421.31,-71.3825 1429.82,-72.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1429.47,-76.3242 1439.92,-74.5544 1430.65,-69.4237 1429.47,-76.3242\"/>\n</g>\n<!-- layer2_conv -->\n<g id=\"node17\" class=\"node\"><title>layer2_conv</title>\n<g id=\"a_node17\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 256, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1572,-97C1572,-97 1542,-97 1542,-97 1536,-97 1530,-91 1530,-85 1530,-85 1530,-73 1530,-73 1530,-67 1536,-61 1542,-61 1542,-61 1572,-61 1572,-61 1578,-61 1584,-67 1584,-73 1584,-73 1584,-85 1584,-85 1584,-91 1578,-97 1572,-97\"/>\n<text text-anchor=\"middle\" x=\"1557\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_add&#45;&gt;layer2_conv -->\n<g id=\"edge18\" class=\"edge\"><title>layer1_residual_add&#45;&gt;layer2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1494.4,-79C1502.39,-79 1511.31,-79 1519.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1519.92,-82.5001 1529.92,-79 1519.92,-75.5001 1519.92,-82.5001\"/>\n</g>\n<!-- layer2_pool -->\n<g id=\"node18\" class=\"node\"><title>layer2_pool</title>\n<g id=\"a_node18\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M1662,-97C1662,-97 1632,-97 1632,-97 1626,-97 1620,-91 1620,-85 1620,-85 1620,-73 1620,-73 1620,-67 1626,-61 1632,-61 1632,-61 1662,-61 1662,-61 1668,-61 1674,-67 1674,-73 1674,-73 1674,-85 1674,-85 1674,-91 1668,-97 1662,-97\"/>\n<text text-anchor=\"middle\" x=\"1647\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer2_conv&#45;&gt;layer2_pool -->\n<g id=\"edge19\" class=\"edge\"><title>layer2_conv&#45;&gt;layer2_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1584.4,-79C1592.39,-79 1601.31,-79 1609.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1609.92,-82.5001 1619.92,-79 1609.92,-75.5001 1609.92,-82.5001\"/>\n</g>\n<!-- layer2_norm -->\n<g id=\"node19\" class=\"node\"><title>layer2_norm</title>\n<g id=\"a_node19\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16) {&#39;num_features&#39;: 256}\">\n<path fill=\"#4dddf8\" stroke=\"black\" d=\"M1752,-97C1752,-97 1722,-97 1722,-97 1716,-97 1710,-91 1710,-85 1710,-85 1710,-73 1710,-73 1710,-67 1716,-61 1722,-61 1722,-61 1752,-61 1752,-61 1758,-61 1764,-67 1764,-73 1764,-73 1764,-85 1764,-85 1764,-91 1758,-97 1752,-97\"/>\n<text text-anchor=\"middle\" x=\"1737\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer2_pool&#45;&gt;layer2_norm -->\n<g id=\"edge20\" class=\"edge\"><title>layer2_pool&#45;&gt;layer2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1674.4,-79C1682.39,-79 1691.31,-79 1699.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1699.92,-82.5001 1709.92,-79 1699.92,-75.5001 1699.92,-82.5001\"/>\n</g>\n<!-- layer2_act -->\n<g id=\"node20\" class=\"node\"><title>layer2_act</title>\n<g id=\"a_node20\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.075) {}\">\n<path fill=\"#e66493\" stroke=\"black\" d=\"M1842,-97C1842,-97 1812,-97 1812,-97 1806,-97 1800,-91 1800,-85 1800,-85 1800,-73 1800,-73 1800,-67 1806,-61 1812,-61 1812,-61 1842,-61 1842,-61 1848,-61 1854,-67 1854,-73 1854,-73 1854,-85 1854,-85 1854,-91 1848,-97 1842,-97\"/>\n<text text-anchor=\"middle\" x=\"1827\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer2_norm&#45;&gt;layer2_act -->\n<g id=\"edge21\" class=\"edge\"><title>layer2_norm&#45;&gt;layer2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1764.4,-79C1772.39,-79 1781.31,-79 1789.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1789.92,-82.5001 1799.92,-79 1789.92,-75.5001 1789.92,-82.5001\"/>\n</g>\n<!-- layer3_conv -->\n<g id=\"node21\" class=\"node\"><title>layer3_conv</title>\n<g id=\"a_node21\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 256, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1932,-97C1932,-97 1902,-97 1902,-97 1896,-97 1890,-91 1890,-85 1890,-85 1890,-73 1890,-73 1890,-67 1896,-61 1902,-61 1902,-61 1932,-61 1932,-61 1938,-61 1944,-67 1944,-73 1944,-73 1944,-85 1944,-85 1944,-91 1938,-97 1932,-97\"/>\n<text text-anchor=\"middle\" x=\"1917\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer2_act&#45;&gt;layer3_conv -->\n<g id=\"edge22\" class=\"edge\"><title>layer2_act&#45;&gt;layer3_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1854.4,-79C1862.39,-79 1871.31,-79 1879.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1879.92,-82.5001 1889.92,-79 1879.92,-75.5001 1879.92,-82.5001\"/>\n</g>\n<!-- layer3_pool -->\n<g id=\"node22\" class=\"node\"><title>layer3_pool</title>\n<g id=\"a_node22\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M2022,-97C2022,-97 1992,-97 1992,-97 1986,-97 1980,-91 1980,-85 1980,-85 1980,-73 1980,-73 1980,-67 1986,-61 1992,-61 1992,-61 2022,-61 2022,-61 2028,-61 2034,-67 2034,-73 2034,-73 2034,-85 2034,-85 2034,-91 2028,-97 2022,-97\"/>\n<text text-anchor=\"middle\" x=\"2007\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_conv&#45;&gt;layer3_pool -->\n<g id=\"edge23\" class=\"edge\"><title>layer3_conv&#45;&gt;layer3_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1944.4,-79C1952.39,-79 1961.31,-79 1969.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1969.92,-82.5001 1979.92,-79 1969.92,-75.5001 1969.92,-82.5001\"/>\n</g>\n<!-- layer3_norm -->\n<g id=\"node23\" class=\"node\"><title>layer3_norm</title>\n<g id=\"a_node23\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16) {&#39;num_features&#39;: 512}\">\n<path fill=\"#4dddf8\" stroke=\"black\" d=\"M2112,-97C2112,-97 2082,-97 2082,-97 2076,-97 2070,-91 2070,-85 2070,-85 2070,-73 2070,-73 2070,-67 2076,-61 2082,-61 2082,-61 2112,-61 2112,-61 2118,-61 2124,-67 2124,-73 2124,-73 2124,-85 2124,-85 2124,-91 2118,-97 2112,-97\"/>\n<text text-anchor=\"middle\" x=\"2097\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_pool&#45;&gt;layer3_norm -->\n<g id=\"edge24\" class=\"edge\"><title>layer3_pool&#45;&gt;layer3_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2034.4,-79C2042.39,-79 2051.31,-79 2059.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2059.92,-82.5001 2069.92,-79 2059.92,-75.5001 2059.92,-82.5001\"/>\n</g>\n<!-- layer3_act -->\n<g id=\"node24\" class=\"node\"><title>layer3_act</title>\n<g id=\"a_node24\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.075) {}\">\n<path fill=\"#e66493\" stroke=\"black\" d=\"M2202,-97C2202,-97 2172,-97 2172,-97 2166,-97 2160,-91 2160,-85 2160,-85 2160,-73 2160,-73 2160,-67 2166,-61 2172,-61 2172,-61 2202,-61 2202,-61 2208,-61 2214,-67 2214,-73 2214,-73 2214,-85 2214,-85 2214,-91 2208,-97 2202,-97\"/>\n<text text-anchor=\"middle\" x=\"2187\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_norm&#45;&gt;layer3_act -->\n<g id=\"edge25\" class=\"edge\"><title>layer3_norm&#45;&gt;layer3_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2124.4,-79C2132.39,-79 2141.31,-79 2149.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2149.92,-82.5001 2159.92,-79 2149.92,-75.5001 2149.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_in -->\n<g id=\"node25\" class=\"node\"><title>layer3_residual_in</title>\n<g id=\"a_node25\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2292,-97C2292,-97 2262,-97 2262,-97 2256,-97 2250,-91 2250,-85 2250,-85 2250,-73 2250,-73 2250,-67 2256,-61 2262,-61 2262,-61 2292,-61 2292,-61 2298,-61 2304,-67 2304,-73 2304,-73 2304,-85 2304,-85 2304,-91 2298,-97 2292,-97\"/>\n<text text-anchor=\"middle\" x=\"2277\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer3_act&#45;&gt;layer3_residual_in -->\n<g id=\"edge26\" class=\"edge\"><title>layer3_act&#45;&gt;layer3_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2214.4,-79C2222.39,-79 2231.31,-79 2239.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2239.92,-82.5001 2249.92,-79 2239.92,-75.5001 2239.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_res1_conv -->\n<g id=\"node26\" class=\"node\"><title>layer3_residual_res1_conv</title>\n<g id=\"a_node26\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2382,-68C2382,-68 2352,-68 2352,-68 2346,-68 2340,-62 2340,-56 2340,-56 2340,-44 2340,-44 2340,-38 2346,-32 2352,-32 2352,-32 2382,-32 2382,-32 2388,-32 2394,-38 2394,-44 2394,-44 2394,-56 2394,-56 2394,-62 2388,-68 2382,-68\"/>\n<text text-anchor=\"middle\" x=\"2367\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_res1_conv -->\n<g id=\"edge27\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.4,-70.299C2312.57,-67.608 2321.7,-64.5979 2330.38,-61.7368\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2331.52,-65.049 2339.92,-58.5949 2329.33,-58.4007 2331.52,-65.049\"/>\n</g>\n<!-- layer3_residual_add -->\n<g id=\"node33\" class=\"node\"><title>layer3_residual_add</title>\n<g id=\"a_node33\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M3012,-93C3012,-93 2982,-93 2982,-93 2976,-93 2970,-87 2970,-81 2970,-81 2970,-69 2970,-69 2970,-63 2976,-57 2982,-57 2982,-57 3012,-57 3012,-57 3018,-57 3024,-63 3024,-69 3024,-69 3024,-81 3024,-81 3024,-87 3018,-93 3012,-93\"/>\n<text text-anchor=\"middle\" x=\"2997\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_add -->\n<g id=\"edge34\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.22,-93.0359C2312.83,-96.9904 2322.61,-100.808 2332,-103 2385.71,-115.535 2400.84,-108 2456,-108 2456,-108 2456,-108 2818,-108 2867.77,-108 2924.08,-95.3711 2959.8,-85.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2961.04,-89.0265 2969.74,-82.9884 2959.17,-82.2796 2961.04,-89.0265\"/>\n</g>\n<!-- layer3_residual_res1_norm -->\n<g id=\"node27\" class=\"node\"><title>layer3_residual_res1_norm</title>\n<g id=\"a_node27\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16) {&#39;num_features&#39;: 512}\">\n<path fill=\"#4dddf8\" stroke=\"black\" d=\"M2472,-68C2472,-68 2442,-68 2442,-68 2436,-68 2430,-62 2430,-56 2430,-56 2430,-44 2430,-44 2430,-38 2436,-32 2442,-32 2442,-32 2472,-32 2472,-32 2478,-32 2484,-38 2484,-44 2484,-44 2484,-56 2484,-56 2484,-62 2478,-68 2472,-68\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm -->\n<g id=\"edge28\" class=\"edge\"><title>layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2394.4,-50C2402.39,-50 2411.31,-50 2419.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2419.92,-53.5001 2429.92,-50 2419.92,-46.5001 2419.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res1_act -->\n<g id=\"node28\" class=\"node\"><title>layer3_residual_res1_act</title>\n<g id=\"a_node28\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.075) {}\">\n<path fill=\"#e66493\" stroke=\"black\" d=\"M2562,-68C2562,-68 2532,-68 2532,-68 2526,-68 2520,-62 2520,-56 2520,-56 2520,-44 2520,-44 2520,-38 2526,-32 2532,-32 2532,-32 2562,-32 2562,-32 2568,-32 2574,-38 2574,-44 2574,-44 2574,-56 2574,-56 2574,-62 2568,-68 2562,-68\"/>\n<text text-anchor=\"middle\" x=\"2547\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act -->\n<g id=\"edge29\" class=\"edge\"><title>layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2484.4,-50C2492.39,-50 2501.31,-50 2509.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2509.92,-53.5001 2519.92,-50 2509.92,-46.5001 2509.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_conv -->\n<g id=\"node29\" class=\"node\"><title>layer3_residual_res2_conv</title>\n<g id=\"a_node29\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2652,-68C2652,-68 2622,-68 2622,-68 2616,-68 2610,-62 2610,-56 2610,-56 2610,-44 2610,-44 2610,-38 2616,-32 2622,-32 2622,-32 2652,-32 2652,-32 2658,-32 2664,-38 2664,-44 2664,-44 2664,-56 2664,-56 2664,-62 2658,-68 2652,-68\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv -->\n<g id=\"edge30\" class=\"edge\"><title>layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2574.4,-50C2582.39,-50 2591.31,-50 2599.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2599.92,-53.5001 2609.92,-50 2599.92,-46.5001 2599.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_norm -->\n<g id=\"node30\" class=\"node\"><title>layer3_residual_res2_norm</title>\n<g id=\"a_node30\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16) {&#39;num_features&#39;: 512}\">\n<path fill=\"#4dddf8\" stroke=\"black\" d=\"M2742,-68C2742,-68 2712,-68 2712,-68 2706,-68 2700,-62 2700,-56 2700,-56 2700,-44 2700,-44 2700,-38 2706,-32 2712,-32 2712,-32 2742,-32 2742,-32 2748,-32 2754,-38 2754,-44 2754,-44 2754,-56 2754,-56 2754,-62 2748,-68 2742,-68\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm -->\n<g id=\"edge31\" class=\"edge\"><title>layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2664.4,-50C2672.39,-50 2681.31,-50 2689.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2689.92,-53.5001 2699.92,-50 2689.92,-46.5001 2689.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_act -->\n<g id=\"node31\" class=\"node\"><title>layer3_residual_res2_act</title>\n<g id=\"a_node31\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.075) {}\">\n<path fill=\"#e66493\" stroke=\"black\" d=\"M2832,-68C2832,-68 2802,-68 2802,-68 2796,-68 2790,-62 2790,-56 2790,-56 2790,-44 2790,-44 2790,-38 2796,-32 2802,-32 2802,-32 2832,-32 2832,-32 2838,-32 2844,-38 2844,-44 2844,-44 2844,-56 2844,-56 2844,-62 2838,-68 2832,-68\"/>\n<text text-anchor=\"middle\" x=\"2817\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act -->\n<g id=\"edge32\" class=\"edge\"><title>layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2754.4,-50C2762.39,-50 2771.31,-50 2779.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2779.92,-53.5001 2789.92,-50 2779.92,-46.5001 2779.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_out -->\n<g id=\"node32\" class=\"node\"><title>layer3_residual_out</title>\n<g id=\"a_node32\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2922,-78C2922,-78 2892,-78 2892,-78 2886,-78 2880,-72 2880,-66 2880,-66 2880,-54 2880,-54 2880,-48 2886,-42 2892,-42 2892,-42 2922,-42 2922,-42 2928,-42 2934,-48 2934,-54 2934,-54 2934,-66 2934,-66 2934,-72 2928,-78 2922,-78\"/>\n<text text-anchor=\"middle\" x=\"2907\" y=\"-56.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_act&#45;&gt;layer3_residual_out -->\n<g id=\"edge33\" class=\"edge\"><title>layer3_residual_res2_act&#45;&gt;layer3_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2844.4,-53.0003C2852.39,-53.9083 2861.31,-54.9217 2869.82,-55.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2869.59,-59.3847 2879.92,-57.0362 2870.38,-52.4294 2869.59,-59.3847\"/>\n</g>\n<!-- layer3_residual_out&#45;&gt;layer3_residual_add -->\n<g id=\"edge35\" class=\"edge\"><title>layer3_residual_out&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2934.4,-64.5005C2942.39,-65.8625 2951.31,-67.3825 2959.82,-68.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2959.47,-72.3242 2969.92,-70.5544 2960.65,-65.4237 2959.47,-72.3242\"/>\n</g>\n<!-- pool -->\n<g id=\"node34\" class=\"node\"><title>pool</title>\n<g id=\"a_node34\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 4}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M3102,-93C3102,-93 3072,-93 3072,-93 3066,-93 3060,-87 3060,-81 3060,-81 3060,-69 3060,-69 3060,-63 3066,-57 3072,-57 3072,-57 3102,-57 3102,-57 3108,-57 3114,-63 3114,-69 3114,-69 3114,-81 3114,-81 3114,-87 3108,-93 3102,-93\"/>\n<text text-anchor=\"middle\" x=\"3087\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_add&#45;&gt;pool -->\n<g id=\"edge36\" class=\"edge\"><title>layer3_residual_add&#45;&gt;pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3024.4,-75C3032.39,-75 3041.31,-75 3049.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3049.92,-78.5001 3059.92,-75 3049.92,-71.5001 3049.92,-78.5001\"/>\n</g>\n<!-- classifier_flatten -->\n<g id=\"node35\" class=\"node\"><title>classifier_flatten</title>\n<g id=\"a_node35\"><a xlink:title=\"&lt;class &#39;__main__.Flatten&#39;&gt; {}\">\n<path fill=\"#b3de69\" stroke=\"black\" d=\"M3192,-93C3192,-93 3162,-93 3162,-93 3156,-93 3150,-87 3150,-81 3150,-81 3150,-69 3150,-69 3150,-63 3156,-57 3162,-57 3162,-57 3192,-57 3192,-57 3198,-57 3204,-63 3204,-69 3204,-69 3204,-81 3204,-81 3204,-87 3198,-93 3192,-93\"/>\n<text text-anchor=\"middle\" x=\"3177\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">flatten</text>\n</a>\n</g>\n</g>\n<!-- pool&#45;&gt;classifier_flatten -->\n<g id=\"edge37\" class=\"edge\"><title>pool&#45;&gt;classifier_flatten</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3114.4,-75C3122.39,-75 3131.31,-75 3139.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3139.92,-78.5001 3149.92,-75 3139.92,-71.5001 3139.92,-78.5001\"/>\n</g>\n<!-- classifier_conv -->\n<g id=\"node36\" class=\"node\"><title>classifier_conv</title>\n<g id=\"a_node36\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.linear.Linear&#39;&gt; {&#39;in_features&#39;: 512, &#39;out_features&#39;: 10, &#39;bias&#39;: False}\">\n<path fill=\"#fccde5\" stroke=\"black\" d=\"M3282,-93C3282,-93 3252,-93 3252,-93 3246,-93 3240,-87 3240,-81 3240,-81 3240,-69 3240,-69 3240,-63 3246,-57 3252,-57 3252,-57 3282,-57 3282,-57 3288,-57 3294,-63 3294,-69 3294,-69 3294,-81 3294,-81 3294,-87 3288,-93 3282,-93\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- classifier_flatten&#45;&gt;classifier_conv -->\n<g id=\"edge38\" class=\"edge\"><title>classifier_flatten&#45;&gt;classifier_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3204.4,-75C3212.39,-75 3221.31,-75 3229.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3229.92,-78.5001 3239.92,-75 3229.92,-71.5001 3229.92,-78.5001\"/>\n</g>\n<!-- classifier_scale -->\n<g id=\"node37\" class=\"node\"><title>classifier_scale</title>\n<g id=\"a_node37\"><a xlink:title=\"&lt;class &#39;__main__.Mul&#39;&gt; {&#39;weight&#39;: 0.125}\">\n<path fill=\"#bc80bd\" stroke=\"black\" d=\"M3372,-93C3372,-93 3342,-93 3342,-93 3336,-93 3330,-87 3330,-81 3330,-81 3330,-69 3330,-69 3330,-63 3336,-57 3342,-57 3342,-57 3372,-57 3372,-57 3378,-57 3384,-63 3384,-69 3384,-69 3384,-81 3384,-81 3384,-87 3378,-93 3372,-93\"/>\n<text text-anchor=\"middle\" x=\"3357\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">scale</text>\n</a>\n</g>\n</g>\n<!-- classifier_conv&#45;&gt;classifier_scale -->\n<g id=\"edge39\" class=\"edge\"><title>classifier_conv&#45;&gt;classifier_scale</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3294.4,-75C3302.39,-75 3311.31,-75 3319.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3319.92,-78.5001 3329.92,-75 3319.92,-71.5001 3319.92,-78.5001\"/>\n</g>\n<!-- logits -->\n<g id=\"node38\" class=\"node\"><title>logits</title>\n<g id=\"a_node38\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M3462,-93C3462,-93 3432,-93 3432,-93 3426,-93 3420,-87 3420,-81 3420,-81 3420,-69 3420,-69 3420,-63 3426,-57 3432,-57 3432,-57 3462,-57 3462,-57 3468,-57 3474,-63 3474,-69 3474,-69 3474,-81 3474,-81 3474,-87 3468,-93 3462,-93\"/>\n<text text-anchor=\"middle\" x=\"3447\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">logits</text>\n</a>\n</g>\n</g>\n<!-- classifier_scale&#45;&gt;logits -->\n<g id=\"edge40\" class=\"edge\"><title>classifier_scale&#45;&gt;logits</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3384.4,-75C3392.39,-75 3401.31,-75 3409.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3409.92,-78.5001 3419.92,-75 3409.92,-71.5001 3409.92,-78.5001\"/>\n</g>\n<!-- input -->\n<g id=\"node39\" class=\"node\"><title>input</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-92C42,-92 12,-92 12,-92 6,-92 0,-86 0,-80 0,-80 0,-68 0,-68 0,-62 6,-56 12,-56 12,-56 42,-56 42,-56 48,-56 54,-62 54,-68 54,-68 54,-80 54,-80 54,-86 48,-92 42,-92\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">input</text>\n</g>\n<!-- input&#45;&gt;prep_conv -->\n<g id=\"edge1\" class=\"edge\"><title>input&#45;&gt;prep_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4029,-74C62.3932,-74 71.3106,-74 79.8241,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"79.919,-77.5001 89.919,-74 79.919,-70.5001 79.919,-77.5001\"/>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YAUyBrtqN7gj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 20, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.4, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs, 'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(ghost_bn_net, label_smoothing_loss(0.2))\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NyvPyD2xc77U",
        "colab_type": "text"
      },
      "source": [
        "This gives a healthy boost to the 20 epoch test accuracy of 94.2%. As training becomes ever shorter, it is occasionally helpful to increase the learning rate to compensate. If we raise the max learning rate by 50% we can achieve 94.1% accuracy in 18 epochs and a training time of 46s."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4-gEHrSYp94Z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 18, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.6, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs, 'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(ghost_bn_net, label_smoothing_loss(0.2))\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nRrXu8g5eoQ1",
        "colab_type": "text"
      },
      "source": [
        "### Frozen batch norm scales (43s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vbafQdo_er-7",
        "colab_type": "text"
      },
      "source": [
        "Batch norm standardises the mean and variance of each channel but is followed by a learnable scale and bias. Our batch norm layers are succeeded by ReLUs, so the learnable biases could allow the network to optimise the level of sparsity per channel. On the other hand, if channel scales vary substantially this might reduce the effective number of channels and introduce a bottleneck. Let's have a look at the dynamics of these parameters during training:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3KbCvLxqq8VI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 18, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.6, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "model = build_model(ghost_bn_net, label_smoothing_loss(0.2))\n",
        "state, timer, logs = {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize), Table(report=every(epochs, 'epoch'))\n",
        "for epoch in range(epochs):\n",
        "    logs.append(union({'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size), \n",
        "                                                      on_epoch_end=partial(log_weights, weights={k: v for k,v in trainable_params(model).items() if 'norm' in k}))))\n",
        "data = pd.DataFrame([{'epoch': epoch, 'type': weight.split('.')[1], 'layer': weight.split('_norm')[0], 'channel': channel, 'value': value} \n",
        "                     for epoch, epoch_vals in enumerate(state[WEIGHT_LOG], 1) \n",
        "                     for weight, weight_vals in epoch_vals.items() \n",
        "                     for channel, value in enumerate(weight_vals[:8], 1)])\n",
        "\n",
        "alt.Chart(data).mark_line(opacity=0.7).encode(x='epoch', y='value', color=alt.Color('layer', sort=['prep']), detail='channel', column='type').resolve_scale(y='independent')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gMvagTTwx1wO",
        "colab_type": "text"
      },
      "source": [
        "There's a lot going on in these plots, but one thing that sticks out is that the scales are not doing much learning and evolve largely under the control of weight decay. Let's try freezing these at a constant value of $1/4$ - roughly their average at the midpoint of training. The learnable scale for the final layer is somewhat larger but we can adjust the scaling of the network output to compensate. \n",
        "\n",
        "Actually we can fix the batch norm scales to $1$ instead if we rescale the $\\alpha$ parameter of CELU by a compensating factor of $4$ and the learning rate and weight decay for the batch norm biases by $4^2$ and $1/4^2$ respectively. We prefer to do things this way since it makes the impact of the channel scales on the learning rate dynamics of the biases more explicit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lkSuchyvLSa4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        },
        "outputId": "10f9be49-cf0a-4d94-b740-0b846d436c85"
      },
      "source": [
        "frozen_bn_scale_net = network(conv_pool_block=conv_pool_block_pre, scale=1/16, types={\n",
        "    nn.ReLU: partial(nn.CELU, 0.3),\n",
        "    BatchNorm: partial(GhostBatchNorm, num_splits=16, weight=False)\n",
        "})\n",
        "show(frozen_bn_scale_net)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<__main__.DotGraph at 0x7f4140a82128>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n -->\n<!-- Title: G Pages: 1 -->\n<svg width=\"1080pt\" height=\"59pt\"\n viewBox=\"0.00 0.00 1080.00 59.24\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(0.310167 0.310167) rotate(0) translate(4 187)\">\n<title>G</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-187 3478,-187 3478,4 -4,4\"/>\n<g id=\"clust1\" class=\"cluster\"><title>cluster_prep</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M94,-48C94,-48 320,-48 320,-48 326,-48 332,-54 332,-60 332,-60 332,-111 332,-111 332,-117 326,-123 320,-123 320,-123 94,-123 94,-123 88,-123 82,-117 82,-111 82,-111 82,-60 82,-60 82,-54 88,-48 94,-48\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-107.8\" font-family=\"Times,serif\" font-size=\"14.00\">prep</text>\n</g>\n<g id=\"clust2\" class=\"cluster\"><title>cluster_layer1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M364,-12C364,-12 1498,-12 1498,-12 1504,-12 1510,-18 1510,-24 1510,-24 1510,-163 1510,-163 1510,-169 1504,-175 1498,-175 1498,-175 364,-175 364,-175 358,-175 352,-169 352,-163 352,-163 352,-24 352,-24 352,-18 358,-12 364,-12\"/>\n<text text-anchor=\"middle\" x=\"931\" y=\"-159.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer1</text>\n</g>\n<g id=\"clust3\" class=\"cluster\"><title>cluster_layer1_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M724,-20C724,-20 1490,-20 1490,-20 1496,-20 1502,-26 1502,-32 1502,-32 1502,-132 1502,-132 1502,-138 1496,-144 1490,-144 1490,-144 724,-144 724,-144 718,-144 712,-138 712,-132 712,-132 712,-32 712,-32 712,-26 718,-20 724,-20\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-128.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust4\" class=\"cluster\"><title>cluster_layer1_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M814,-28C814,-28 1040,-28 1040,-28 1046,-28 1052,-34 1052,-40 1052,-40 1052,-91 1052,-91 1052,-97 1046,-103 1040,-103 1040,-103 814,-103 814,-103 808,-103 802,-97 802,-91 802,-91 802,-40 802,-40 802,-34 808,-28 814,-28\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust5\" class=\"cluster\"><title>cluster_layer1_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1084,-28C1084,-28 1310,-28 1310,-28 1316,-28 1322,-34 1322,-40 1322,-40 1322,-91 1322,-91 1322,-97 1316,-103 1310,-103 1310,-103 1084,-103 1084,-103 1078,-103 1072,-97 1072,-91 1072,-91 1072,-40 1072,-40 1072,-34 1078,-28 1084,-28\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust6\" class=\"cluster\"><title>cluster_layer2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1534,-53C1534,-53 1850,-53 1850,-53 1856,-53 1862,-59 1862,-65 1862,-65 1862,-116 1862,-116 1862,-122 1856,-128 1850,-128 1850,-128 1534,-128 1534,-128 1528,-128 1522,-122 1522,-116 1522,-116 1522,-65 1522,-65 1522,-59 1528,-53 1534,-53\"/>\n<text text-anchor=\"middle\" x=\"1692\" y=\"-112.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer2</text>\n</g>\n<g id=\"clust7\" class=\"cluster\"><title>cluster_layer3</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1894,-8C1894,-8 3028,-8 3028,-8 3034,-8 3040,-14 3040,-20 3040,-20 3040,-159 3040,-159 3040,-165 3034,-171 3028,-171 3028,-171 1894,-171 1894,-171 1888,-171 1882,-165 1882,-159 1882,-159 1882,-20 1882,-20 1882,-14 1888,-8 1894,-8\"/>\n<text text-anchor=\"middle\" x=\"2461\" y=\"-155.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer3</text>\n</g>\n<g id=\"clust8\" class=\"cluster\"><title>cluster_layer3_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2254,-16C2254,-16 3020,-16 3020,-16 3026,-16 3032,-22 3032,-28 3032,-28 3032,-128 3032,-128 3032,-134 3026,-140 3020,-140 3020,-140 2254,-140 2254,-140 2248,-140 2242,-134 2242,-128 2242,-128 2242,-28 2242,-28 2242,-22 2248,-16 2254,-16\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust9\" class=\"cluster\"><title>cluster_layer3_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2344,-24C2344,-24 2570,-24 2570,-24 2576,-24 2582,-30 2582,-36 2582,-36 2582,-87 2582,-87 2582,-93 2576,-99 2570,-99 2570,-99 2344,-99 2344,-99 2338,-99 2332,-93 2332,-87 2332,-87 2332,-36 2332,-36 2332,-30 2338,-24 2344,-24\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust10\" class=\"cluster\"><title>cluster_layer3_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2614,-24C2614,-24 2840,-24 2840,-24 2846,-24 2852,-30 2852,-36 2852,-36 2852,-87 2852,-87 2852,-93 2846,-99 2840,-99 2840,-99 2614,-99 2614,-99 2608,-99 2602,-93 2602,-87 2602,-87 2602,-36 2602,-36 2602,-30 2608,-24 2614,-24\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust11\" class=\"cluster\"><title>cluster_classifier</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M3154,-49C3154,-49 3380,-49 3380,-49 3386,-49 3392,-55 3392,-61 3392,-61 3392,-112 3392,-112 3392,-118 3386,-124 3380,-124 3380,-124 3154,-124 3154,-124 3148,-124 3142,-118 3142,-112 3142,-112 3142,-61 3142,-61 3142,-55 3148,-49 3154,-49\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-108.8\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n</g>\n<!-- prep_conv -->\n<g id=\"node1\" class=\"node\"><title>prep_conv</title>\n<g id=\"a_node1\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 3, &#39;out_channels&#39;: 64, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M132,-92C132,-92 102,-92 102,-92 96,-92 90,-86 90,-80 90,-80 90,-68 90,-68 90,-62 96,-56 102,-56 102,-56 132,-56 132,-56 138,-56 144,-62 144,-68 144,-68 144,-80 144,-80 144,-86 138,-92 132,-92\"/>\n<text text-anchor=\"middle\" x=\"117\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_norm -->\n<g id=\"node2\" class=\"node\"><title>prep_norm</title>\n<g id=\"a_node2\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 64}\">\n<path fill=\"#b07b87\" stroke=\"black\" d=\"M222,-92C222,-92 192,-92 192,-92 186,-92 180,-86 180,-80 180,-80 180,-68 180,-68 180,-62 186,-56 192,-56 192,-56 222,-56 222,-56 228,-56 234,-62 234,-68 234,-68 234,-80 234,-80 234,-86 228,-92 222,-92\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- prep_conv&#45;&gt;prep_norm -->\n<g id=\"edge2\" class=\"edge\"><title>prep_conv&#45;&gt;prep_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M144.403,-74C152.393,-74 161.311,-74 169.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"169.919,-77.5001 179.919,-74 169.919,-70.5001 169.919,-77.5001\"/>\n</g>\n<!-- prep_act -->\n<g id=\"node3\" class=\"node\"><title>prep_act</title>\n<g id=\"a_node3\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#4e90e3\" stroke=\"black\" d=\"M312,-92C312,-92 282,-92 282,-92 276,-92 270,-86 270,-80 270,-80 270,-68 270,-68 270,-62 276,-56 282,-56 282,-56 312,-56 312,-56 318,-56 324,-62 324,-68 324,-68 324,-80 324,-80 324,-86 318,-92 312,-92\"/>\n<text text-anchor=\"middle\" x=\"297\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- prep_norm&#45;&gt;prep_act -->\n<g id=\"edge3\" class=\"edge\"><title>prep_norm&#45;&gt;prep_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M234.403,-74C242.393,-74 251.311,-74 259.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"259.919,-77.5001 269.919,-74 259.919,-70.5001 259.919,-77.5001\"/>\n</g>\n<!-- layer1_conv -->\n<g id=\"node4\" class=\"node\"><title>layer1_conv</title>\n<g id=\"a_node4\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 64, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M402,-92C402,-92 372,-92 372,-92 366,-92 360,-86 360,-80 360,-80 360,-68 360,-68 360,-62 366,-56 372,-56 372,-56 402,-56 402,-56 408,-56 414,-62 414,-68 414,-68 414,-80 414,-80 414,-86 408,-92 402,-92\"/>\n<text text-anchor=\"middle\" x=\"387\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_act&#45;&gt;layer1_conv -->\n<g id=\"edge4\" class=\"edge\"><title>prep_act&#45;&gt;layer1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M324.403,-74C332.393,-74 341.311,-74 349.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"349.919,-77.5001 359.919,-74 349.919,-70.5001 349.919,-77.5001\"/>\n</g>\n<!-- layer1_pool -->\n<g id=\"node5\" class=\"node\"><title>layer1_pool</title>\n<g id=\"a_node5\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M492,-92C492,-92 462,-92 462,-92 456,-92 450,-86 450,-80 450,-80 450,-68 450,-68 450,-62 456,-56 462,-56 462,-56 492,-56 492,-56 498,-56 504,-62 504,-68 504,-68 504,-80 504,-80 504,-86 498,-92 492,-92\"/>\n<text text-anchor=\"middle\" x=\"477\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer1_conv&#45;&gt;layer1_pool -->\n<g id=\"edge5\" class=\"edge\"><title>layer1_conv&#45;&gt;layer1_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M414.403,-74C422.393,-74 431.311,-74 439.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"439.919,-77.5001 449.919,-74 439.919,-70.5001 439.919,-77.5001\"/>\n</g>\n<!-- layer1_norm -->\n<g id=\"node6\" class=\"node\"><title>layer1_norm</title>\n<g id=\"a_node6\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 128}\">\n<path fill=\"#b07b87\" stroke=\"black\" d=\"M582,-92C582,-92 552,-92 552,-92 546,-92 540,-86 540,-80 540,-80 540,-68 540,-68 540,-62 546,-56 552,-56 552,-56 582,-56 582,-56 588,-56 594,-62 594,-68 594,-68 594,-80 594,-80 594,-86 588,-92 582,-92\"/>\n<text text-anchor=\"middle\" x=\"567\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_pool&#45;&gt;layer1_norm -->\n<g id=\"edge6\" class=\"edge\"><title>layer1_pool&#45;&gt;layer1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M504.403,-74C512.393,-74 521.311,-74 529.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"529.919,-77.5001 539.919,-74 529.919,-70.5001 529.919,-77.5001\"/>\n</g>\n<!-- layer1_act -->\n<g id=\"node7\" class=\"node\"><title>layer1_act</title>\n<g id=\"a_node7\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#4e90e3\" stroke=\"black\" d=\"M672,-92C672,-92 642,-92 642,-92 636,-92 630,-86 630,-80 630,-80 630,-68 630,-68 630,-62 636,-56 642,-56 642,-56 672,-56 672,-56 678,-56 684,-62 684,-68 684,-68 684,-80 684,-80 684,-86 678,-92 672,-92\"/>\n<text text-anchor=\"middle\" x=\"657\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_norm&#45;&gt;layer1_act -->\n<g id=\"edge7\" class=\"edge\"><title>layer1_norm&#45;&gt;layer1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M594.403,-74C602.393,-74 611.311,-74 619.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"619.919,-77.5001 629.919,-74 619.919,-70.5001 619.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_in -->\n<g id=\"node8\" class=\"node\"><title>layer1_residual_in</title>\n<g id=\"a_node8\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M762,-92C762,-92 732,-92 732,-92 726,-92 720,-86 720,-80 720,-80 720,-68 720,-68 720,-62 726,-56 732,-56 732,-56 762,-56 762,-56 768,-56 774,-62 774,-68 774,-68 774,-80 774,-80 774,-86 768,-92 762,-92\"/>\n<text text-anchor=\"middle\" x=\"747\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer1_act&#45;&gt;layer1_residual_in -->\n<g id=\"edge8\" class=\"edge\"><title>layer1_act&#45;&gt;layer1_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M684.403,-74C692.393,-74 701.311,-74 709.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"709.919,-77.5001 719.919,-74 709.919,-70.5001 709.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_res1_conv -->\n<g id=\"node9\" class=\"node\"><title>layer1_residual_res1_conv</title>\n<g id=\"a_node9\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M852,-72C852,-72 822,-72 822,-72 816,-72 810,-66 810,-60 810,-60 810,-48 810,-48 810,-42 816,-36 822,-36 822,-36 852,-36 852,-36 858,-36 864,-42 864,-48 864,-48 864,-60 864,-60 864,-66 858,-72 852,-72\"/>\n<text text-anchor=\"middle\" x=\"837\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_res1_conv -->\n<g id=\"edge9\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M774.403,-67.9993C782.481,-66.1634 791.507,-64.1121 800.105,-62.158\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"800.943,-65.5568 809.919,-59.9275 799.392,-58.7308 800.943,-65.5568\"/>\n</g>\n<!-- layer1_residual_add -->\n<g id=\"node16\" class=\"node\"><title>layer1_residual_add</title>\n<g id=\"a_node16\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M1482,-97C1482,-97 1452,-97 1452,-97 1446,-97 1440,-91 1440,-85 1440,-85 1440,-73 1440,-73 1440,-67 1446,-61 1452,-61 1452,-61 1482,-61 1482,-61 1488,-61 1494,-67 1494,-73 1494,-73 1494,-85 1494,-85 1494,-91 1488,-97 1482,-97\"/>\n<text text-anchor=\"middle\" x=\"1467\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_add -->\n<g id=\"edge16\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M771.737,-92.1018C780.735,-97.9639 791.381,-103.78 802,-107 854.782,-123.006 870.844,-112 926,-112 926,-112 926,-112 1288,-112 1337.77,-112 1394.08,-99.3711 1429.8,-89.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1431.04,-93.0265 1439.74,-86.9884 1429.17,-86.2796 1431.04,-93.0265\"/>\n</g>\n<!-- layer1_residual_res1_norm -->\n<g id=\"node10\" class=\"node\"><title>layer1_residual_res1_norm</title>\n<g id=\"a_node10\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 128}\">\n<path fill=\"#b07b87\" stroke=\"black\" d=\"M942,-72C942,-72 912,-72 912,-72 906,-72 900,-66 900,-60 900,-60 900,-48 900,-48 900,-42 906,-36 912,-36 912,-36 942,-36 942,-36 948,-36 954,-42 954,-48 954,-48 954,-60 954,-60 954,-66 948,-72 942,-72\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm -->\n<g id=\"edge10\" class=\"edge\"><title>layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M864.403,-54C872.393,-54 881.311,-54 889.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"889.919,-57.5001 899.919,-54 889.919,-50.5001 889.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res1_act -->\n<g id=\"node11\" class=\"node\"><title>layer1_residual_res1_act</title>\n<g id=\"a_node11\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#4e90e3\" stroke=\"black\" d=\"M1032,-72C1032,-72 1002,-72 1002,-72 996,-72 990,-66 990,-60 990,-60 990,-48 990,-48 990,-42 996,-36 1002,-36 1002,-36 1032,-36 1032,-36 1038,-36 1044,-42 1044,-48 1044,-48 1044,-60 1044,-60 1044,-66 1038,-72 1032,-72\"/>\n<text text-anchor=\"middle\" x=\"1017\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act -->\n<g id=\"edge11\" class=\"edge\"><title>layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M954.403,-54C962.393,-54 971.311,-54 979.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"979.919,-57.5001 989.919,-54 979.919,-50.5001 979.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_conv -->\n<g id=\"node12\" class=\"node\"><title>layer1_residual_res2_conv</title>\n<g id=\"a_node12\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1122,-72C1122,-72 1092,-72 1092,-72 1086,-72 1080,-66 1080,-60 1080,-60 1080,-48 1080,-48 1080,-42 1086,-36 1092,-36 1092,-36 1122,-36 1122,-36 1128,-36 1134,-42 1134,-48 1134,-48 1134,-60 1134,-60 1134,-66 1128,-72 1122,-72\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv -->\n<g id=\"edge12\" class=\"edge\"><title>layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1044.4,-54C1052.39,-54 1061.31,-54 1069.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1069.92,-57.5001 1079.92,-54 1069.92,-50.5001 1069.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_norm -->\n<g id=\"node13\" class=\"node\"><title>layer1_residual_res2_norm</title>\n<g id=\"a_node13\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 128}\">\n<path fill=\"#b07b87\" stroke=\"black\" d=\"M1212,-72C1212,-72 1182,-72 1182,-72 1176,-72 1170,-66 1170,-60 1170,-60 1170,-48 1170,-48 1170,-42 1176,-36 1182,-36 1182,-36 1212,-36 1212,-36 1218,-36 1224,-42 1224,-48 1224,-48 1224,-60 1224,-60 1224,-66 1218,-72 1212,-72\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm -->\n<g id=\"edge13\" class=\"edge\"><title>layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1134.4,-54C1142.39,-54 1151.31,-54 1159.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1159.92,-57.5001 1169.92,-54 1159.92,-50.5001 1159.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_act -->\n<g id=\"node14\" class=\"node\"><title>layer1_residual_res2_act</title>\n<g id=\"a_node14\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#4e90e3\" stroke=\"black\" d=\"M1302,-72C1302,-72 1272,-72 1272,-72 1266,-72 1260,-66 1260,-60 1260,-60 1260,-48 1260,-48 1260,-42 1266,-36 1272,-36 1272,-36 1302,-36 1302,-36 1308,-36 1314,-42 1314,-48 1314,-48 1314,-60 1314,-60 1314,-66 1308,-72 1302,-72\"/>\n<text text-anchor=\"middle\" x=\"1287\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act -->\n<g id=\"edge14\" class=\"edge\"><title>layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1224.4,-54C1232.39,-54 1241.31,-54 1249.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1249.92,-57.5001 1259.92,-54 1249.92,-50.5001 1249.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_out -->\n<g id=\"node15\" class=\"node\"><title>layer1_residual_out</title>\n<g id=\"a_node15\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M1392,-82C1392,-82 1362,-82 1362,-82 1356,-82 1350,-76 1350,-70 1350,-70 1350,-58 1350,-58 1350,-52 1356,-46 1362,-46 1362,-46 1392,-46 1392,-46 1398,-46 1404,-52 1404,-58 1404,-58 1404,-70 1404,-70 1404,-76 1398,-82 1392,-82\"/>\n<text text-anchor=\"middle\" x=\"1377\" y=\"-60.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_act&#45;&gt;layer1_residual_out -->\n<g id=\"edge15\" class=\"edge\"><title>layer1_residual_res2_act&#45;&gt;layer1_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1314.4,-57.0003C1322.39,-57.9083 1331.31,-58.9217 1339.82,-59.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1339.59,-63.3847 1349.92,-61.0362 1340.38,-56.4294 1339.59,-63.3847\"/>\n</g>\n<!-- layer1_residual_out&#45;&gt;layer1_residual_add -->\n<g id=\"edge17\" class=\"edge\"><title>layer1_residual_out&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1404.4,-68.5005C1412.39,-69.8625 1421.31,-71.3825 1429.82,-72.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1429.47,-76.3242 1439.92,-74.5544 1430.65,-69.4237 1429.47,-76.3242\"/>\n</g>\n<!-- layer2_conv -->\n<g id=\"node17\" class=\"node\"><title>layer2_conv</title>\n<g id=\"a_node17\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 256, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1572,-97C1572,-97 1542,-97 1542,-97 1536,-97 1530,-91 1530,-85 1530,-85 1530,-73 1530,-73 1530,-67 1536,-61 1542,-61 1542,-61 1572,-61 1572,-61 1578,-61 1584,-67 1584,-73 1584,-73 1584,-85 1584,-85 1584,-91 1578,-97 1572,-97\"/>\n<text text-anchor=\"middle\" x=\"1557\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_add&#45;&gt;layer2_conv -->\n<g id=\"edge18\" class=\"edge\"><title>layer1_residual_add&#45;&gt;layer2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1494.4,-79C1502.39,-79 1511.31,-79 1519.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1519.92,-82.5001 1529.92,-79 1519.92,-75.5001 1519.92,-82.5001\"/>\n</g>\n<!-- layer2_pool -->\n<g id=\"node18\" class=\"node\"><title>layer2_pool</title>\n<g id=\"a_node18\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M1662,-97C1662,-97 1632,-97 1632,-97 1626,-97 1620,-91 1620,-85 1620,-85 1620,-73 1620,-73 1620,-67 1626,-61 1632,-61 1632,-61 1662,-61 1662,-61 1668,-61 1674,-67 1674,-73 1674,-73 1674,-85 1674,-85 1674,-91 1668,-97 1662,-97\"/>\n<text text-anchor=\"middle\" x=\"1647\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer2_conv&#45;&gt;layer2_pool -->\n<g id=\"edge19\" class=\"edge\"><title>layer2_conv&#45;&gt;layer2_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1584.4,-79C1592.39,-79 1601.31,-79 1609.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1609.92,-82.5001 1619.92,-79 1609.92,-75.5001 1609.92,-82.5001\"/>\n</g>\n<!-- layer2_norm -->\n<g id=\"node19\" class=\"node\"><title>layer2_norm</title>\n<g id=\"a_node19\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 256}\">\n<path fill=\"#b07b87\" stroke=\"black\" d=\"M1752,-97C1752,-97 1722,-97 1722,-97 1716,-97 1710,-91 1710,-85 1710,-85 1710,-73 1710,-73 1710,-67 1716,-61 1722,-61 1722,-61 1752,-61 1752,-61 1758,-61 1764,-67 1764,-73 1764,-73 1764,-85 1764,-85 1764,-91 1758,-97 1752,-97\"/>\n<text text-anchor=\"middle\" x=\"1737\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer2_pool&#45;&gt;layer2_norm -->\n<g id=\"edge20\" class=\"edge\"><title>layer2_pool&#45;&gt;layer2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1674.4,-79C1682.39,-79 1691.31,-79 1699.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1699.92,-82.5001 1709.92,-79 1699.92,-75.5001 1699.92,-82.5001\"/>\n</g>\n<!-- layer2_act -->\n<g id=\"node20\" class=\"node\"><title>layer2_act</title>\n<g id=\"a_node20\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#4e90e3\" stroke=\"black\" d=\"M1842,-97C1842,-97 1812,-97 1812,-97 1806,-97 1800,-91 1800,-85 1800,-85 1800,-73 1800,-73 1800,-67 1806,-61 1812,-61 1812,-61 1842,-61 1842,-61 1848,-61 1854,-67 1854,-73 1854,-73 1854,-85 1854,-85 1854,-91 1848,-97 1842,-97\"/>\n<text text-anchor=\"middle\" x=\"1827\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer2_norm&#45;&gt;layer2_act -->\n<g id=\"edge21\" class=\"edge\"><title>layer2_norm&#45;&gt;layer2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1764.4,-79C1772.39,-79 1781.31,-79 1789.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1789.92,-82.5001 1799.92,-79 1789.92,-75.5001 1789.92,-82.5001\"/>\n</g>\n<!-- layer3_conv -->\n<g id=\"node21\" class=\"node\"><title>layer3_conv</title>\n<g id=\"a_node21\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 256, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1932,-97C1932,-97 1902,-97 1902,-97 1896,-97 1890,-91 1890,-85 1890,-85 1890,-73 1890,-73 1890,-67 1896,-61 1902,-61 1902,-61 1932,-61 1932,-61 1938,-61 1944,-67 1944,-73 1944,-73 1944,-85 1944,-85 1944,-91 1938,-97 1932,-97\"/>\n<text text-anchor=\"middle\" x=\"1917\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer2_act&#45;&gt;layer3_conv -->\n<g id=\"edge22\" class=\"edge\"><title>layer2_act&#45;&gt;layer3_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1854.4,-79C1862.39,-79 1871.31,-79 1879.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1879.92,-82.5001 1889.92,-79 1879.92,-75.5001 1879.92,-82.5001\"/>\n</g>\n<!-- layer3_pool -->\n<g id=\"node22\" class=\"node\"><title>layer3_pool</title>\n<g id=\"a_node22\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M2022,-97C2022,-97 1992,-97 1992,-97 1986,-97 1980,-91 1980,-85 1980,-85 1980,-73 1980,-73 1980,-67 1986,-61 1992,-61 1992,-61 2022,-61 2022,-61 2028,-61 2034,-67 2034,-73 2034,-73 2034,-85 2034,-85 2034,-91 2028,-97 2022,-97\"/>\n<text text-anchor=\"middle\" x=\"2007\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_conv&#45;&gt;layer3_pool -->\n<g id=\"edge23\" class=\"edge\"><title>layer3_conv&#45;&gt;layer3_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1944.4,-79C1952.39,-79 1961.31,-79 1969.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1969.92,-82.5001 1979.92,-79 1969.92,-75.5001 1969.92,-82.5001\"/>\n</g>\n<!-- layer3_norm -->\n<g id=\"node23\" class=\"node\"><title>layer3_norm</title>\n<g id=\"a_node23\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 512}\">\n<path fill=\"#b07b87\" stroke=\"black\" d=\"M2112,-97C2112,-97 2082,-97 2082,-97 2076,-97 2070,-91 2070,-85 2070,-85 2070,-73 2070,-73 2070,-67 2076,-61 2082,-61 2082,-61 2112,-61 2112,-61 2118,-61 2124,-67 2124,-73 2124,-73 2124,-85 2124,-85 2124,-91 2118,-97 2112,-97\"/>\n<text text-anchor=\"middle\" x=\"2097\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_pool&#45;&gt;layer3_norm -->\n<g id=\"edge24\" class=\"edge\"><title>layer3_pool&#45;&gt;layer3_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2034.4,-79C2042.39,-79 2051.31,-79 2059.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2059.92,-82.5001 2069.92,-79 2059.92,-75.5001 2059.92,-82.5001\"/>\n</g>\n<!-- layer3_act -->\n<g id=\"node24\" class=\"node\"><title>layer3_act</title>\n<g id=\"a_node24\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#4e90e3\" stroke=\"black\" d=\"M2202,-97C2202,-97 2172,-97 2172,-97 2166,-97 2160,-91 2160,-85 2160,-85 2160,-73 2160,-73 2160,-67 2166,-61 2172,-61 2172,-61 2202,-61 2202,-61 2208,-61 2214,-67 2214,-73 2214,-73 2214,-85 2214,-85 2214,-91 2208,-97 2202,-97\"/>\n<text text-anchor=\"middle\" x=\"2187\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_norm&#45;&gt;layer3_act -->\n<g id=\"edge25\" class=\"edge\"><title>layer3_norm&#45;&gt;layer3_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2124.4,-79C2132.39,-79 2141.31,-79 2149.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2149.92,-82.5001 2159.92,-79 2149.92,-75.5001 2149.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_in -->\n<g id=\"node25\" class=\"node\"><title>layer3_residual_in</title>\n<g id=\"a_node25\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2292,-97C2292,-97 2262,-97 2262,-97 2256,-97 2250,-91 2250,-85 2250,-85 2250,-73 2250,-73 2250,-67 2256,-61 2262,-61 2262,-61 2292,-61 2292,-61 2298,-61 2304,-67 2304,-73 2304,-73 2304,-85 2304,-85 2304,-91 2298,-97 2292,-97\"/>\n<text text-anchor=\"middle\" x=\"2277\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer3_act&#45;&gt;layer3_residual_in -->\n<g id=\"edge26\" class=\"edge\"><title>layer3_act&#45;&gt;layer3_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2214.4,-79C2222.39,-79 2231.31,-79 2239.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2239.92,-82.5001 2249.92,-79 2239.92,-75.5001 2239.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_res1_conv -->\n<g id=\"node26\" class=\"node\"><title>layer3_residual_res1_conv</title>\n<g id=\"a_node26\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2382,-68C2382,-68 2352,-68 2352,-68 2346,-68 2340,-62 2340,-56 2340,-56 2340,-44 2340,-44 2340,-38 2346,-32 2352,-32 2352,-32 2382,-32 2382,-32 2388,-32 2394,-38 2394,-44 2394,-44 2394,-56 2394,-56 2394,-62 2388,-68 2382,-68\"/>\n<text text-anchor=\"middle\" x=\"2367\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_res1_conv -->\n<g id=\"edge27\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.4,-70.299C2312.57,-67.608 2321.7,-64.5979 2330.38,-61.7368\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2331.52,-65.049 2339.92,-58.5949 2329.33,-58.4007 2331.52,-65.049\"/>\n</g>\n<!-- layer3_residual_add -->\n<g id=\"node33\" class=\"node\"><title>layer3_residual_add</title>\n<g id=\"a_node33\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M3012,-93C3012,-93 2982,-93 2982,-93 2976,-93 2970,-87 2970,-81 2970,-81 2970,-69 2970,-69 2970,-63 2976,-57 2982,-57 2982,-57 3012,-57 3012,-57 3018,-57 3024,-63 3024,-69 3024,-69 3024,-81 3024,-81 3024,-87 3018,-93 3012,-93\"/>\n<text text-anchor=\"middle\" x=\"2997\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_add -->\n<g id=\"edge34\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.22,-93.0359C2312.83,-96.9904 2322.61,-100.808 2332,-103 2385.71,-115.535 2400.84,-108 2456,-108 2456,-108 2456,-108 2818,-108 2867.77,-108 2924.08,-95.3711 2959.8,-85.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2961.04,-89.0265 2969.74,-82.9884 2959.17,-82.2796 2961.04,-89.0265\"/>\n</g>\n<!-- layer3_residual_res1_norm -->\n<g id=\"node27\" class=\"node\"><title>layer3_residual_res1_norm</title>\n<g id=\"a_node27\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 512}\">\n<path fill=\"#b07b87\" stroke=\"black\" d=\"M2472,-68C2472,-68 2442,-68 2442,-68 2436,-68 2430,-62 2430,-56 2430,-56 2430,-44 2430,-44 2430,-38 2436,-32 2442,-32 2442,-32 2472,-32 2472,-32 2478,-32 2484,-38 2484,-44 2484,-44 2484,-56 2484,-56 2484,-62 2478,-68 2472,-68\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm -->\n<g id=\"edge28\" class=\"edge\"><title>layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2394.4,-50C2402.39,-50 2411.31,-50 2419.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2419.92,-53.5001 2429.92,-50 2419.92,-46.5001 2419.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res1_act -->\n<g id=\"node28\" class=\"node\"><title>layer3_residual_res1_act</title>\n<g id=\"a_node28\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#4e90e3\" stroke=\"black\" d=\"M2562,-68C2562,-68 2532,-68 2532,-68 2526,-68 2520,-62 2520,-56 2520,-56 2520,-44 2520,-44 2520,-38 2526,-32 2532,-32 2532,-32 2562,-32 2562,-32 2568,-32 2574,-38 2574,-44 2574,-44 2574,-56 2574,-56 2574,-62 2568,-68 2562,-68\"/>\n<text text-anchor=\"middle\" x=\"2547\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act -->\n<g id=\"edge29\" class=\"edge\"><title>layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2484.4,-50C2492.39,-50 2501.31,-50 2509.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2509.92,-53.5001 2519.92,-50 2509.92,-46.5001 2509.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_conv -->\n<g id=\"node29\" class=\"node\"><title>layer3_residual_res2_conv</title>\n<g id=\"a_node29\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2652,-68C2652,-68 2622,-68 2622,-68 2616,-68 2610,-62 2610,-56 2610,-56 2610,-44 2610,-44 2610,-38 2616,-32 2622,-32 2622,-32 2652,-32 2652,-32 2658,-32 2664,-38 2664,-44 2664,-44 2664,-56 2664,-56 2664,-62 2658,-68 2652,-68\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv -->\n<g id=\"edge30\" class=\"edge\"><title>layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2574.4,-50C2582.39,-50 2591.31,-50 2599.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2599.92,-53.5001 2609.92,-50 2599.92,-46.5001 2599.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_norm -->\n<g id=\"node30\" class=\"node\"><title>layer3_residual_res2_norm</title>\n<g id=\"a_node30\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 512}\">\n<path fill=\"#b07b87\" stroke=\"black\" d=\"M2742,-68C2742,-68 2712,-68 2712,-68 2706,-68 2700,-62 2700,-56 2700,-56 2700,-44 2700,-44 2700,-38 2706,-32 2712,-32 2712,-32 2742,-32 2742,-32 2748,-32 2754,-38 2754,-44 2754,-44 2754,-56 2754,-56 2754,-62 2748,-68 2742,-68\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm -->\n<g id=\"edge31\" class=\"edge\"><title>layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2664.4,-50C2672.39,-50 2681.31,-50 2689.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2689.92,-53.5001 2699.92,-50 2689.92,-46.5001 2689.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_act -->\n<g id=\"node31\" class=\"node\"><title>layer3_residual_res2_act</title>\n<g id=\"a_node31\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#4e90e3\" stroke=\"black\" d=\"M2832,-68C2832,-68 2802,-68 2802,-68 2796,-68 2790,-62 2790,-56 2790,-56 2790,-44 2790,-44 2790,-38 2796,-32 2802,-32 2802,-32 2832,-32 2832,-32 2838,-32 2844,-38 2844,-44 2844,-44 2844,-56 2844,-56 2844,-62 2838,-68 2832,-68\"/>\n<text text-anchor=\"middle\" x=\"2817\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act -->\n<g id=\"edge32\" class=\"edge\"><title>layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2754.4,-50C2762.39,-50 2771.31,-50 2779.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2779.92,-53.5001 2789.92,-50 2779.92,-46.5001 2779.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_out -->\n<g id=\"node32\" class=\"node\"><title>layer3_residual_out</title>\n<g id=\"a_node32\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2922,-78C2922,-78 2892,-78 2892,-78 2886,-78 2880,-72 2880,-66 2880,-66 2880,-54 2880,-54 2880,-48 2886,-42 2892,-42 2892,-42 2922,-42 2922,-42 2928,-42 2934,-48 2934,-54 2934,-54 2934,-66 2934,-66 2934,-72 2928,-78 2922,-78\"/>\n<text text-anchor=\"middle\" x=\"2907\" y=\"-56.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_act&#45;&gt;layer3_residual_out -->\n<g id=\"edge33\" class=\"edge\"><title>layer3_residual_res2_act&#45;&gt;layer3_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2844.4,-53.0003C2852.39,-53.9083 2861.31,-54.9217 2869.82,-55.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2869.59,-59.3847 2879.92,-57.0362 2870.38,-52.4294 2869.59,-59.3847\"/>\n</g>\n<!-- layer3_residual_out&#45;&gt;layer3_residual_add -->\n<g id=\"edge35\" class=\"edge\"><title>layer3_residual_out&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2934.4,-64.5005C2942.39,-65.8625 2951.31,-67.3825 2959.82,-68.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2959.47,-72.3242 2969.92,-70.5544 2960.65,-65.4237 2959.47,-72.3242\"/>\n</g>\n<!-- pool -->\n<g id=\"node34\" class=\"node\"><title>pool</title>\n<g id=\"a_node34\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 4}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M3102,-93C3102,-93 3072,-93 3072,-93 3066,-93 3060,-87 3060,-81 3060,-81 3060,-69 3060,-69 3060,-63 3066,-57 3072,-57 3072,-57 3102,-57 3102,-57 3108,-57 3114,-63 3114,-69 3114,-69 3114,-81 3114,-81 3114,-87 3108,-93 3102,-93\"/>\n<text text-anchor=\"middle\" x=\"3087\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_add&#45;&gt;pool -->\n<g id=\"edge36\" class=\"edge\"><title>layer3_residual_add&#45;&gt;pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3024.4,-75C3032.39,-75 3041.31,-75 3049.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3049.92,-78.5001 3059.92,-75 3049.92,-71.5001 3049.92,-78.5001\"/>\n</g>\n<!-- classifier_flatten -->\n<g id=\"node35\" class=\"node\"><title>classifier_flatten</title>\n<g id=\"a_node35\"><a xlink:title=\"&lt;class &#39;__main__.Flatten&#39;&gt; {}\">\n<path fill=\"#b3de69\" stroke=\"black\" d=\"M3192,-93C3192,-93 3162,-93 3162,-93 3156,-93 3150,-87 3150,-81 3150,-81 3150,-69 3150,-69 3150,-63 3156,-57 3162,-57 3162,-57 3192,-57 3192,-57 3198,-57 3204,-63 3204,-69 3204,-69 3204,-81 3204,-81 3204,-87 3198,-93 3192,-93\"/>\n<text text-anchor=\"middle\" x=\"3177\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">flatten</text>\n</a>\n</g>\n</g>\n<!-- pool&#45;&gt;classifier_flatten -->\n<g id=\"edge37\" class=\"edge\"><title>pool&#45;&gt;classifier_flatten</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3114.4,-75C3122.39,-75 3131.31,-75 3139.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3139.92,-78.5001 3149.92,-75 3139.92,-71.5001 3139.92,-78.5001\"/>\n</g>\n<!-- classifier_conv -->\n<g id=\"node36\" class=\"node\"><title>classifier_conv</title>\n<g id=\"a_node36\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.linear.Linear&#39;&gt; {&#39;in_features&#39;: 512, &#39;out_features&#39;: 10, &#39;bias&#39;: False}\">\n<path fill=\"#fccde5\" stroke=\"black\" d=\"M3282,-93C3282,-93 3252,-93 3252,-93 3246,-93 3240,-87 3240,-81 3240,-81 3240,-69 3240,-69 3240,-63 3246,-57 3252,-57 3252,-57 3282,-57 3282,-57 3288,-57 3294,-63 3294,-69 3294,-69 3294,-81 3294,-81 3294,-87 3288,-93 3282,-93\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- classifier_flatten&#45;&gt;classifier_conv -->\n<g id=\"edge38\" class=\"edge\"><title>classifier_flatten&#45;&gt;classifier_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3204.4,-75C3212.39,-75 3221.31,-75 3229.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3229.92,-78.5001 3239.92,-75 3229.92,-71.5001 3229.92,-78.5001\"/>\n</g>\n<!-- classifier_scale -->\n<g id=\"node37\" class=\"node\"><title>classifier_scale</title>\n<g id=\"a_node37\"><a xlink:title=\"&lt;class &#39;__main__.Mul&#39;&gt; {&#39;weight&#39;: 0.0625}\">\n<path fill=\"#bc80bd\" stroke=\"black\" d=\"M3372,-93C3372,-93 3342,-93 3342,-93 3336,-93 3330,-87 3330,-81 3330,-81 3330,-69 3330,-69 3330,-63 3336,-57 3342,-57 3342,-57 3372,-57 3372,-57 3378,-57 3384,-63 3384,-69 3384,-69 3384,-81 3384,-81 3384,-87 3378,-93 3372,-93\"/>\n<text text-anchor=\"middle\" x=\"3357\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">scale</text>\n</a>\n</g>\n</g>\n<!-- classifier_conv&#45;&gt;classifier_scale -->\n<g id=\"edge39\" class=\"edge\"><title>classifier_conv&#45;&gt;classifier_scale</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3294.4,-75C3302.39,-75 3311.31,-75 3319.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3319.92,-78.5001 3329.92,-75 3319.92,-71.5001 3319.92,-78.5001\"/>\n</g>\n<!-- logits -->\n<g id=\"node38\" class=\"node\"><title>logits</title>\n<g id=\"a_node38\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M3462,-93C3462,-93 3432,-93 3432,-93 3426,-93 3420,-87 3420,-81 3420,-81 3420,-69 3420,-69 3420,-63 3426,-57 3432,-57 3432,-57 3462,-57 3462,-57 3468,-57 3474,-63 3474,-69 3474,-69 3474,-81 3474,-81 3474,-87 3468,-93 3462,-93\"/>\n<text text-anchor=\"middle\" x=\"3447\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">logits</text>\n</a>\n</g>\n</g>\n<!-- classifier_scale&#45;&gt;logits -->\n<g id=\"edge40\" class=\"edge\"><title>classifier_scale&#45;&gt;logits</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3384.4,-75C3392.39,-75 3401.31,-75 3409.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3409.92,-78.5001 3419.92,-75 3409.92,-71.5001 3409.92,-78.5001\"/>\n</g>\n<!-- input -->\n<g id=\"node39\" class=\"node\"><title>input</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-92C42,-92 12,-92 12,-92 6,-92 0,-86 0,-80 0,-80 0,-68 0,-68 0,-62 6,-56 12,-56 12,-56 42,-56 42,-56 48,-56 54,-62 54,-68 54,-68 54,-80 54,-80 54,-86 48,-92 42,-92\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">input</text>\n</g>\n<!-- input&#45;&gt;prep_conv -->\n<g id=\"edge1\" class=\"edge\"><title>input&#45;&gt;prep_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4029,-74C62.3932,-74 71.3106,-74 79.8241,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"79.919,-77.5001 89.919,-74 79.919,-70.5001 79.919,-77.5001\"/>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oe5ByKdin8ZX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 18, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.6, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "opt_params_bias = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.6*16, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size/16), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(frozen_bn_scale_net, label_smoothing_loss(0.2))\n",
        "    is_bias = group_by_key(('bias' in k, v) for k, v in trainable_params(model).items())\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(is_bias[False], opt_params), SGD(is_bias[True], opt_params_bias)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F29j8KyvenL_",
        "colab_type": "text"
      },
      "source": [
        "Test accuracy improves to 94.2%. Interestingly, had we not increased the learning rate of the batch norm biases, we would have achieved a substantially lower accuracy as the reader can verify. This suggests that the learnable biases are indeed doing something useful - either learning appropriate levels of sparsity or perhaps just adding regularisation noise. Indeed we can improve things slightly by increasing the learning rate of the biases even further:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "USqjaYDnpogJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 18, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.6, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "opt_params_bias = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.6*64, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size/64), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(frozen_bn_scale_net, label_smoothing_loss(0.2))\n",
        "    is_bias = group_by_key(('bias' in k, v) for k, v in trainable_params(model).items())\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(is_bias[False], opt_params), SGD(is_bias[True], opt_params_bias)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpwXfP3sRIgU",
        "colab_type": "text"
      },
      "source": [
        "Finally we can use the increased accuracy to reduce training to 17 epochs. The new test accuracy is 94.1% and most importantly we've overtaken the 8 GPUs of BaiduNet9P with a time of 43s, placing us second on the leaderboard!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MDGYbRmvQVts",
        "colab_type": "text"
      },
      "source": [
        "### Input patch whitening (36s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gmfdPrAIVhUE",
        "colab_type": "text"
      },
      "source": [
        "Batch norm does a good job at controlling distributions of individual channels but doesn't tackle covariance between channels and pixels. Controlling the covariance at internal layers, using a 'whitening' version of batch norm, might be helpful but would entail extra computation as well as non-trivial implementation effort. We are going to focus on the easier problem at the input layer.\n",
        "\n",
        "The classic way to remove input correlations is to perform global PCA (or ZCA) whitening. We propose a patch-based approach which is agnostic to the total image size and more in keeping with the structure of a conv net. We are going to apply PCA whitening to 3×3 patches of inputs as an initial 3×3 convolution with fixed (non-learnable) weights. We will follow this with a learnable 1×1 convolution. The 27 input channels to this layer are a transformed version of the original 3×3×3 input patches whose covariance matrix is approximately the identity, which should make optimisation easier.\n",
        "\n",
        "First let's plot the leading eigenvectors of the covariance matrix of 3×3 patches of the input data. The numbers in brackets are the square root of the corresponding eigenvalues to show the relative scales of variation along these directions and we plot the eigenvector with both signs to illustrate the direction of variation. As we might expect, variations in local brightness dominate.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zY1hEAoONY0P",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190
        },
        "outputId": "7f148b17-14fe-4a7a-a00a-bfd5052c44cc"
      },
      "source": [
        "def cov(X):\n",
        "    X = X/np.sqrt(X.size(0) - 1)\n",
        "    return X.t() @ X\n",
        "\n",
        "def patches(data, patch_size=(3, 3), dtype=torch.float32):\n",
        "    h, w = patch_size\n",
        "    c = data.size(1)\n",
        "    return data.unfold(2,h,1).unfold(3,w,1).transpose(1,3).reshape(-1, c, h, w).to(dtype)\n",
        "\n",
        "def eigens(patches):\n",
        "    n,c,h,w = patches.shape\n",
        "    Σ = cov(patches.reshape(n, c*h*w))\n",
        "    Λ, V = torch.symeig(Σ, eigenvectors=True)\n",
        "    return Λ.flip(0), V.t().reshape(c*h*w, c, h, w).flip(0)\n",
        "\n",
        "Λ, V = eigens(patches(train_set['data'][:10000,:,4:-4,4:-4])) #center crop to remove padding\n",
        "\n",
        "layout([\n",
        "    [partial(image_plot, img=V[i].to(torch.float16)*3, title=f'{i+1} ({torch.sqrt(Λ[i]):.2f})') for i in range(len(V))],\n",
        "    [partial(image_plot, img=-V[i].to(torch.float16)*3, title='') for i in range(len(V))],\n",
        "], col_width=1.0, row_height=1.0\n",
        ");"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1944x144 with 54 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABgYAAACZCAYAAADkSYGdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xm4FMW9//FPcYADalwwLogL7rtRkyCiiBHFPVFBjZgILlGjxtyYiN6rEoMmKmiSm0TjchWVaDDu0bgQ3EBx119wI7iAW8BEUaOIgFC/P6qOt2mmp7urzszx0u/X85wHZrq+Xf2dru6u7uqeMdZaAQAAAAAAAACAaujU0QsAAAAAAAAAAACah4EBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAq5AsxMGCMWc0Y83djTLcm17uNMWZKE+rpqPzWMMa8ZIxpbUJdy/o63MIY81Sj66lR7zeNMeObUM+ynl9Htc9lPb9mbX+txpgXjTFrNrquVL1NWX++rmV6Hdao9zxjzH90QL23GGP2akI9xxljft3oemrU+0tjzPFNqIf8GlNvU9qnr2tZ3waX9fyW6TZagfwGGWNua3Q9Neo92RhzfhPqWabXn6+r6fsY3x+eZoxZvQl1ddQ6fMIYs2UT6iG/xtTbrPw66hhPfu1TzzJ9jPg/mZ+1tt3/JJ0k6SlJ8yVdXaD8RZJOr/F+D0n/kvRwndjhkhZJ+jjxt2ti+raSJkv6UNJbkkam4u+StH+J3FolXSnpdUkfSXpW0t5l8pN0iKQpkj6R9GBO7DckPSfpA0nvSbpVUq/E9NGS3pT0b79MZ6TiL5H0g4B1+AdJs/x8p0s6poE57ivpYZ/jbElXSPpS6jO/yi/LbEmnxKzDVOzGkj6V9IeccjdL+nZEG/+RX/YPfS6tqek/lDRD0lxJL0naJDHteUnblMzrQZ9X2zbx95L59fBtba5vV0Nz4reXNMnX9Y6kH/r3V5f0R0n/8Lk/ImmHVGzp/Hzct/1nNVfSq5L6l2ifddtUKna4MvYxjcovVdfHvv7fNiK/1Hzul2QldU681677UB/T28e975fvd8k6a5T/gaRLE6+NpAvk9onvye0HTYF6x/r8Nqoxrea+IKJ9bu4/zw8lvSLpwJzyMW10mKSnfdm3/OfROVUmc3spsg6Vs8+TNFDSNLl9/gOS1qszr9UkvS2pe9n4gtvcapKulzumvC/pusS0PpKeLpOfpK6SbpI007efXXM+q65yx+XksXpbv44+8f9umxGb28eQtJzcsf1d/xlMSkzr6evuWiK/vpL+KmmOXJ/rRkk9OyI/SVv45Xzf/02UtEVkfnXn2Z75FdlWEuV+6tvT7nntM3a+7bkN+rLnyPVNP5N0dmpaXr81ZBvs7XNKHhfP6qD88vqspfPL266b3UbrfVaB6+/w1Lr7xK/PrzYov8w+rFz/4QxJb8gdM8dLWrEd1t8hcsfYjyS9KOmArOXz5Z+S1DfVxh/w+U1T/e0377woc1kkdZPrJ6xeMr9j5PoyH0u6R9JaDWyfdc9BFHaML3JsbUo/Jm99K//axghJF5XJT03sxxTYBneVtDiV37BU+725ZH5N68cUaaOJckuddwTm1+x+TFPzS5VtRh+mt7K3v29L+rvccfifkq7RkseIkPXXW83tw9TLb5jqnDOGrj81tw9Tb/9Suo9WYP19kfowPSX9We482ErqnYrNPI/I+ysdUGim0kGSDpD0e+VcNPUr4l1Ja9eYdoXcxca8gYF601+U9HNJLZI2lLvY/c3Uir6zRG7LSzrbb3CdJO3nG1DvjPJL5Sdpd7/RjVT+RfM15Dtffl6jJf05MX1TScv7//eS9IKkgxLTd5L0fMA63FL+4rWkzfyGldX4Y3McKmkvuR3KKpLu1pIXAc+TuzC5itzFttmS9gpdh6m6J/h5Zw4M+A1wjqRugW18T7mL5Vv6HB6UdH5i+jGSpsod9I1vpz0S08+Q9LuSeT2onMGcnPz+KOkGSStI2lluB79lRvyX5Q6ch/u28CVJm/tpG0g6xdfRIulY31ZWiMxvD7kdZV+57bCXEjvfAu2zbptKxQ9Xxj6mUfml6lhe7iC0SyPyS21Hk7T0wEC77kN9zF2SrpY7QV1T7gLMyXXKPy9pp8Tr4+Q6bWv7df+ipONz6tw5kV+tgYGa+4LA9tlZbkD1FP+57SZ3cN8ko3xsG/2+pP5yHZFech2O5CBD3e2lyDpUnX2e3D7gQ0kH+3U6RtJjdeZ1qqQrQuJVbJubLOmXklaS1EXSdql5vCzpayXy6yrpP3wbmqX8E+qDJf01Ff+63ABxq6ST/euuNWJz+xhyA/fj5U4cWpQ6NsudHA8pkd/efplXlDsOXyXpno7IT9LKfprxuZ0saWpkfrnzbK/88pYlUWZDuf3eP5Q6+a3VPttjvu21Dfryw3y7uV1LXziv228N3AZ7K3VsytkGG5lf3T5rSH5FtutmtdEin1VIfqmyw+UGqGsO6LdDfpl9WL9up0lax0+/XdI1ke2zl6QFvs0YuQsTnyh18T1R/uuSXk6996jccau7pMFyFzVWy4jP7B8UWRa58+uflMhvgFw/f0u/Ln4v6aEGts+65yAKO8bnHXua1o/JW9/Kv7axtly/p7VEfk3rxxTYBneV9FadurvJnZf2LJFf0/oxRdqoL1PzvCMwv2b3Y5qaX6Jcs/ow9ba/dSR92f9/BUnXSfpN5Prrreb2Yerll3fOGLT+1Nw+TL39S+k+WpH8UvHD1XF9mDUknSBpR9UYGKiVX9G/UoVLz1w6V/mdw10kvVLj/R19oz5ScQMDn2jJu81ulPSfide9JM1T6g7uknlOlTS4TH5+2jHKuWieKt8q1xl8MWN6L7md6YjEe539Z7BeRH6bynUiDml0jj7mIEnPJV6/LWlQ4vU5ksbHrkO5EeE/+R1BvYGBIyRNzJhWpI1fL+kXidcDJc32/+8kN6I4sE78TpJmlMztQRUfGFgiP7md4wIt+dTCOCUGM1Lxv5A0rsSy/VuJg0VgflMkHV2w7FLtM69NpcrW3cc0Ir/U/IZJek3ZB5+o/Pz0leQuZPfV0gMD7b4PlbuTbZ/E6zGSLssou66ff3KZpkg6NvH6aNXvIHWWG/3fRjUGBurtCwLb51Zygzkm8d4ESec0ah2mYk+RdEfq88rcXsqsQ9XY58ldnJ+SeL28n99mGfO4X9J3QuNrzO/zbU7SILk74lrqlL9C0k+L5pea/pbyT6ivknRm4vUgvz6T7eEN5QzWJcp+3seQOx7/W4m7l2qUP0PS2JD8fJntJX3UEfml3u8s6URJn7RXflnzbER+9ZZF7kRlH99W0wMDme0zZr6pcu2yDcqdCJ5dZ3rNfmvZbVDlT6qbkp8vs0SfNTC/3O26WW20yGdVNr8aZR7IaePB+SmnDyt31/SpiWn95J4WXC5i/e0g6Z+p9/4laceMeYyU9D+J15vI3amfvKtxsjJueFCd/kGRZZG7GeCBEvldKOnixOu1/Pa4YQesv6hjfKpc8tjatH5M3vpWgfMOuQs/A4rml3q/of2YAutwV9UZGPBl/qrEUwRl8vPTGtaPycvPv8477wjOTw3ux3RkfmpCHyZv+0uVXUHStZLuislPTezDlMnPT1vinDEwv6b1YYq0z1T5Un20eu0zMa3D+jCJ9zore2Cg0DEw/fdF+I2BreXu/PycMaZF0sVyjzXaAvPYzhjzrjFmujHmLGNM58S0X0s6whjTxRizqdyAw8S2idbatyUtlGvQpRlj1pDbAF/IKLJUfgF1rGuM+UBuB/ATubuvktNPN8Z8LHegX17uQrQkyVr7mdyjn18JqPcSY0zbI0iz5O7yrSU6x5Rd5D9PY8wqch3QvyWm/03urhVJYevQGLOipFGSflygeGx+W2rp5V/DGLOq3F0fa0vayhjzpjFmhjHmZ8aY5Lb5kqTefpnLOM9vF48YY3atUy6d3yaSFllrp6eWOev75vpKmmOMmWKM+acx5g5jzLq1ChpjtpUbNX0l8Xap/Pz+4WuSVjPGvGKMecsY8ztjTPeMkCXyK9Kmaqi3j0kuW3R+NQyTdK31e/oa2iO/X8jdATa7xrRG7EP/W9K3jTHLGWN6yd3pc09G2a0lveb3ZW1qbVP18vuR3CONU9MTCuwLQtafyXhvq4zy7bEOk5L70NztJfY4qNT6sNa2fV1R1vKm9zll4z9XY5vr6+d9jTHmPWPMk8aYAamwlxRwTCyhVn5TU9vwVBXLL93H2EHuLpOf+X3Sc8aYwamw2Pw+bz8ZGplf2/sfyF2s+63c/ikpKL+ceSa1W34Zy3GwpAXW2qw+VWh+efNNardtMGNZ6vZbFd5GX/f7sLHGmC/XKdfQ/FJqbS9l8yuyXSc1so0W+ayC9zHGmPXkPrNr6xSLyS+vD2u05DHayA1gbZx4r2x+T0l6ybjfJWoxxhwgd2FmqT6HVyu/16y1H2Us8/8ubH7/oMiylM2v1mcmFezTqH3XX7sc42sce5rZjymyvvPOO+rmWOD6RJ5GrkNJWt0Y844/7/2VMWb51Dxi82tkP6ZIfpnnHV5Qfk3qx3RIfk3sw+Ruf8aYnY0xH8rdNT5Y7lw4KbR9NqMPU/h44pXuwwSemyQ1un0mtUd+yWkd3YcpIqiP9kUYGFhZbqNLOlnS49bapwvET5LrmKwut+EeJvf4TZs7JQ2ROzmZJulKa+2TqXl85JejFGNMF7nHi66x1k7LKFYrv1KstW9Ya1eWe4zoTLk8ktPPl/v6lu3lRpQ+TM0iKD9r7Ql+vv0l3SLXsawlOsc2xpg95C6EjvRvreD/Teb0oV+upLI5niPXFt4sUDY2vxW09PJLLoe1/f8Hye1EviHXho9OlG+ru0x+p8l95UYvSZdLusMYs2FG2XR+6eVtW+b0Z95mbbl19kO5u7tnyD0CtQR/YXWcpJ9Za5PzL5vfGnKPDg+Ra5vbStpObtuopVZ+Un6bapO3j5HUrvkl57mu3CPc19QpFpWfMeZrcnfF/zZj/o3Yhz4kd4Br+37DpyRl/Qhfre2v1ja1gjFmqQvyxph15L56aGR6mpe3LwhZf9PkHrs/1Q+oDJJbj8tllI9to58zxhwpNxBwoX+r6PYSdJxILG+ZfUbsPkdS5ja3ttz+9AG5r6m6SNLtqQ54TK5FtFd+tfoYa8vtjz6Uuzh0ktwFks0TocH5GWO2kdtWltrHJTQyP0mS7/OsJJffs6nQ0D5NvXkmtUt+tRhjVpA7ma/3g3Kl8ys436SG5Sjl91tVPsd35b5+ZT1JX/XLdV2d8g3Nr02NPmubsvkV2a6TGplfkXnF7EOPkDTZWjujTpmY/PLK3i3pGGNMb2PMSnL9ZWnJ43Op/Ky1i+QuElwvd650vaTj/MWbWmLza5u+VNmCy/KR3L6wqLskHWKM2cbfVDBS7sa9Mn2a9lp/0cf4jGNPM/sxeWWLnHdk5ljw+kSeRuY3Ta4v2lPuqza/Kve1J0nB+TWhH1O3bIHzDikwvyb1Y5qeX5P7MLllrbUPW2tXktvfjJF7giGp7PprZh+mcNka54xtyubXzD5MmfxK99EK7D87ug9TRFAf7YswMPC+EokaY9aSGxg4o0iwtfY1a+0Ma+1ia+1zcnd+DvHz6iF3F+ooue/LWkfSnsaYE1Kz+ZLcd28VZtwd3ePkHvU4qU7RJfKLYa2dI3eB8Pb0nQPWeVbu4t3PUqGl80vMd5G19mG5Df77GcXaJUdjTF+5DuyQxCjZx/7f5N26K2rpC4WFc/R3mO4u6VcFFy02v4+19PJLLod5/v+jrbUfWGtnSrpM7jG6Nm11F16H1trHrbUfWWvnW2uvkfuBzn0yiqfzSy9v2zJnDY7Mk3SrtfZJa+2ncu2vnz/hkiT5E4k75L7u5bxUfNn82j6z31prZ1lr35XrUJbJT8pvU5Lq72PatHN+SUfIPU5c7+ATnJ/fj10i92PRn9WY3u77UF/nvXKDjcvLXThaRe7HhGuptf3V2qY+zniq4teSRqUGa9qWpci+IGT7Wyj3Xb37yj2F8WO5ryp6KyMkqo228XcGni/3g0nv+reLbi/BxwmV32fE7nPqbXPzJM201l5prV1orR0v93VtOyXKxORaRHvkl9XHmCf3dMe51toF1tqH5C6QDEqUCcrPGLOR3EWzH1prJ9cp2sj8PucvZF0q6VpjzOqJSTF9mqx5JkXnV8fP5L56r94+PSS/IvNNamSOn6vTby2Vo7X2Y2vtU9baz6y178i1mUEm+0muhueX0WdtU3YdFtmukxqZX5F5xexDj1D9mx2kuPzyyl4ld/PKg3J3Az7g308en0vlZ4zZXe6pmF3lnmAbIOl/fB+jltj82qYvVbbgsnxJS194yGStvU/uB0FvlrsrdKavr0yfpr3WX9Qxvs6xp5n9mLpli5x3KCPHEtcn8jQyv9nW2hd9fjPkfky5XfJrUj8mr2zmeUdC8PprQj+mI/JrZh+mcFnrnqi+R+6785NK5dfkPkyhshnnjG3Krr9m9mGK5le6j1Zw/9nRfZgigvpoX4SBgalyj0y06SM3gvyiMWa23FdO9DHGzDbuKxHyWP3vI44byD2Kca3fEN+S27A/vyDiByK6qsRXxfi7Uq+UuxNzsL8IlCWdX6zOcncQZO1IOsv9cIskyZ+IbaQlHzkNrTfrjvPoHI0x28n9wvZRvgMqSbLWvi/3NUbJx2G+oiUfPSu7DneV+663N3wb+4mkwcaYZzLKx+b3gpZe/neste/JLfMC1f/KrM3lOsH/jliG5HaRls5vuqTOxpjkY9VLfOY14pPL3/Z/I0nGmFa5u8HflrvDIK1Ufr5NvKX6n1l6+T7Pr0ibylsEJT7L9s4vpcjBJya/FeXuFLjBbwttTwK8ZYzpr8bsQ3vIDTD8zg9cvSdprLIHdqZK2iB1UanWNpW1/gZKGuOPIW1flfSoMWaoiu0LgtaftXaqtXaAtXZVa+2ecp/lExnFo9uoMWYvue8U3N+fSCbnVXd7CTkOpiyxPox7JHzDOsub3ueUis/Z5tL7o1o2V/wxsZ5a+W2TeqJlG2XnV6+PkfXYdlLp/PyjsRPlfgdjXE7xRuaX1knurtReifdi11+teSZF5ZdjoKSTE/ujdST9yRhzWqJMSH5F5psUtQ2WVKvfGrsOl+hn1NDQ/LL6rAll8yuyXafLN6qNFvmsgtafMWYnubsJb8opGpNf3T6svxj5U2ttb2vt2v79t/1fm7L5bSv3tRpP+fk/KelxuRsPaqmV3wbGmOSFhJrH/AL9gyLLUnr9WWsvttZubK1dXW6AoLOk5zOKN2z9KeIYn3PsaWY/pvD69mqdwy2VY8lja55GrsO0dsmvif2YvPzqnXcE55fSyH5MR+TXzD5M2e2v1jWw2PXXyD5Mbn5Z54wJZfNrZh8md/8S0kcrsv6+CH2YgsL62LbkjxIU+ZPbgLrJ/eDYOP//mj+2IXcx4l+SevnXrXKPBrb9/VCuQ7NmRvzektbw/99MrpPyU/96RbnRkqFyO9A15X7Q+OeJ+KFK/aBIgfwulfSYpBUKlF0iP/9ei/9Mjpd7XLCbpC4Z8QfJfe9zJ7lf+f6TpGf8tE5yF0VWkdux9JHrLJ6ciO+njB8rrrPMq8v9GOcKfln3lDRX0rcalONWkt6RdGjG9PPlvnpkFb+OZynxYx1l16HcgTTZxi6U28BXyyi/hqT3JHULbON7yd01vIXP4X4t+QM+18p9XUvbVwtNU+KHQiX9l6RLSuS3sl9n3fxyHu7X36Yl8hsvd0fV8nJ34Xwo/2voNeJ3kxsZ3VbuK0t+JfeIlfzrO+Qu4mV9PqXy8zGj5C5ir+4/08nK/mHXWu2zbptKxdfbxzQkv8S2O1eJHw9q7/zk9hvJbeHrcp2VXn6+jdqHvibpdN8+V5Z0q6Tr6pSfKqlf4vXxct+f10vuAP2Csn9UafVUjlbuO2q7q8C+IGL9bSO3DS4nN+AwQxk/7tsObXQ3uW14l5Dtpcg6VJ19ntyx6UO5R967yT39Ue/HoE+RdHnideF45WxzcgNP78s9OtoidxfaHElfTpSZLqlP0fz89Fb/3ltyd8B0kzJ/EPwgSRNS6/d1uf5Mq9xdKK9L6poRn9nH8Pm/Iuksv8w7yd1Fkvxh0AmSDimx/nrJfVfpqbWWp8n57SH3VVctcvuf30j6h5Y8PpXNL3ee7ZxfvWVZVUvuc96UdHDys6jVPttjvu21DSbaYTe5u7HO9f9vSXx+Nfutodug3PfXts1zVUk3KOPHU5uQX90+a2B+udt1E9to7mdVNr9EmcvlfjOp0fuYzD6s3DFiQ7n+zxZyfbpjI9ffALmvitjWv95O7pg8KGP5tpc0PfXeY3J9kG6SDpTre2Wdl2T2D4osi18PI0rk102u3Ru5rwx9UNIvOnD9lT7G+/frHXua1o/JW9+qc97h3+vl12lr0fz89Gb2Y+qtw119OzJyF4AfkDQ2tZxzJK1VYv01ux9TL7/M846I/Jrdj2l2fs3uw9Tb/g5PtM/15Pa1t0S2z2b3Yerll3fOGJJfs/sw9dpnaB8t9/quvgB9GD+9m59mfbvqlpdfkb9ShQvPVDrbL2jy7+w65cdIOi1j2nC5r9Joe72u3CMW6/rXF/qVP1fuYtMoJS5A+8b/pP9AZ8uNji2XmP4XSd8skdt6Pp9P/XK0/R1eND+fU/rzuTox/WNJ/f3/fyB3QWmuX/7xktbz0zrJPd40x8dMl7uIlfzF64uVGCgomONqcjvBD+S+A/w5Sd/LiYnJcaykxanP84VE2Va5R3//7df1Kam6S63DjPb6h5wyNyqxg6nXxtNt1L93il/2f/t8WxPTVvTr9SO5A+HI1Dp8TtJXSq6/J/38PpDbye1RMr8echfe5sr9avrQxLT+cl/bkoz/vtzdVu/LXbRbx78/wH82n6TWb//Q/HxMF7mvwPnAbxe/UUbnKKN9Zrap9PpTnX1Mo/LzcZfJPVZZpGxwfqn59Pb5JC+Itus+1MdsK3di+b7cCeyNklavU/5ESb9PvDZyj8rP8X+jU9vMEusgNS8raaOMaWcrtS+IWH9jfH4fyz3WXLPOdmqjD0j6LNUG7y66vRRZh8o5rsvdjThN7nHSByX1rjOvL8udmHYvEi/XWbu0xDbX36+3j+V+vyI57euSng3Ib2aN6TVz9J/3G0p0quVO6J72+T0jabvEtP9qW18q0MeQ+32OR+X2SS9KOjAxraf/bLsWzU/uKyJsqr6Pa+XW6PzkTgSn+ff+Jffd1ttE5ld3nu2ZX5G2lKprpqTd89pn7Hzbcxv0r6+usSzD/bTMfmvoNij3Hdtt85wld0NFzRuGmpDfWNXvs4buYzK362a30ZzPKjS/bnLHoIFZebVjfvX6sJvIPR33idyJefqcIjS/k+QujHwk11f8cU6OT0raIfG6t/+s5/nlS+4XDle586LMZdH/Xhheo8T2t7LcDRpt2/R58gNlzV5/fnrIMb7IsbUp/ZgC6zvv2sapkn4ZkN/MGuu4Zo6NXIdy58Rvy22Db8r9xtmXEtMPVuJCbJH81MR+TJE2mqrLKnEOEJhfs/sxTc2vxjxnqrF9mN7K3v5+7uc91/97uaRVI9dfs/sw9fJ7QPXPGYPWn5rbh6m3fxmrkn20gvl9IfowiW1uib96+RX9M34GHcoYs5rcHYzbWWvn5ZVvx3q3lhuN27HB9XRUfqvLXeDfzrrvfm9kXcv6OtxC7itd+tgmbjTGmP0lfddae0iD61nW8+uo9rms59es7a9V7ke2BlprZzWyrlS9TVl/vq5leh3WqPcXkv5prf11k+u9We7Hpu9qcD3HStrCWlv0h9Taq96LJL1qrb2kwfWQX2PqbUr79HUt69vgsp7fMt1GK5DfIEknWGsPaGQ9Ner9gdzNOyMaXM8yvf58XU3fx/j+8N/k7vb9Z4Pr6qh1+Ljck/NZX1XVXvWQX2PqbVZ+HXWMJ7/2qWeZPkb8X8zvCzEwAAAAAAAAAAAAmuOL8OPDAAAAAAAAAACgSRgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQjo3YqbXXH1p1C8aG2OCY2N/THnY8ONzKx//X9+Iyy8idnFMxZIO+8UDudXf9McrovLr0rlLcOzChQtjqtaQod/Lze+GP/5PVH4vPP9ieOwLL8VUrZtvuzs3v0v3/WVUfnZReLhpiWnd0vF/OSV3BnffeUNUfo8+9kRw7OMRsZJ078TJuflddeWvo/KbPfud4NhevdaKqVrDhv+gUAP40cnHRuW4x6Ddg2MnTrgvpmr98jeX5eZ4+Le/FZXfkUcOC4695ppxMVVr3PW35ua3989vjMpvoW0Jju302byYqjXh7MNz8xt31X9H5Tdzxszg2OnTX46pWuNuuDM3vz+f2j8qvye79gmPbd0hpmrdM/KQ3PyO+953ovLbZUD/4NhJD02OqVqXXfGH3PwuG3B5VH6dPgvv+n/WbUFM1fr+ffl9bEk65o5vRuXYa8om4bGPbhxTtY598LjcHPuee29UfoPm3Rscu9unccfAXS/6W25+Q0bFnUdsP/+Z4NgnIvcxt40cnpvff555elR++879S3DsncvvF1O1zj/3vNz8hj62a1R+G92xfXDs+vd+JaZqHfnUsNz8TvjjIVH5rTN5s+DYGXtMjalalx94W6F96MgzfhyV44x9ng6OXf/u8PUvSaPO/WVujoN/dmVUfqd/MDo4dvTKP4mpWjf+NP9axZBRcdcqNp8/LTh264XPx1StQ0bfk5vf8TcdFJXfTiMPDY599MybY6rWJUP/VGAfc3BUfqtMDz8f7/l4XB/mpLtOys3vyFFjovI7YM7E4NjbeoRfA5CksSNPzc3vgiMuiMrvjQMfCo5d59YBMVXr9GtPC74YxxMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAB7Q0EFAAAPa0lEQVQAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUCAMDAAAAAAAAAABUSOdGzNQY06HxjdbSKW48pVNEeottVNWFPP/8i3EzsE1YyAxDCpRpbW1t+HJkmT9/fsPrMIsjZ7AofP01Y9udMuWxqPiY1vn1Hb4WVXcRDzzwUFT8Cy+8FBy75ZabR9U9bPgPCpV77dUZUfXM+ses4NhXX30tqu4iNt5446j4SZMeDo5dr3fvqLqL2PajR6Liu7SEH0NjYp3Dc0tcdfX1UTX0779TcOx3jzoqqu4iutm445CJ2IsuMF2i6i6if/+do+Ivu+Sy4Nhjjz82qu4i/tFvelS87RTeSbAtzenfLVwuro3G5NiyoCGnRksYNO/eqPgWLQqOvb/7wKi6dy1QZo9PJkbVsee8vwbHrrpoTlTd0vDcEpO7h+/jJWnVxf8Kjn24e7+ouovY6M9xfd1OC1uCY2cMei6q7iLWfmiLqPjNxoevA2sj+zAHFiu2/l3bRVVjO4Xv69f/y1ej6ta5+UW2WvB8VBV3LL9vcOymC6dF1V3EiPcvjIq/c/l9gmPPW/nUqLoPKVBmzSc2iqrj5QMfD47t8XLPqLqL6HtOkStS2V458Ing2Cnn/DGq7pN0Um6Zb865L6qOEeuNCI49//UxUXVL+e37zW9Njqph+xEnBsc+e/7vo+qWTguO5IkBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqhIEBAAAAAAAAAAAqpHMjZrp48eKoeGNMcKy1NqruQjq1NL6OLOEfTWEvPP9CVPyCBQuDY7t06RJVdxHz58+Pm0FE+2xtbY2ruwDTEjneF5Gf6dT4BvrEE09Fxe/Qt09wbN8dw2OLGrj7blHxm2++WXDsGmuuEVV3URtutEFUfM+eawbHbrBhXN1FrNe7d1T8NVePC4498qhhUXUX0d/+v6j4lk/nBccusI0//n73qKOi4l+e/nJw7Nirr4+qe9CeB+WWWWC6RtURcYhQq/00qu4iJk96OCr+uO8fFxw76aHJUXV/54jjc8u83e/vUXUs6r4gOLbTwub0f1sWxJ2e2Jbwc4GFy4V/PkUN/PS+qPj7ug0Mjp3QfVBU3aMKlLl7ub2j6ni3ZdXg2Gdbt4uqu8jWv/O8KVF19P30ieDY9zqtFlW3tH9uifUnbB1Vw4w9pwbHvrr/01F1FzFzz7g+jOkUfp3jrZ2nRdVd1Ix9n4mL3+vZ4NiIj8c7PLfEZgvjPsfRK40Ijj39wwui6i7i/JVPjYrfesHzwbGnvX9hVN1S/vGpx8s9o2qYctbNwbE7n3VoVN1FTDlnfFR8z8c3Do7d4ZyDo+rWbflF7ugR3geRpAteHx0ce3uP3aPqPrBAmXVu7x9VxzOjLwmOXffWXaLq1uDwUJ4YAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQhgYAAAAAAAAAACgQoy1tqOXAQAAAAAAAAAANAlPDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCEMDAAAAAAAAAAAUCGdGzHTww/d38bEWxseboyJqVrX3XBH7gyGnPWHuPwiYuOyk2465zu5szjggH2i8lv42cLg2C5dusRUrdtuvSs3v8EH7B2V35ZbbREeu+XmMVXr0MOOyc3v6JsOjcrPtIS3MrsoqmpdOeSG3Mr3GLhzVCV9+/aJiP16TNXad//DcvM7YujBUfmtueYawbFvv/2PmKp13Q23FGo8Yy4YGZXjhAn3BcfuMWhgTNUacdqo3ByvHntxVH5Xj702OHbYsO/GVK0jjz4pN787TukXlV8Xsyg4dlHn7jFVa98LHszN79uHDYnKb/31ewfHbrLJRjFV68jhx+fmt8+o8VH5fX3B4+Gx85+IqVr7jXkkN7+Lf3NBVH6THpocHLvLgP4xVevEk0/LP8bfE9dHW9Q5fPvr8mnXmKp1xX75fWxJunT330fl+Ha/6eGxO74cU7Wu2vvO3Bwn/3iruGNgtz2DY+/vHncMfOTMfXLzG39qXBt9pnX74Ng+88P3T5I0ZMxfc/M754yTovK7c/n9gmP3n3tnTNU68+e/y83vmq9cF5XfK/s/HRw7c8+pMVVrXP+Jufn95oBfReX3Zv9pwbHr/3WbmKp1wj0nFtqHjjzzlLh+zF1fDY6dsfczMVVr1M8vys3xhhF7ReV3/sojgmNHfHBhTNU6bHT+tYrxp8Zdq3ipdbPg2Oe6bBVTtW756dG5+f1u399G5ffIqBuCY/ueOzimav3w1h/l5vffB/w6Kr/3Nwk/H5+9Q1wf5tLBt+bmd/mpQ6Pyu73H7sGx35ozMaZqHTvm+tz8fnzZd6LyW+fWXYJj3zxwUkzVuui4PwRfyOOJAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKoSBAQAAAAAAAAAAKqRzI2ZqrY2KX2wXB8d2asJYx6LF4csnSYsjPp5OJqrqQrbcaouoeGOasJAR5s+fHzmH8BXY2to1su4CYjeBlojYuE2/kH79+kbFx7TOJx5/Kqruffc/LLfMN74xIKqOLbbcPDj2xRdeiqq7qA023CAqfq21XgyO3TCy7iJeefmVqPhddtk5OHbmzJlRdRfx7Jd2ior/bFH4MXThZ3HH330LlDlq+NCoOh6e9HBw7LVXjY2q+8jhx+eW+dS0RtVhI/aiXe3CqLqLmDw5/POXpONOODY49vJLr4iq+8STT8st03PKplF1dFoc0UlYFNm/269Ysc6fxPWVzOLw5VzUtfFtdEL3PaPiF0d01Habd19U3dI+uSUmLLd7VA0Tuu8RHPteS4+ouocUKNN/3pSoOt7rtFpw7M6RdRfxyjfj+rqLuywKju09YeuoutU/v8hbu4T3ISVp2mHh68CYuD5MUTP2eSYqPmYfOmPfp6PqLuL5rltFxe8/987g2GldNouqu4jRq/wkKn6/uX8Jjj39gzFRdUtH55aY3SfuPGnjW3YIjn1/41lRdRfx2Fk3RcVvfEuf4Nh+Z+Vfa6hrcH6RO1YZGFXF6NcvCI49bb0RUXUX6eGvc3uBA0kdz46+ODh2u9NPiKpbx4WH8sQAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAVwsAAAAAAAAAAAAAV0rkRMzWd4sYbOtmIuk0TxjoWL2p8HVkiPpuittpqi6j4rl27BscuXLgwqu4iuraGL5+kqHUwf/6CuLoLsIsWx8VbEx68uPENtE+fr0XFP/bYE8Gxjz8aHlvUxIn3R8W/9NK04NjZs9+JqvvI751UqNyrr7wWVc+sWbODY197Na7uIl6fOTMqftjw7wbHjr3qmqi6i3jYbBsVv6hb9+DYLqbxx99xV10VFb/JJhsFxx41fGhU3UV0tXHHoZhjxKemW1TdRfTfZaeo+Et/f1lw7C4D+kfVXcTaUzaNim+ZF94HWtylOf3fxV3j6jGLws8FunwS2Ucs4P5uA6Pid/v0vuDYPedNiKq7iH0+uTsq/suL3guO3X7+M1F1S6fklni4e7+oGh7rtkNw7KqL342qe9cCZWYOei6qjvXv3SY4dsM7vhpVt87JL9J7Qlwfxtrw/UuvhzeLqrtA85Qkrf+XuM+x9z3bBccujrmQI0kFNq9pXeI+xxEfjg6OPX+l06LqLuL0D8KXT5Ke67p1cOwFq/wkqu6bC5SZs/GsqDp2PGdwcOwj59wQVXcRO511aFT8rB1eCY59/Kwbo+o+Qofkltn//fA+iCSNWC98G/rWnIlRdUtH55Z481uTo2rYfsSJwbFvHDgpqm7puOBInhgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCGBgAAAAAAAAAAKBCjLW2o5cBAAAAAAAAAAA0CU8MAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIQwMAAAAAAAAAABQIf8frpnNXocsUqEAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j6qlex4zYaGm",
        "colab_type": "text"
      },
      "source": [
        "Now let's replace the first 3×3 convolution of the network with a fixed 3×3 whitening convolution to equalise the scales of the eigenpatches above, followed by a learnable 1×1 convolution. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ckzKy37AY1gq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "b63310fe-2ae7-4fce-883e-c99f2ac3204c"
      },
      "source": [
        "def whitening_block(c_in, c_out, Λ=None, V=None, eps=1e-2):\n",
        "    filt = nn.Conv2d(3, 27, kernel_size=(3,3), padding=(1,1), bias=False)\n",
        "    filt.weight.data = (V/torch.sqrt(Λ+eps)[:,None,None,None])\n",
        "    filt.weight.requires_grad = False \n",
        "                                   \n",
        "    return {\n",
        "        'whiten': (identity, {'value': filt}),\n",
        "        'conv': conv(27, c_out, kernel_size=(1, 1), bias=False),\n",
        "        'norm': batch_norm(c_out), \n",
        "        'act':  relu(),\n",
        "    }\n",
        "\n",
        "input_whitening_net = network(conv_pool_block=conv_pool_block_pre, prep_block=partial(whitening_block, Λ=Λ, V=V), scale=1/16, types={\n",
        "    nn.ReLU: partial(nn.CELU, 0.3),\n",
        "    BatchNorm: partial(GhostBatchNorm, num_splits=16, weight=False)\n",
        "})\n",
        "show(input_whitening_net)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<__main__.DotGraph at 0x7f4140167cf8>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n -->\n<!-- Title: G Pages: 1 -->\n<svg width=\"1080pt\" height=\"58pt\"\n viewBox=\"0.00 0.00 1080.00 57.75\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(0.302352 0.302352) rotate(0) translate(4 187)\">\n<title>G</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-187 3568,-187 3568,4 -4,4\"/>\n<g id=\"clust1\" class=\"cluster\"><title>cluster_prep</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M94,-48C94,-48 410,-48 410,-48 416,-48 422,-54 422,-60 422,-60 422,-111 422,-111 422,-117 416,-123 410,-123 410,-123 94,-123 94,-123 88,-123 82,-117 82,-111 82,-111 82,-60 82,-60 82,-54 88,-48 94,-48\"/>\n<text text-anchor=\"middle\" x=\"252\" y=\"-107.8\" font-family=\"Times,serif\" font-size=\"14.00\">prep</text>\n</g>\n<g id=\"clust2\" class=\"cluster\"><title>cluster_layer1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M454,-12C454,-12 1588,-12 1588,-12 1594,-12 1600,-18 1600,-24 1600,-24 1600,-163 1600,-163 1600,-169 1594,-175 1588,-175 1588,-175 454,-175 454,-175 448,-175 442,-169 442,-163 442,-163 442,-24 442,-24 442,-18 448,-12 454,-12\"/>\n<text text-anchor=\"middle\" x=\"1021\" y=\"-159.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer1</text>\n</g>\n<g id=\"clust3\" class=\"cluster\"><title>cluster_layer1_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M814,-20C814,-20 1580,-20 1580,-20 1586,-20 1592,-26 1592,-32 1592,-32 1592,-132 1592,-132 1592,-138 1586,-144 1580,-144 1580,-144 814,-144 814,-144 808,-144 802,-138 802,-132 802,-132 802,-32 802,-32 802,-26 808,-20 814,-20\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-128.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust4\" class=\"cluster\"><title>cluster_layer1_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M904,-28C904,-28 1130,-28 1130,-28 1136,-28 1142,-34 1142,-40 1142,-40 1142,-91 1142,-91 1142,-97 1136,-103 1130,-103 1130,-103 904,-103 904,-103 898,-103 892,-97 892,-91 892,-91 892,-40 892,-40 892,-34 898,-28 904,-28\"/>\n<text text-anchor=\"middle\" x=\"1017\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust5\" class=\"cluster\"><title>cluster_layer1_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1174,-28C1174,-28 1400,-28 1400,-28 1406,-28 1412,-34 1412,-40 1412,-40 1412,-91 1412,-91 1412,-97 1406,-103 1400,-103 1400,-103 1174,-103 1174,-103 1168,-103 1162,-97 1162,-91 1162,-91 1162,-40 1162,-40 1162,-34 1168,-28 1174,-28\"/>\n<text text-anchor=\"middle\" x=\"1287\" y=\"-87.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust6\" class=\"cluster\"><title>cluster_layer2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1624,-53C1624,-53 1940,-53 1940,-53 1946,-53 1952,-59 1952,-65 1952,-65 1952,-116 1952,-116 1952,-122 1946,-128 1940,-128 1940,-128 1624,-128 1624,-128 1618,-128 1612,-122 1612,-116 1612,-116 1612,-65 1612,-65 1612,-59 1618,-53 1624,-53\"/>\n<text text-anchor=\"middle\" x=\"1782\" y=\"-112.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer2</text>\n</g>\n<g id=\"clust7\" class=\"cluster\"><title>cluster_layer3</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M1984,-8C1984,-8 3118,-8 3118,-8 3124,-8 3130,-14 3130,-20 3130,-20 3130,-159 3130,-159 3130,-165 3124,-171 3118,-171 3118,-171 1984,-171 1984,-171 1978,-171 1972,-165 1972,-159 1972,-159 1972,-20 1972,-20 1972,-14 1978,-8 1984,-8\"/>\n<text text-anchor=\"middle\" x=\"2551\" y=\"-155.8\" font-family=\"Times,serif\" font-size=\"14.00\">layer3</text>\n</g>\n<g id=\"clust8\" class=\"cluster\"><title>cluster_layer3_residual</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2344,-16C2344,-16 3110,-16 3110,-16 3116,-16 3122,-22 3122,-28 3122,-28 3122,-128 3122,-128 3122,-134 3116,-140 3110,-140 3110,-140 2344,-140 2344,-140 2338,-140 2332,-134 2332,-128 2332,-128 2332,-28 2332,-28 2332,-22 2338,-16 2344,-16\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">residual</text>\n</g>\n<g id=\"clust9\" class=\"cluster\"><title>cluster_layer3_residual_res1</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2434,-24C2434,-24 2660,-24 2660,-24 2666,-24 2672,-30 2672,-36 2672,-36 2672,-87 2672,-87 2672,-93 2666,-99 2660,-99 2660,-99 2434,-99 2434,-99 2428,-99 2422,-93 2422,-87 2422,-87 2422,-36 2422,-36 2422,-30 2428,-24 2434,-24\"/>\n<text text-anchor=\"middle\" x=\"2547\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res1</text>\n</g>\n<g id=\"clust10\" class=\"cluster\"><title>cluster_layer3_residual_res2</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M2704,-24C2704,-24 2930,-24 2930,-24 2936,-24 2942,-30 2942,-36 2942,-36 2942,-87 2942,-87 2942,-93 2936,-99 2930,-99 2930,-99 2704,-99 2704,-99 2698,-99 2692,-93 2692,-87 2692,-87 2692,-36 2692,-36 2692,-30 2698,-24 2704,-24\"/>\n<text text-anchor=\"middle\" x=\"2817\" y=\"-83.8\" font-family=\"Times,serif\" font-size=\"14.00\">res2</text>\n</g>\n<g id=\"clust11\" class=\"cluster\"><title>cluster_classifier</title>\n<path fill=\"#777777\" fill-opacity=\"0.266667\" stroke=\"black\" d=\"M3244,-49C3244,-49 3470,-49 3470,-49 3476,-49 3482,-55 3482,-61 3482,-61 3482,-112 3482,-112 3482,-118 3476,-124 3470,-124 3470,-124 3244,-124 3244,-124 3238,-124 3232,-118 3232,-112 3232,-112 3232,-61 3232,-61 3232,-55 3238,-49 3244,-49\"/>\n<text text-anchor=\"middle\" x=\"3357\" y=\"-108.8\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n</g>\n<!-- prep_whiten -->\n<g id=\"node1\" class=\"node\"><title>prep_whiten</title>\n<g id=\"a_node1\"><a xlink:title=\"&lt;function identity at 0x7f414a26c6a8&gt; {&#39;value&#39;: Conv2d(3, 27, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)}\">\n<path fill=\"#dea05e\" stroke=\"black\" d=\"M132,-92C132,-92 102,-92 102,-92 96,-92 90,-86 90,-80 90,-80 90,-68 90,-68 90,-62 96,-56 102,-56 102,-56 132,-56 132,-56 138,-56 144,-62 144,-68 144,-68 144,-80 144,-80 144,-86 138,-92 132,-92\"/>\n<text text-anchor=\"middle\" x=\"117\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">whiten</text>\n</a>\n</g>\n</g>\n<!-- prep_conv -->\n<g id=\"node2\" class=\"node\"><title>prep_conv</title>\n<g id=\"a_node2\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 27, &#39;out_channels&#39;: 64, &#39;kernel_size&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M222,-92C222,-92 192,-92 192,-92 186,-92 180,-86 180,-80 180,-80 180,-68 180,-68 180,-62 186,-56 192,-56 192,-56 222,-56 222,-56 228,-56 234,-62 234,-68 234,-68 234,-80 234,-80 234,-86 228,-92 222,-92\"/>\n<text text-anchor=\"middle\" x=\"207\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_whiten&#45;&gt;prep_conv -->\n<g id=\"edge2\" class=\"edge\"><title>prep_whiten&#45;&gt;prep_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M144.403,-74C152.393,-74 161.311,-74 169.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"169.919,-77.5001 179.919,-74 169.919,-70.5001 169.919,-77.5001\"/>\n</g>\n<!-- prep_norm -->\n<g id=\"node3\" class=\"node\"><title>prep_norm</title>\n<g id=\"a_node3\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 64}\">\n<path fill=\"#d0c281\" stroke=\"black\" d=\"M312,-92C312,-92 282,-92 282,-92 276,-92 270,-86 270,-80 270,-80 270,-68 270,-68 270,-62 276,-56 282,-56 282,-56 312,-56 312,-56 318,-56 324,-62 324,-68 324,-68 324,-80 324,-80 324,-86 318,-92 312,-92\"/>\n<text text-anchor=\"middle\" x=\"297\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- prep_conv&#45;&gt;prep_norm -->\n<g id=\"edge3\" class=\"edge\"><title>prep_conv&#45;&gt;prep_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M234.403,-74C242.393,-74 251.311,-74 259.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"259.919,-77.5001 269.919,-74 259.919,-70.5001 259.919,-77.5001\"/>\n</g>\n<!-- prep_act -->\n<g id=\"node4\" class=\"node\"><title>prep_act</title>\n<g id=\"a_node4\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#f0e189\" stroke=\"black\" d=\"M402,-92C402,-92 372,-92 372,-92 366,-92 360,-86 360,-80 360,-80 360,-68 360,-68 360,-62 366,-56 372,-56 372,-56 402,-56 402,-56 408,-56 414,-62 414,-68 414,-68 414,-80 414,-80 414,-86 408,-92 402,-92\"/>\n<text text-anchor=\"middle\" x=\"387\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- prep_norm&#45;&gt;prep_act -->\n<g id=\"edge4\" class=\"edge\"><title>prep_norm&#45;&gt;prep_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M324.403,-74C332.393,-74 341.311,-74 349.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"349.919,-77.5001 359.919,-74 349.919,-70.5001 349.919,-77.5001\"/>\n</g>\n<!-- layer1_conv -->\n<g id=\"node5\" class=\"node\"><title>layer1_conv</title>\n<g id=\"a_node5\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 64, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M492,-92C492,-92 462,-92 462,-92 456,-92 450,-86 450,-80 450,-80 450,-68 450,-68 450,-62 456,-56 462,-56 462,-56 492,-56 492,-56 498,-56 504,-62 504,-68 504,-68 504,-80 504,-80 504,-86 498,-92 492,-92\"/>\n<text text-anchor=\"middle\" x=\"477\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- prep_act&#45;&gt;layer1_conv -->\n<g id=\"edge5\" class=\"edge\"><title>prep_act&#45;&gt;layer1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M414.403,-74C422.393,-74 431.311,-74 439.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"439.919,-77.5001 449.919,-74 439.919,-70.5001 439.919,-77.5001\"/>\n</g>\n<!-- layer1_pool -->\n<g id=\"node6\" class=\"node\"><title>layer1_pool</title>\n<g id=\"a_node6\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M582,-92C582,-92 552,-92 552,-92 546,-92 540,-86 540,-80 540,-80 540,-68 540,-68 540,-62 546,-56 552,-56 552,-56 582,-56 582,-56 588,-56 594,-62 594,-68 594,-68 594,-80 594,-80 594,-86 588,-92 582,-92\"/>\n<text text-anchor=\"middle\" x=\"567\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer1_conv&#45;&gt;layer1_pool -->\n<g id=\"edge6\" class=\"edge\"><title>layer1_conv&#45;&gt;layer1_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M504.403,-74C512.393,-74 521.311,-74 529.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"529.919,-77.5001 539.919,-74 529.919,-70.5001 529.919,-77.5001\"/>\n</g>\n<!-- layer1_norm -->\n<g id=\"node7\" class=\"node\"><title>layer1_norm</title>\n<g id=\"a_node7\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 128}\">\n<path fill=\"#d0c281\" stroke=\"black\" d=\"M672,-92C672,-92 642,-92 642,-92 636,-92 630,-86 630,-80 630,-80 630,-68 630,-68 630,-62 636,-56 642,-56 642,-56 672,-56 672,-56 678,-56 684,-62 684,-68 684,-68 684,-80 684,-80 684,-86 678,-92 672,-92\"/>\n<text text-anchor=\"middle\" x=\"657\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_pool&#45;&gt;layer1_norm -->\n<g id=\"edge7\" class=\"edge\"><title>layer1_pool&#45;&gt;layer1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M594.403,-74C602.393,-74 611.311,-74 619.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"619.919,-77.5001 629.919,-74 619.919,-70.5001 619.919,-77.5001\"/>\n</g>\n<!-- layer1_act -->\n<g id=\"node8\" class=\"node\"><title>layer1_act</title>\n<g id=\"a_node8\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#f0e189\" stroke=\"black\" d=\"M762,-92C762,-92 732,-92 732,-92 726,-92 720,-86 720,-80 720,-80 720,-68 720,-68 720,-62 726,-56 732,-56 732,-56 762,-56 762,-56 768,-56 774,-62 774,-68 774,-68 774,-80 774,-80 774,-86 768,-92 762,-92\"/>\n<text text-anchor=\"middle\" x=\"747\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_norm&#45;&gt;layer1_act -->\n<g id=\"edge8\" class=\"edge\"><title>layer1_norm&#45;&gt;layer1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M684.403,-74C692.393,-74 701.311,-74 709.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"709.919,-77.5001 719.919,-74 709.919,-70.5001 709.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_in -->\n<g id=\"node9\" class=\"node\"><title>layer1_residual_in</title>\n<g id=\"a_node9\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M852,-92C852,-92 822,-92 822,-92 816,-92 810,-86 810,-80 810,-80 810,-68 810,-68 810,-62 816,-56 822,-56 822,-56 852,-56 852,-56 858,-56 864,-62 864,-68 864,-68 864,-80 864,-80 864,-86 858,-92 852,-92\"/>\n<text text-anchor=\"middle\" x=\"837\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer1_act&#45;&gt;layer1_residual_in -->\n<g id=\"edge9\" class=\"edge\"><title>layer1_act&#45;&gt;layer1_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M774.403,-74C782.393,-74 791.311,-74 799.824,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"799.919,-77.5001 809.919,-74 799.919,-70.5001 799.919,-77.5001\"/>\n</g>\n<!-- layer1_residual_res1_conv -->\n<g id=\"node10\" class=\"node\"><title>layer1_residual_res1_conv</title>\n<g id=\"a_node10\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M942,-72C942,-72 912,-72 912,-72 906,-72 900,-66 900,-60 900,-60 900,-48 900,-48 900,-42 906,-36 912,-36 912,-36 942,-36 942,-36 948,-36 954,-42 954,-48 954,-48 954,-60 954,-60 954,-66 948,-72 942,-72\"/>\n<text text-anchor=\"middle\" x=\"927\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_res1_conv -->\n<g id=\"edge10\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M864.403,-67.9993C872.481,-66.1634 881.507,-64.1121 890.105,-62.158\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"890.943,-65.5568 899.919,-59.9275 889.392,-58.7308 890.943,-65.5568\"/>\n</g>\n<!-- layer1_residual_add -->\n<g id=\"node17\" class=\"node\"><title>layer1_residual_add</title>\n<g id=\"a_node17\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M1572,-97C1572,-97 1542,-97 1542,-97 1536,-97 1530,-91 1530,-85 1530,-85 1530,-73 1530,-73 1530,-67 1536,-61 1542,-61 1542,-61 1572,-61 1572,-61 1578,-61 1584,-67 1584,-73 1584,-73 1584,-85 1584,-85 1584,-91 1578,-97 1572,-97\"/>\n<text text-anchor=\"middle\" x=\"1557\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_in&#45;&gt;layer1_residual_add -->\n<g id=\"edge17\" class=\"edge\"><title>layer1_residual_in&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M861.737,-92.1018C870.735,-97.9639 881.381,-103.78 892,-107 944.782,-123.006 960.844,-112 1016,-112 1016,-112 1016,-112 1378,-112 1427.77,-112 1484.08,-99.3711 1519.8,-89.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1521.04,-93.0265 1529.74,-86.9884 1519.17,-86.2796 1521.04,-93.0265\"/>\n</g>\n<!-- layer1_residual_res1_norm -->\n<g id=\"node11\" class=\"node\"><title>layer1_residual_res1_norm</title>\n<g id=\"a_node11\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 128}\">\n<path fill=\"#d0c281\" stroke=\"black\" d=\"M1032,-72C1032,-72 1002,-72 1002,-72 996,-72 990,-66 990,-60 990,-60 990,-48 990,-48 990,-42 996,-36 1002,-36 1002,-36 1032,-36 1032,-36 1038,-36 1044,-42 1044,-48 1044,-48 1044,-60 1044,-60 1044,-66 1038,-72 1032,-72\"/>\n<text text-anchor=\"middle\" x=\"1017\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm -->\n<g id=\"edge11\" class=\"edge\"><title>layer1_residual_res1_conv&#45;&gt;layer1_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M954.403,-54C962.393,-54 971.311,-54 979.824,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"979.919,-57.5001 989.919,-54 979.919,-50.5001 979.919,-57.5001\"/>\n</g>\n<!-- layer1_residual_res1_act -->\n<g id=\"node12\" class=\"node\"><title>layer1_residual_res1_act</title>\n<g id=\"a_node12\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#f0e189\" stroke=\"black\" d=\"M1122,-72C1122,-72 1092,-72 1092,-72 1086,-72 1080,-66 1080,-60 1080,-60 1080,-48 1080,-48 1080,-42 1086,-36 1092,-36 1092,-36 1122,-36 1122,-36 1128,-36 1134,-42 1134,-48 1134,-48 1134,-60 1134,-60 1134,-66 1128,-72 1122,-72\"/>\n<text text-anchor=\"middle\" x=\"1107\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act -->\n<g id=\"edge12\" class=\"edge\"><title>layer1_residual_res1_norm&#45;&gt;layer1_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1044.4,-54C1052.39,-54 1061.31,-54 1069.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1069.92,-57.5001 1079.92,-54 1069.92,-50.5001 1069.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_conv -->\n<g id=\"node13\" class=\"node\"><title>layer1_residual_res2_conv</title>\n<g id=\"a_node13\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 128, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1212,-72C1212,-72 1182,-72 1182,-72 1176,-72 1170,-66 1170,-60 1170,-60 1170,-48 1170,-48 1170,-42 1176,-36 1182,-36 1182,-36 1212,-36 1212,-36 1218,-36 1224,-42 1224,-48 1224,-48 1224,-60 1224,-60 1224,-66 1218,-72 1212,-72\"/>\n<text text-anchor=\"middle\" x=\"1197\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv -->\n<g id=\"edge13\" class=\"edge\"><title>layer1_residual_res1_act&#45;&gt;layer1_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1134.4,-54C1142.39,-54 1151.31,-54 1159.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1159.92,-57.5001 1169.92,-54 1159.92,-50.5001 1159.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_norm -->\n<g id=\"node14\" class=\"node\"><title>layer1_residual_res2_norm</title>\n<g id=\"a_node14\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 128}\">\n<path fill=\"#d0c281\" stroke=\"black\" d=\"M1302,-72C1302,-72 1272,-72 1272,-72 1266,-72 1260,-66 1260,-60 1260,-60 1260,-48 1260,-48 1260,-42 1266,-36 1272,-36 1272,-36 1302,-36 1302,-36 1308,-36 1314,-42 1314,-48 1314,-48 1314,-60 1314,-60 1314,-66 1308,-72 1302,-72\"/>\n<text text-anchor=\"middle\" x=\"1287\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm -->\n<g id=\"edge14\" class=\"edge\"><title>layer1_residual_res2_conv&#45;&gt;layer1_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1224.4,-54C1232.39,-54 1241.31,-54 1249.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1249.92,-57.5001 1259.92,-54 1249.92,-50.5001 1249.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_res2_act -->\n<g id=\"node15\" class=\"node\"><title>layer1_residual_res2_act</title>\n<g id=\"a_node15\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#f0e189\" stroke=\"black\" d=\"M1392,-72C1392,-72 1362,-72 1362,-72 1356,-72 1350,-66 1350,-60 1350,-60 1350,-48 1350,-48 1350,-42 1356,-36 1362,-36 1362,-36 1392,-36 1392,-36 1398,-36 1404,-42 1404,-48 1404,-48 1404,-60 1404,-60 1404,-66 1398,-72 1392,-72\"/>\n<text text-anchor=\"middle\" x=\"1377\" y=\"-50.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act -->\n<g id=\"edge15\" class=\"edge\"><title>layer1_residual_res2_norm&#45;&gt;layer1_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1314.4,-54C1322.39,-54 1331.31,-54 1339.82,-54\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1339.92,-57.5001 1349.92,-54 1339.92,-50.5001 1339.92,-57.5001\"/>\n</g>\n<!-- layer1_residual_out -->\n<g id=\"node16\" class=\"node\"><title>layer1_residual_out</title>\n<g id=\"a_node16\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M1482,-82C1482,-82 1452,-82 1452,-82 1446,-82 1440,-76 1440,-70 1440,-70 1440,-58 1440,-58 1440,-52 1446,-46 1452,-46 1452,-46 1482,-46 1482,-46 1488,-46 1494,-52 1494,-58 1494,-58 1494,-70 1494,-70 1494,-76 1488,-82 1482,-82\"/>\n<text text-anchor=\"middle\" x=\"1467\" y=\"-60.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_res2_act&#45;&gt;layer1_residual_out -->\n<g id=\"edge16\" class=\"edge\"><title>layer1_residual_res2_act&#45;&gt;layer1_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1404.4,-57.0003C1412.39,-57.9083 1421.31,-58.9217 1429.82,-59.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1429.59,-63.3847 1439.92,-61.0362 1430.38,-56.4294 1429.59,-63.3847\"/>\n</g>\n<!-- layer1_residual_out&#45;&gt;layer1_residual_add -->\n<g id=\"edge18\" class=\"edge\"><title>layer1_residual_out&#45;&gt;layer1_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1494.4,-68.5005C1502.39,-69.8625 1511.31,-71.3825 1519.82,-72.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1519.47,-76.3242 1529.92,-74.5544 1520.65,-69.4237 1519.47,-76.3242\"/>\n</g>\n<!-- layer2_conv -->\n<g id=\"node18\" class=\"node\"><title>layer2_conv</title>\n<g id=\"a_node18\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 128, &#39;out_channels&#39;: 256, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M1662,-97C1662,-97 1632,-97 1632,-97 1626,-97 1620,-91 1620,-85 1620,-85 1620,-73 1620,-73 1620,-67 1626,-61 1632,-61 1632,-61 1662,-61 1662,-61 1668,-61 1674,-67 1674,-73 1674,-73 1674,-85 1674,-85 1674,-91 1668,-97 1662,-97\"/>\n<text text-anchor=\"middle\" x=\"1647\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer1_residual_add&#45;&gt;layer2_conv -->\n<g id=\"edge19\" class=\"edge\"><title>layer1_residual_add&#45;&gt;layer2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1584.4,-79C1592.39,-79 1601.31,-79 1609.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1609.92,-82.5001 1619.92,-79 1609.92,-75.5001 1609.92,-82.5001\"/>\n</g>\n<!-- layer2_pool -->\n<g id=\"node19\" class=\"node\"><title>layer2_pool</title>\n<g id=\"a_node19\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M1752,-97C1752,-97 1722,-97 1722,-97 1716,-97 1710,-91 1710,-85 1710,-85 1710,-73 1710,-73 1710,-67 1716,-61 1722,-61 1722,-61 1752,-61 1752,-61 1758,-61 1764,-67 1764,-73 1764,-73 1764,-85 1764,-85 1764,-91 1758,-97 1752,-97\"/>\n<text text-anchor=\"middle\" x=\"1737\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer2_conv&#45;&gt;layer2_pool -->\n<g id=\"edge20\" class=\"edge\"><title>layer2_conv&#45;&gt;layer2_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1674.4,-79C1682.39,-79 1691.31,-79 1699.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1699.92,-82.5001 1709.92,-79 1699.92,-75.5001 1699.92,-82.5001\"/>\n</g>\n<!-- layer2_norm -->\n<g id=\"node20\" class=\"node\"><title>layer2_norm</title>\n<g id=\"a_node20\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 256}\">\n<path fill=\"#d0c281\" stroke=\"black\" d=\"M1842,-97C1842,-97 1812,-97 1812,-97 1806,-97 1800,-91 1800,-85 1800,-85 1800,-73 1800,-73 1800,-67 1806,-61 1812,-61 1812,-61 1842,-61 1842,-61 1848,-61 1854,-67 1854,-73 1854,-73 1854,-85 1854,-85 1854,-91 1848,-97 1842,-97\"/>\n<text text-anchor=\"middle\" x=\"1827\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer2_pool&#45;&gt;layer2_norm -->\n<g id=\"edge21\" class=\"edge\"><title>layer2_pool&#45;&gt;layer2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1764.4,-79C1772.39,-79 1781.31,-79 1789.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1789.92,-82.5001 1799.92,-79 1789.92,-75.5001 1789.92,-82.5001\"/>\n</g>\n<!-- layer2_act -->\n<g id=\"node21\" class=\"node\"><title>layer2_act</title>\n<g id=\"a_node21\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#f0e189\" stroke=\"black\" d=\"M1932,-97C1932,-97 1902,-97 1902,-97 1896,-97 1890,-91 1890,-85 1890,-85 1890,-73 1890,-73 1890,-67 1896,-61 1902,-61 1902,-61 1932,-61 1932,-61 1938,-61 1944,-67 1944,-73 1944,-73 1944,-85 1944,-85 1944,-91 1938,-97 1932,-97\"/>\n<text text-anchor=\"middle\" x=\"1917\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer2_norm&#45;&gt;layer2_act -->\n<g id=\"edge22\" class=\"edge\"><title>layer2_norm&#45;&gt;layer2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1854.4,-79C1862.39,-79 1871.31,-79 1879.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1879.92,-82.5001 1889.92,-79 1879.92,-75.5001 1879.92,-82.5001\"/>\n</g>\n<!-- layer3_conv -->\n<g id=\"node22\" class=\"node\"><title>layer3_conv</title>\n<g id=\"a_node22\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 256, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2022,-97C2022,-97 1992,-97 1992,-97 1986,-97 1980,-91 1980,-85 1980,-85 1980,-73 1980,-73 1980,-67 1986,-61 1992,-61 1992,-61 2022,-61 2022,-61 2028,-61 2034,-67 2034,-73 2034,-73 2034,-85 2034,-85 2034,-91 2028,-97 2022,-97\"/>\n<text text-anchor=\"middle\" x=\"2007\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer2_act&#45;&gt;layer3_conv -->\n<g id=\"edge23\" class=\"edge\"><title>layer2_act&#45;&gt;layer3_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M1944.4,-79C1952.39,-79 1961.31,-79 1969.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"1969.92,-82.5001 1979.92,-79 1969.92,-75.5001 1969.92,-82.5001\"/>\n</g>\n<!-- layer3_pool -->\n<g id=\"node23\" class=\"node\"><title>layer3_pool</title>\n<g id=\"a_node23\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 2}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M2112,-97C2112,-97 2082,-97 2082,-97 2076,-97 2070,-91 2070,-85 2070,-85 2070,-73 2070,-73 2070,-67 2076,-61 2082,-61 2082,-61 2112,-61 2112,-61 2118,-61 2124,-67 2124,-73 2124,-73 2124,-85 2124,-85 2124,-91 2118,-97 2112,-97\"/>\n<text text-anchor=\"middle\" x=\"2097\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_conv&#45;&gt;layer3_pool -->\n<g id=\"edge24\" class=\"edge\"><title>layer3_conv&#45;&gt;layer3_pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2034.4,-79C2042.39,-79 2051.31,-79 2059.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2059.92,-82.5001 2069.92,-79 2059.92,-75.5001 2059.92,-82.5001\"/>\n</g>\n<!-- layer3_norm -->\n<g id=\"node24\" class=\"node\"><title>layer3_norm</title>\n<g id=\"a_node24\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 512}\">\n<path fill=\"#d0c281\" stroke=\"black\" d=\"M2202,-97C2202,-97 2172,-97 2172,-97 2166,-97 2160,-91 2160,-85 2160,-85 2160,-73 2160,-73 2160,-67 2166,-61 2172,-61 2172,-61 2202,-61 2202,-61 2208,-61 2214,-67 2214,-73 2214,-73 2214,-85 2214,-85 2214,-91 2208,-97 2202,-97\"/>\n<text text-anchor=\"middle\" x=\"2187\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_pool&#45;&gt;layer3_norm -->\n<g id=\"edge25\" class=\"edge\"><title>layer3_pool&#45;&gt;layer3_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2124.4,-79C2132.39,-79 2141.31,-79 2149.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2149.92,-82.5001 2159.92,-79 2149.92,-75.5001 2149.92,-82.5001\"/>\n</g>\n<!-- layer3_act -->\n<g id=\"node25\" class=\"node\"><title>layer3_act</title>\n<g id=\"a_node25\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#f0e189\" stroke=\"black\" d=\"M2292,-97C2292,-97 2262,-97 2262,-97 2256,-97 2250,-91 2250,-85 2250,-85 2250,-73 2250,-73 2250,-67 2256,-61 2262,-61 2262,-61 2292,-61 2292,-61 2298,-61 2304,-67 2304,-73 2304,-73 2304,-85 2304,-85 2304,-91 2298,-97 2292,-97\"/>\n<text text-anchor=\"middle\" x=\"2277\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_norm&#45;&gt;layer3_act -->\n<g id=\"edge26\" class=\"edge\"><title>layer3_norm&#45;&gt;layer3_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2214.4,-79C2222.39,-79 2231.31,-79 2239.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2239.92,-82.5001 2249.92,-79 2239.92,-75.5001 2239.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_in -->\n<g id=\"node26\" class=\"node\"><title>layer3_residual_in</title>\n<g id=\"a_node26\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M2382,-97C2382,-97 2352,-97 2352,-97 2346,-97 2340,-91 2340,-85 2340,-85 2340,-73 2340,-73 2340,-67 2346,-61 2352,-61 2352,-61 2382,-61 2382,-61 2388,-61 2394,-67 2394,-73 2394,-73 2394,-85 2394,-85 2394,-91 2388,-97 2382,-97\"/>\n<text text-anchor=\"middle\" x=\"2367\" y=\"-75.3\" font-family=\"Times,serif\" font-size=\"14.00\">in</text>\n</a>\n</g>\n</g>\n<!-- layer3_act&#45;&gt;layer3_residual_in -->\n<g id=\"edge27\" class=\"edge\"><title>layer3_act&#45;&gt;layer3_residual_in</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2304.4,-79C2312.39,-79 2321.31,-79 2329.82,-79\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2329.92,-82.5001 2339.92,-79 2329.92,-75.5001 2329.92,-82.5001\"/>\n</g>\n<!-- layer3_residual_res1_conv -->\n<g id=\"node27\" class=\"node\"><title>layer3_residual_res1_conv</title>\n<g id=\"a_node27\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2472,-68C2472,-68 2442,-68 2442,-68 2436,-68 2430,-62 2430,-56 2430,-56 2430,-44 2430,-44 2430,-38 2436,-32 2442,-32 2442,-32 2472,-32 2472,-32 2478,-32 2484,-38 2484,-44 2484,-44 2484,-56 2484,-56 2484,-62 2478,-68 2472,-68\"/>\n<text text-anchor=\"middle\" x=\"2457\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_res1_conv -->\n<g id=\"edge28\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_res1_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2394.4,-70.299C2402.57,-67.608 2411.7,-64.5979 2420.38,-61.7368\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2421.52,-65.049 2429.92,-58.5949 2419.33,-58.4007 2421.52,-65.049\"/>\n</g>\n<!-- layer3_residual_add -->\n<g id=\"node34\" class=\"node\"><title>layer3_residual_add</title>\n<g id=\"a_node34\"><a xlink:title=\"&lt;class &#39;__main__.Add&#39;&gt; {}\">\n<path fill=\"#fdb462\" stroke=\"black\" d=\"M3102,-93C3102,-93 3072,-93 3072,-93 3066,-93 3060,-87 3060,-81 3060,-81 3060,-69 3060,-69 3060,-63 3066,-57 3072,-57 3072,-57 3102,-57 3102,-57 3108,-57 3114,-63 3114,-69 3114,-69 3114,-81 3114,-81 3114,-87 3108,-93 3102,-93\"/>\n<text text-anchor=\"middle\" x=\"3087\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">add</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_in&#45;&gt;layer3_residual_add -->\n<g id=\"edge35\" class=\"edge\"><title>layer3_residual_in&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2394.22,-93.0359C2402.83,-96.9904 2412.61,-100.808 2422,-103 2475.71,-115.535 2490.84,-108 2546,-108 2546,-108 2546,-108 2908,-108 2957.77,-108 3014.08,-95.3711 3049.8,-85.7384\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3051.04,-89.0265 3059.74,-82.9884 3049.17,-82.2796 3051.04,-89.0265\"/>\n</g>\n<!-- layer3_residual_res1_norm -->\n<g id=\"node28\" class=\"node\"><title>layer3_residual_res1_norm</title>\n<g id=\"a_node28\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 512}\">\n<path fill=\"#d0c281\" stroke=\"black\" d=\"M2562,-68C2562,-68 2532,-68 2532,-68 2526,-68 2520,-62 2520,-56 2520,-56 2520,-44 2520,-44 2520,-38 2526,-32 2532,-32 2532,-32 2562,-32 2562,-32 2568,-32 2574,-38 2574,-44 2574,-44 2574,-56 2574,-56 2574,-62 2568,-68 2562,-68\"/>\n<text text-anchor=\"middle\" x=\"2547\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm -->\n<g id=\"edge29\" class=\"edge\"><title>layer3_residual_res1_conv&#45;&gt;layer3_residual_res1_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2484.4,-50C2492.39,-50 2501.31,-50 2509.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2509.92,-53.5001 2519.92,-50 2509.92,-46.5001 2509.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res1_act -->\n<g id=\"node29\" class=\"node\"><title>layer3_residual_res1_act</title>\n<g id=\"a_node29\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#f0e189\" stroke=\"black\" d=\"M2652,-68C2652,-68 2622,-68 2622,-68 2616,-68 2610,-62 2610,-56 2610,-56 2610,-44 2610,-44 2610,-38 2616,-32 2622,-32 2622,-32 2652,-32 2652,-32 2658,-32 2664,-38 2664,-44 2664,-44 2664,-56 2664,-56 2664,-62 2658,-68 2652,-68\"/>\n<text text-anchor=\"middle\" x=\"2637\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act -->\n<g id=\"edge30\" class=\"edge\"><title>layer3_residual_res1_norm&#45;&gt;layer3_residual_res1_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2574.4,-50C2582.39,-50 2591.31,-50 2599.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2599.92,-53.5001 2609.92,-50 2599.92,-46.5001 2599.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_conv -->\n<g id=\"node30\" class=\"node\"><title>layer3_residual_res2_conv</title>\n<g id=\"a_node30\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.conv.Conv2d&#39;&gt; {&#39;in_channels&#39;: 512, &#39;out_channels&#39;: 512, &#39;kernel_size&#39;: (3, 3), &#39;stride&#39;: (1, 1), &#39;padding&#39;: (1, 1), &#39;bias&#39;: False}\">\n<path fill=\"#bebada\" stroke=\"black\" d=\"M2742,-68C2742,-68 2712,-68 2712,-68 2706,-68 2700,-62 2700,-56 2700,-56 2700,-44 2700,-44 2700,-38 2706,-32 2712,-32 2712,-32 2742,-32 2742,-32 2748,-32 2754,-38 2754,-44 2754,-44 2754,-56 2754,-56 2754,-62 2748,-68 2742,-68\"/>\n<text text-anchor=\"middle\" x=\"2727\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv -->\n<g id=\"edge31\" class=\"edge\"><title>layer3_residual_res1_act&#45;&gt;layer3_residual_res2_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2664.4,-50C2672.39,-50 2681.31,-50 2689.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2689.92,-53.5001 2699.92,-50 2689.92,-46.5001 2689.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_norm -->\n<g id=\"node31\" class=\"node\"><title>layer3_residual_res2_norm</title>\n<g id=\"a_node31\"><a xlink:title=\"functools.partial(&lt;class &#39;__main__.GhostBatchNorm&#39;&gt;, num_splits=16, weight=False) {&#39;num_features&#39;: 512}\">\n<path fill=\"#d0c281\" stroke=\"black\" d=\"M2832,-68C2832,-68 2802,-68 2802,-68 2796,-68 2790,-62 2790,-56 2790,-56 2790,-44 2790,-44 2790,-38 2796,-32 2802,-32 2802,-32 2832,-32 2832,-32 2838,-32 2844,-38 2844,-44 2844,-44 2844,-56 2844,-56 2844,-62 2838,-68 2832,-68\"/>\n<text text-anchor=\"middle\" x=\"2817\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">norm</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm -->\n<g id=\"edge32\" class=\"edge\"><title>layer3_residual_res2_conv&#45;&gt;layer3_residual_res2_norm</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2754.4,-50C2762.39,-50 2771.31,-50 2779.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2779.92,-53.5001 2789.92,-50 2779.92,-46.5001 2779.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_res2_act -->\n<g id=\"node32\" class=\"node\"><title>layer3_residual_res2_act</title>\n<g id=\"a_node32\"><a xlink:title=\"functools.partial(&lt;class &#39;torch.nn.modules.activation.CELU&#39;&gt;, 0.3) {}\">\n<path fill=\"#f0e189\" stroke=\"black\" d=\"M2922,-68C2922,-68 2892,-68 2892,-68 2886,-68 2880,-62 2880,-56 2880,-56 2880,-44 2880,-44 2880,-38 2886,-32 2892,-32 2892,-32 2922,-32 2922,-32 2928,-32 2934,-38 2934,-44 2934,-44 2934,-56 2934,-56 2934,-62 2928,-68 2922,-68\"/>\n<text text-anchor=\"middle\" x=\"2907\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\">act</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act -->\n<g id=\"edge33\" class=\"edge\"><title>layer3_residual_res2_norm&#45;&gt;layer3_residual_res2_act</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2844.4,-50C2852.39,-50 2861.31,-50 2869.82,-50\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2869.92,-53.5001 2879.92,-50 2869.92,-46.5001 2869.92,-53.5001\"/>\n</g>\n<!-- layer3_residual_out -->\n<g id=\"node33\" class=\"node\"><title>layer3_residual_out</title>\n<g id=\"a_node33\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M3012,-78C3012,-78 2982,-78 2982,-78 2976,-78 2970,-72 2970,-66 2970,-66 2970,-54 2970,-54 2970,-48 2976,-42 2982,-42 2982,-42 3012,-42 3012,-42 3018,-42 3024,-48 3024,-54 3024,-54 3024,-66 3024,-66 3024,-72 3018,-78 3012,-78\"/>\n<text text-anchor=\"middle\" x=\"2997\" y=\"-56.3\" font-family=\"Times,serif\" font-size=\"14.00\">out</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_res2_act&#45;&gt;layer3_residual_out -->\n<g id=\"edge34\" class=\"edge\"><title>layer3_residual_res2_act&#45;&gt;layer3_residual_out</title>\n<path fill=\"none\" stroke=\"black\" d=\"M2934.4,-53.0003C2942.39,-53.9083 2951.31,-54.9217 2959.82,-55.8891\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"2959.59,-59.3847 2969.92,-57.0362 2960.38,-52.4294 2959.59,-59.3847\"/>\n</g>\n<!-- layer3_residual_out&#45;&gt;layer3_residual_add -->\n<g id=\"edge36\" class=\"edge\"><title>layer3_residual_out&#45;&gt;layer3_residual_add</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3024.4,-64.5005C3032.39,-65.8625 3041.31,-67.3825 3049.82,-68.8336\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3049.47,-72.3242 3059.92,-70.5544 3050.65,-65.4237 3049.47,-72.3242\"/>\n</g>\n<!-- pool -->\n<g id=\"node35\" class=\"node\"><title>pool</title>\n<g id=\"a_node35\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.pooling.MaxPool2d&#39;&gt; {&#39;kernel_size&#39;: 4}\">\n<path fill=\"#8dd3c7\" stroke=\"black\" d=\"M3192,-93C3192,-93 3162,-93 3162,-93 3156,-93 3150,-87 3150,-81 3150,-81 3150,-69 3150,-69 3150,-63 3156,-57 3162,-57 3162,-57 3192,-57 3192,-57 3198,-57 3204,-63 3204,-69 3204,-69 3204,-81 3204,-81 3204,-87 3198,-93 3192,-93\"/>\n<text text-anchor=\"middle\" x=\"3177\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">pool</text>\n</a>\n</g>\n</g>\n<!-- layer3_residual_add&#45;&gt;pool -->\n<g id=\"edge37\" class=\"edge\"><title>layer3_residual_add&#45;&gt;pool</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3114.4,-75C3122.39,-75 3131.31,-75 3139.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3139.92,-78.5001 3149.92,-75 3139.92,-71.5001 3139.92,-78.5001\"/>\n</g>\n<!-- classifier_flatten -->\n<g id=\"node36\" class=\"node\"><title>classifier_flatten</title>\n<g id=\"a_node36\"><a xlink:title=\"&lt;class &#39;__main__.Flatten&#39;&gt; {}\">\n<path fill=\"#b3de69\" stroke=\"black\" d=\"M3282,-93C3282,-93 3252,-93 3252,-93 3246,-93 3240,-87 3240,-81 3240,-81 3240,-69 3240,-69 3240,-63 3246,-57 3252,-57 3252,-57 3282,-57 3282,-57 3288,-57 3294,-63 3294,-69 3294,-69 3294,-81 3294,-81 3294,-87 3288,-93 3282,-93\"/>\n<text text-anchor=\"middle\" x=\"3267\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">flatten</text>\n</a>\n</g>\n</g>\n<!-- pool&#45;&gt;classifier_flatten -->\n<g id=\"edge38\" class=\"edge\"><title>pool&#45;&gt;classifier_flatten</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3204.4,-75C3212.39,-75 3221.31,-75 3229.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3229.92,-78.5001 3239.92,-75 3229.92,-71.5001 3229.92,-78.5001\"/>\n</g>\n<!-- classifier_conv -->\n<g id=\"node37\" class=\"node\"><title>classifier_conv</title>\n<g id=\"a_node37\"><a xlink:title=\"&lt;class &#39;torch.nn.modules.linear.Linear&#39;&gt; {&#39;in_features&#39;: 512, &#39;out_features&#39;: 10, &#39;bias&#39;: False}\">\n<path fill=\"#fccde5\" stroke=\"black\" d=\"M3372,-93C3372,-93 3342,-93 3342,-93 3336,-93 3330,-87 3330,-81 3330,-81 3330,-69 3330,-69 3330,-63 3336,-57 3342,-57 3342,-57 3372,-57 3372,-57 3378,-57 3384,-63 3384,-69 3384,-69 3384,-81 3384,-81 3384,-87 3378,-93 3372,-93\"/>\n<text text-anchor=\"middle\" x=\"3357\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">conv</text>\n</a>\n</g>\n</g>\n<!-- classifier_flatten&#45;&gt;classifier_conv -->\n<g id=\"edge39\" class=\"edge\"><title>classifier_flatten&#45;&gt;classifier_conv</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3294.4,-75C3302.39,-75 3311.31,-75 3319.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3319.92,-78.5001 3329.92,-75 3319.92,-71.5001 3319.92,-78.5001\"/>\n</g>\n<!-- classifier_scale -->\n<g id=\"node38\" class=\"node\"><title>classifier_scale</title>\n<g id=\"a_node38\"><a xlink:title=\"&lt;class &#39;__main__.Mul&#39;&gt; {&#39;weight&#39;: 0.0625}\">\n<path fill=\"#bc80bd\" stroke=\"black\" d=\"M3462,-93C3462,-93 3432,-93 3432,-93 3426,-93 3420,-87 3420,-81 3420,-81 3420,-69 3420,-69 3420,-63 3426,-57 3432,-57 3432,-57 3462,-57 3462,-57 3468,-57 3474,-63 3474,-69 3474,-69 3474,-81 3474,-81 3474,-87 3468,-93 3462,-93\"/>\n<text text-anchor=\"middle\" x=\"3447\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">scale</text>\n</a>\n</g>\n</g>\n<!-- classifier_conv&#45;&gt;classifier_scale -->\n<g id=\"edge40\" class=\"edge\"><title>classifier_conv&#45;&gt;classifier_scale</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3384.4,-75C3392.39,-75 3401.31,-75 3409.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3409.92,-78.5001 3419.92,-75 3409.92,-71.5001 3409.92,-78.5001\"/>\n</g>\n<!-- logits -->\n<g id=\"node39\" class=\"node\"><title>logits</title>\n<g id=\"a_node39\"><a xlink:title=\"&lt;class &#39;__main__.Identity&#39;&gt; {}\">\n<path fill=\"#80b1d3\" stroke=\"black\" d=\"M3552,-93C3552,-93 3522,-93 3522,-93 3516,-93 3510,-87 3510,-81 3510,-81 3510,-69 3510,-69 3510,-63 3516,-57 3522,-57 3522,-57 3552,-57 3552,-57 3558,-57 3564,-63 3564,-69 3564,-69 3564,-81 3564,-81 3564,-87 3558,-93 3552,-93\"/>\n<text text-anchor=\"middle\" x=\"3537\" y=\"-71.3\" font-family=\"Times,serif\" font-size=\"14.00\">logits</text>\n</a>\n</g>\n</g>\n<!-- classifier_scale&#45;&gt;logits -->\n<g id=\"edge41\" class=\"edge\"><title>classifier_scale&#45;&gt;logits</title>\n<path fill=\"none\" stroke=\"black\" d=\"M3474.4,-75C3482.39,-75 3491.31,-75 3499.82,-75\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"3499.92,-78.5001 3509.92,-75 3499.92,-71.5001 3499.92,-78.5001\"/>\n</g>\n<!-- input -->\n<g id=\"node40\" class=\"node\"><title>input</title>\n<path fill=\"#ffffff\" stroke=\"black\" d=\"M42,-92C42,-92 12,-92 12,-92 6,-92 0,-86 0,-80 0,-80 0,-68 0,-68 0,-62 6,-56 12,-56 12,-56 42,-56 42,-56 48,-56 54,-62 54,-68 54,-68 54,-80 54,-80 54,-86 48,-92 42,-92\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-70.3\" font-family=\"Times,serif\" font-size=\"14.00\">input</text>\n</g>\n<!-- input&#45;&gt;prep_whiten -->\n<g id=\"edge1\" class=\"edge\"><title>input&#45;&gt;prep_whiten</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.4029,-74C62.3932,-74 71.3106,-74 79.8241,-74\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"79.919,-77.5001 89.919,-74 79.919,-70.5001 79.919,-77.5001\"/>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IqAvZkzucX9W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 17, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(8, 8))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.6, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "opt_params_bias = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 0.6*64, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size/64), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(input_whitening_net, label_smoothing_loss(0.2))\n",
        "    is_bias = group_by_key(('bias' in k, v) for k, v in trainable_params(model).items())\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(is_bias[False], opt_params), SGD(is_bias[True], opt_params_bias)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IfjLj-HORxWe",
        "colab_type": "text"
      },
      "source": [
        "17 epoch test accuracy jumps to 94.4% allowing a further 2 epoch reduction in training time. 15 epochs brings a test accuracy of 94.1% in 39s, closing in on the 4-GPU, test-time-augmentation assisted DAWNBench leader! If we increase the maximum learning rate by a further ~50% and reduce the amount of cutout augmentation, from 8×8 to 5×5 patches, to compensate for the extra regularisation that the higher learning rate brings, we can remove a further epoch and reach 94.1% test accuracy in 36s, moving us narrowly into top spot on the leaderboard!!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_TouhGnnKIBn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size = 14, 512\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(5, 5))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 1.0, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "opt_params_bias = {'lr': lr_schedule([0, epochs/5, epochs], [0.0, 1.0*64, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size/64), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(input_whitening_net, label_smoothing_loss(0.2))\n",
        "    is_bias = group_by_key(('bias' in k, v) for k, v in trainable_params(model).items())\n",
        "    state, timer = {MODEL: model, OPTS: [SGD(is_bias[False], opt_params), SGD(is_bias[True], opt_params_bias)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bz_9Q9MapvH7",
        "colab_type": "text"
      },
      "source": [
        "### Exponential moving averages (33.5s)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DD_F0A7fpzBb",
        "colab_type": "text"
      },
      "source": [
        "High learning rates are necessary for rapid training since they allow stochastic gradient descent to traverse the necessary distances in parameter space in a limited amount of time. On the other hand, learning rates need to be annealed towards the end of training to enable optimisation along the steeper and noisier directions in parameter space. Parameter averaging methods allow training to continue at a higher rate whilst potentially approaching minima along noisy or oscillatory directions by averaging over multiple iterates. \n",
        "\n",
        "We shall investigate exponential moving averaging of parameters which is a standard approach. For efficiency reasons we update the moving average every 5 batches since we find that more frequent updates don't improve things. We need to choose a new learning rate schedule with higher learning rates towards the end of training, and a momentum for the moving average. For the learning rate, a simple choice is to stick with the piecewise linear schedule that we've been using throughout, floored at a low fixed value for the last 2 epochs and we choose a momentum of 0.99 so that averaging takes place over a timescale of roughly the last epoch.\n",
        "\n",
        "Test accuracy improves to 94.3% allowing us to trim a further epoch. 13 epoch training reaches a test accuracy of 94.1%, achieving a training time below 34s and a 10× improvement over the single-GPU state-of-the-art at the outset of the series!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Jef-6wWUcpH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Final training setup\n",
        "epochs, batch_size, ema_epochs=13, 512, 2\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(5, 5))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs - ema_epochs], [0.0, 1.0, 0.1], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "opt_params_bias = {'lr': lr_schedule([0, epochs/5, epochs - ema_epochs], [0.0, 1.0*64, 0.1*64], batch_size), 'weight_decay': Const(5e-4*batch_size/64), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(input_whitening_net, label_smoothing_loss(0.2))\n",
        "    is_bias = group_by_key(('bias' in k, v) for k, v in trainable_params(model).items())\n",
        "    state, timer = {MODEL: model, VALID_MODEL: copy.deepcopy(model), OPTS: [SGD(is_bias[False], opt_params), SGD(is_bias[True], opt_params_bias)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size), train_steps=(*train_steps, update_ema(momentum=0.99, update_freq=5)))))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dv-Q0jUA-oHi",
        "colab_type": "text"
      },
      "source": [
        "### Test-time augmentation (26s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O-2DG-xvVr2g",
        "colab_type": "text"
      },
      "source": [
        "Suppose that you'd like your network to classify images the same way under horizontal flips of the input. One possibility, that we've been using till now, is to present the network with a large amount of data, possibly augmented by label preserving left-right flips, and hope that the network will eventually learn the invariance through extensive training. \n",
        "\n",
        "A second approach, which doesn't leave things to chance, is to present both the input image and its horizontally flipped version and come to a consensus by averaging network outputs for the two versions, thus guaranteeing invariance. This eminently sensible approach goes by the name of test-time augmentation.\n",
        "\n",
        "At training time, we still present the network with a single version of each image - potentially subject to random flipping as a data augmentation so that different versions are presented on different training epochs. An alternative, would be to use the same procedure at training time as at test time and present each image along with its mirror. In this case, we could claim to have changed the network by splitting into two identical branches, one of which sees the flipped image, and then merging at the end. Through this lens, the original training can be viewed as a stochastic training procedure for a weight-tied, two branch network in which a single branch is 'dropped-out' for each training example.\n",
        "\n",
        "This dropout-training viewpoint makes it clear that any attempt to introduce a rule disallowing TTA from a benchmark is going to be fraught with difficulties. From this point of view, we have just introduced a larger network for which we have an efficient stochastic training methodology. On the other hand, if we don't limit the amount of work that we are prepared to do at test time then there are some obvious degenerate solutions in which training takes as little time as is required to store the dataset!\n",
        "\n",
        "These arguments are not only relevant to artificial benchmarks but also to end use-cases. In some applications, classification accuracy is all that is desired and in that case TTA should most definitely be used. In other cases, inference time is also a constraint and a sensible approach is to maximise accuracy subject to such constraints. This is probably a good approach for training benchmarks too. \n",
        "\n",
        "In the case at hand, the Kakao Brain team has applied the simple form of TTA described here - presenting an image and its left-right mirror at inference time, thus doubling the computational load. More extensive forms of TTA are of course possible for other symmetries (such as translational symmetry, variations in brightness/colour etc.) but these would come at a higher computational cost. \n",
        "\n",
        "Now because these entries are based of a computationally light 9-layer ResNet _total inference time including TTA_ is likely to be much lower for these entries than for some of the 100+ layer networks that have been entered at earlier stages of the competition! According to our discussion above, any reasonable rule to limit this kind of approach should be based on inference time constraints and not an arbitrary feature of the implementation and so from this point-of-view, we should accept the approach.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6GcFJZJeg__W",
        "colab_type": "text"
      },
      "source": [
        "Let's see what improvement TTA brings. We shall restrict ourselves to horizontal flip TTA for consistency with the current DAWNBench submissions and because this seems a sweet spot between accuracy and inference cost. With our current network and 13 epoch training setup, the test accuracy with TTA rises to 94.6%, making this the largest individual effect we've studied today. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xjS1RJ11oLef",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "valid_steps_tta = (forward_tta([identity, flip_lr]), log_activations(('loss', 'acc')))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J3r9fcUEnlk_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size, ema_epochs=13, 512, 2\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(5, 5))\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs - ema_epochs], [0.0, 1.0, 0.1], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "opt_params_bias = {'lr': lr_schedule([0, epochs/5, epochs - ema_epochs], [0.0, 1.0*64, 0.1*64], batch_size), 'weight_decay': Const(5e-4*batch_size/64), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(input_whitening_net, label_smoothing_loss(0.2))\n",
        "    is_bias = group_by_key(('bias' in k, v) for k, v in trainable_params(model).items())\n",
        "    state, timer = {MODEL: model, VALID_MODEL: copy.deepcopy(model), OPTS: [SGD(is_bias[False], opt_params), SGD(is_bias[True], opt_params_bias)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size), \n",
        "                                                                        train_steps=(*train_steps, update_ema(momentum=0.99, update_freq=5)),\n",
        "                                                                        valid_steps=valid_steps_tta)))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vH4hq224nqQh",
        "colab_type": "text"
      },
      "source": [
        "If we remove the remaining cutout data augmentation - which is getting in the way on such a short training schedule - we can reduce training to 10 epochs (!) and achieve a TTA test accuracy of 94.1% in 26s!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oewy20lcoBVK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs, batch_size, ema_epochs=10, 512, 2\n",
        "transforms = (Crop(32, 32), FlipLR())\n",
        "opt_params = {'lr': lr_schedule([0, epochs/5, epochs - ema_epochs], [0.0, 1.0, 0.1], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "opt_params_bias = {'lr': lr_schedule([0, epochs/5, epochs - ema_epochs], [0.0, 1.0*64, 0.1*64], batch_size), 'weight_decay': Const(5e-4*batch_size/64), 'momentum': Const(0.9)}\n",
        "\n",
        "logs = Table(report=every(epochs,'epoch'))\n",
        "for run in range(N_RUNS):\n",
        "    model = build_model(input_whitening_net, label_smoothing_loss(0.2))\n",
        "    is_bias = group_by_key(('bias' in k, v) for k, v in trainable_params(model).items())\n",
        "    state, timer = {MODEL: model, VALID_MODEL: copy.deepcopy(model), OPTS: [SGD(is_bias[False], opt_params), SGD(is_bias[True], opt_params_bias)]}, Timer(torch.cuda.synchronize)\n",
        "    for epoch in range(epochs):\n",
        "        logs.append(union({'run': run+1, 'epoch': epoch+1}, train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size), \n",
        "                                                                        train_steps=(*train_steps, update_ema(momentum=0.99, update_freq=5)),\n",
        "                                                                        valid_steps=valid_steps_tta)))\n",
        "summary(logs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dD7-fE8Xoq_r",
        "colab_type": "text"
      },
      "source": [
        "### Training to convergence"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yO8eWpWOdFS-",
        "colab_type": "text"
      },
      "source": [
        "Here is a simple experiment to investigate whether the gains in training speed that we have collected also translate into gains in final accuracy for the model if it is trained to convergence. We have every reason to believe that this should be the case, if only because many of the techniques that we have been using today were originally proposed as techniques to improve converged accuracy on ImageNet! If it is the case that the same techniques which speed up training time to 94% accuracy on CIFAR10 also improve converged accuracy on ImageNet, then this suggests a rather effective way to accelerate research on the latter problem!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K9G4gjGueEAZ",
        "colab_type": "text"
      },
      "source": [
        "Unlike the previous experiments, this is going to be very rough and ready and we leave it to future work to do this experiment more carefully. We are going to pick a fixed learning rate schedule with lower learning rates appropriate for longer training and increase the amount of cutout augmentation to 12×12 patches to allow training for longer without overfitting. We will fix the other hyperparameters as they were above and train both the baseline network and the final network for a range of different times from 24 to 100 epochs. Finally we're going to break all the rules and only run each experiment 5 times! Here are the results: "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5HShJN2xh22e",
        "colab_type": "text"
      },
      "source": [
        "#### Convergence experiment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V2LQrVTMgTSq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "valid_steps_tta = (forward_tta([identity, flip_lr]), log_activations(('loss', 'acc')))\n",
        "\n",
        "def train_epoch_tta(state, timer, train_batches, valid_batches, train_steps=train_steps, \n",
        "                    valid_steps=valid_steps, valid_steps_tta=valid_steps_tta, on_epoch_end=identity):\n",
        "    train_summary, train_time = epoch_stats(on_epoch_end(reduce(train_batches, state, train_steps))), timer()\n",
        "    valid_summary, valid_time = epoch_stats(reduce(valid_batches, state, valid_steps)), timer(update_total=False) #DAWNBench rules\n",
        "    valid_summary_tta, valid_time_tta = epoch_stats(reduce(valid_batches, state, valid_steps_tta)), timer(update_total=False) #DAWNBench rules\n",
        "    return {\n",
        "        'train': union({'time': train_time}, train_summary), \n",
        "        'valid': union({'time': valid_time}, valid_summary), \n",
        "        'tta': union({'time': valid_time}, valid_summary_tta), \n",
        "        'total time': timer.total_time\n",
        "    }\n",
        "\n",
        "#baseline model\n",
        "transforms = (Crop(32, 32), FlipLR(), Cutout(12, 12))\n",
        "logs = Table()\n",
        "for run in range(5):\n",
        "    for epochs in [24, 40, 60, 80, 100]:\n",
        "        opt_params = {'lr': lr_schedule([0, 5, epochs], [0.0, 0.4, 0.0], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "\n",
        "        model = build_model(network(), x_ent_loss)\n",
        "        state, timer = {MODEL: model, OPTS: [SGD(trainable_params(model).values(), opt_params)]}, Timer(torch.cuda.synchronize)\n",
        "        for epoch in range(epochs-1):\n",
        "            train_epoch(state, timer, train_batches(batch_size, transforms),  valid_batches(batch_size)) \n",
        "        logs.append(union({'run': run+1, 'epoch': epochs, 'experiment': baseline}, \n",
        "                          train_epoch_tta(state, timer, train_batches(batch_size, transforms),  valid_batches(batch_size), \n",
        "                                          valid_steps_tta=valid_steps_tta)))   \n",
        "#final model\n",
        "ema_epochs=2\n",
        "for run in range(5):\n",
        "    for epochs in [24, 40, 60, 80, 100]:\n",
        "        opt_params = {'lr': lr_schedule([0, epochs/5, epochs - ema_epochs], [0.0, 0.4, 0.04], batch_size), 'weight_decay': Const(5e-4*batch_size), 'momentum': Const(0.9)}\n",
        "        opt_params_bias = {'lr': lr_schedule([0, epochs/5, epochs - ema_epochs], [0.0, 0.4*64, 0.04\n",
        "                                                                          *64], batch_size), 'weight_decay': Const(5e-4*batch_size/64), 'momentum': Const(0.9)}\n",
        "\n",
        "        model = build_model(input_whitening_net, label_smoothing_loss(0.2))\n",
        "        is_bias = group_by_key(('bias' in k, v) for k, v in trainable_params(model).items())\n",
        "        state, timer = {MODEL: model, VALID_MODEL: copy.deepcopy(model), OPTS: [SGD(is_bias[False], opt_params), SGD(is_bias[True], opt_params_bias)]}, Timer(torch.cuda.synchronize)\n",
        "        for epoch in range(epochs-1):\n",
        "            train_epoch(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size), \n",
        "                                                                            train_steps=(*train_steps, update_ema(momentum=0.99, update_freq=5)))\n",
        "        logs.append(union({'run': run+1, 'epoch': epochs, 'experiment': final}, \n",
        "                          train_epoch_tta(state, timer, train_batches(batch_size, transforms), valid_batches(batch_size), \n",
        "                                          train_steps=(*train_steps, update_ema(momentum=0.99, update_freq=5)))))  \n",
        "        \n",
        "data = logs.df()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GI_eqtvSh6Ng",
        "colab_type": "text"
      },
      "source": [
        "#### Results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t-5yQ-E3g-xt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 371
        },
        "outputId": "ce602892-b006-466a-e743-d6ce2314a463"
      },
      "source": [
        "scale = alt.Scale(zero=False)\n",
        "c = alt.Chart(data).encode(x=alt.X('epoch', scale=scale), color='experiment')\n",
        "\n",
        "c1=(c.mark_point().encode(y=alt.Y('valid_acc',scale=scale, axis=alt.Axis(title='valid acc'))) + \n",
        "c.mark_line().encode(y=alt.Y('mean(valid_acc)',scale=scale)))\n",
        "\n",
        "c2=(c.mark_point().encode(y=alt.Y('tta_acc',scale=scale, axis=alt.Axis(title='valid acc (tta)'))) + \n",
        "c.mark_line().encode(y=alt.Y('mean(tta_acc)',scale=scale)))\n",
        "\n",
        "c1 | c2"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<!DOCTYPE html>\n",
              "<html>\n",
              "<head>\n",
              "  <style>\n",
              "    .vega-actions a {\n",
              "        margin-right: 12px;\n",
              "        color: #757575;\n",
              "        font-weight: normal;\n",
              "        font-size: 13px;\n",
              "    }\n",
              "    .error {\n",
              "        color: red;\n",
              "    }\n",
              "  </style>\n",
              "  <script type=\"text/javascript\" src=\"https://cdn.jsdelivr.net/npm//vega@5\"></script>\n",
              "  <script type=\"text/javascript\" src=\"https://cdn.jsdelivr.net/npm//vega-lite@3.4.0\"></script>\n",
              "  <script type=\"text/javascript\" src=\"https://cdn.jsdelivr.net/npm//vega-embed@4\"></script>\n",
              "</head>\n",
              "<body>\n",
              "  <div id=\"altair-viz\"></div>\n",
              "  <script>\n",
              "    (function(vegaEmbed) {\n",
              "      var spec = {\"config\": {\"view\": {\"width\": 400, \"height\": 300}, \"mark\": {\"tooltip\": null}}, \"hconcat\": [{\"layer\": [{\"mark\": \"point\", \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"experiment\"}, \"x\": {\"type\": \"quantitative\", \"field\": \"epoch\", \"scale\": {\"zero\": false}}, \"y\": {\"type\": \"quantitative\", \"axis\": {\"title\": \"valid acc\"}, \"field\": \"valid_acc\", \"scale\": {\"zero\": false}}}}, {\"mark\": \"line\", \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"experiment\"}, \"x\": {\"type\": \"quantitative\", \"field\": \"epoch\", \"scale\": {\"zero\": false}}, \"y\": {\"type\": \"quantitative\", \"aggregate\": \"mean\", \"field\": \"valid_acc\", \"scale\": {\"zero\": false}}}}]}, {\"layer\": [{\"mark\": \"point\", \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"experiment\"}, \"x\": {\"type\": \"quantitative\", \"field\": \"epoch\", \"scale\": {\"zero\": false}}, \"y\": {\"type\": \"quantitative\", \"axis\": {\"title\": \"valid acc (tta)\"}, \"field\": \"tta_acc\", \"scale\": {\"zero\": false}}}}, {\"mark\": \"line\", \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"experiment\"}, \"x\": {\"type\": \"quantitative\", \"field\": \"epoch\", \"scale\": {\"zero\": false}}, \"y\": {\"type\": \"quantitative\", \"aggregate\": \"mean\", \"field\": \"tta_acc\", \"scale\": {\"zero\": false}}}}]}], \"data\": {\"name\": \"data-14c72c6d62a735a120867cb9d5a4c575\"}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v3.4.0.json\", \"datasets\": {\"data-14c72c6d62a735a120867cb9d5a4c575\": [{\"epoch\": 40, \"run\": 1, \"total time\": 115.77539189700474, \"train_acc\": 0.9724750322164949, \"train_loss\": 0.08945317443498631, \"train_time\": 2.8944502609992924, \"tta_acc\": 0.9522, \"tta_loss\": 0.14731700954437255, \"tta_time\": 0.191972672000702, \"valid_acc\": 0.9454, \"valid_loss\": 0.1637589175224304, \"valid_time\": 0.191972672000702, \"experiment\": \"baseline\"}, {\"epoch\": 60, \"run\": 1, \"total time\": 173.7991279079979, \"train_acc\": 0.9833279639175257, \"train_loss\": 0.05970388147787949, \"train_time\": 2.892037823999999, \"tta_acc\": 0.9561, \"tta_loss\": 0.13223285465240478, \"tta_time\": 0.191634595998039, \"valid_acc\": 0.9526, \"valid_loss\": 0.14667956314086913, \"valid_time\": 0.191634595998039, \"experiment\": \"baseline\"}, {\"epoch\": 80, \"run\": 1, \"total time\": 231.56872692101751, \"train_acc\": 0.9870127255154639, \"train_loss\": 0.04798114148074204, \"train_time\": 2.8922985280005378, \"tta_acc\": 0.9575, \"tta_loss\": 0.13690186977386476, \"tta_time\": 0.1917704199986474, \"valid_acc\": 0.9522, \"valid_loss\": 0.1517398440361023, \"valid_time\": 0.1917704199986474, \"experiment\": \"baseline\"}, {\"epoch\": 100, \"run\": 1, \"total time\": 289.0958471929989, \"train_acc\": 0.9906974871134021, \"train_loss\": 0.03765406883945785, \"train_time\": 2.886209833999601, \"tta_acc\": 0.957, \"tta_loss\": 0.13574402532577515, \"tta_time\": 0.1920333970010688, \"valid_acc\": 0.9535, \"valid_loss\": 0.14849681997299194, \"valid_time\": 0.1920333970010688, \"experiment\": \"baseline\"}, {\"epoch\": 40, \"run\": 2, \"total time\": 115.93349109901101, \"train_acc\": 0.973401256443299, \"train_loss\": 0.08912419904138624, \"train_time\": 2.893640766000317, \"tta_acc\": 0.9515, \"tta_loss\": 0.1469169704437256, \"tta_time\": 0.19260841499635717, \"valid_acc\": 0.9459, \"valid_loss\": 0.161656810092926, \"valid_time\": 0.19260841499635717, \"experiment\": \"baseline\"}, {\"epoch\": 60, \"run\": 2, \"total time\": 173.81213349894824, \"train_acc\": 0.9840125644329897, \"train_loss\": 0.05957732916108726, \"train_time\": 2.892742789001204, \"tta_acc\": 0.9551, \"tta_loss\": 0.13827357215881347, \"tta_time\": 0.19220788899838226, \"valid_acc\": 0.9498, \"valid_loss\": 0.15355262699127198, \"valid_time\": 0.19220788899838226, \"experiment\": \"baseline\"}, {\"epoch\": 80, \"run\": 2, \"total time\": 231.54911988700042, \"train_acc\": 0.98828125, \"train_loss\": 0.04477034353641505, \"train_time\": 2.891720998995879, \"tta_acc\": 0.9562, \"tta_loss\": 0.1326014790534973, \"tta_time\": 0.19183066400000826, \"valid_acc\": 0.9508, \"valid_loss\": 0.15004254236221312, \"valid_time\": 0.19183066400000826, \"experiment\": \"baseline\"}, {\"epoch\": 100, \"run\": 2, \"total time\": 289.1528977750204, \"train_acc\": 0.990576675257732, \"train_loss\": 0.03862031384073582, \"train_time\": 2.885488786996575, \"tta_acc\": 0.9563, \"tta_loss\": 0.1359883903503418, \"tta_time\": 0.1906894510029815, \"valid_acc\": 0.9539, \"valid_loss\": 0.1490915186882019, \"valid_time\": 0.1906894510029815, \"experiment\": \"baseline\"}, {\"epoch\": 40, \"run\": 3, \"total time\": 115.94290988198918, \"train_acc\": 0.9740455863402062, \"train_loss\": 0.08699826053199694, \"train_time\": 2.8944724870016216, \"tta_acc\": 0.9513, \"tta_loss\": 0.14736389656066895, \"tta_time\": 0.19228573099826463, \"valid_acc\": 0.9476, \"valid_loss\": 0.16150463714599608, \"valid_time\": 0.19228573099826463, \"experiment\": \"baseline\"}, {\"epoch\": 60, \"run\": 3, \"total time\": 173.79642055099248, \"train_acc\": 0.9826634987113402, \"train_loss\": 0.060447668176643626, \"train_time\": 2.8918151920006494, \"tta_acc\": 0.9554, \"tta_loss\": 0.13378720598220825, \"tta_time\": 0.19201715199596947, \"valid_acc\": 0.9523, \"valid_loss\": 0.14677832469940186, \"valid_time\": 0.19201715199596947, \"experiment\": \"baseline\"}, {\"epoch\": 80, \"run\": 3, \"total time\": 231.33222012402257, \"train_acc\": 0.9876167847938144, \"train_loss\": 0.04650055932015488, \"train_time\": 2.8850283520005178, \"tta_acc\": 0.9571, \"tta_loss\": 0.1345288269996643, \"tta_time\": 0.19103913399885641, \"valid_acc\": 0.9514, \"valid_loss\": 0.15048525075912475, \"valid_time\": 0.19103913399885641, \"experiment\": \"baseline\"}, {\"epoch\": 100, \"run\": 3, \"total time\": 289.1790519290007, \"train_acc\": 0.9903753221649485, \"train_loss\": 0.03815095111266854, \"train_time\": 2.886819564999314, \"tta_acc\": 0.9571, \"tta_loss\": 0.13442925338745118, \"tta_time\": 0.19067772800190141, \"valid_acc\": 0.9519, \"valid_loss\": 0.15329569854736327, \"valid_time\": 0.19067772800190141, \"experiment\": \"baseline\"}, {\"epoch\": 40, \"run\": 4, \"total time\": 115.98809706900647, \"train_acc\": 0.974206668814433, \"train_loss\": 0.08665592160037498, \"train_time\": 2.898000020002655, \"tta_acc\": 0.9534, \"tta_loss\": 0.14771631813049316, \"tta_time\": 0.1922089979998418, \"valid_acc\": 0.949, \"valid_loss\": 0.16354228534698487, \"valid_time\": 0.1922089979998418, \"experiment\": \"baseline\"}, {\"epoch\": 60, \"run\": 4, \"total time\": 173.8017946649852, \"train_acc\": 0.9843145940721649, \"train_loss\": 0.05849805401311707, \"train_time\": 2.8925928979952005, \"tta_acc\": 0.9566, \"tta_loss\": 0.1356637414932251, \"tta_time\": 0.19205911600147374, \"valid_acc\": 0.9513, \"valid_loss\": 0.1493300030708313, \"valid_time\": 0.19205911600147374, \"experiment\": \"baseline\"}, {\"epoch\": 80, \"run\": 4, \"total time\": 231.53127032201883, \"train_acc\": 0.9885430090206185, \"train_loss\": 0.04646093713254044, \"train_time\": 2.8897476560014184, \"tta_acc\": 0.9569, \"tta_loss\": 0.1331675503730774, \"tta_time\": 0.19233000699750846, \"valid_acc\": 0.9516, \"valid_loss\": 0.15248397092819213, \"valid_time\": 0.19233000699750846, \"experiment\": \"baseline\"}, {\"epoch\": 100, \"run\": 4, \"total time\": 289.1731662430029, \"train_acc\": 0.9904558634020618, \"train_loss\": 0.039665708952036104, \"train_time\": 2.887360422006168, \"tta_acc\": 0.9576, \"tta_loss\": 0.13489235219955445, \"tta_time\": 0.19174574600037886, \"valid_acc\": 0.9536, \"valid_loss\": 0.1465499900817871, \"valid_time\": 0.19174574600037886, \"experiment\": \"baseline\"}, {\"epoch\": 40, \"run\": 5, \"total time\": 115.90960084103426, \"train_acc\": 0.9747905927835051, \"train_loss\": 0.08495002355157714, \"train_time\": 2.8942276980014867, \"tta_acc\": 0.9505, \"tta_loss\": 0.14900790967941285, \"tta_time\": 0.19179014800465666, \"valid_acc\": 0.9475, \"valid_loss\": 0.1620381199836731, \"valid_time\": 0.19179014800465666, \"experiment\": \"baseline\"}, {\"epoch\": 60, \"run\": 5, \"total time\": 173.73591956799646, \"train_acc\": 0.9835293170103093, \"train_loss\": 0.0589248721593434, \"train_time\": 2.8926016470068134, \"tta_acc\": 0.9554, \"tta_loss\": 0.1400909294128418, \"tta_time\": 0.19176351399801206, \"valid_acc\": 0.9514, \"valid_loss\": 0.1536110694885254, \"valid_time\": 0.19176351399801206, \"experiment\": \"baseline\"}, {\"epoch\": 80, \"run\": 5, \"total time\": 231.572428033076, \"train_acc\": 0.9887040914948454, \"train_loss\": 0.045570830766534065, \"train_time\": 2.891090276003524, \"tta_acc\": 0.9574, \"tta_loss\": 0.1329388027191162, \"tta_time\": 0.19153811800060794, \"valid_acc\": 0.9542, \"valid_loss\": 0.14818038806915282, \"valid_time\": 0.19153811800060794, \"experiment\": \"baseline\"}, {\"epoch\": 100, \"run\": 5, \"total time\": 289.2042186130275, \"train_acc\": 0.9903753221649485, \"train_loss\": 0.03801192454609674, \"train_time\": 2.8860927780042402, \"tta_acc\": 0.9573, \"tta_loss\": 0.1373170760154724, \"tta_time\": 0.19177757999568712, \"valid_acc\": 0.9522, \"valid_loss\": 0.15495181436538696, \"valid_time\": 0.19177757999568712, \"experiment\": \"baseline\"}, {\"epoch\": 24, \"run\": 1, \"total time\": 69.47972413498792, \"train_acc\": 0.9537290592783505, \"train_loss\": 0.14411357749906398, \"train_time\": 2.897183038003277, \"tta_acc\": 0.9469, \"tta_loss\": 0.1606788829803467, \"tta_time\": 0.1921295049978653, \"valid_acc\": 0.9401, \"valid_loss\": 0.17633453397750853, \"valid_time\": 0.1921295049978653, \"experiment\": \"baseline\"}, {\"epoch\": 24, \"run\": 2, \"total time\": 69.62557950800692, \"train_acc\": 0.953870006443299, \"train_loss\": 0.14425174651917108, \"train_time\": 2.8987395690055564, \"tta_acc\": 0.9435, \"tta_loss\": 0.16524562730789186, \"tta_time\": 0.19240747199364705, \"valid_acc\": 0.9397, \"valid_loss\": 0.18035643215179442, \"valid_time\": 0.19240747199364705, \"experiment\": \"baseline\"}, {\"epoch\": 24, \"run\": 3, \"total time\": 69.65048073300568, \"train_acc\": 0.9536887886597938, \"train_loss\": 0.14596862908567965, \"train_time\": 2.90208780499961, \"tta_acc\": 0.9424, \"tta_loss\": 0.16533652591705322, \"tta_time\": 0.1919627120005316, \"valid_acc\": 0.9384, \"valid_loss\": 0.1796941129684448, \"valid_time\": 0.1919627120005316, \"experiment\": \"baseline\"}, {\"epoch\": 24, \"run\": 4, \"total time\": 69.65110485800687, \"train_acc\": 0.9530645940721649, \"train_loss\": 0.1461100154124277, \"train_time\": 2.9011953369990806, \"tta_acc\": 0.9432, \"tta_loss\": 0.17059293451309204, \"tta_time\": 0.19200909399660304, \"valid_acc\": 0.9396, \"valid_loss\": 0.18514128446578978, \"valid_time\": 0.19200909399660304, \"experiment\": \"baseline\"}, {\"epoch\": 24, \"run\": 5, \"total time\": 69.64399432497885, \"train_acc\": 0.9548163659793815, \"train_loss\": 0.1410928838646289, \"train_time\": 2.900024989998201, \"tta_acc\": 0.9445, \"tta_loss\": 0.16654890432357788, \"tta_time\": 0.19193982199794846, \"valid_acc\": 0.9384, \"valid_loss\": 0.1841355944633484, \"valid_time\": 0.19193982199794846, \"experiment\": \"baseline\"}, {\"epoch\": 40, \"run\": 1, \"total time\": 102.84597889299039, \"train_acc\": 0.9693741945876289, \"train_loss\": 0.9459850665220281, \"train_time\": 2.5670782279994455, \"tta_acc\": 0.9579, \"tta_loss\": 0.9583810546875, \"tta_time\": 0.181817176999175, \"valid_acc\": 0.9538, \"valid_loss\": 0.96737177734375, \"valid_time\": 0.181817176999175, \"experiment\": \"final\"}, {\"epoch\": 60, \"run\": 1, \"total time\": 154.31354360000114, \"train_acc\": 0.9786767074742269, \"train_loss\": 0.928253222986595, \"train_time\": 2.57236493799428, \"tta_acc\": 0.9587, \"tta_loss\": 0.951898486328125, \"tta_time\": 0.1827594560018042, \"valid_acc\": 0.9546, \"valid_loss\": 0.96043193359375, \"valid_time\": 0.1827594560018042, \"experiment\": \"final\"}, {\"epoch\": 80, \"run\": 1, \"total time\": 205.7615784409645, \"train_acc\": 0.9815157860824743, \"train_loss\": 0.9220649384960686, \"train_time\": 2.569626550997782, \"tta_acc\": 0.9609, \"tta_loss\": 0.949437255859375, \"tta_time\": 0.18310099600057583, \"valid_acc\": 0.9568, \"valid_loss\": 0.95888212890625, \"valid_time\": 0.18310099600057583, \"experiment\": \"final\"}, {\"epoch\": 100, \"run\": 1, \"total time\": 257.2018751499709, \"train_acc\": 0.9832675579896907, \"train_loss\": 0.9179677373355197, \"train_time\": 2.5681756339981803, \"tta_acc\": 0.9601, \"tta_loss\": 0.947637109375, \"tta_time\": 0.18255041600059485, \"valid_acc\": 0.9575, \"valid_loss\": 0.956150048828125, \"valid_time\": 0.18255041600059485, \"experiment\": \"final\"}, {\"epoch\": 40, \"run\": 2, \"total time\": 102.95762568798818, \"train_acc\": 0.9720521907216495, \"train_loss\": 0.9434325228032377, \"train_time\": 2.577212516000145, \"tta_acc\": 0.958, \"tta_loss\": 0.957646435546875, \"tta_time\": 0.18276126800628845, \"valid_acc\": 0.954, \"valid_loss\": 0.965418359375, \"valid_time\": 0.18276126800628845, \"experiment\": \"final\"}, {\"epoch\": 60, \"run\": 2, \"total time\": 154.3870944369919, \"train_acc\": 0.9785357603092784, \"train_loss\": 0.9288256733687883, \"train_time\": 2.5691759129986167, \"tta_acc\": 0.9574, \"tta_loss\": 0.953997314453125, \"tta_time\": 0.18221270800131606, \"valid_acc\": 0.9556, \"valid_loss\": 0.962770068359375, \"valid_time\": 0.18221270800131606, \"experiment\": \"final\"}, {\"epoch\": 80, \"run\": 2, \"total time\": 205.821977417967, \"train_acc\": 0.9823413337628866, \"train_loss\": 0.9203307948161646, \"train_time\": 2.5703018869971856, \"tta_acc\": 0.9611, \"tta_loss\": 0.950356884765625, \"tta_time\": 0.18253108699718723, \"valid_acc\": 0.9566, \"valid_loss\": 0.958939599609375, \"valid_time\": 0.18253108699718723, \"experiment\": \"final\"}, {\"epoch\": 100, \"run\": 2, \"total time\": 257.2657502310103, \"train_acc\": 0.9837910760309279, \"train_loss\": 0.9158867904820394, \"train_time\": 2.568709031002072, \"tta_acc\": 0.9623, \"tta_loss\": 0.945064013671875, \"tta_time\": 0.182770311999775, \"valid_acc\": 0.9593, \"valid_loss\": 0.955058447265625, \"valid_time\": 0.182770311999775, \"experiment\": \"final\"}, {\"epoch\": 40, \"run\": 3, \"total time\": 102.91727831303433, \"train_acc\": 0.9696560889175257, \"train_loss\": 0.9443699256661012, \"train_time\": 2.5694828940031584, \"tta_acc\": 0.9568, \"tta_loss\": 0.958073681640625, \"tta_time\": 0.18202030599786667, \"valid_acc\": 0.9535, \"valid_loss\": 0.966338623046875, \"valid_time\": 0.18202030599786667, \"experiment\": \"final\"}, {\"epoch\": 60, \"run\": 3, \"total time\": 154.38148195800022, \"train_acc\": 0.9781934600515464, \"train_loss\": 0.9290023587413669, \"train_time\": 2.572302505999687, \"tta_acc\": 0.9601, \"tta_loss\": 0.952961572265625, \"tta_time\": 0.18237990399939008, \"valid_acc\": 0.9547, \"valid_loss\": 0.962862646484375, \"valid_time\": 0.18237990399939008, \"experiment\": \"final\"}, {\"epoch\": 80, \"run\": 3, \"total time\": 205.85524166001414, \"train_acc\": 0.9814352448453608, \"train_loss\": 0.9220564045856908, \"train_time\": 2.5693484250004985, \"tta_acc\": 0.96, \"tta_loss\": 0.9477572265625, \"tta_time\": 0.18291645099816378, \"valid_acc\": 0.9566, \"valid_loss\": 0.956994482421875, \"valid_time\": 0.18291645099816378, \"experiment\": \"final\"}, {\"epoch\": 100, \"run\": 3, \"total time\": 257.400613299993, \"train_acc\": 0.9848179768041238, \"train_loss\": 0.9156462285936493, \"train_time\": 2.571818030999566, \"tta_acc\": 0.9618, \"tta_loss\": 0.947703662109375, \"tta_time\": 0.18251981300272746, \"valid_acc\": 0.9584, \"valid_loss\": 0.95703037109375, \"valid_time\": 0.18251981300272746, \"experiment\": \"final\"}, {\"epoch\": 40, \"run\": 4, \"total time\": 102.93862039600208, \"train_acc\": 0.9711259664948454, \"train_loss\": 0.9436643639790643, \"train_time\": 2.5738141760011786, \"tta_acc\": 0.9581, \"tta_loss\": 0.957923681640625, \"tta_time\": 0.18292383100197185, \"valid_acc\": 0.9539, \"valid_loss\": 0.96743447265625, \"valid_time\": 0.18292383100197185, \"experiment\": \"final\"}, {\"epoch\": 60, \"run\": 4, \"total time\": 154.41663919502753, \"train_acc\": 0.9784954896907216, \"train_loss\": 0.9284879841755346, \"train_time\": 2.5739044319998357, \"tta_acc\": 0.9602, \"tta_loss\": 0.95122548828125, \"tta_time\": 0.18250545299815712, \"valid_acc\": 0.9561, \"valid_loss\": 0.96009091796875, \"valid_time\": 0.18250545299815712, \"experiment\": \"final\"}, {\"epoch\": 80, \"run\": 4, \"total time\": 205.85838152302313, \"train_acc\": 0.9814352448453608, \"train_loss\": 0.9217412496350476, \"train_time\": 2.572255806000612, \"tta_acc\": 0.9619, \"tta_loss\": 0.947907763671875, \"tta_time\": 0.18321384899900295, \"valid_acc\": 0.9578, \"valid_loss\": 0.95641865234375, \"valid_time\": 0.18321384899900295, \"experiment\": \"final\"}, {\"epoch\": 100, \"run\": 4, \"total time\": 257.2927283469835, \"train_acc\": 0.9829655283505154, \"train_loss\": 0.9180972797354472, \"train_time\": 2.5725200929955463, \"tta_acc\": 0.9601, \"tta_loss\": 0.946186474609375, \"tta_time\": 0.18265706300007878, \"valid_acc\": 0.9571, \"valid_loss\": 0.95430537109375, \"valid_time\": 0.18265706300007878, \"experiment\": \"final\"}, {\"epoch\": 40, \"run\": 5, \"total time\": 102.95283322800242, \"train_acc\": 0.9719313788659794, \"train_loss\": 0.9427000517697678, \"train_time\": 2.5722945279994747, \"tta_acc\": 0.957, \"tta_loss\": 0.958058056640625, \"tta_time\": 0.18345675599994138, \"valid_acc\": 0.9541, \"valid_loss\": 0.9679720703125, \"valid_time\": 0.18345675599994138, \"experiment\": \"final\"}, {\"epoch\": 60, \"run\": 5, \"total time\": 154.39701501696254, \"train_acc\": 0.9778108891752577, \"train_loss\": 0.9301550560390827, \"train_time\": 2.57575754699792, \"tta_acc\": 0.9584, \"tta_loss\": 0.9511826171875, \"tta_time\": 0.18216746600228362, \"valid_acc\": 0.9553, \"valid_loss\": 0.96147744140625, \"valid_time\": 0.18216746600228362, \"experiment\": \"final\"}, {\"epoch\": 80, \"run\": 5, \"total time\": 205.8281666639814, \"train_acc\": 0.9823211984536082, \"train_loss\": 0.9212127567566547, \"train_time\": 2.570514618004381, \"tta_acc\": 0.9624, \"tta_loss\": 0.947673046875, \"tta_time\": 0.18234977099928074, \"valid_acc\": 0.958, \"valid_loss\": 0.955671728515625, \"valid_time\": 0.18234977099928074, \"experiment\": \"final\"}, {\"epoch\": 100, \"run\": 5, \"total time\": 257.22457723898697, \"train_acc\": 0.9840729703608248, \"train_loss\": 0.9173678270320302, \"train_time\": 2.5702690160032944, \"tta_acc\": 0.9618, \"tta_loss\": 0.946293212890625, \"tta_time\": 0.18217301399999997, \"valid_acc\": 0.9569, \"valid_loss\": 0.95438818359375, \"valid_time\": 0.18217301399999997, \"experiment\": \"final\"}, {\"epoch\": 24, \"run\": 1, \"total time\": 61.77687859902653, \"train_acc\": 0.9563667847938144, \"train_loss\": 0.9728790656807497, \"train_time\": 2.5735945090054884, \"tta_acc\": 0.9521, \"tta_loss\": 0.973640478515625, \"tta_time\": 0.18347028499556473, \"valid_acc\": 0.9473, \"valid_loss\": 0.982691748046875, \"valid_time\": 0.18347028499556473, \"experiment\": \"final\"}, {\"epoch\": 24, \"run\": 2, \"total time\": 61.786338885976875, \"train_acc\": 0.953990818298969, \"train_loss\": 0.9761515450231808, \"train_time\": 2.572059215999616, \"tta_acc\": 0.9539, \"tta_loss\": 0.97068408203125, \"tta_time\": 0.18265191599493846, \"valid_acc\": 0.9471, \"valid_loss\": 0.9804400390625, \"valid_time\": 0.18265191599493846, \"experiment\": \"final\"}, {\"epoch\": 24, \"run\": 3, \"total time\": 61.781463586994505, \"train_acc\": 0.9569507087628866, \"train_loss\": 0.9721518742669489, \"train_time\": 2.5763632460002555, \"tta_acc\": 0.9506, \"tta_loss\": 0.971709228515625, \"tta_time\": 0.18320887300069444, \"valid_acc\": 0.9482, \"valid_loss\": 0.9808876953125, \"valid_time\": 0.18320887300069444, \"experiment\": \"final\"}, {\"epoch\": 24, \"run\": 4, \"total time\": 61.80873339098616, \"train_acc\": 0.9562661082474226, \"train_loss\": 0.9727197332480519, \"train_time\": 2.573691127996426, \"tta_acc\": 0.949, \"tta_loss\": 0.97357724609375, \"tta_time\": 0.1826382819999708, \"valid_acc\": 0.9451, \"valid_loss\": 0.982364697265625, \"valid_time\": 0.1826382819999708, \"experiment\": \"final\"}, {\"epoch\": 24, \"run\": 5, \"total time\": 61.78838591801468, \"train_acc\": 0.9554002899484536, \"train_loss\": 0.973067352452229, \"train_time\": 2.5750719900024706, \"tta_acc\": 0.9506, \"tta_loss\": 0.97138369140625, \"tta_time\": 0.1822957199983648, \"valid_acc\": 0.9461, \"valid_loss\": 0.98204267578125, \"valid_time\": 0.1822957199983648, \"experiment\": \"final\"}]}};\n",
              "      var embedOpt = {\"mode\": \"vega-lite\"};\n",
              "\n",
              "      function showError(el, error){\n",
              "          el.innerHTML = ('<div class=\"error\" style=\"color:red;\">'\n",
              "                          + '<p>JavaScript Error: ' + error.message + '</p>'\n",
              "                          + \"<p>This usually means there's a typo in your chart specification. \"\n",
              "                          + \"See the javascript console for the full traceback.</p>\"\n",
              "                          + '</div>');\n",
              "          throw error;\n",
              "      }\n",
              "      const el = document.getElementById('altair-viz');\n",
              "      vegaEmbed(\"#altair-viz\", spec, embedOpt)\n",
              "        .catch(error => showError(el, error));\n",
              "    })(vegaEmbed);\n",
              "\n",
              "  </script>\n",
              "</body>\n",
              "</html>"
            ],
            "text/plain": [
              "alt.HConcatChart(...)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tExlB_cQiADd",
        "colab_type": "text"
      },
      "source": [
        "Despite the lack of tuning of the various extra hyperparameters of the final training setup for longer runs, it appears to maintain a healthy lead over the baseline even out to 100 epochs of training and approximate convergence. The final TTA accuracy of our little 9-layer ResNet at 80 epochs is 96.1% even though we never optimised anything for training above 94% accuracy! We could presumably go quite a bit higher with proper hyperparameter optimisation. \n",
        "\n",
        "It appears that 96% accuracy is reached in about 70 epochs and 3 minutes of total training time, answering a question that I've been asked several times by people who (perhaps rightly) believe that the 94% threshold of DAWNBench is too low. Note that we have made almost no attempt to optimise the 96% time and we would expect it to come down considerably from here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WXPU8vZJlgpC",
        "colab_type": "text"
      },
      "source": [
        "### Final thoughts"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f9qabx_UlkoN",
        "colab_type": "text"
      },
      "source": [
        "Thanks to everyone who contributed to, supported or provided feedback on the project. Special thanks to Sam Davis, to Thomas Read for his work last summer on what became the post on weight decay and to everyone at Myrtle.\n",
        "\n",
        "It has been tremendous fun working on this project, exploring dynamics of neural network training and extending the work of others to bring training times to a level where rapid experimentation becomes possible. I hope that the reader will find this useful in their work and believe that training times have a long way to fall yet (or accuracies improve if that's your thing!) through further algorithmic developments.\n",
        "\n",
        "At the outset of the series I half joked that if we could achieve 100% compute efficiency, training should take 40s. I would have been surprised to find that target surpassed by the end of the series with compute efficiency little better than it ever was! There is much scope for improvement on that front as well."
      ]
    }
  ]
}