mead_2_pytorch.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "mead-2-pytorch.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dpressel/mead-tutorials/blob/master/mead_2_pytorch.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ObdkwD8V0cb",
        "colab_type": "text"
      },
      "source": [
        "# Tagging for Named Entity Recognition with MEAD\n",
        "\n",
        "In this example, we will use [mead-baseline](https://github.com/dpressel/mead-baseline) with PyTorch as a backend to create deep learning tagger models using a `CNN-BiLSTM-CRF` architecture and look at the idiomatic way that this model is defined in Baseline and how we can achieve other models by switching out different components.\n",
        "\n",
        "We will train a tagger on the CONLL2003 dataset and use `baseline.services` to reload the model and use it to predict a sample.\n",
        "\n",
        "Finally, we will use an alternate model for training, using BERT-Base and fine-tuning it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VsubIBXXrozS",
        "colab_type": "code",
        "outputId": "96546252-3760-4ebb-9129-63f5a36c0393",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "!pip install wheel\n",
        "!pip install torch\n",
        "!pip install mead-baseline[yaml]"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: wheel in /usr/local/lib/python3.6/dist-packages (0.34.2)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.5.0+cu101)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch) (0.16.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch) (1.18.4)\n",
            "Collecting mead-baseline[yaml]\n",
            "  Using cached https://files.pythonhosted.org/packages/bf/fb/f81e8cfd141c900729f609db9ae77699f6c684a5c91205a9f5bcbaa9fdca/mead_baseline-2.0.1-py3-none-any.whl\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from mead-baseline[yaml]) (1.18.4)\n",
            "Requirement already satisfied: mead-layers in /usr/local/lib/python3.6/dist-packages (from mead-baseline[yaml]) (2.0.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from mead-baseline[yaml]) (1.12.0)\n",
            "Requirement already satisfied: pyyaml; extra == \"yaml\" in /usr/local/lib/python3.6/dist-packages (from mead-baseline[yaml]) (3.13)\n",
            "Installing collected packages: mead-baseline\n",
            "Successfully installed mead-baseline-2.0.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LVyy90xxXONp",
        "colab_type": "text"
      },
      "source": [
        "For starters, we are just going to try to train a basic tagger model using a JSON configuration file.  In the previous tutorial, we showed YAML examples which are a bit easier to digest, but in this case, the example we want to run is in JSON.  Other than the format, its laid out exactly the same as before.  The example we will use is [from the repository](https://github.com/dpressel/mead-baseline/blob/master/mead/config/conll.json).\n",
        "\n",
        "For starters, we will look at the \"features\" definition\n",
        "\n",
        "## Features\n",
        "```\n",
        "\"features\": [\n",
        "    {\n",
        "      \"name\": \"word\",\n",
        "      \"vectorizer\": {\n",
        "        \"type\": \"dict1d\",\n",
        "        \"fields\": \"text\",\n",
        "        \"transform\": \"baseline.lowercase\"\n",
        "      },\n",
        "      \"embeddings\": {\n",
        "        \"label\": \"glove-6B-100\"\n",
        "      }\n",
        "    },\n",
        "    {\n",
        "      \"name\": \"senna\",\n",
        "      \"vectorizer\": {\n",
        "        \"type\": \"dict1d\",\n",
        "        \"fields\": \"text\",\n",
        "        \"transform\": \"baseline.lowercase\"\n",
        "      },\n",
        "      \"embeddings\": {\n",
        "        \"label\": \"senna\"\n",
        "      }\n",
        "    },\n",
        "    {\n",
        "      \"name\": \"char\",\n",
        "      \"vectorizer\": {\n",
        "        \"type\": \"dict2d\"\n",
        "      },\n",
        "      \"embeddings\": { \"dsz\": 30, \"wsz\": 30, \"type\": \"char-conv\" }\n",
        "    }\n",
        "\n",
        "```\n",
        "\n",
        "### Vectorizers\n",
        "\n",
        "You might have noticed that the vectorizers are defined as a different type than in our previous tutorial.  Here, there are 2 types of vectorizers that are referenced, `dict1d` and `dict2d`.  To understand what's going on here, lets first take a glance at our training data:\n",
        "\n",
        "```\n",
        "EU NNP I-NP S-ORG\n",
        "rejects VBZ I-VP O\n",
        "German JJ I-NP S-MISC\n",
        "call NN I-NP O\n",
        "to TO I-VP O\n",
        "boycott VB I-VP O\n",
        "British JJ I-NP S-MISC\n",
        "lamb NN I-NP O\n",
        ". . O O\n",
        "\n",
        "Peter NNP I-NP B-PER\n",
        "Blackburn NNP I-NP E-PER\n",
        "\n",
        "BRUSSELS NNP I-NP S-LOC\n",
        "1996-08-22 CD I-NP O\n",
        "\n",
        "The DT I-NP O\n",
        "European NNP I-NP B-ORG\n",
        "Commission NNP I-NP E-ORG\n",
        "said VBD I-VP O\n",
        "\n",
        "```\n",
        "\n",
        "This is a common file format for tagging, parsing and some other tasks.  It can be space or tab-delimited with one feature per column, and the last column represents the target label we wish to learn to predict.\n",
        "\n",
        "The second column here contains the Part-of-Speech chunks in IOB1 format.  For our example run, this is not important at all.  Since this is deep learning and we wish to learn good representations from just the surface data we are going to ignore this feature entirely and try to learn to predict the last column only from the surface text.  The final column here is in IOBES format, which is a specific formulation of tags known to work well for NER.  The idea is that single word spans will be annotated `S-<label>`, two-word spans will be annotated `B-<label> E-<label>` and any larger length span is annotated `B-<label> I-<label>+ E-<label>`. Anything that is not one of the labels of interest is annotated as `O` (short for Outside).\n",
        "\n",
        "### Embeddings\n",
        "\n",
        "The first 2 features defined here are using 2 sets of word embeddings, which are identified here as `word` and `senna`, and referencing `glove-6B-100` and `senna` pretrained word embeddings, respectively.  These are pretty well-known word embeddings, both of which perform quite well on the CONLL2003 Named Entity Recognition task.  The way we have defined them, they will be concatenated together by word.  The `senna` embeddings are 50-dimensional, and the `glove-6B-100` are 100-dimensional, so together this yields a 150-dimensional word vector.  In addition, we have another feature `char` that is of type `char-conv` and this is actually where the first name in our `CNN-BiLSTM-CRF` comes from.  The `char-conv` type embedding is a convolution over the characters in a word, followed by max-over-time pooling, and followed by one or more \"gating\" layers, defined as either a highway layer or a residual connection (with the default being the latter).  This produces a fixed dimensional vector of size `wsz` above (30 in this example).  We will also concatenate this to our `word` and `senna` features yielding a 180-dimensional hidden unit vector.  Remember from the last tutorial that the way this works is that these features are placed into an `EmbeddingsStack` and the default `reduction`, `concat` is applied to them.\n",
        "\n",
        "## Model\n",
        "\n",
        "Here is the `model` block of the JSON file:\n",
        "\n",
        "```\n",
        "  \"model\": {\n",
        "    \"type\": \"default\",\n",
        "    \"cfiltsz\": [\n",
        "      3\n",
        "    ],\n",
        "    \"hsz\": 400,\n",
        "    \"dropout\": 0.5,\n",
        "    \"dropin\": {\"word\": 0.1,\"senna\": 0.1},\n",
        "    \"rnntype\": \"blstm\",\n",
        "    \"layers\": 1,\n",
        "    \"constrain_decode\": true,\n",
        "    \"crf\": 1\n",
        "  },\n",
        "```\n",
        "\n",
        "This defines the second and third part of the model architecture name we are using here (`BiLSTM` is given as `blstm` under the `rnntype`, and the `crf` parameter is given as well).  Once the input features are projected through the embeddings and concatenated together, this embedding output is sent into an encoder layer, defined here as a `BiLSTM`.  The output of this layer will be in the label dimension space -- in other words, the number of output units is equal to the number of tag-pieces (IBES x Labels + O).\n",
        "\n",
        "The `hsz` specifies the total number of hidden units that will be used for the `BiLSTM` encoder -- 200 for the forward direction, and 200 for the backward direction.\n",
        "\n",
        "It turns out that a rather flat model works just fine for the `CONLL2003` NER dataset -- a single layer of `BiLSTM`s is all we need!\n",
        "\n",
        "Once we have encoded or \"transduced\" our input data into our our label space output, we decode it to form an output.\n",
        "\n",
        "The simplest form of decoding is greedy -- we are given the label output space from our encoder, so just pick the `argmax` at each timestep to get the output.  A linear-chain CRF is an alternative approach, where we learn a transition matrix from a current label state to a future label state, and we apply this to our encoder output to yield a final, globally coherent output, learning the most likely path through a sequence.  The `crf: 1` above is a `boolean` and it specifies that we would like to use a `CRF` to decode.\n",
        "\n",
        "One thing that is worth noticing.  Previously, we described the way in which the labels are annotated in IOBES format.  Some transitions through the label space are not possible, and we could actually place a hard-constraint on our model to prevent it from generating those transitions.  For example, if we have a single token of some label surrounded by tokens of a different label, its impossible for the prefix of that token to be a `B-`. We can see in the example above, for instance, that the first token is a single token, and so its prefixed with `S-`:\n",
        "\n",
        "```\n",
        "EU NNP I-NP S-ORG\n",
        "```\n",
        "\n",
        "Setting `constrain_decode` to `true` (or `1` or anything \"truthy\") ensures that our model cannot generate invalid sequences.  In fact, it turns out that we can usually get nearly the same performance from a model with greedy, constrained decoding as we can from using a CRF layer for the output!\n",
        "\n",
        "## Training\n",
        "\n",
        "The training portion is quite similar to in our previous examples.  A key difference is that `CNN-BiLSTM-CRF` models are often trained with relatively small batches and using SGD with momentum for a large number of epochs with early stopping on F1, which is the target metric typically used for NER:\n",
        "\n",
        "```\n",
        "  \"train\": {\n",
        "    \"batchsz\": 10,\n",
        "    \"epochs\": 100,\n",
        "    \"optim\": \"sgd\",\n",
        "    \"eta\": 0.015,\n",
        "    \"mom\": 0.9,\n",
        "    \"patience\": 40,\n",
        "    \"early_stopping_metric\": \"f1\",\n",
        "    \"clip\": 5.0,\n",
        "    \"span_type\": \"iobes\"\n",
        "  }\n",
        "```\n",
        "\n",
        "This training procedure will run up to 100 epochs (unless it does not improve for `40` epochs due to the `patience` criteria), will take quite a bit of time.\n",
        "\n",
        "Since the `epoch` parameter is inside the `train` block, we can override its values as `--x:train.epoch`.  So for instance, if you do not feel like waiting, modify the command line below to pass in `--x:train.epoch 10` to make it run for 10 epochs, which should be good enough for the purposes of the tutorial."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JoVDLd3DekhA",
        "colab_type": "code",
        "outputId": "001d4097-2b54-49fa-c576-cccf4925e572",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!mead-train --config https://raw.githubusercontent.com/dpressel/mead-baseline/master/mead/config/conll.json"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading config file '/tmp/tmp7ks3__py'\n",
            "Reading config file '/usr/local/lib/python3.6/dist-packages/mead/config/logging.json'\n",
            "No file found '/usr/local/l...', loading as string\n",
            "\u001b[33;1mWarning: no mead-settings file was found at [/usr/local/lib/python3.6/dist-packages/mead/config/mead-settings.json]\u001b[0m\n",
            "Reading config file '/usr/local/lib/python3.6/dist-packages/mead/config/datasets.json'\n",
            "Reading config file '/usr/local/lib/python3.6/dist-packages/mead/config/embeddings.json'\n",
            "Reading config file '/usr/local/lib/python3.6/dist-packages/mead/config/vecs.json'\n",
            "Task: [tagger]\n",
            "using /root/.bl-data as data/embeddings cache\n",
            "Progress: [========================================] 100%\n",
            "extracting file..\n",
            "downloaded data saved in /root/.bl-data/80b0a839a7edd4c99f54537aa83327340592a4e8\n",
            "[train file]: /root/.bl-data/80b0a839a7edd4c99f54537aa83327340592a4e8/eng.train.iobes\n",
            "[valid file]: /root/.bl-data/80b0a839a7edd4c99f54537aa83327340592a4e8/eng.testa.iobes\n",
            "[test file]: /root/.bl-data/80b0a839a7edd4c99f54537aa83327340592a4e8/eng.testb.iobes\n",
            "Progress: [========================================] 100%\n",
            "extracting file..\n",
            "downloaded data saved in /root/.bl-data/a483a44d4414a18c7b10b36dd6daa59195eb292b\n",
            "embedding file location: /root/.bl-data/a483a44d4414a18c7b10b36dd6daa59195eb292b\n",
            "Progress: [========================================] 100%\n",
            "extracting file..\n",
            "downloaded data saved in /root/.bl-data/5258af089f40f9071bbb8f4f5ac2b2f35b18fea2\n",
            "embedding file location: /root/.bl-data/5258af089f40f9071bbb8f4f5ac2b2f35b18fea2\n",
            "model file [./tagger/tagger-model-1199.pyt]\n",
            "Doing early stopping on [f1] with patience [40]\n",
            "reporting [<bound method EpochReportingHook.step of <baseline.reporting.LoggingReporting object at 0x7f5305c5da20>>]\n",
            "Calling model <function register_model.<locals>.create at 0x7f52bac1b0d0>\n",
            "Setting span type iobes\n",
            "sgd(eta=0.015000, mom=0.900000, wd=0.000000)\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 1, \"phase\": \"Train\", \"avg_loss\": 1.4180291367832318}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 1, \"phase\": \"Valid\", \"acc\": 0.9656442669355152, \"f1\": 0.8699255751014884}\n",
            "New best 0.870\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 2, \"phase\": \"Train\", \"avg_loss\": 0.8103339549627038}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 2, \"phase\": \"Valid\", \"acc\": 0.9776648958858428, \"f1\": 0.9078870170145}\n",
            "New best 0.908\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 3, \"phase\": \"Train\", \"avg_loss\": 0.683124783812297}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 3, \"phase\": \"Valid\", \"acc\": 0.9791577804490287, \"f1\": 0.9150491555331485}\n",
            "New best 0.915\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 4, \"phase\": \"Train\", \"avg_loss\": 0.5849697273281919}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 4, \"phase\": \"Valid\", \"acc\": 0.9828027453565473, \"f1\": 0.926329921127706}\n",
            "New best 0.926\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 5, \"phase\": \"Train\", \"avg_loss\": 0.5251844100172727}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 5, \"phase\": \"Valid\", \"acc\": 0.9841017488076311, \"f1\": 0.9320127580997147}\n",
            "New best 0.932\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 6, \"phase\": \"Train\", \"avg_loss\": 0.465776717144567}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 6, \"phase\": \"Valid\", \"acc\": 0.9850323781457211, \"f1\": 0.9354811503753057}\n",
            "New best 0.935\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 7, \"phase\": \"Train\", \"avg_loss\": 0.4494532032233282}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 7, \"phase\": \"Valid\", \"acc\": 0.9852262592578231, \"f1\": 0.9357196874212251}\n",
            "New best 0.936\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 8, \"phase\": \"Train\", \"avg_loss\": 0.4105625636231139}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 8, \"phase\": \"Valid\", \"acc\": 0.9871650703788437, \"f1\": 0.9411170928667564}\n",
            "New best 0.941\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 9, \"phase\": \"Train\", \"avg_loss\": 0.38571479084177773}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 9, \"phase\": \"Valid\", \"acc\": 0.9866803675985886, \"f1\": 0.9403851004792735}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 10, \"phase\": \"Train\", \"avg_loss\": 0.3663359766483724}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 10, \"phase\": \"Valid\", \"acc\": 0.9860017837062314, \"f1\": 0.9391596638655462}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 11, \"phase\": \"Train\", \"avg_loss\": 0.3454117663073395}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 11, \"phase\": \"Valid\", \"acc\": 0.9874946682694172, \"f1\": 0.9445146080660098}\n",
            "New best 0.945\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 12, \"phase\": \"Train\", \"avg_loss\": 0.3155369599649271}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 12, \"phase\": \"Valid\", \"acc\": 0.9869324130443212, \"f1\": 0.9425519387669274}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 13, \"phase\": \"Train\", \"avg_loss\": 0.30725143225735174}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 13, \"phase\": \"Valid\", \"acc\": 0.9870099654891621, \"f1\": 0.9427445720513035}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 14, \"phase\": \"Train\", \"avg_loss\": 0.2923808529072901}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 14, \"phase\": \"Valid\", \"acc\": 0.9871262941564233, \"f1\": 0.9425712604052805}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 15, \"phase\": \"Train\", \"avg_loss\": 0.2702169031184402}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 15, \"phase\": \"Valid\", \"acc\": 0.987572220714258, \"f1\": 0.9445378151260504}\n",
            "New best 0.945\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 16, \"phase\": \"Train\", \"avg_loss\": 0.26131078621106246}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 16, \"phase\": \"Valid\", \"acc\": 0.9873783396021559, \"f1\": 0.9429122306982982}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 17, \"phase\": \"Train\", \"avg_loss\": 0.26091671100576636}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 17, \"phase\": \"Valid\", \"acc\": 0.9878242661599906, \"f1\": 0.9449533574249936}\n",
            "New best 0.945\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 18, \"phase\": \"Train\", \"avg_loss\": 0.24606848787223584}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 18, \"phase\": \"Valid\", \"acc\": 0.98776610182636, \"f1\": 0.9449109842122944}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 19, \"phase\": \"Train\", \"avg_loss\": 0.2409954936897758}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 19, \"phase\": \"Valid\", \"acc\": 0.9876303850478886, \"f1\": 0.9453781512605042}\n",
            "New best 0.945\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 20, \"phase\": \"Train\", \"avg_loss\": 0.23356218075285873}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 20, \"phase\": \"Valid\", \"acc\": 0.9876109969366784, \"f1\": 0.9439165895905155}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 21, \"phase\": \"Train\", \"avg_loss\": 0.22556421086003461}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 21, \"phase\": \"Valid\", \"acc\": 0.9881538640505642, \"f1\": 0.9461118116855822}\n",
            "New best 0.946\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 22, \"phase\": \"Train\", \"avg_loss\": 0.2080525249158065}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 22, \"phase\": \"Valid\", \"acc\": 0.9878048780487805, \"f1\": 0.9445564516129032}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 23, \"phase\": \"Train\", \"avg_loss\": 0.20894893171479387}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 23, \"phase\": \"Valid\", \"acc\": 0.9878630423824111, \"f1\": 0.9463324360699865}\n",
            "New best 0.946\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 24, \"phase\": \"Train\", \"avg_loss\": 0.19577316360675284}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 24, \"phase\": \"Valid\", \"acc\": 0.9879599829384621, \"f1\": 0.944645107097858}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 25, \"phase\": \"Train\", \"avg_loss\": 0.19560043567281174}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 25, \"phase\": \"Valid\", \"acc\": 0.9878242661599906, \"f1\": 0.9442485306465155}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 26, \"phase\": \"Train\", \"avg_loss\": 0.1933228118876349}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 26, \"phase\": \"Valid\", \"acc\": 0.9879405948272519, \"f1\": 0.945613005949887}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 27, \"phase\": \"Train\", \"avg_loss\": 0.1870302246213194}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 27, \"phase\": \"Valid\", \"acc\": 0.9876497731590989, \"f1\": 0.9440806045340051}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 28, \"phase\": \"Train\", \"avg_loss\": 0.17229326149029647}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 28, \"phase\": \"Valid\", \"acc\": 0.9882508046066152, \"f1\": 0.9472268907563024}\n",
            "New best 0.947\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 29, \"phase\": \"Train\", \"avg_loss\": 0.17152495739170975}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 29, \"phase\": \"Valid\", \"acc\": 0.9881150878281438, \"f1\": 0.9468647695794511}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 30, \"phase\": \"Train\", \"avg_loss\": 0.17194625640600666}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 30, \"phase\": \"Valid\", \"acc\": 0.9881150878281438, \"f1\": 0.9460322797579018}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 31, \"phase\": \"Train\", \"avg_loss\": 0.1636051531261512}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 31, \"phase\": \"Valid\", \"acc\": 0.9883089689402459, \"f1\": 0.9470059628789786}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 32, \"phase\": \"Train\", \"avg_loss\": 0.16801185395785453}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 32, \"phase\": \"Valid\", \"acc\": 0.9879987591608825, \"f1\": 0.9463340891912321}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 33, \"phase\": \"Train\", \"avg_loss\": 0.15769582205241994}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 33, \"phase\": \"Valid\", \"acc\": 0.98776610182636, \"f1\": 0.9459005376344086}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 34, \"phase\": \"Train\", \"avg_loss\": 0.1541334125397847}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 34, \"phase\": \"Valid\", \"acc\": 0.9879405948272519, \"f1\": 0.946307033022435}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 35, \"phase\": \"Train\", \"avg_loss\": 0.15373572014962025}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 35, \"phase\": \"Valid\", \"acc\": 0.9876497731590989, \"f1\": 0.9452527121352283}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 36, \"phase\": \"Train\", \"avg_loss\": 0.15369669874106898}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 36, \"phase\": \"Valid\", \"acc\": 0.988231416495405, \"f1\": 0.9477756286266925}\n",
            "New best 0.948\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 37, \"phase\": \"Train\", \"avg_loss\": 0.1372351335507948}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 37, \"phase\": \"Valid\", \"acc\": 0.9877854899375703, \"f1\": 0.9467058329836341}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 38, \"phase\": \"Train\", \"avg_loss\": 0.13112455601799058}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 38, \"phase\": \"Valid\", \"acc\": 0.9875334444918376, \"f1\": 0.9444677812316223}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 39, \"phase\": \"Train\", \"avg_loss\": 0.13520802213244523}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 39, \"phase\": \"Valid\", \"acc\": 0.9882120283841949, \"f1\": 0.9468022522901084}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 40, \"phase\": \"Train\", \"avg_loss\": 0.1363733505289475}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 40, \"phase\": \"Valid\", \"acc\": 0.9880181472720928, \"f1\": 0.9468031548917604}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 41, \"phase\": \"Train\", \"avg_loss\": 0.12290764156709344}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 41, \"phase\": \"Valid\", \"acc\": 0.9878048780487805, \"f1\": 0.9460232049772995}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 42, \"phase\": \"Train\", \"avg_loss\": 0.12668175636081302}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 42, \"phase\": \"Valid\", \"acc\": 0.988134475939354, \"f1\": 0.9475894506971274}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 43, \"phase\": \"Train\", \"avg_loss\": 0.13233704584345196}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 43, \"phase\": \"Valid\", \"acc\": 0.9880569234945131, \"f1\": 0.9467047747141897}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 44, \"phase\": \"Train\", \"avg_loss\": 0.12309898997004412}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 44, \"phase\": \"Valid\", \"acc\": 0.9881538640505642, \"f1\": 0.9472357586960175}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 45, \"phase\": \"Train\", \"avg_loss\": 0.12016550079248584}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 45, \"phase\": \"Valid\", \"acc\": 0.9882701927178255, \"f1\": 0.947244623655914}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 46, \"phase\": \"Train\", \"avg_loss\": 0.12495588445650907}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 46, \"phase\": \"Valid\", \"acc\": 0.9884059094962969, \"f1\": 0.9488645920941969}\n",
            "New best 0.949\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 47, \"phase\": \"Train\", \"avg_loss\": 0.11555347294748425}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 47, \"phase\": \"Valid\", \"acc\": 0.9882120283841949, \"f1\": 0.9458392812158872}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 48, \"phase\": \"Train\", \"avg_loss\": 0.12215668179556624}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 48, \"phase\": \"Valid\", \"acc\": 0.9879599829384621, \"f1\": 0.945671341002603}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 49, \"phase\": \"Train\", \"avg_loss\": 0.11761283581771516}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 49, \"phase\": \"Valid\", \"acc\": 0.9888712241653418, \"f1\": 0.9501135694456129}\n",
            "New best 0.950\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 50, \"phase\": \"Train\", \"avg_loss\": 0.11621083555851472}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 50, \"phase\": \"Valid\", \"acc\": 0.988134475939354, \"f1\": 0.9469435802572943}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 51, \"phase\": \"Train\", \"avg_loss\": 0.11190945639215849}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 51, \"phase\": \"Valid\", \"acc\": 0.9884446857187172, \"f1\": 0.9491582491582492}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 52, \"phase\": \"Train\", \"avg_loss\": 0.11340649393960398}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 52, \"phase\": \"Valid\", \"acc\": 0.9882895808290356, \"f1\": 0.9471206389239176}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 53, \"phase\": \"Train\", \"avg_loss\": 0.11172299752480414}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 53, \"phase\": \"Valid\", \"acc\": 0.9882895808290356, \"f1\": 0.9461913569867161}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 54, \"phase\": \"Train\", \"avg_loss\": 0.11851042649202732}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 54, \"phase\": \"Valid\", \"acc\": 0.9886191787196091, \"f1\": 0.9489418878065167}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 55, \"phase\": \"Train\", \"avg_loss\": 0.10762033507566342}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 55, \"phase\": \"Valid\", \"acc\": 0.9884834619411377, \"f1\": 0.9488278295941517}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 56, \"phase\": \"Train\", \"avg_loss\": 0.10715105204956335}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 56, \"phase\": \"Valid\", \"acc\": 0.9888130598317112, \"f1\": 0.9499664204163869}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 57, \"phase\": \"Train\", \"avg_loss\": 0.10653921805988958}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 57, \"phase\": \"Valid\", \"acc\": 0.9882895808290356, \"f1\": 0.9478019668824073}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 58, \"phase\": \"Train\", \"avg_loss\": 0.10509884136890123}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 58, \"phase\": \"Valid\", \"acc\": 0.988134475939354, \"f1\": 0.94776684330053}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 59, \"phase\": \"Train\", \"avg_loss\": 0.10210703538478934}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 59, \"phase\": \"Valid\", \"acc\": 0.988231416495405, \"f1\": 0.947996303452911}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 60, \"phase\": \"Train\", \"avg_loss\": 0.10172095760098622}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 60, \"phase\": \"Valid\", \"acc\": 0.9878824304936213, \"f1\": 0.9473241332884551}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 61, \"phase\": \"Train\", \"avg_loss\": 0.09697113758897766}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 61, \"phase\": \"Valid\", \"acc\": 0.9881732521617744, \"f1\": 0.9471558397845843}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 62, \"phase\": \"Train\", \"avg_loss\": 0.0987837919347402}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 62, \"phase\": \"Valid\", \"acc\": 0.9881732521617744, \"f1\": 0.9471472985463406}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 63, \"phase\": \"Train\", \"avg_loss\": 0.10032212805429969}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 63, \"phase\": \"Valid\", \"acc\": 0.9879405948272519, \"f1\": 0.9452560873215784}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 64, \"phase\": \"Train\", \"avg_loss\": 0.09659942595880974}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 64, \"phase\": \"Valid\", \"acc\": 0.9881926402729846, \"f1\": 0.946793309237623}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 65, \"phase\": \"Train\", \"avg_loss\": 0.09586916011277978}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 65, \"phase\": \"Valid\", \"acc\": 0.9883089689402459, \"f1\": 0.9477223062699613}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 66, \"phase\": \"Train\", \"avg_loss\": 0.0885829213146703}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 66, \"phase\": \"Valid\", \"acc\": 0.9883671332738765, \"f1\": 0.947412634408602}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 67, \"phase\": \"Train\", \"avg_loss\": 0.09217916220033888}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 67, \"phase\": \"Valid\", \"acc\": 0.9881926402729846, \"f1\": 0.9474569146700293}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 68, \"phase\": \"Train\", \"avg_loss\": 0.09507047994053934}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 68, \"phase\": \"Valid\", \"acc\": 0.9877854899375703, \"f1\": 0.9460050462573592}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 69, \"phase\": \"Train\", \"avg_loss\": 0.09507204451144943}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 69, \"phase\": \"Valid\", \"acc\": 0.9882701927178255, \"f1\": 0.9479875640702463}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 70, \"phase\": \"Train\", \"avg_loss\": 0.08633048328388458}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 70, \"phase\": \"Valid\", \"acc\": 0.9876885493815193, \"f1\": 0.9459436738125263}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 71, \"phase\": \"Train\", \"avg_loss\": 0.08827150877350404}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 71, \"phase\": \"Valid\", \"acc\": 0.9882120283841949, \"f1\": 0.9480759536212401}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 72, \"phase\": \"Train\", \"avg_loss\": 0.09090014574773242}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 72, \"phase\": \"Valid\", \"acc\": 0.98869673116445, \"f1\": 0.949331988908495}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 73, \"phase\": \"Train\", \"avg_loss\": 0.09049012186103181}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 73, \"phase\": \"Valid\", \"acc\": 0.9884640738299275, \"f1\": 0.9486856470983456}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 74, \"phase\": \"Train\", \"avg_loss\": 0.0877947760174585}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 74, \"phase\": \"Valid\", \"acc\": 0.98869673116445, \"f1\": 0.948782535684299}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 75, \"phase\": \"Train\", \"avg_loss\": 0.09118724878863291}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 75, \"phase\": \"Valid\", \"acc\": 0.9887548954980806, \"f1\": 0.9488278295941517}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 76, \"phase\": \"Train\", \"avg_loss\": 0.08473504106770521}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 76, \"phase\": \"Valid\", \"acc\": 0.9882120283841949, \"f1\": 0.9469525010508617}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 77, \"phase\": \"Train\", \"avg_loss\": 0.08591796568694704}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 77, \"phase\": \"Valid\", \"acc\": 0.9883089689402459, \"f1\": 0.9468728984532616}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 78, \"phase\": \"Train\", \"avg_loss\": 0.08354226775909075}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 78, \"phase\": \"Valid\", \"acc\": 0.9884252976075071, \"f1\": 0.9484484063577494}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 79, \"phase\": \"Train\", \"avg_loss\": 0.08136711815575615}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 79, \"phase\": \"Valid\", \"acc\": 0.9882508046066152, \"f1\": 0.9473242039821895}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 80, \"phase\": \"Train\", \"avg_loss\": 0.09005040211599308}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 80, \"phase\": \"Valid\", \"acc\": 0.9886773430532397, \"f1\": 0.9495628782784129}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 81, \"phase\": \"Train\", \"avg_loss\": 0.08336475118379401}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 81, \"phase\": \"Valid\", \"acc\": 0.9888906122765521, \"f1\": 0.950147120638924}\n",
            "New best 0.950\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 82, \"phase\": \"Train\", \"avg_loss\": 0.081114359888753}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 82, \"phase\": \"Valid\", \"acc\": 0.9882895808290356, \"f1\": 0.9482889094425292}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 83, \"phase\": \"Train\", \"avg_loss\": 0.0761845373704698}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 83, \"phase\": \"Valid\", \"acc\": 0.988502850052348, \"f1\": 0.9483425879185596}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 84, \"phase\": \"Train\", \"avg_loss\": 0.08130140855416235}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 84, \"phase\": \"Valid\", \"acc\": 0.9881732521617744, \"f1\": 0.9466700260351054}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 85, \"phase\": \"Train\", \"avg_loss\": 0.0878623530302397}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 85, \"phase\": \"Valid\", \"acc\": 0.9883671332738765, \"f1\": 0.9478641103262697}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 86, \"phase\": \"Train\", \"avg_loss\": 0.08152952820894115}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 86, \"phase\": \"Valid\", \"acc\": 0.9882895808290356, \"f1\": 0.948271511481201}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 87, \"phase\": \"Train\", \"avg_loss\": 0.07356415508060138}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 87, \"phase\": \"Valid\", \"acc\": 0.988599790608399, \"f1\": 0.9483425879185596}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 88, \"phase\": \"Train\", \"avg_loss\": 0.08292405678800067}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 88, \"phase\": \"Valid\", \"acc\": 0.9889875528326031, \"f1\": 0.9501767379229087}\n",
            "New best 0.950\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 89, \"phase\": \"Train\", \"avg_loss\": 0.07716355641514833}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 89, \"phase\": \"Valid\", \"acc\": 0.9884834619411377, \"f1\": 0.9478991596638656}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 90, \"phase\": \"Train\", \"avg_loss\": 0.074730506699189}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 90, \"phase\": \"Valid\", \"acc\": 0.9889487766101827, \"f1\": 0.9491126251156532}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 91, \"phase\": \"Train\", \"avg_loss\": 0.07430639307909014}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 91, \"phase\": \"Valid\", \"acc\": 0.9886385668308193, \"f1\": 0.9481917577796467}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 92, \"phase\": \"Train\", \"avg_loss\": 0.07564265321762814}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 92, \"phase\": \"Valid\", \"acc\": 0.9888130598317112, \"f1\": 0.9493947545393409}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 93, \"phase\": \"Train\", \"avg_loss\": 0.06328259530552495}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 93, \"phase\": \"Valid\", \"acc\": 0.9889487766101827, \"f1\": 0.9496257673870995}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 94, \"phase\": \"Train\", \"avg_loss\": 0.07652842848324606}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 94, \"phase\": \"Valid\", \"acc\": 0.9885804024971887, \"f1\": 0.9490841875315074}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 95, \"phase\": \"Train\", \"avg_loss\": 0.07010883609180808}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 95, \"phase\": \"Valid\", \"acc\": 0.9887355073868703, \"f1\": 0.9485454851185472}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 96, \"phase\": \"Train\", \"avg_loss\": 0.07379614505481465}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 96, \"phase\": \"Valid\", \"acc\": 0.9886385668308193, \"f1\": 0.9484639919422528}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 97, \"phase\": \"Train\", \"avg_loss\": 0.06550683828831787}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 97, \"phase\": \"Valid\", \"acc\": 0.9884640738299275, \"f1\": 0.9472621751198587}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 98, \"phase\": \"Train\", \"avg_loss\": 0.06850729091089669}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 98, \"phase\": \"Valid\", \"acc\": 0.9882701927178255, \"f1\": 0.9471558397845843}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 99, \"phase\": \"Train\", \"avg_loss\": 0.0741865034104748}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 99, \"phase\": \"Valid\", \"acc\": 0.9880181472720928, \"f1\": 0.9462528387585163}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 100, \"phase\": \"Train\", \"avg_loss\": 0.06684667571660594}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 100, \"phase\": \"Valid\", \"acc\": 0.9882120283841949, \"f1\": 0.9467306235799041}\n",
            "Best performance on f1: 0.950 at epoch 87\n",
            "Reloading best checkpoint\n",
            "Setting span type iobes\n",
            "sgd(eta=0.015000, mom=0.900000, wd=0.000000)\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 0, \"phase\": \"Test\", \"acc\": 0.9804354347919256, \"f1\": 0.9154879773691657}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZfYrge62Rmu",
        "colab_type": "text"
      },
      "source": [
        "Our model got a final score of **~91.55** which is a pretty strong result on this dataset, and within [the expected results for this model](https://github.com/dpressel/mead-baseline/blob/master/docs/tagging.md#model-performance).  The resulting model checkpoint, along with the vectorizers was stored below in `tagger-1199.zip` "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V0zPgMAfyM7z",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "50cd91f1-229d-4705-dd89-0ed852a2279b"
      },
      "source": [
        "!ls"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "conllresults.conll  info.log\t\tsample_data  tagger-1199.zip\n",
            "errors.log\t    reporting-1199.log\ttagger\t     timing-1199.log\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fPaWS88johIZ",
        "colab_type": "text"
      },
      "source": [
        "## Using our trained model\n",
        "\n",
        "Now that we have created an NER tagger, lets use it to tag some sentences.\n",
        "\n",
        "`mead-baseline` has the concept of a \"service\", which is something that orchestrates underlying components like a `model` and a `vectorizer` to perform inference.  Each `mead` Task has a corresponding service object. \n",
        "\n",
        "These services can be wired up in many different ways to support remote execution (where the `model` itself is part of a remote gRPC service like TensorFlow Serving), or a local model checkpoint that was trained like in the previous example.\n",
        "\n",
        "There is an example of how to use `baseline.services` for each Task type in the `api-examples` in `mead-baseline`, but in this example, we will create the code ourselves and run inference right in the notebook.\n",
        "\n",
        "We just trained a `tagger`, and now we will reload the model with the `TaggerService` locally in Colab.\n",
        "\n",
        "For this example, we are not going to use any special tokenizer, lets just assume that the tokens are white-space delimited.  Note that for a real use-case, you want to make sure that the tokenizer used to train looks as much like the one used for inference as possible, otherwise you might find that it does worse in inference than your metrics might lead you to believe.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9MrZ7aeY1HLE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        },
        "outputId": "36d4c09a-e08d-4465-b1c0-9fbcfb4bdae3"
      },
      "source": [
        "from baseline.services import TaggerService\n",
        "CHECKPOINT_PATH = './tagger-1199.zip'\n",
        "model = TaggerService.load(CHECKPOINT_PATH, backend='pytorch')\n",
        "\n",
        "EXAMPLE_SENTENCE = \"Mr. Jones thinks Las Vegas is not as fun as NYC !\".split()\n",
        "\n",
        "model.predict(EXAMPLE_SENTENCE)\n",
        "\n"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "unzipping model\n",
            "/tmp/39751301005946022bfb0cd6d0ff42a657ccc198/tagger/tagger-model-1199.pyt\n",
            "/tmp/39751301005946022bfb0cd6d0ff42a657ccc198/tagger/vocabs-char-1199.json\n",
            "/tmp/39751301005946022bfb0cd6d0ff42a657ccc198/tagger/vocabs-senna-1199.json\n",
            "/tmp/39751301005946022bfb0cd6d0ff42a657ccc198/tagger/vocabs-word-1199.json\n",
            "Calling model <function TaggerModelBase.load at 0x7fa7829018c8>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[{'label': 'O', 'text': 'Mr.'},\n",
              "  {'label': 'S-PER', 'text': 'Jones'},\n",
              "  {'label': 'O', 'text': 'thinks'},\n",
              "  {'label': 'B-LOC', 'text': 'Las'},\n",
              "  {'label': 'E-LOC', 'text': 'Vegas'},\n",
              "  {'label': 'O', 'text': 'is'},\n",
              "  {'label': 'O', 'text': 'not'},\n",
              "  {'label': 'O', 'text': 'as'},\n",
              "  {'label': 'O', 'text': 'fun'},\n",
              "  {'label': 'O', 'text': 'as'},\n",
              "  {'label': 'S-LOC', 'text': 'NYC'},\n",
              "  {'label': 'O', 'text': '!'}]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y-ZjvZYjSOt0",
        "colab_type": "text"
      },
      "source": [
        "## Fine-Tuning BERT for NER\n",
        "\n",
        "In the last tutorial, we saw that `mead-baseline` supports BERT through its Transformer API.  We will use that again to train a model, but this time, we are going to train an NER model on the same dataset as our `CNN-BiLSTM-CRF`.\n",
        "\n",
        "As in the case of classification, when we fine-tune BERT, all we usually do on top is provide a top layer that projects from the BERT hidden dimension output (or \"bottleneck\" to the final number of units).\n",
        "\n",
        "BERT and other Transformers typically are trained with sub-word embeddings, like BPE or WordPiece.  When we go to predict a label, we have to be careful with this, as each word may be fragmented.  This fragmenting of the word into sub-words could happen inside a label, or outside of it.  How should the model compensate for this issue?\n",
        "\n",
        "The easiest thing (arguably), is to pad the sub-word suffixes and teach the model to predict the head of the sub-word.  This the approach outlined in the BERT paper, and the approach we use within MEAD.  For any word-piece suffixes, we pass in a special label `<PAD>` and then we mask this in the output.\n",
        "\n",
        "All of the metric computations in MEAD take into account that `<PAD>` labels are ignored, and thus the metrics can be as usual without any modification.\n",
        "\n",
        "Our [example config is using the cased BERT-Base model](https://raw.githubusercontent.com/dpressel/mead-baseline/master/mead/config/conll-bert.yml)\n",
        "\n",
        "The features look much the same as in our classification, again with the difference that we are using a vectorizer with the label `bert-base-cased-dict1d`.  You might be wondering where this comes from.  The answer is that, like other `label` values in MEAD, this field comes from the `vectorizer` index file, which defaults to [vecs.json](https://github.com/dpressel/mead-baseline/blob/master/mead/config/vecs.json)\n",
        "\n",
        "That `vectorizer` is defined as a tokenizer that uses a Dictionary of inputs from each line int the CONLL file to produce `WordPiece` tokens as output, and uses the pre-defined BERT-Base cased vocabulary.  Our `WordPiece` tokenization is based on, and produces identical output to its definition in the BERT official repository\n",
        "\n",
        "```\n",
        " {\n",
        "    \"label\": \"bert-base-cased-dict1d\",\n",
        "    \"type\": \"wordpiece-dict1d\",\n",
        "    \"vocab_file\": \"https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt\"\n",
        "  },\n",
        "```\n",
        "\n",
        "\n",
        "### Features and Embeddings\n",
        "```\n",
        "\n",
        "features:\n",
        " - name: word\n",
        "   vectorizer:\n",
        "     label: bert-base-cased-dict1d\n",
        "   embeddings:\n",
        "     type: tlm-words-embed \n",
        "     word_embed_type: learned-positional-w-bias\n",
        "     label: bert-base-cased-npz\n",
        "     reduction: sum-layer-norm\n",
        "     layer_norms_after: true\n",
        "     finetune: true\n",
        "     mlm: true\n",
        "\n",
        "```\n",
        "\n",
        "The reader is a bit different, and worth digging into:\n",
        "\n",
        "```\n",
        "loader:\n",
        "  reader_type: default\n",
        "  named_fields: {\"0\": \"text\", \"-1\": \"y\"}\n",
        "  label_vectorizer:\n",
        "    label: y\n",
        "    type: wordpiece-label-dict1d\n",
        "```\n",
        "\n",
        "The `named_fields` is a mapping, which assigns a dictionary field name to each column by integer.  Using negative indexing is allowed, just like in Python.  So in our example, the field in column 0 is named `text` and the last column field is named `y`.\n",
        "\n",
        "The `label_vectorizer`, which typically defaults to a `dict1d` is defined here as a special `wordpiece-label-dict1d` vectorizer -- for every WordPiece initial sub-word, this generates the label corresponding to the word, and for ever subsequent sub-word, as mentioned before, this vectorizer generates `<PAD>` labels.  To work properly, this vectorizer has to actually tokenize the surface terms to decide whether to emit the label in the file or a `<PAD>` token.\n",
        "\n",
        "### The Model\n",
        "\n",
        "We defined our `CNN-BiLSTM-CRF` model as a sequence of phases:\n",
        "1. embeddings\n",
        "1. encoding (or transduction)\n",
        "1. output to label space\n",
        "1. decoding to output labels\n",
        "\n",
        "In our code above, the embeddings themselves are the full BERT stack (minus the LM head), and we just want a linear layer on top, so there isnt a lot left to do.  The model itself just needs a linear projection to the label space.  It is possible to add a CRF decoder, but here we just employ greedy decoding:\n",
        "\n",
        "```\n",
        "model:\n",
        "  model_type: pass\n",
        "  constrain_decode: 0\n",
        "  crf: 0\n",
        "```\n",
        "\n",
        "### The Training Parameters\n",
        "\n",
        "```\n",
        "train:\n",
        "  batchsz: 32\n",
        "  epochs: 50\n",
        "  optim: adam\n",
        "  eta: 1.0e-5\n",
        "  patience: 15\n",
        "  early_stopping_metric: f1\n",
        "  clip: 5.0\n",
        "  span_type: iobes \n",
        "```\n",
        "\n",
        "In the BERT fine-tuning example, the `adam` optimizer is typically used.  Here we keep the `early_stopping_metric` the same as before (`f1`).\n",
        "\n",
        "As in the previous example, if you do not wish to wait around for the results, try modifying the command line to pass `--x:train.epochs` with some smaller number of epochs.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9fNrvpa2RpCU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "e7912d36-7ed1-4d13-a631-c51539f8e37e"
      },
      "source": [
        "!mead-train --config https://raw.githubusercontent.com/dpressel/mead-baseline/master/mead/config/conll-bert.yml"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading config file '/tmp/tmp4l63kpc1'\n",
            "Reading config file '/usr/local/lib/python3.6/dist-packages/mead/config/logging.json'\n",
            "No file found '/usr/local/l...', loading as string\n",
            "\u001b[33;1mWarning: no mead-settings file was found at [/usr/local/lib/python3.6/dist-packages/mead/config/mead-settings.json]\u001b[0m\n",
            "Reading config file '/usr/local/lib/python3.6/dist-packages/mead/config/datasets.json'\n",
            "Reading config file '/usr/local/lib/python3.6/dist-packages/mead/config/embeddings.json'\n",
            "Reading config file '/usr/local/lib/python3.6/dist-packages/mead/config/vecs.json'\n",
            "Task: [tagger]\n",
            "using /root/.bl-data as data/embeddings cache\n",
            "Downloading https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt\n",
            "Progress: [========================================] 100%\n",
            "extracting file..\n",
            "downloaded data saved in /root/.bl-data/80b0a839a7edd4c99f54537aa83327340592a4e8\n",
            "[train file]: /root/.bl-data/80b0a839a7edd4c99f54537aa83327340592a4e8/eng.train.iobes\n",
            "[valid file]: /root/.bl-data/80b0a839a7edd4c99f54537aa83327340592a4e8/eng.testa.iobes\n",
            "[test file]: /root/.bl-data/80b0a839a7edd4c99f54537aa83327340592a4e8/eng.testb.iobes\n",
            "Progress: [========================================] 100%\n",
            "extracting file..\n",
            "downloaded data saved in /root/.bl-data/b28358911296329a1623a8d8d00517bbf7177fe0\n",
            "embedding file location: /root/.bl-data/b28358911296329a1623a8d8d00517bbf7177fe0\n",
            "model file [./tagger/tagger-model-223.pyt]\n",
            "Doing early stopping on [f1] with patience [15]\n",
            "reporting [<bound method EpochReportingHook.step of <baseline.reporting.LoggingReporting object at 0x7fd05192ba58>>]\n",
            "Calling model <function register_model.<locals>.create at 0x7fd0068e8598>\n",
            "Setting span type iobes\n",
            "adam(eta=0.000010, beta1=0.900000, beta2=0.999000, epsilon=0.000000, wd=0.000000)\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 1, \"phase\": \"Train\", \"avg_loss\": 2.6148412382578035}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 1, \"phase\": \"Valid\", \"acc\": 0.9768732295991619, \"f1\": 0.8698635869110386}\n",
            "New best 0.870\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 2, \"phase\": \"Train\", \"avg_loss\": 0.9340948096474693}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 2, \"phase\": \"Valid\", \"acc\": 0.9851383337860385, \"f1\": 0.9163739755812009}\n",
            "New best 0.916\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 3, \"phase\": \"Train\", \"avg_loss\": 0.5483942897513716}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 3, \"phase\": \"Valid\", \"acc\": 0.9874277288424974, \"f1\": 0.9287629902782434}\n",
            "New best 0.929\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 4, \"phase\": \"Train\", \"avg_loss\": 0.37546397392344283}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 4, \"phase\": \"Valid\", \"acc\": 0.9881455900042684, \"f1\": 0.9310547038619419}\n",
            "New best 0.931\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 5, \"phase\": \"Train\", \"avg_loss\": 0.26542128247691177}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 5, \"phase\": \"Valid\", \"acc\": 0.9886112296767685, \"f1\": 0.9305009617797106}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 6, \"phase\": \"Train\", \"avg_loss\": 0.1915307781699024}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 6, \"phase\": \"Valid\", \"acc\": 0.9884366147995809, \"f1\": 0.9333669016448473}\n",
            "New best 0.933\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 7, \"phase\": \"Train\", \"avg_loss\": 0.14051421018071383}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 7, \"phase\": \"Valid\", \"acc\": 0.9890574676962477, \"f1\": 0.9373114314448543}\n",
            "New best 0.937\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 8, \"phase\": \"Train\", \"avg_loss\": 0.09682761203111621}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 8, \"phase\": \"Valid\", \"acc\": 0.9890574676962477, \"f1\": 0.935184410805386}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 9, \"phase\": \"Train\", \"avg_loss\": 0.08699057480643349}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 9, \"phase\": \"Valid\", \"acc\": 0.9882425982693726, \"f1\": 0.9315595716198125}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 10, \"phase\": \"Train\", \"avg_loss\": 0.06446995755481442}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 10, \"phase\": \"Valid\", \"acc\": 0.9884754181056226, \"f1\": 0.9334783700108777}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 11, \"phase\": \"Train\", \"avg_loss\": 0.05931680762623672}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 11, \"phase\": \"Valid\", \"acc\": 0.9899693453882271, \"f1\": 0.9450457253125263}\n",
            "New best 0.945\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 12, \"phase\": \"Train\", \"avg_loss\": 0.036992363657949244}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 12, \"phase\": \"Valid\", \"acc\": 0.9897559272049978, \"f1\": 0.9428835024742095}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 13, \"phase\": \"Train\", \"avg_loss\": 0.03549839108100154}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 13, \"phase\": \"Valid\", \"acc\": 0.9897947305110395, \"f1\": 0.9456312306274608}\n",
            "New best 0.946\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 14, \"phase\": \"Train\", \"avg_loss\": 0.031244984030731154}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 14, \"phase\": \"Valid\", \"acc\": 0.9894066974506228, \"f1\": 0.9431017119838871}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 15, \"phase\": \"Train\", \"avg_loss\": 0.029457802466150276}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 15, \"phase\": \"Valid\", \"acc\": 0.989620115633852, \"f1\": 0.9434341741389424}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 16, \"phase\": \"Train\", \"avg_loss\": 0.023979926591391622}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 16, \"phase\": \"Valid\", \"acc\": 0.9895425090217687, \"f1\": 0.9450697127498741}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 17, \"phase\": \"Train\", \"avg_loss\": 0.031091647303298484}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 17, \"phase\": \"Valid\", \"acc\": 0.9898723371231228, \"f1\": 0.9464240798188984}\n",
            "New best 0.946\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 18, \"phase\": \"Train\", \"avg_loss\": 0.02009400451542441}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 18, \"phase\": \"Valid\", \"acc\": 0.9901439602654146, \"f1\": 0.9478187919463088}\n",
            "New best 0.948\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 19, \"phase\": \"Train\", \"avg_loss\": 0.019171412507981975}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 19, \"phase\": \"Valid\", \"acc\": 0.9900275503472896, \"f1\": 0.9471565173628587}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 20, \"phase\": \"Train\", \"avg_loss\": 0.017522968825922504}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 20, \"phase\": \"Valid\", \"acc\": 0.9899693453882271, \"f1\": 0.9489222511112976}\n",
            "New best 0.949\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 21, \"phase\": \"Train\", \"avg_loss\": 0.018299938242782067}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 21, \"phase\": \"Valid\", \"acc\": 0.9898723371231228, \"f1\": 0.9466621938946661}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 22, \"phase\": \"Train\", \"avg_loss\": 0.018267686828218246}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 22, \"phase\": \"Valid\", \"acc\": 0.9901827635714563, \"f1\": 0.9477398756511511}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 23, \"phase\": \"Train\", \"avg_loss\": 0.015852666787277748}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 23, \"phase\": \"Valid\", \"acc\": 0.9900663536533313, \"f1\": 0.9473860589812333}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 24, \"phase\": \"Train\", \"avg_loss\": 0.017940919649187623}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 24, \"phase\": \"Valid\", \"acc\": 0.9897559272049978, \"f1\": 0.9469792454415595}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 25, \"phase\": \"Train\", \"avg_loss\": 0.021123119293672898}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 25, \"phase\": \"Valid\", \"acc\": 0.9898723371231228, \"f1\": 0.9483033095936322}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 26, \"phase\": \"Train\", \"avg_loss\": 0.016587971543316368}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 26, \"phase\": \"Valid\", \"acc\": 0.9902603701835396, \"f1\": 0.9500335345405767}\n",
            "New best 0.950\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 27, \"phase\": \"Train\", \"avg_loss\": 0.01976229686818598}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 27, \"phase\": \"Valid\", \"acc\": 0.9905707966318731, \"f1\": 0.9515100671140941}\n",
            "New best 0.952\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 28, \"phase\": \"Train\", \"avg_loss\": 0.013134965125378902}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 28, \"phase\": \"Valid\", \"acc\": 0.9899693453882271, \"f1\": 0.9453177257525083}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 29, \"phase\": \"Train\", \"avg_loss\": 0.012124325154168114}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 29, \"phase\": \"Valid\", \"acc\": 0.9901827635714563, \"f1\": 0.9484985740647542}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 30, \"phase\": \"Train\", \"avg_loss\": 0.018610463702691316}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 30, \"phase\": \"Valid\", \"acc\": 0.9899499437352063, \"f1\": 0.9463283932010382}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 31, \"phase\": \"Train\", \"avg_loss\": 0.014425886904675325}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 31, \"phase\": \"Valid\", \"acc\": 0.9905125916728105, \"f1\": 0.9488707917051464}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 32, \"phase\": \"Train\", \"avg_loss\": 0.015445712426111357}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 32, \"phase\": \"Valid\", \"acc\": 0.9899305420821854, \"f1\": 0.9479691171534073}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 33, \"phase\": \"Train\", \"avg_loss\": 0.011917820238982225}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 33, \"phase\": \"Valid\", \"acc\": 0.9904349850607271, \"f1\": 0.9510067114093959}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 34, \"phase\": \"Train\", \"avg_loss\": 0.011538200447778415}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 34, \"phase\": \"Valid\", \"acc\": 0.9904931900197896, \"f1\": 0.9502140878179834}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 35, \"phase\": \"Train\", \"avg_loss\": 0.01074471144073422}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 35, \"phase\": \"Valid\", \"acc\": 0.9902021652244771, \"f1\": 0.9472536687631028}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 36, \"phase\": \"Train\", \"avg_loss\": 0.012705201613351076}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 36, \"phase\": \"Valid\", \"acc\": 0.9899693453882271, \"f1\": 0.9475448298977711}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 37, \"phase\": \"Train\", \"avg_loss\": 0.008575328030983096}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 37, \"phase\": \"Valid\", \"acc\": 0.9902797718365605, \"f1\": 0.9483497102544721}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 38, \"phase\": \"Train\", \"avg_loss\": 0.007864618520925987}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 38, \"phase\": \"Valid\", \"acc\": 0.9902021652244771, \"f1\": 0.9482238605898123}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 39, \"phase\": \"Train\", \"avg_loss\": 0.012387273277729214}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 39, \"phase\": \"Valid\", \"acc\": 0.9903573784486438, \"f1\": 0.9494254801643883}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 40, \"phase\": \"Train\", \"avg_loss\": 0.012055296047444779}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 40, \"phase\": \"Valid\", \"acc\": 0.9905125916728105, \"f1\": 0.9524289796604472}\n",
            "New best 0.952\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 41, \"phase\": \"Train\", \"avg_loss\": 0.008430880638651292}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 41, \"phase\": \"Valid\", \"acc\": 0.9901633619184355, \"f1\": 0.949911905361188}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 42, \"phase\": \"Train\", \"avg_loss\": 0.014469159821096691}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 42, \"phase\": \"Valid\", \"acc\": 0.9902991734895813, \"f1\": 0.9509161203563624}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 43, \"phase\": \"Train\", \"avg_loss\": 0.0077162278599722965}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 43, \"phase\": \"Valid\", \"acc\": 0.9902991734895813, \"f1\": 0.9505166764681173}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 44, \"phase\": \"Train\", \"avg_loss\": 0.008124644501872877}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 44, \"phase\": \"Valid\", \"acc\": 0.9903573784486438, \"f1\": 0.9496342386277643}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 45, \"phase\": \"Train\", \"avg_loss\": 0.009557785405293775}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 45, \"phase\": \"Valid\", \"acc\": 0.9901245586123938, \"f1\": 0.9486212387897075}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 46, \"phase\": \"Train\", \"avg_loss\": 0.009296203984775195}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 46, \"phase\": \"Valid\", \"acc\": 0.990454386713748, \"f1\": 0.9510149303808085}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 47, \"phase\": \"Train\", \"avg_loss\": 0.007504824920074837}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 47, \"phase\": \"Valid\", \"acc\": 0.9908424197741648, \"f1\": 0.9513830678960603}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 48, \"phase\": \"Train\", \"avg_loss\": 0.00774602191001443}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 48, \"phase\": \"Valid\", \"acc\": 0.9903961817546855, \"f1\": 0.9524289796604472}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 49, \"phase\": \"Train\", \"avg_loss\": 0.0059471396517955044}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 49, \"phase\": \"Valid\", \"acc\": 0.9903573784486438, \"f1\": 0.9503784693019345}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 50, \"phase\": \"Train\", \"avg_loss\": 0.008302955171030581}\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 50, \"phase\": \"Valid\", \"acc\": 0.9900857553063521, \"f1\": 0.9492777964393684}\n",
            "Best performance on f1: 0.952 at epoch 39\n",
            "Reloading best checkpoint\n",
            "Setting span type iobes\n",
            "adam(eta=0.000010, beta1=0.900000, beta2=0.999000, epsilon=0.000000, wd=0.000000)\n",
            "Progress: [========================================] 100%\n",
            "{\"tick_type\": \"EPOCH\", \"tick\": 0, \"phase\": \"Test\", \"acc\": 0.9816716328324294, \"f1\": 0.9109920042175556}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUTVLaVAYpOX",
        "colab_type": "text"
      },
      "source": [
        "We can see that our final BERT-Base model also did fairly well on the dataset, with an F1 around 91.1, a little lower but in the same rough range, and with a bit higher dev set performance, without any CRF or constrained decoding.\n",
        "\n",
        "## Wrapup\n",
        "\n",
        "In this tutorial we explored the `mead-baseline` Task for tagging (`tagger`) and we build trained 2 very common deep learning models to create Named Entity Recognizers using a very common NER dataset.\n",
        "\n",
        "We also looked at how to load these models within MEAD for inference.  In future tutorials we will show a more production-ready approach for running these models, using `mead-export`"
      ]
    }
  ]
}