PELIC_spelling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PELIC spelling\n",
    "\n",
    "This notebook adds further processing to `PELIC_compiled.csv`  in the [`PELIC-dataset`](https://github.com/ELI-Data-Mining-Group/PELIC-dataset) repo by creating a column of tok_POS whose spelling has been automatically corrected.\n",
    "\n",
    "**Notebook contents:**\n",
    "- [Building `non_words_df`](#Building-non_words_df)\n",
    "- [Building `misspell_df`](#Building-misspell_df)\n",
    "- [Possible segmentation](#Applying-segmentation)\n",
    "- [Applying spelling correction](#Applying-spelling-correction)\n",
    "- [Incorporating corrections into `pelic_df`](#Incorporating-corrections-into-pelic_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building non_words_df\n",
    "In this section, we build a dataframe, `non_words_df`, which collects all of the non-words from the PELIC dataset (in `PELIC_compiled.csv`). The final dataframe has the following columns:\n",
    "- `non_word`: tuples with the non-words and their parts of speech\n",
    "- `sentence`: the complete sentence containing the non-word to provide context\n",
    "- `answer_id`: the id of the text they come from"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "\n",
    "import pandas as pd\n",
    "import pprint\n",
    "import numpy as np\n",
    "from ast import literal_eval\n",
    "import nltk\n",
    "import random\n",
    "from pelitk import lex\n",
    "import string\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>course_id</th>\n",
       "      <th>level_id</th>\n",
       "      <th>class_id</th>\n",
       "      <th>question_id</th>\n",
       "      <th>version</th>\n",
       "      <th>text_len</th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>eq0</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>149</td>\n",
       "      <td>4</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>177</td>\n",
       "      <td>I met my friend Nife while I was studying in a...</td>\n",
       "      <td>[I, met, my, friend, Nife, while, I, was, stud...</td>\n",
       "      <td>[(I, I, PRP), (met, meet, VBD), (my, my, PRP$)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>am8</td>\n",
       "      <td>Thai</td>\n",
       "      <td>Female</td>\n",
       "      <td>149</td>\n",
       "      <td>4</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>137</td>\n",
       "      <td>Ten years ago, I met a women on the train betw...</td>\n",
       "      <td>[Ten, years, ago, ,, I, met, a, women, on, the...</td>\n",
       "      <td>[(Ten, ten, CD), (years, year, NNS), (ago, ago...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dk5</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>63</td>\n",
       "      <td>In my country we usually don't use tea bags. F...</td>\n",
       "      <td>[In, my, country, we, usually, do, n't, use, t...</td>\n",
       "      <td>[(In, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dk5</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>I organized the instructions by time.</td>\n",
       "      <td>[I, organized, the, instructions, by, time, .]</td>\n",
       "      <td>[(I, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ad1</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>59</td>\n",
       "      <td>First, prepare a port, loose tea, and cup.\\nSe...</td>\n",
       "      <td>[First, ,, prepare, a, port, ,, loose, tea, ,,...</td>\n",
       "      <td>[(First, first, RB), (,, ,, ,), (prepare, prep...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id       L1  gender course_id level_id class_id question_id  \\\n",
       "answer_id                                                                    \n",
       "1             eq0   Arabic    Male       149        4        g           5   \n",
       "2             am8     Thai  Female       149        4        g           5   \n",
       "3             dk5  Turkish  Female       115        4        w          12   \n",
       "4             dk5  Turkish  Female       115        4        w          13   \n",
       "5             ad1   Korean  Female       115        4        w          12   \n",
       "\n",
       "          version  text_len  \\\n",
       "answer_id                     \n",
       "1               1       177   \n",
       "2               1       137   \n",
       "3               1        63   \n",
       "4               1         6   \n",
       "5               1        59   \n",
       "\n",
       "                                                        text  \\\n",
       "answer_id                                                      \n",
       "1          I met my friend Nife while I was studying in a...   \n",
       "2          Ten years ago, I met a women on the train betw...   \n",
       "3          In my country we usually don't use tea bags. F...   \n",
       "4                      I organized the instructions by time.   \n",
       "5          First, prepare a port, loose tea, and cup.\\nSe...   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "1          [I, met, my, friend, Nife, while, I, was, stud...   \n",
       "2          [Ten, years, ago, ,, I, met, a, women, on, the...   \n",
       "3          [In, my, country, we, usually, do, n't, use, t...   \n",
       "4             [I, organized, the, instructions, by, time, .]   \n",
       "5          [First, ,, prepare, a, port, ,, loose, tea, ,,...   \n",
       "\n",
       "                                                 tok_lem_POS  \n",
       "answer_id                                                     \n",
       "1          [(I, I, PRP), (met, meet, VBD), (my, my, PRP$)...  \n",
       "2          [(Ten, ten, CD), (years, year, NNS), (ago, ago...  \n",
       "3          [(In, in, IN), (my, my, PRP$), (country, count...  \n",
       "4          [(I, I, PRP), (organized, organize, VBD), (the...  \n",
       "5          [(First, first, RB), (,, ,, ,), (prepare, prep...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in PELIC_compiled.csv\n",
    "\n",
    "pelic_df = pd.read_csv(\"../PELIC-dataset/PELIC_compiled.csv\", index_col = 'answer_id', # answer_id is unique\n",
    "                      dtype = {'level_id':'object','question_id':'object','version':'object','course_id':'object'}, # str not ints\n",
    "                               converters={'tokens':literal_eval,'tok_lem_POS':literal_eval}) # read in as lists\n",
    "pelic_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The focus here is the `tok_lem_POS` column, but all columns will be kept as the entire df will be written out at the end of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Creating small dataframe to be used for finding non-words\n",
    "\n",
    "non_words = pelic_df[['text','tok_lem_POS']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** For spelling correction, it is necessary to decide what list of words will be used for determining if a word is real or not.\n",
    "\n",
    "Here, we use the [`SCOWL_condensed.txt`](https://github.com/ELI-Data-Mining-Group/PELIC-spelling/blob/master/SCOWL_condensed.txt) file which is a combination of wordlists available for download at http://wordlist.aspell.net/. We include items from all the dictionaries _except_ the abbreviations dictionary. For a detailed look at the compilation of this dictionary, please see the [SCOWL_wordlist](https://github.com/ELI-Data-Mining-Group/PELIC-spelling/blob/master/SCOWL_wordlist.ipynb) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['melonry', 'gyrostabilizers', 'pixing', 'yttrocolumbite', 'brattishes']\n"
     ]
    }
   ],
   "source": [
    "#Reading in SCOWL_condensed as a set as a lookup list for spelling (500k words)\n",
    "\n",
    "scowl = set(open(\"SCOWL_condensed.txt\", \"r\").read().split('\\n'))\n",
    "print(random.sample(scowl,5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "497552"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scowl = set([x.lower() for x in scowl])\n",
    "len(scowl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following is a list of words which should be considered words but which were previously being labelled as non-words. These items have been manually added to this list based on output later in this notebook. Most of these items are food items, names, or abbreviations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "241\n",
      "['adha', 'adj', 'ahamed', 'alaikum', 'anonurlpage', 'antiretroviral', 'arpa', 'atm', 'ave', 'beyonce', 'bibimbap', 'bio', 'biodiesel', 'bioethanol', 'bulgogi', 'bundang', 'cafe', 'carnaval', 'cds', 'cf', 'co', 'comscore', 'cyber', 'ddukboggi', 'def', 'dr', 'eg', 'eid', 'electrospray', 'entrees', 'erectus', 'etc', 'fiance', 'fiancee', 'fiter', 'fitir', 'fitr', 'fl', 'freediving', 'fukubukuro', 'geolinguist', 'hikikomori', 'hp', 'ibt', 'icu', 'iq', 'ipods', 'iriver', 'jetta', 'jul', 'kabsa', 'kaled', 'kawader', 'kennywood', 'km', 'leisureville', 'll', 'maamool', 'mayumi', 'mcdonalds', 'min', 'mongongo', 'nc', 'neuro', 'nian', 'notting', 'okroshka', 'onsen', 'pajeon', 'pbt', 'pc', 'pcs', 'pp', 'pudim', 'puket', 'samear', 'shui', 'sq', 'st', 'staycation', 'sth', 'taoyuan', 'toefl', 'trans', 'transgene', 'tv', 'unsub', 'va', 'vol', 'vs', 'webaholic', 'webaholics', 'webaholism', 'wenjing', 'woong', 'yaoming', 'ying', 'yingdong', 'yugong', 'yuval', 'zi', 'abha', 'achuar', 'ae', 'afandi', 'aladha', 'alfater', 'alfeter', 'alfetr', 'alfter', 'alftr', 'ansan', 'apci', 'arial', 'ayumu', 'aziz', 'bartercard', 'bbc', 'bbq', 'beckham', 'bennigan', 'bmi', 'burkina', 'busan', 'camela', 'caral', 'ceos', 'cmu', 'cnn', 'cpu', 'cyworld', 'daegu', 'dammam', 'dc', 'dvd', 'dvds', 'eaid', 'esl', 'etat', 'ets', 'etsuko', 'eun', 'fargana', 'fda', 'federer', 'feng', 'frisbee', 'gaviotas', 'gmo', 'gpa', 'gunsan', 'hadza', 'hanshin', 'hd', 'hofstra', 'hsk', 'hyun', 'iaad', 'ielts', 'incheon', 'jang', 'japchae', 'jazan', 'jeddah', 'jeju', 'jiri', 'jong', 'jr', 'junine', 'kabsah', 'kakike', 'kanye', 'kimbab', 'kmt', 'koreas', 'krabi', 'ksa', 'kyung', 'lcd', 'lelan', 'lenovo', 'lg', 'lippman', 'longman', 'matsui', 'meltzer', 'mishori', 'mlb', 'morakot', 'mp', 'msn', 'mt', 'myung', 'nadal', 'najran', 'namwon', 'nano', 'nba', 'neimark', 'ngondo', 'noha', 'noor', 'nouwen', 'nyc', 'obon', 'occitane', 'oishi', 'paterno', 'pattak', 'pattee', 'pausini', 'pnc', 'qdoba', 'ritu', 'roh', 'ronaldo', 'rsa', 'sakarya', 'saletan', 'saudia', 'sawiris', 'sc', 'sep', 'sf', 'southside', 'sujin', 'tgif', 'toeic', 'tvs', 'uae', 'upmc', 'usd', 'usda', 'vaio', 'wto', 'xbox', 'yokoso', 'yut', 'zawia', 'zhou', 'zong']\n"
     ]
    }
   ],
   "source": [
    "scowl_supp = open(\"SCOWL_supp.txt\", \"r\").read().split(',')\n",
    "scowl_supp = [x[2:-1] for x in scowl_supp]\n",
    "print(len(scowl_supp))\n",
    "print(scowl_supp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py:5303: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[name] = value\n"
     ]
    }
   ],
   "source": [
    "# Lower case all toks\n",
    "\n",
    "non_words.tok_lem_POS = non_words.tok_lem_POS.apply(lambda row: [(x[0].lower(),x[1],x[2]) for x in row])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to find non-words\n",
    "\n",
    "def spell_check(tok_lem_POS_list):\n",
    "    word_list = scowl # Choose word_list here. Default is scowl described above.\n",
    "    not_in_word_list = []\n",
    "    for tok_lem_POS in tok_lem_POS_list:\n",
    "        if tok_lem_POS[0] not in word_list and tok_lem_POS[0] not in scowl_supp:\n",
    "            not_in_word_list.append(tok_lem_POS)\n",
    "    return not_in_word_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    }
   ],
   "source": [
    "# Apply spell check function to find all misspelled-words. \n",
    "\n",
    "non_words['misspelled_words'] = non_words.tok_lem_POS.apply(spell_check)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelled_words</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I met my friend Nife while I was studying in a...</td>\n",
       "      <td>[(i, I, PRP), (met, meet, VBD), (my, my, PRP$)...</td>\n",
       "      <td>[(., ., .), (., ., .), (., ., .), (;, ;, :), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ten years ago, I met a women on the train betw...</td>\n",
       "      <td>[(ten, ten, CD), (years, year, NNS), (ago, ago...</td>\n",
       "      <td>[(,, ,, ,), (,, ,, ,), (., ., .), (;, ;, :), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>In my country we usually don't use tea bags. F...</td>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>[(., ., .), (,, ,, ,), (., ., .), (., ., .), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I organized the instructions by time.</td>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>[(., ., .)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>First, prepare a port, loose tea, and cup.\\nSe...</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (prepare, prep...</td>\n",
       "      <td>[(,, ,, ,), (,, ,, ,), (,, ,, ,), (., ., .), (...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                        text  \\\n",
       "answer_id                                                      \n",
       "1          I met my friend Nife while I was studying in a...   \n",
       "2          Ten years ago, I met a women on the train betw...   \n",
       "3          In my country we usually don't use tea bags. F...   \n",
       "4                      I organized the instructions by time.   \n",
       "5          First, prepare a port, loose tea, and cup.\\nSe...   \n",
       "\n",
       "                                                 tok_lem_POS  \\\n",
       "answer_id                                                      \n",
       "1          [(i, I, PRP), (met, meet, VBD), (my, my, PRP$)...   \n",
       "2          [(ten, ten, CD), (years, year, NNS), (ago, ago...   \n",
       "3          [(in, in, IN), (my, my, PRP$), (country, count...   \n",
       "4          [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "5          [(first, first, RB), (,, ,, ,), (prepare, prep...   \n",
       "\n",
       "                                            misspelled_words  \n",
       "answer_id                                                     \n",
       "1          [(., ., .), (., ., .), (., ., .), (;, ;, :), (...  \n",
       "2          [(,, ,, ,), (,, ,, ,), (., ., .), (;, ;, :), (...  \n",
       "3          [(., ., .), (,, ,, ,), (., ., .), (., ., .), (...  \n",
       "4                                                [(., ., .)]  \n",
       "5          [(,, ,, ,), (,, ,, ,), (,, ,, ,), (., ., .), (...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "non_words.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding context to the dataframe\n",
    "Seeing the mistakes in the context of a sentence will allow for better manual checking if required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    }
   ],
   "source": [
    "# Sent-tokenizing the text\n",
    "\n",
    "non_words['sentence'] = non_words['text'].apply(lambda x: nltk.sent_tokenize(x))\n",
    "\n",
    "# And delete text column which is no longer needed\n",
    "\n",
    "del non_words['text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1182\n",
      "['dog-eat-dog', 'euro-americans', 'prize-winning', 'concert-hall', 'quart-size', 'modern-day', 'life-span', 'non-muslims', 'life-threatening', 'ready-made']\n"
     ]
    }
   ],
   "source": [
    "# Checking for hyphenated words tagged as misspellings because SCOWL doesn't contain hypenated words\n",
    "\n",
    "hyphenated = set([x[0] for x in [x for y in non_words.misspelled_words.to_list() for x in y] if '-' in x[0]])\n",
    "print(len(hyphenated))\n",
    "print(list(hyphenated)[:10])\n",
    "\n",
    "# These need to be removed from the non-words dataframe if composed of valid words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['', \"'\"],\n",
       " ['', '***', '****'],\n",
       " ['', '+'],\n",
       " ['', '.'],\n",
       " ['',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  '.',\n",
       "  ''],\n",
       " [\"'\", ''],\n",
       " ['.', ''],\n",
       " ['/', ''],\n",
       " ['\\\\\\\\', ''],\n",
       " ['^', '^'],\n",
       " ['al', 'qaida'],\n",
       " ['austro', 'hungarian'],\n",
       " ['cd', 'rom'],\n",
       " ['co', 'authored'],\n",
       " ['co', 'ed'],\n",
       " ['co', 'educational'],\n",
       " ['co', 'exist'],\n",
       " ['co', 'existence'],\n",
       " ['co', 'founded'],\n",
       " ['co', 'founder'],\n",
       " ['co', 'founders'],\n",
       " ['co', 'host'],\n",
       " ['co', 'op'],\n",
       " ['co', 'operate'],\n",
       " ['co', 'operation'],\n",
       " ['co', 'pay'],\n",
       " ['co', 'pilot'],\n",
       " ['co', 'sleeping'],\n",
       " ['co', 'star'],\n",
       " ['co', 'worker'],\n",
       " ['co', 'workers'],\n",
       " ['co', 'written'],\n",
       " ['co', 'wrote'],\n",
       " ['mah', 'jong'],\n",
       " ['mid', '80s'],\n",
       " ['pay', 'tv'],\n",
       " ['roly', 'poly'],\n",
       " ['socio', 'cultural'],\n",
       " ['socio', 'economic'],\n",
       " ['trans', 'fat'],\n",
       " ['vis', 'a', 'vis'],\n",
       " ['wal', 'mart']]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Hyphenated items whose components are not in scowl - possible misspellings or punctuation strings\n",
    "\n",
    "sorted([y for y in [x.split('-') for x in hyphenated] if y[0] not in scowl or y[1] not in scowl])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After manual checking, all the hypenated words are punctuation, real words (or true productive use of affixes) and can be removed from the non-words df.\n",
    "\n",
    "The following two cells \n",
    "1. remove all the hypenated words from the dataframe\n",
    "2. remove all words that don't contain a letter\n",
    "\n",
    "However, as all hyphenated word are fine, we will instead just eliminate all words that are not purely composed of letters. This will have the effect of removing the following categories from the dataframe:\n",
    "- punctuation\n",
    "- hyphenated words (e.g. well-known)\n",
    "- contractions (e.g. 'll, 've)\n",
    "- years (e.g. 1950s)\n",
    "- ordinals (e.g. 1st, 2nd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tangent\n",
    "The next three cells use the above information to create a more complete `PELIC-SCOWL.txt` wordlist for use with PELIC data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "# Creating a text file of hyphenated list for use elsewhere in creating an PELIC-SCOWL wordlist\n",
    "\n",
    "hyphenated = {word for word in hyphenated if any(x.isalpha() for x in word)}\n",
    "with open('hyphens.txt', 'w') as f:\n",
    "    for item in hyphenated:\n",
    "        f.write(\"%s\\n\" % item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "# Creating a text file of contractions for use elsewhere in creating an PELIC-SCOWL wordlist\n",
    "\n",
    "contractions = {\"'ll\",\"'ve\",\"n't\",\"'m\",\"'s\",\"'d\",\"'re\",\"'ve\"}\n",
    "with open('contractions.txt', 'w') as f:\n",
    "    for item in contractions:\n",
    "        f.write(\"%s\\n\" % item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Kock', 'Czachor', 'Omdahl', 'Hawkinberry', 'Conney']\n"
     ]
    }
   ],
   "source": [
    "# Reading in 'all_names' file\n",
    "\n",
    "all_names = set(open(\"all_names.txt\", \"r\").read().split('\\n'))\n",
    "all_names = {x.capitalize() for x in all_names}\n",
    "print(random.sample(all_names,5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "# Combining SCOWL_condensed, hyphenated, and contraction lists\n",
    "\n",
    "pelic_scowl = scowl|hyphenated|contractions|all_names\n",
    "pelic_scowl.remove('')\n",
    "pelic_scowl = sorted(list(pelic_scowl))\n",
    "\n",
    "with open('PELIC-SCOWL.txt', 'w') as f:\n",
    "    for item in pelic_scowl:\n",
    "        f.write(\"%s\\n\" % item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing hypenated words\n",
    "\n",
    "# non_words.misspelled_words = non_words.misspelled_words.apply(lambda row: [x for x in row if x[0] not in hyphenated])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing items that are only numbers or punctuation\n",
    "# .isalpha() cannot be used without 'any' as this also removes hyphenated words\n",
    "\n",
    "# non_words.misspelled_words = non_words.misspelled_words.apply(lambda row: [x for x in row if any(y.isalpha() for y in x[0])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "680878\n",
      "20620\n"
     ]
    }
   ],
   "source": [
    "# Checking initial length of non_words list\n",
    "\n",
    "print(len([x for y in non_words.misspelled_words.to_list() for x in y]))\n",
    "print(len(set([x for y in non_words.misspelled_words.to_list() for x in y])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py:5303: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[name] = value\n"
     ]
    }
   ],
   "source": [
    "# Removing items that are not purely alpha\n",
    "\n",
    "non_words.misspelled_words = non_words.misspelled_words.apply(lambda row: [x for x in row if x[0].isalpha()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26630\n",
      "15959\n"
     ]
    }
   ],
   "source": [
    "# Checking affect of removal\n",
    "\n",
    "print(len([x for y in non_words.misspelled_words.to_list() for x in y]))\n",
    "print(len(set([x for y in non_words.misspelled_words.to_list() for x in y])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing proper names - NNP, NNPS\n",
    "\n",
    "# non_words.misspelled_words = non_words.misspelled_words.apply(lambda row: [x for x in row if x[2] != 'NNP' and x[1] != 'NNPS'])\n",
    "#print(len([x for y in non_words.misspelled_words.to_list() for x in y]))\n",
    "#print(len(set([x for y in non_words.misspelled_words.to_list() for x in y])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After manual checking, it was decided to keep in items tagged as NNP and NNPS as some items were in fact mistagged and were general capitalized nouns (NN) which were misspelled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing all words with length 1\n",
    "\n",
    "non_words.misspelled_words = non_words.misspelled_words.apply(lambda row: [x for x in row if len(x[0]) > 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26618\n",
      "15950\n"
     ]
    }
   ],
   "source": [
    "# Checking affect of removal\n",
    "\n",
    "print(len([x for y in non_words.misspelled_words.to_list() for x in y]))\n",
    "print(len(set([x for y in non_words.misspelled_words.to_list() for x in y])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing all words with special characters (non_ascii)\n",
    "# after checking, these are all foreign words with accents and other non-latin characters\n",
    "\n",
    "# Creating function to check\n",
    "def is_ascii(s):\n",
    "    return all(ord(c) < 128 for c in s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('são', 'São', 'NNP'),\n",
       " ('ideás', 'ideás', 'NN'),\n",
       " ('áã', 'áã', 'NNP'),\n",
       " ('jørgensen', 'Jørgensen', 'NNP'),\n",
       " ('ôåñ', 'ÔåÑ', 'NNP'),\n",
       " ('décor', 'décor', 'NN'),\n",
       " ('sugestión', 'sugestión', 'NN'),\n",
       " ('íóçè', 'ÍÓÇÈ', 'NNP'),\n",
       " ('çáèííñì', 'ÇáÈÍíÑì', 'JJ'),\n",
       " ('ããßä', 'ããßä', 'NNP'),\n",
       " ('êñóá', 'ÊÑÓá', 'NNP'),\n",
       " ('ïúçá', 'ÏÚÇÁ', 'NNP'),\n",
       " ('paraíso', 'Paraíso', 'NNP'),\n",
       " ('íßæä', 'íßæä', 'NNP'),\n",
       " ('çáçêõçá', 'ÇáÇÊÕÇá', 'NNP'),\n",
       " ('íçãï', 'ÍÇãÏ', 'NNP'),\n",
       " ('renée', 'Renée', 'NNP'),\n",
       " ('méxico', 'México', 'NNP'),\n",
       " ('çáííæì', 'ÇáÍíæì', 'NNP'),\n",
       " ('compiègne', 'Compiègne', 'NNP')]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "non_word_list = set([x for y in non_words.misspelled_words.to_list() for x in y])\n",
    "foreign_words = [x for x in non_word_list if is_ascii(x[0]) == False ]\n",
    "foreign_words[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing foreign words\n",
    "non_words.misspelled_words = non_words.misspelled_words.apply(lambda row: [x for x in row if x not in foreign_words])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26295\n",
      "15710\n"
     ]
    }
   ],
   "source": [
    "# Checking affect of removal\n",
    "\n",
    "print(len([x for y in non_words.misspelled_words.to_list() for x in y]))\n",
    "print(len(set([x for y in non_words.misspelled_words.to_list() for x in y])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing names from names list words\n",
    "all_names = {y.lower() for y in all_names}\n",
    "non_words.misspelled_words = non_words.misspelled_words.apply(lambda row: [x for x in row if x[0] not in all_names])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24526\n",
      "14874\n"
     ]
    }
   ],
   "source": [
    "# Checking affect of removal\n",
    "\n",
    "print(len([x for y in non_words.misspelled_words.to_list() for x in y]))\n",
    "print(len(set([x for y in non_words.misspelled_words.to_list() for x in y])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelled_words</th>\n",
       "      <th>sentence</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[(i, I, PRP), (met, meet, VBD), (my, my, PRP$)...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[I met my friend Nife while I was studying in ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[(ten, ten, CD), (years, year, NNS), (ago, ago...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[Ten years ago, I met a women on the train bet...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[In my country we usually don't use tea bags.,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[I organized the instructions by time.]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (prepare, prep...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[First, prepare a port, loose tea, and cup., S...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 tok_lem_POS misspelled_words  \\\n",
       "answer_id                                                                       \n",
       "1          [(i, I, PRP), (met, meet, VBD), (my, my, PRP$)...               []   \n",
       "2          [(ten, ten, CD), (years, year, NNS), (ago, ago...               []   \n",
       "3          [(in, in, IN), (my, my, PRP$), (country, count...               []   \n",
       "4          [(i, I, PRP), (organized, organize, VBD), (the...               []   \n",
       "5          [(first, first, RB), (,, ,, ,), (prepare, prep...               []   \n",
       "\n",
       "                                                    sentence  \n",
       "answer_id                                                     \n",
       "1          [I met my friend Nife while I was studying in ...  \n",
       "2          [Ten years ago, I met a women on the train bet...  \n",
       "3          [In my country we usually don't use tea bags.,...  \n",
       "4                    [I organized the instructions by time.]  \n",
       "5          [First, prepare a port, loose tea, and cup., S...  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "non_words.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create new dataframe so that each misspelling token is a separate row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing rows with no misspellings\n",
    "\n",
    "non_words2 = non_words.loc[non_words.misspelled_words.str.len() > 0,:].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exploding the lists in misspelled words so that each misspelling gets its own row\n",
    "\n",
    "non_words2 = non_words2.explode('misspelled_words')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keeping only the sentence containing the error (the first occurence of the error if repeated)\n",
    "\n",
    "non_words2['sentence'] = list(zip([x[0] for x in non_words2.misspelled_words], non_words2.sentence))\n",
    "non_words2['sentence'] = non_words2['sentence'].apply(\n",
    "    lambda row: [i for i in row[1] if row[0] in lex.re_tokenize(i) or row[0]+\"n't\" in i.lower()])\n",
    "non_words2['sentence'] = [x[0] for x in non_words2['sentence']]\n",
    "non_words2 = non_words2.drop_duplicates(subset = ['misspelled_words','sentence'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelled_words</th>\n",
       "      <th>sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>(beacause, beacause, NN)</td>\n",
       "      <td>I organized the instructions by time, beacause...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11</td>\n",
       "      <td>[(to, to, TO), (make, make, VB), (tea, tea, NN...</td>\n",
       "      <td>(wallmart, wallmart, NN)</td>\n",
       "      <td>next, you need to buy a box of tea in wallmart...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(dovn, dovn, NN)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(mircowave, mircowave, VBP)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15</td>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>(fitst, Fitst, NNP)</td>\n",
       "      <td>Fitst, boil a water in a pot.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id                                        tok_lem_POS  \\\n",
       "0          8  [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "1         11  [(to, to, TO), (make, make, VB), (tea, tea, NN...   \n",
       "2         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "3         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "4         15  [(in, in, IN), (my, my, PRP$), (country, count...   \n",
       "\n",
       "              misspelled_words  \\\n",
       "0     (beacause, beacause, NN)   \n",
       "1     (wallmart, wallmart, NN)   \n",
       "2             (dovn, dovn, NN)   \n",
       "3  (mircowave, mircowave, VBP)   \n",
       "4          (fitst, Fitst, NNP)   \n",
       "\n",
       "                                            sentence  \n",
       "0  I organized the instructions by time, beacause...  \n",
       "1  next, you need to buy a box of tea in wallmart...  \n",
       "2  First, you should take some hot water, you can...  \n",
       "3  First, you should take some hot water, you can...  \n",
       "4                      Fitst, boil a water in a pot.  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Keeping the answer_id (which is no longer unique) as a separate column\n",
    "\n",
    "non_words2 = non_words2.reset_index(drop = False)\n",
    "non_words2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding a bigrams column, i.e. one token left and right of the misspelled word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelled_words</th>\n",
       "      <th>sentence</th>\n",
       "      <th>enumerated</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>(beacause, beacause, NN)</td>\n",
       "      <td>I organized the instructions by time, beacause...</td>\n",
       "      <td>[(0, i), (1, organized), (2, the), (3, instruc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11</td>\n",
       "      <td>[(to, to, TO), (make, make, VB), (tea, tea, NN...</td>\n",
       "      <td>(wallmart, wallmart, NN)</td>\n",
       "      <td>next, you need to buy a box of tea in wallmart...</td>\n",
       "      <td>[(0, next), (1, you), (2, need), (3, to), (4, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(dovn, dovn, NN)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "      <td>[(0, first), (1, you), (2, should), (3, take),...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(mircowave, mircowave, VBP)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "      <td>[(0, first), (1, you), (2, should), (3, take),...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15</td>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>(fitst, Fitst, NNP)</td>\n",
       "      <td>Fitst, boil a water in a pot.</td>\n",
       "      <td>[(0, fitst), (1, boil), (2, a), (3, water), (4...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id                                        tok_lem_POS  \\\n",
       "0          8  [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "1         11  [(to, to, TO), (make, make, VB), (tea, tea, NN...   \n",
       "2         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "3         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "4         15  [(in, in, IN), (my, my, PRP$), (country, count...   \n",
       "\n",
       "              misspelled_words  \\\n",
       "0     (beacause, beacause, NN)   \n",
       "1     (wallmart, wallmart, NN)   \n",
       "2             (dovn, dovn, NN)   \n",
       "3  (mircowave, mircowave, VBP)   \n",
       "4          (fitst, Fitst, NNP)   \n",
       "\n",
       "                                            sentence  \\\n",
       "0  I organized the instructions by time, beacause...   \n",
       "1  next, you need to buy a box of tea in wallmart...   \n",
       "2  First, you should take some hot water, you can...   \n",
       "3  First, you should take some hot water, you can...   \n",
       "4                      Fitst, boil a water in a pot.   \n",
       "\n",
       "                                          enumerated  \n",
       "0  [(0, i), (1, organized), (2, the), (3, instruc...  \n",
       "1  [(0, next), (1, you), (2, need), (3, to), (4, ...  \n",
       "2  [(0, first), (1, you), (2, should), (3, take),...  \n",
       "3  [(0, first), (1, you), (2, should), (3, take),...  \n",
       "4  [(0, fitst), (1, boil), (2, a), (3, water), (4...  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a tokenized version of the sentence without punctuation and with the index for each token\n",
    "\n",
    "non_words2['enumerated'] = non_words2.sentence.apply(lambda x: lex.re_tokenize(x)).apply(enumerate).apply(list)\n",
    "non_words2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a function to extract the bigrams (1 word either side of misspelling)\n",
    "\n",
    "def get_bigrams(misspelled_word, enumerated_list):\n",
    "    if len(enumerated_list) <2:\n",
    "        return []\n",
    "    for tup in enumerated_list:\n",
    "        if tup[1] == misspelled_word[0]:\n",
    "            if tup[0] == 0:\n",
    "                bigram = ' '.join([x[1] for x in (enumerated_list[tup[0]],enumerated_list[tup[0]+1])])\n",
    "                return [(bigram,2)]\n",
    "            if tup[0] == len(enumerated_list)-1:\n",
    "                bigram = ' '.join([x[1] for x in (enumerated_list[tup[0]-1],enumerated_list[tup[0]])])\n",
    "                return [(bigram,1)]\n",
    "            else:\n",
    "                bigram1 = ' '.join([x[1] for x in (enumerated_list[tup[0]-1],enumerated_list[tup[0]])])\n",
    "                bigram2 = ' '.join([x[1] for x in (enumerated_list[tup[0]],enumerated_list[tup[0]+1])])\n",
    "                return [(bigram1,1), (bigram2,2)]\n",
    "            \n",
    "# Notes:\n",
    "# '1' and '2' added as identifiers to show which word in bigram is the misspelling\n",
    "# Type 1 bigram = (word, misspelling), Type 2 bigram = (misspelling, word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 'every'), (1, 'paragragh'), (2, 's'), (3, 'instructions'), (4, 'depend'), (5, 'on'), (6, 'a'), (7, 'main'), (8, 'idea')]\n",
      "\n",
      " ('every', 'every', 'DT') ('depend', 'depend', 'VBP') ('idea', 'idea', 'NN')\n"
     ]
    }
   ],
   "source": [
    "# Testing the function\n",
    "\n",
    "test_list = non_words2.iloc[5,4]\n",
    "print(test_list)\n",
    "\n",
    "first_item = non_words2.iloc[5,1][0] # first item in list\n",
    "middle_item = non_words2.iloc[5,1][4] # item in in middle of list\n",
    "last_item = non_words2.iloc[5,1][8] # item at end of list\n",
    "print('\\n',first_item, middle_item, last_item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('every paragragh', 2)]\n",
      "[('instructions depend', 1), ('depend on', 2)]\n",
      "[('main idea', 1)]\n"
     ]
    }
   ],
   "source": [
    "#getbigrams\n",
    "\n",
    "print(get_bigrams(first_item,test_list)) # One possible bigram\n",
    "print(get_bigrams(middle_item,test_list)) # Two possible bigrams\n",
    "print(get_bigrams(last_item,test_list)) # One possible bigram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying getbigrams2\n",
    "\n",
    "non_words2['bigrams'] = non_words2[['misspelled_words','enumerated']].apply(lambda x: get_bigrams(x[0],x[1]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelled_words</th>\n",
       "      <th>sentence</th>\n",
       "      <th>enumerated</th>\n",
       "      <th>bigrams</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>(beacause, beacause, NN)</td>\n",
       "      <td>I organized the instructions by time, beacause...</td>\n",
       "      <td>[(0, i), (1, organized), (2, the), (3, instruc...</td>\n",
       "      <td>[(time beacause, 1), (beacause to, 2)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11</td>\n",
       "      <td>[(to, to, TO), (make, make, VB), (tea, tea, NN...</td>\n",
       "      <td>(wallmart, wallmart, NN)</td>\n",
       "      <td>next, you need to buy a box of tea in wallmart...</td>\n",
       "      <td>[(0, next), (1, you), (2, need), (3, to), (4, ...</td>\n",
       "      <td>[(in wallmart, 1), (wallmart or, 2)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(dovn, dovn, NN)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "      <td>[(0, first), (1, you), (2, should), (3, take),...</td>\n",
       "      <td>[(use dovn, 1), (dovn mircowave, 2)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(mircowave, mircowave, VBP)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "      <td>[(0, first), (1, you), (2, should), (3, take),...</td>\n",
       "      <td>[(dovn mircowave, 1), (mircowave or, 2)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15</td>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>(fitst, Fitst, NNP)</td>\n",
       "      <td>Fitst, boil a water in a pot.</td>\n",
       "      <td>[(0, fitst), (1, boil), (2, a), (3, water), (4...</td>\n",
       "      <td>[(fitst boil, 2)]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id                                        tok_lem_POS  \\\n",
       "0          8  [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "1         11  [(to, to, TO), (make, make, VB), (tea, tea, NN...   \n",
       "2         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "3         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "4         15  [(in, in, IN), (my, my, PRP$), (country, count...   \n",
       "\n",
       "              misspelled_words  \\\n",
       "0     (beacause, beacause, NN)   \n",
       "1     (wallmart, wallmart, NN)   \n",
       "2             (dovn, dovn, NN)   \n",
       "3  (mircowave, mircowave, VBP)   \n",
       "4          (fitst, Fitst, NNP)   \n",
       "\n",
       "                                            sentence  \\\n",
       "0  I organized the instructions by time, beacause...   \n",
       "1  next, you need to buy a box of tea in wallmart...   \n",
       "2  First, you should take some hot water, you can...   \n",
       "3  First, you should take some hot water, you can...   \n",
       "4                      Fitst, boil a water in a pot.   \n",
       "\n",
       "                                          enumerated  \\\n",
       "0  [(0, i), (1, organized), (2, the), (3, instruc...   \n",
       "1  [(0, next), (1, you), (2, need), (3, to), (4, ...   \n",
       "2  [(0, first), (1, you), (2, should), (3, take),...   \n",
       "3  [(0, first), (1, you), (2, should), (3, take),...   \n",
       "4  [(0, fitst), (1, boil), (2, a), (3, water), (4...   \n",
       "\n",
       "                                    bigrams  \n",
       "0    [(time beacause, 1), (beacause to, 2)]  \n",
       "1      [(in wallmart, 1), (wallmart or, 2)]  \n",
       "2      [(use dovn, 1), (dovn mircowave, 2)]  \n",
       "3  [(dovn mircowave, 1), (mircowave or, 2)]  \n",
       "4                         [(fitst boil, 2)]  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "non_words2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelled_words</th>\n",
       "      <th>sentence</th>\n",
       "      <th>enumerated</th>\n",
       "      <th>bigrams</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>872</th>\n",
       "      <td>1469</td>\n",
       "      <td>[(about, about, IN), (4, 4, CD), (days, day, N...</td>\n",
       "      <td>(shoudl, shoudl, VBP)</td>\n",
       "      <td>Anyway, we went to a hospital and the doctor s...</td>\n",
       "      <td>[(0, anyway), (1, we), (2, went), (3, to), (4,...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1057</th>\n",
       "      <td>1797</td>\n",
       "      <td>[(i, I, PRP), (failed, fail, VBD), (the, the, ...</td>\n",
       "      <td>(idid, Idid, NNP)</td>\n",
       "      <td>I failed the test because Ididn't study.</td>\n",
       "      <td>[(0, i), (1, failed), (2, the), (3, test), (4,...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2337</th>\n",
       "      <td>4804</td>\n",
       "      <td>[(bill, Bill, NNP), (shoul, shoul, VBP), (n't,...</td>\n",
       "      <td>(shoul, shoul, VBP)</td>\n",
       "      <td>Bill shouln't have lied about the price of the...</td>\n",
       "      <td>[(0, bill), (1, shouln), (2, t), (3, have), (4...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2340</th>\n",
       "      <td>4830</td>\n",
       "      <td>[(caroline, Caroline, NNP), (must, must, MD), ...</td>\n",
       "      <td>(sould, sould, VBP)</td>\n",
       "      <td>Caroline souldn't have insulted Bill in front ...</td>\n",
       "      <td>[(0, caroline), (1, souldn), (2, t), (3, have)...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2602</th>\n",
       "      <td>6346</td>\n",
       "      <td>[(last, last, JJ), (year, year, NN), (,, ,, ,)...</td>\n",
       "      <td>(sould, sould, MD)</td>\n",
       "      <td>I told the officer that he caused the accident...</td>\n",
       "      <td>[(0, i), (1, told), (2, the), (3, officer), (4...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2605</th>\n",
       "      <td>6346</td>\n",
       "      <td>[(last, last, JJ), (year, year, NN), (,, ,, ,)...</td>\n",
       "      <td>(sould, sould, VBP)</td>\n",
       "      <td>I told the officer that he caused the accident...</td>\n",
       "      <td>[(0, i), (1, told), (2, the), (3, officer), (4...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3589</th>\n",
       "      <td>8640</td>\n",
       "      <td>[(before, before, IN), (i, I, PRP), (came, com...</td>\n",
       "      <td>(daoe, daoe, VBP)</td>\n",
       "      <td>I like to play soccer,but in Korea daoen't hav...</td>\n",
       "      <td>[(0, i), (1, like), (2, to), (3, play), (4, so...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3868</th>\n",
       "      <td>10001</td>\n",
       "      <td>[(you, You, PRP), (shoud, shoud, VBP), (n't, n...</td>\n",
       "      <td>(shoud, shoud, VBP)</td>\n",
       "      <td>You shoudn't</td>\n",
       "      <td>[(0, you), (1, shoudn), (2, t)]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9735</th>\n",
       "      <td>23939</td>\n",
       "      <td>[(i, I, PRP), (have, have, VBP), (an, a, DT), ...</td>\n",
       "      <td>(sould, sould, VBP)</td>\n",
       "      <td>I souldn't have gone to his wedding party.</td>\n",
       "      <td>[(0, i), (1, souldn), (2, t), (3, have), (4, g...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10753</th>\n",
       "      <td>26126</td>\n",
       "      <td>[(i, I, PRP), (like, like, VBP), (cooking, coo...</td>\n",
       "      <td>(doed, doed, VBP)</td>\n",
       "      <td>But In pittsburgh doedn't have good traditiona...</td>\n",
       "      <td>[(0, but), (1, in), (2, pittsburgh), (3, doedn...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11242</th>\n",
       "      <td>27409</td>\n",
       "      <td>[(on, on, IN), (saturday, saturday, NN), (,, ,...</td>\n",
       "      <td>(shoul, shoul, VBP)</td>\n",
       "      <td>I shouln't have gone Giant Eagle or I could ha...</td>\n",
       "      <td>[(0, i), (1, shouln), (2, t), (3, have), (4, g...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11402</th>\n",
       "      <td>27911</td>\n",
       "      <td>[(maui, Maui, NNP), (is, be, VBZ), (the, the, ...</td>\n",
       "      <td>(counld, counld, VBP)</td>\n",
       "      <td>I counldn't forget the experience when I was i...</td>\n",
       "      <td>[(0, i), (1, counldn), (2, t), (3, forget), (4...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12694</th>\n",
       "      <td>30978</td>\n",
       "      <td>[(as, as, IN), (being, be, VBG), (a, a, DT), (...</td>\n",
       "      <td>(woould, woould, VBP)</td>\n",
       "      <td>All of them are native speakers, I was pretty ...</td>\n",
       "      <td>[(0, all), (1, of), (2, them), (3, are), (4, n...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15659</th>\n",
       "      <td>37339</td>\n",
       "      <td>[(you, You, PRP), (should, should, MD), (visit...</td>\n",
       "      <td>(shold, shold, VBP)</td>\n",
       "      <td>You sholdn't shake hands when you first meet s...</td>\n",
       "      <td>[(0, you), (1, sholdn), (2, t), (3, shake), (4...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       answer_id                                        tok_lem_POS  \\\n",
       "872         1469  [(about, about, IN), (4, 4, CD), (days, day, N...   \n",
       "1057        1797  [(i, I, PRP), (failed, fail, VBD), (the, the, ...   \n",
       "2337        4804  [(bill, Bill, NNP), (shoul, shoul, VBP), (n't,...   \n",
       "2340        4830  [(caroline, Caroline, NNP), (must, must, MD), ...   \n",
       "2602        6346  [(last, last, JJ), (year, year, NN), (,, ,, ,)...   \n",
       "2605        6346  [(last, last, JJ), (year, year, NN), (,, ,, ,)...   \n",
       "3589        8640  [(before, before, IN), (i, I, PRP), (came, com...   \n",
       "3868       10001  [(you, You, PRP), (shoud, shoud, VBP), (n't, n...   \n",
       "9735       23939  [(i, I, PRP), (have, have, VBP), (an, a, DT), ...   \n",
       "10753      26126  [(i, I, PRP), (like, like, VBP), (cooking, coo...   \n",
       "11242      27409  [(on, on, IN), (saturday, saturday, NN), (,, ,...   \n",
       "11402      27911  [(maui, Maui, NNP), (is, be, VBZ), (the, the, ...   \n",
       "12694      30978  [(as, as, IN), (being, be, VBG), (a, a, DT), (...   \n",
       "15659      37339  [(you, You, PRP), (should, should, MD), (visit...   \n",
       "\n",
       "            misspelled_words  \\\n",
       "872    (shoudl, shoudl, VBP)   \n",
       "1057       (idid, Idid, NNP)   \n",
       "2337     (shoul, shoul, VBP)   \n",
       "2340     (sould, sould, VBP)   \n",
       "2602      (sould, sould, MD)   \n",
       "2605     (sould, sould, VBP)   \n",
       "3589       (daoe, daoe, VBP)   \n",
       "3868     (shoud, shoud, VBP)   \n",
       "9735     (sould, sould, VBP)   \n",
       "10753      (doed, doed, VBP)   \n",
       "11242    (shoul, shoul, VBP)   \n",
       "11402  (counld, counld, VBP)   \n",
       "12694  (woould, woould, VBP)   \n",
       "15659    (shold, shold, VBP)   \n",
       "\n",
       "                                                sentence  \\\n",
       "872    Anyway, we went to a hospital and the doctor s...   \n",
       "1057            I failed the test because Ididn't study.   \n",
       "2337   Bill shouln't have lied about the price of the...   \n",
       "2340   Caroline souldn't have insulted Bill in front ...   \n",
       "2602   I told the officer that he caused the accident...   \n",
       "2605   I told the officer that he caused the accident...   \n",
       "3589   I like to play soccer,but in Korea daoen't hav...   \n",
       "3868                                        You shoudn't   \n",
       "9735          I souldn't have gone to his wedding party.   \n",
       "10753  But In pittsburgh doedn't have good traditiona...   \n",
       "11242  I shouln't have gone Giant Eagle or I could ha...   \n",
       "11402  I counldn't forget the experience when I was i...   \n",
       "12694  All of them are native speakers, I was pretty ...   \n",
       "15659  You sholdn't shake hands when you first meet s...   \n",
       "\n",
       "                                              enumerated bigrams  \n",
       "872    [(0, anyway), (1, we), (2, went), (3, to), (4,...    None  \n",
       "1057   [(0, i), (1, failed), (2, the), (3, test), (4,...    None  \n",
       "2337   [(0, bill), (1, shouln), (2, t), (3, have), (4...    None  \n",
       "2340   [(0, caroline), (1, souldn), (2, t), (3, have)...    None  \n",
       "2602   [(0, i), (1, told), (2, the), (3, officer), (4...    None  \n",
       "2605   [(0, i), (1, told), (2, the), (3, officer), (4...    None  \n",
       "3589   [(0, i), (1, like), (2, to), (3, play), (4, so...    None  \n",
       "3868                     [(0, you), (1, shoudn), (2, t)]    None  \n",
       "9735   [(0, i), (1, souldn), (2, t), (3, have), (4, g...    None  \n",
       "10753  [(0, but), (1, in), (2, pittsburgh), (3, doedn...    None  \n",
       "11242  [(0, i), (1, shouln), (2, t), (3, have), (4, g...    None  \n",
       "11402  [(0, i), (1, counldn), (2, t), (3, forget), (4...    None  \n",
       "12694  [(0, all), (1, of), (2, them), (3, are), (4, n...    None  \n",
       "15659  [(0, you), (1, sholdn), (2, t), (3, shake), (4...    None  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The 14 misspellings with contracted words did not return any suggestions as the n't is handled differently by \n",
    "# the re_tokenize function which strips punctuation and the nltk tokenizer used to create the tok_lem_POS column\n",
    "\n",
    "no_bigrams = non_words2.loc[(non_words2['bigrams'].isnull()) & (non_words2.enumerated.str.len() >1),:]\n",
    "print(len(no_bigrams))\n",
    "no_bigrams.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating function to fix these items\n",
    "\n",
    "misspelled_contractions = [x[0] for x in no_bigrams.misspelled_words]\n",
    "\n",
    "def fix_contractions(word):\n",
    "    if word == \"t\":\n",
    "        word = \"n't\"\n",
    "    if word[:-1] in misspelled_contractions:\n",
    "        word = word[:-1]\n",
    "    return word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying the above function to the selected rows\n",
    "\n",
    "mask = non_words2.index.isin(no_bigrams.index)\n",
    "non_words2.loc[mask, 'enumerated'] = non_words2.loc[mask, 'enumerated'].apply(\n",
    "    lambda row: [(x[0],fix_contractions(x[1])) for x in row])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reapplying the bigrams function\n",
    "\n",
    "non_words2['bigrams'] = non_words2[['misspelled_words','enumerated']].apply(lambda x: get_bigrams(x[0],x[1]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelled_words</th>\n",
       "      <th>sentence</th>\n",
       "      <th>enumerated</th>\n",
       "      <th>bigrams</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [answer_id, tok_lem_POS, misspelled_words, sentence, enumerated, bigrams]\n",
       "Index: []"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Re-checking that all rows have bigrams\n",
    "non_words2.loc[(non_words2['bigrams'].isnull()) & (non_words2.enumerated.str.len() >1),:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deleting the enumerated column as no longer necessary\n",
    "\n",
    "del non_words2['enumerated']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Renaming the 'misspelled_words' column as there is only one word in each row\n",
    "\n",
    "non_words2 = non_words2.rename(columns={\"misspelled_words\": \"misspelling\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelling</th>\n",
       "      <th>sentence</th>\n",
       "      <th>bigrams</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>(beacause, beacause, NN)</td>\n",
       "      <td>I organized the instructions by time, beacause...</td>\n",
       "      <td>[(time beacause, 1), (beacause to, 2)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11</td>\n",
       "      <td>[(to, to, TO), (make, make, VB), (tea, tea, NN...</td>\n",
       "      <td>(wallmart, wallmart, NN)</td>\n",
       "      <td>next, you need to buy a box of tea in wallmart...</td>\n",
       "      <td>[(in wallmart, 1), (wallmart or, 2)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(dovn, dovn, NN)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "      <td>[(use dovn, 1), (dovn mircowave, 2)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(mircowave, mircowave, VBP)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "      <td>[(dovn mircowave, 1), (mircowave or, 2)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15</td>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>(fitst, Fitst, NNP)</td>\n",
       "      <td>Fitst, boil a water in a pot.</td>\n",
       "      <td>[(fitst boil, 2)]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id                                        tok_lem_POS  \\\n",
       "0          8  [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "1         11  [(to, to, TO), (make, make, VB), (tea, tea, NN...   \n",
       "2         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "3         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "4         15  [(in, in, IN), (my, my, PRP$), (country, count...   \n",
       "\n",
       "                   misspelling  \\\n",
       "0     (beacause, beacause, NN)   \n",
       "1     (wallmart, wallmart, NN)   \n",
       "2             (dovn, dovn, NN)   \n",
       "3  (mircowave, mircowave, VBP)   \n",
       "4          (fitst, Fitst, NNP)   \n",
       "\n",
       "                                            sentence  \\\n",
       "0  I organized the instructions by time, beacause...   \n",
       "1  next, you need to buy a box of tea in wallmart...   \n",
       "2  First, you should take some hot water, you can...   \n",
       "3  First, you should take some hot water, you can...   \n",
       "4                      Fitst, boil a water in a pot.   \n",
       "\n",
       "                                    bigrams  \n",
       "0    [(time beacause, 1), (beacause to, 2)]  \n",
       "1      [(in wallmart, 1), (wallmart or, 2)]  \n",
       "2      [(use dovn, 1), (dovn mircowave, 2)]  \n",
       "3  [(dovn mircowave, 1), (mircowave or, 2)]  \n",
       "4                         [(fitst boil, 2)]  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking final non_words2 dataframe\n",
    "\n",
    "non_words2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19814\n",
      "14874\n"
     ]
    }
   ],
   "source": [
    "# Total number of non-words (tokens)\n",
    "print(len(non_words2))\n",
    "\n",
    "# Total number of non-words (types)\n",
    "print(non_words2.misspelling.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a dataframe of misspellings\n",
    "In the `non-words2` dataframe above, each row is an occurrence of a misspelling (i.e. _tokens_ ). We also want a dataframe where each row is a misspelling _type_ with frequency information attached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gathering the total misspellings and bigrams\n",
    "\n",
    "total_unigram_misspell = [x for x in non_words2['misspelling']]\n",
    "total_bigram_misspell = [x[0] for y in non_words2['bigrams'] for x in y] #flattened list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('beacause', 'beacause', 'NN'), ('wallmart', 'wallmart', 'NN'), ('dovn', 'dovn', 'NN'), ('mircowave', 'mircowave', 'VBP'), ('fitst', 'Fitst', 'NNP')]\n",
      "['time beacause', 'beacause to', 'in wallmart', 'wallmart or', 'use dovn']\n"
     ]
    }
   ],
   "source": [
    "print(total_unigram_misspell[:5])\n",
    "print(total_bigram_misspell[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating frequency dictionaries for unigrams and bigrams\n",
    "\n",
    "unigram_misspell_freq_dict = {}\n",
    "for word in total_unigram_misspell:\n",
    "    if word not in unigram_misspell_freq_dict:\n",
    "        unigram_misspell_freq_dict[word] = 1\n",
    "    else:\n",
    "        unigram_misspell_freq_dict[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigram_misspell_freq_dict = {}\n",
    "for bigram in total_bigram_misspell:\n",
    "    if bigram not in bigram_misspell_freq_dict:\n",
    "        bigram_misspell_freq_dict[bigram] = 1\n",
    "    else:\n",
    "        bigram_misspell_freq_dict[bigram] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('abdulwahed', 'Abdulwahed', 'NNP'), ('cremony', 'cremony', 'NN'), ('abroading', 'abroading', 'VBG'), ('studty', 'studty', 'VB'), ('fishs', 'fishs', 'NN')]\n",
      "['movels plays', 'awaew of', 'the mosr', 'brtter than', 'of twix']\n"
     ]
    }
   ],
   "source": [
    "# Checking dictionaries\n",
    "\n",
    "print(random.sample(list(unigram_misspell_freq_dict),5))\n",
    "print(random.sample(list(bigram_misspell_freq_dict),5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14874\n",
      "30188\n"
     ]
    }
   ],
   "source": [
    "# Remove duplicates\n",
    "\n",
    "final_unigram_misspellings = sorted(list(set(total_unigram_misspell)))\n",
    "final_bigram_misspellings = sorted(list(set(total_bigram_misspell)))\n",
    "print(len(final_unigram_misspellings))\n",
    "print(len(final_bigram_misspellings))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aa</td>\n",
       "      <td>AA</td>\n",
       "      <td>NNP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>aa</td>\n",
       "      <td>Aa</td>\n",
       "      <td>NNP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>aa</td>\n",
       "      <td>Aa</td>\n",
       "      <td>VB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aabout</td>\n",
       "      <td>Aabout</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>aad</td>\n",
       "      <td>aad</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        0       1    2\n",
       "0      aa      AA  NNP\n",
       "1      aa      Aa  NNP\n",
       "2      aa      Aa   VB\n",
       "3  aabout  Aabout   IN\n",
       "4     aad     aad   JJ"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Constructing misspell_df\n",
    "\n",
    "misspell_df = pd.DataFrame(final_unigram_misspellings)\n",
    "misspell_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Renaming columns to match other DataFrames in this notebook\n",
    "\n",
    "misspell_df.rename(columns = {0: 'misspelling',1:'lemma',2:'POS'}, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>misspelling</th>\n",
       "      <th>lemma</th>\n",
       "      <th>POS</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aa</td>\n",
       "      <td>AA</td>\n",
       "      <td>NNP</td>\n",
       "      <td>(aa, AA, NNP)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>aa</td>\n",
       "      <td>Aa</td>\n",
       "      <td>NNP</td>\n",
       "      <td>(aa, Aa, NNP)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>aa</td>\n",
       "      <td>Aa</td>\n",
       "      <td>VB</td>\n",
       "      <td>(aa, Aa, VB)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aabout</td>\n",
       "      <td>Aabout</td>\n",
       "      <td>IN</td>\n",
       "      <td>(aabout, Aabout, IN)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>aad</td>\n",
       "      <td>aad</td>\n",
       "      <td>JJ</td>\n",
       "      <td>(aad, aad, JJ)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  misspelling   lemma  POS           tok_lem_POS\n",
       "0          aa      AA  NNP         (aa, AA, NNP)\n",
       "1          aa      Aa  NNP         (aa, Aa, NNP)\n",
       "2          aa      Aa   VB          (aa, Aa, VB)\n",
       "3      aabout  Aabout   IN  (aabout, Aabout, IN)\n",
       "4         aad     aad   JJ        (aad, aad, JJ)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Recreating tok_lem_POS column to match dictionary\n",
    "\n",
    "misspell_df['tok_lem_POS'] = list(zip(misspell_df.misspelling, misspell_df.lemma, misspell_df.POS))\n",
    "misspell_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mapping dictionary to DataFrame\n",
    "\n",
    "misspell_df['freq'] = misspell_df['tok_lem_POS'].map(unigram_misspell_freq_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sorting by frequency\n",
    "\n",
    "misspell_df = misspell_df.sort_values(by=['freq'], ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>misspelling</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>alot</td>\n",
       "      <td>(alot, alot, NN)</td>\n",
       "      <td>103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>studing</td>\n",
       "      <td>(studing, studing, VBG)</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>goverment</td>\n",
       "      <td>(goverment, goverment, NN)</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tofel</td>\n",
       "      <td>(tofel, TOFEL, NNP)</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iam</td>\n",
       "      <td>(iam, Iam, NNP)</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>finaly</td>\n",
       "      <td>(finaly, Finaly, NNP)</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>becuase</td>\n",
       "      <td>(becuase, becuase, NN)</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>beatiful</td>\n",
       "      <td>(beatiful, beatiful, JJ)</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>oss</td>\n",
       "      <td>(oss, OSS, NNP)</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>shoud</td>\n",
       "      <td>(shoud, shoud, VBP)</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>differents</td>\n",
       "      <td>(differents, differents, NNS)</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>somthing</td>\n",
       "      <td>(somthing, somthing, VBG)</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>eilperin</td>\n",
       "      <td>(eilperin, Eilperin, NNP)</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>befor</td>\n",
       "      <td>(befor, befor, NN)</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>okland</td>\n",
       "      <td>(okland, Okland, NNP)</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>sould</td>\n",
       "      <td>(sould, sould, VBP)</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>resturant</td>\n",
       "      <td>(resturant, resturant, NN)</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>begining</td>\n",
       "      <td>(begining, begining, NN)</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>becouse</td>\n",
       "      <td>(becouse, becouse, IN)</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>referes</td>\n",
       "      <td>(referes, referes, VBZ)</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   misspelling                    tok_lem_POS  freq\n",
       "0         alot               (alot, alot, NN)   103\n",
       "1      studing        (studing, studing, VBG)    62\n",
       "2    goverment     (goverment, goverment, NN)    36\n",
       "3        tofel            (tofel, TOFEL, NNP)    29\n",
       "4          iam                (iam, Iam, NNP)    28\n",
       "5       finaly          (finaly, Finaly, NNP)    26\n",
       "6      becuase         (becuase, becuase, NN)    24\n",
       "7     beatiful       (beatiful, beatiful, JJ)    24\n",
       "8          oss                (oss, OSS, NNP)    21\n",
       "9        shoud            (shoud, shoud, VBP)    21\n",
       "10  differents  (differents, differents, NNS)    21\n",
       "11    somthing      (somthing, somthing, VBG)    20\n",
       "12    eilperin      (eilperin, Eilperin, NNP)    20\n",
       "13       befor             (befor, befor, NN)    20\n",
       "14      okland          (okland, Okland, NNP)    20\n",
       "15       sould            (sould, sould, VBP)    18\n",
       "16   resturant     (resturant, resturant, NN)    18\n",
       "17    begining       (begining, begining, NN)    16\n",
       "18     becouse         (becouse, becouse, IN)    16\n",
       "19     referes        (referes, referes, VBZ)    16"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Resetting index and deleting unnecesary columns\n",
    "\n",
    "misspell_df = misspell_df.reset_index(drop = True)\n",
    "del misspell_df['lemma']\n",
    "del misspell_df['POS']\n",
    "\n",
    "misspell_df.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### scowl_supp\n",
    "The following is the basis for the 'scowl_supp' list used earlier. Here, errors with a frequency of 10 or more were manually checked, and if determined to be a real word, were added to the scowl_supp list. There were originally 267 items which met this criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "54\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>misspelling</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>alot</td>\n",
       "      <td>(alot, alot, NN)</td>\n",
       "      <td>103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>studing</td>\n",
       "      <td>(studing, studing, VBG)</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>goverment</td>\n",
       "      <td>(goverment, goverment, NN)</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tofel</td>\n",
       "      <td>(tofel, TOFEL, NNP)</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iam</td>\n",
       "      <td>(iam, Iam, NNP)</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>finaly</td>\n",
       "      <td>(finaly, Finaly, NNP)</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>becuase</td>\n",
       "      <td>(becuase, becuase, NN)</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>beatiful</td>\n",
       "      <td>(beatiful, beatiful, JJ)</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>oss</td>\n",
       "      <td>(oss, OSS, NNP)</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>shoud</td>\n",
       "      <td>(shoud, shoud, VBP)</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>differents</td>\n",
       "      <td>(differents, differents, NNS)</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>somthing</td>\n",
       "      <td>(somthing, somthing, VBG)</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>eilperin</td>\n",
       "      <td>(eilperin, Eilperin, NNP)</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>befor</td>\n",
       "      <td>(befor, befor, NN)</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>okland</td>\n",
       "      <td>(okland, Okland, NNP)</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>sould</td>\n",
       "      <td>(sould, sould, VBP)</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>resturant</td>\n",
       "      <td>(resturant, resturant, NN)</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>begining</td>\n",
       "      <td>(begining, begining, NN)</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>becouse</td>\n",
       "      <td>(becouse, becouse, IN)</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>referes</td>\n",
       "      <td>(referes, referes, VBZ)</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>mikyoung</td>\n",
       "      <td>(mikyoung, Mikyoung, NNP)</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>confortable</td>\n",
       "      <td>(confortable, confortable, JJ)</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>coffe</td>\n",
       "      <td>(coffe, coffe, NN)</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>lifes</td>\n",
       "      <td>(lifes, lifes, NN)</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>becouse</td>\n",
       "      <td>(becouse, becouse, NN)</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>jop</td>\n",
       "      <td>(jop, jop, NN)</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>decasia</td>\n",
       "      <td>(decasia, Decasia, NNP)</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>roomate</td>\n",
       "      <td>(roomate, roomate, NN)</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>freind</td>\n",
       "      <td>(freind, freind, NN)</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>healty</td>\n",
       "      <td>(healty, healty, NN)</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>eventhough</td>\n",
       "      <td>(eventhough, eventhough, IN)</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>becasue</td>\n",
       "      <td>(becasue, becasue, NN)</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>frinds</td>\n",
       "      <td>(frinds, frinds, NNS)</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>vic</td>\n",
       "      <td>(vic, Vic, NNP)</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>poeple</td>\n",
       "      <td>(poeple, poeple, NN)</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>experince</td>\n",
       "      <td>(experince, experince, NN)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>favorit</td>\n",
       "      <td>(favorit, favorit, NN)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>dailer</td>\n",
       "      <td>(dailer, Dailer, NNP)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>tofel</td>\n",
       "      <td>(tofel, Tofel, NNP)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>everythings</td>\n",
       "      <td>(everythings, everythings, NNS)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>misurata</td>\n",
       "      <td>(misurata, Misurata, NNP)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>improtant</td>\n",
       "      <td>(improtant, improtant, JJ)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>belived</td>\n",
       "      <td>(belived, belived, VBN)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>frind</td>\n",
       "      <td>(frind, frind, NN)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>occured</td>\n",
       "      <td>(occured, occur, VBN)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>ustr</td>\n",
       "      <td>(ustr, USTR, NNP)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>sentance</td>\n",
       "      <td>(sentance, sentance, NN)</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>differnt</td>\n",
       "      <td>(differnt, differnt, JJ)</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>monthes</td>\n",
       "      <td>(monthes, monthes, NNS)</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>favorit</td>\n",
       "      <td>(favorit, favorit, JJ)</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>eatting</td>\n",
       "      <td>(eatting, eatting, VBG)</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>childrens</td>\n",
       "      <td>(childrens, child, NNS)</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>freinds</td>\n",
       "      <td>(freinds, freinds, NNS)</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>stangor</td>\n",
       "      <td>(stangor, Stangor, NNP)</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    misspelling                      tok_lem_POS  freq\n",
       "0          alot                 (alot, alot, NN)   103\n",
       "1       studing          (studing, studing, VBG)    62\n",
       "2     goverment       (goverment, goverment, NN)    36\n",
       "3         tofel              (tofel, TOFEL, NNP)    29\n",
       "4           iam                  (iam, Iam, NNP)    28\n",
       "5        finaly            (finaly, Finaly, NNP)    26\n",
       "6       becuase           (becuase, becuase, NN)    24\n",
       "7      beatiful         (beatiful, beatiful, JJ)    24\n",
       "8           oss                  (oss, OSS, NNP)    21\n",
       "9         shoud              (shoud, shoud, VBP)    21\n",
       "10   differents    (differents, differents, NNS)    21\n",
       "11     somthing        (somthing, somthing, VBG)    20\n",
       "12     eilperin        (eilperin, Eilperin, NNP)    20\n",
       "13        befor               (befor, befor, NN)    20\n",
       "14       okland            (okland, Okland, NNP)    20\n",
       "15        sould              (sould, sould, VBP)    18\n",
       "16    resturant       (resturant, resturant, NN)    18\n",
       "17     begining         (begining, begining, NN)    16\n",
       "18      becouse           (becouse, becouse, IN)    16\n",
       "19      referes          (referes, referes, VBZ)    16\n",
       "20     mikyoung        (mikyoung, Mikyoung, NNP)    15\n",
       "21  confortable   (confortable, confortable, JJ)    15\n",
       "22        coffe               (coffe, coffe, NN)    15\n",
       "23        lifes               (lifes, lifes, NN)    15\n",
       "24      becouse           (becouse, becouse, NN)    14\n",
       "25          jop                   (jop, jop, NN)    14\n",
       "26      decasia          (decasia, Decasia, NNP)    13\n",
       "27      roomate           (roomate, roomate, NN)    13\n",
       "28       freind             (freind, freind, NN)    13\n",
       "29       healty             (healty, healty, NN)    13\n",
       "30   eventhough     (eventhough, eventhough, IN)    12\n",
       "31      becasue           (becasue, becasue, NN)    12\n",
       "32       frinds            (frinds, frinds, NNS)    12\n",
       "33          vic                  (vic, Vic, NNP)    12\n",
       "34       poeple             (poeple, poeple, NN)    12\n",
       "35    experince       (experince, experince, NN)    11\n",
       "36      favorit           (favorit, favorit, NN)    11\n",
       "37       dailer            (dailer, Dailer, NNP)    11\n",
       "38        tofel              (tofel, Tofel, NNP)    11\n",
       "39  everythings  (everythings, everythings, NNS)    11\n",
       "40     misurata        (misurata, Misurata, NNP)    11\n",
       "41    improtant       (improtant, improtant, JJ)    11\n",
       "42      belived          (belived, belived, VBN)    11\n",
       "43        frind               (frind, frind, NN)    11\n",
       "44      occured            (occured, occur, VBN)    11\n",
       "45         ustr                (ustr, USTR, NNP)    11\n",
       "46     sentance         (sentance, sentance, NN)    10\n",
       "47     differnt         (differnt, differnt, JJ)    10\n",
       "48      monthes          (monthes, monthes, NNS)    10\n",
       "49      favorit           (favorit, favorit, JJ)    10\n",
       "50      eatting          (eatting, eatting, VBG)    10\n",
       "51    childrens          (childrens, child, NNS)    10\n",
       "52      freinds          (freinds, freinds, NNS)    10\n",
       "53      stangor          (stangor, Stangor, NNP)    10"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(len(misspell_df.loc[misspell_df.freq >= 10]))\n",
    "misspell_df.loc[misspell_df.freq >= 10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Possible segmentation\n",
    "\n",
    "Selected segmenter and spellchecker: SymSpell https://github.com/mammothb/symspellpy\n",
    "\n",
    "There is a dictionary file which which needs to be installed (saved to repo):\n",
    "[frequency_dictionary_en_82_765.txt](https://symspellpy.readthedocs.io/en/latest/users/installing.html)\n",
    "\n",
    "To install symspellpy the first time, use pip in command line: `pip install -U symspellpy`\n",
    "\n",
    "Prior to spelling correct, we first consider using the segmenter. This is a potentially useful first step as misspellings like 'alot' or 'dogmeat' will be separated into 'a lot' and 'dog meat' rather than corrected to a single word like 'lot'.  \n",
    "\n",
    "However, when segementing misspellings, the segmenter over performs, segmenting non-words into real words where it was clearly not intended, e.g. _improtant_ into _imp rot ant_ or _befor_ into _be for_. As such, the segmenting will not be automated. \n",
    "\n",
    "Instead, we rely on the bigram spelling correction which correctly segments items like _alot, iam, everytime,_ etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 23135851162),\n",
       " ('of', 13151942776),\n",
       " ('and', 12997637966),\n",
       " ('to', 12136980858),\n",
       " ('a', 9081174698)]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setting up symspell\n",
    "\n",
    "from itertools import islice\n",
    "import pkg_resources\n",
    "from symspellpy import SymSpell\n",
    "from symspellpy import Verbosity\n",
    "sym_spell = SymSpell()\n",
    "dictionary_path = pkg_resources.resource_filename(\n",
    "    \"symspellpy\", \"frequency_dictionary_en_82_765.txt\")\n",
    "sym_spell.load_dictionary(dictionary_path, 0, 1)\n",
    "\n",
    "# Print out first 5 elements to demonstrate that dictionary is successfully loaded\n",
    "list(islice(sym_spell.words.items(), 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Testing segmenter with 'alot' and 'dogmeat'\n",
    "\n",
    "# Set max_dictionary_edit_distance to avoid spelling correction\n",
    "sym_spell = SymSpell(max_dictionary_edit_distance=0, prefix_length=7)\n",
    "sym_spell.load_dictionary(dictionary_path, term_index=0, count_index=1)\n",
    "\n",
    "# It is also possible to display frequency with result.distance_sum and edit distance with .log_prob_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating function for applying the above code\n",
    "\n",
    "def get_segments(word):\n",
    "    segments = sym_spell.word_segmentation(word)\n",
    "    if len(segments.corrected_string.split(' ')) > 1 \\\n",
    "    and segments.corrected_string.split(' ')[0] in scowl and segments.corrected_string.split(' ')[1] in scowl:\n",
    "        return segments.corrected_string\n",
    "    else:\n",
    "        return word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dog meat\n",
      "fireplace\n",
      "becuase\n"
     ]
    }
   ],
   "source": [
    "# Testing function\n",
    "\n",
    "print(get_segments('dogmeat')) # Should be segmented\n",
    "print(get_segments('fireplace')) # Should not be segmented\n",
    "print(get_segments('becuase')) # Should not be segmented"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>misspelling</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>freq</th>\n",
       "      <th>segments</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>alot</td>\n",
       "      <td>(alot, alot, NN)</td>\n",
       "      <td>103</td>\n",
       "      <td>a lot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>studing</td>\n",
       "      <td>(studing, studing, VBG)</td>\n",
       "      <td>62</td>\n",
       "      <td>stu ding</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>goverment</td>\n",
       "      <td>(goverment, goverment, NN)</td>\n",
       "      <td>36</td>\n",
       "      <td>g over men t</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tofel</td>\n",
       "      <td>(tofel, TOFEL, NNP)</td>\n",
       "      <td>29</td>\n",
       "      <td>tofel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iam</td>\n",
       "      <td>(iam, Iam, NNP)</td>\n",
       "      <td>28</td>\n",
       "      <td>i am</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>finaly</td>\n",
       "      <td>(finaly, Finaly, NNP)</td>\n",
       "      <td>26</td>\n",
       "      <td>final y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>becuase</td>\n",
       "      <td>(becuase, becuase, NN)</td>\n",
       "      <td>24</td>\n",
       "      <td>becuase</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>beatiful</td>\n",
       "      <td>(beatiful, beatiful, JJ)</td>\n",
       "      <td>24</td>\n",
       "      <td>beat if ul</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>oss</td>\n",
       "      <td>(oss, OSS, NNP)</td>\n",
       "      <td>21</td>\n",
       "      <td>oss</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>shoud</td>\n",
       "      <td>(shoud, shoud, VBP)</td>\n",
       "      <td>21</td>\n",
       "      <td>sh oud</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  misspelling                 tok_lem_POS  freq      segments\n",
       "0        alot            (alot, alot, NN)   103         a lot\n",
       "1     studing     (studing, studing, VBG)    62      stu ding\n",
       "2   goverment  (goverment, goverment, NN)    36  g over men t\n",
       "3       tofel         (tofel, TOFEL, NNP)    29         tofel\n",
       "4         iam             (iam, Iam, NNP)    28          i am\n",
       "5      finaly       (finaly, Finaly, NNP)    26       final y\n",
       "6     becuase      (becuase, becuase, NN)    24       becuase\n",
       "7    beatiful    (beatiful, beatiful, JJ)    24    beat if ul\n",
       "8         oss             (oss, OSS, NNP)    21           oss\n",
       "9       shoud         (shoud, shoud, VBP)    21        sh oud"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Applying the function to create a new column\n",
    "\n",
    "misspell_df['segments'] =  misspell_df['misspelling'].apply(get_segments)\n",
    "misspell_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deleting this new column as segmentation creates false segments of misspelled words\n",
    "\n",
    "del misspell_df['segments']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying spelling correction\n",
    "\n",
    "In some ways SymSpell is not ideal as full sentence context is not considered, only general frequencies. However, other well-known spellcheckers (hunspell, pyspell, etc.) use the same strategy - frequency based criteria for suggestions, without considering immediate cotext. As such, we have followed this common practice, but it is important to remember that accuracy of corrected tokens will not be 100% and must be taken into consideration.\n",
    "\n",
    "As a compromise and to consider context, spelling corrections based on bigrams is first implemented. If no suggestions are available, spelling corrections based on unigrams are implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "because, 1, 271323986\n"
     ]
    }
   ],
   "source": [
    "# Testing spelling suggestions with 'becuase'\n",
    "\n",
    "sym_spell = SymSpell(max_dictionary_edit_distance=2, prefix_length=7)\n",
    "dictionary_path = pkg_resources.resource_filename(\n",
    "    \"symspellpy\", \"frequency_dictionary_en_82_765.txt\")\n",
    "# term_index is the column of the term and count_index is the column of the term frequency\n",
    "sym_spell.load_dictionary(dictionary_path, term_index=0, count_index=1)\n",
    "\n",
    "input_term = \"becuase\"\n",
    "suggestions = sym_spell.lookup(input_term, Verbosity.CLOSEST, max_edit_distance=2, #Edit distance can be adjusted\n",
    "                               transfer_casing=True, #Optional argument set to ignore case\n",
    "                              include_unknown=True) #Return same word if unknown\n",
    "for suggestion in suggestions:\n",
    "    print(suggestion)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating function for finding unigram suggestions\n",
    "\n",
    "def get_unigram_suggestions(word):\n",
    "    if len(word) >= 4:\n",
    "        suggestions = sym_spell.lookup(word, Verbosity.CLOSEST,max_edit_distance=2, transfer_casing=True)\n",
    "    else:\n",
    "        suggestions = sym_spell.lookup(word, Verbosity.CLOSEST,max_edit_distance=1, transfer_casing=True)\n",
    "    return [str(x).split(',') for x in suggestions]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['because', ' 1', ' 271323986']]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Testing function\n",
    "\n",
    "get_unigram_suggestions('becuase')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: The function has a variable edit distance: words of length 4 or more get edit distance of 2, shorter words get edit distance of 1. These preferences can be adjusted in the function if desired."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "because of, 1, 3481714\n"
     ]
    }
   ],
   "source": [
    "# Testing spelling suggestions with 'becuase of'\n",
    "\n",
    "max_edit_distance_dictionary = 2\n",
    "prefix_length = 7\n",
    "sym_spell = SymSpell(max_edit_distance_dictionary, prefix_length)\n",
    "bigram_path = pkg_resources.resource_filename(\"symspellpy\", \"frequency_bigramdictionary_en_243_342.txt\")\n",
    "if not sym_spell.load_dictionary(dictionary_path, term_index=0, count_index=1):\n",
    "    print(\"Dictionary file not found\")\n",
    "if not sym_spell.load_bigram_dictionary(bigram_path, term_index=0, count_index=2):\n",
    "    print(\"Bigram dictionary file not found\")\n",
    "input_term = 'becuase of'\n",
    "max_edit_distance_lookup = 2\n",
    "suggestions = sym_spell.lookup_compound(input_term, max_edit_distance_lookup)\n",
    "for suggestion in suggestions:\n",
    "    print(suggestion) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating function for finding bigram suggestions\n",
    "\n",
    "def get_bigram_suggestions(bigram):\n",
    "    suggestions = sym_spell.lookup_compound(bigram, max_edit_distance_lookup)\n",
    "    for suggestion in suggestions:\n",
    "        return [str(x).split(',') for x in suggestions] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['work hard', ' 2', ' 53229']]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Testing function\n",
    "get_bigram_suggestions('worq harg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Returing to non_words2 dataframe and applying functions to create new column\n",
    "\n",
    "# Creating unigram suggestions column\n",
    "\n",
    "non_words2['unigram_suggestions'] =  non_words2['misspelling'].apply(\n",
    "    lambda x: get_unigram_suggestions(x[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turning into tuples for easier processing\n",
    "\n",
    "non_words2.unigram_suggestions = non_words2.unigram_suggestions.apply(\n",
    "    lambda row: [tuple(x) for x in row])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating bigram suggestions column\n",
    "\n",
    "non_words2['bigram_suggestions'] =  non_words2['bigrams'].apply(\n",
    "    lambda row: [(get_bigram_suggestions(x[0]),x[1]) for x in row])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flattening and turning into tuples for easier processing\n",
    "\n",
    "non_words2['bigram_suggestions'] = non_words2.bigram_suggestions.apply(\n",
    "    lambda row: [(tuple([i for j in x[0] for i in j]),x[1]) for x in row])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1900\n",
      "114\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "# Checking how many items without suggestions\n",
    "\n",
    "print(len(non_words2.loc[(non_words2.unigram_suggestions.str.len() == 0),:]))\n",
    "print(len(non_words2.loc[(non_words2.bigram_suggestions.str.len() == 0),:]))\n",
    "print(len(non_words2.loc[(non_words2.bigram_suggestions.str.len() == 0) & (non_words2.unigram_suggestions.str.len() == 0),:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelling</th>\n",
       "      <th>sentence</th>\n",
       "      <th>bigrams</th>\n",
       "      <th>unigram_suggestions</th>\n",
       "      <th>bigram_suggestions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>105</td>\n",
       "      <td>[(here, here, RB), (are, be, VBP), (a, a, DT),...</td>\n",
       "      <td>(asdfkjdlkfjadlfjalsdf, asdfkjdlkfjadlfjalsdf,...</td>\n",
       "      <td>asdfkjdlkfjadlfjalsdf</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3919</th>\n",
       "      <td>10490</td>\n",
       "      <td>[(getting, get, VBG), (good, Good, NNP), (grad...</td>\n",
       "      <td>(quintcareers, Quintcareers, NNS)</td>\n",
       "      <td>Quintcareers.</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6832</th>\n",
       "      <td>18380</td>\n",
       "      <td>[(hi, Hi, NNP), (,, ,, ,), (adrian, Adrian, NN...</td>\n",
       "      <td>(yeonjea, Yeonjea, NN)</td>\n",
       "      <td>- Yeonjea.</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7851</th>\n",
       "      <td>20089</td>\n",
       "      <td>[(uuuuuuu, uuuuuuu, NN)]</td>\n",
       "      <td>(uuuuuuu, uuuuuuu, NN)</td>\n",
       "      <td>uuuuuuu</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7852</th>\n",
       "      <td>20096</td>\n",
       "      <td>[(yhryr, yhryr, NN)]</td>\n",
       "      <td>(yhryr, yhryr, NN)</td>\n",
       "      <td>yhryr</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8472</th>\n",
       "      <td>21585</td>\n",
       "      <td>[(1, 1, CD), (., ., .), (individual, Individua...</td>\n",
       "      <td>(shadizubeidi, Shadizubeidi, NNP)</td>\n",
       "      <td>Shadizubeidi.</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8634</th>\n",
       "      <td>21845</td>\n",
       "      <td>[(when, when, WRB), (i, I, PRP), ('m, be, VBP)...</td>\n",
       "      <td>(jaaaaaaaaaaaaaaaaaaajajjajajajaja, Jaaaaaaaaa...</td>\n",
       "      <td>Jaaaaaaaaaaaaaaaaaaajajjajajajaja ;)</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9846</th>\n",
       "      <td>24136</td>\n",
       "      <td>[(hsthethghryjyjy, hsthethghryjyjy, NN)]</td>\n",
       "      <td>(hsthethghryjyjy, hsthethghryjyjy, NN)</td>\n",
       "      <td>hsthethghryjyjy</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10653</th>\n",
       "      <td>25988</td>\n",
       "      <td>[(fibromyalgia, Fibromyalgia, NNP), (?, ?, .),...</td>\n",
       "      <td>(fibromyalgia, Fibromyalgia, NNP)</td>\n",
       "      <td>Fibromyalgia ?</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12481</th>\n",
       "      <td>30421</td>\n",
       "      <td>[(stayhealthy, StayHealthy, NN)]</td>\n",
       "      <td>(stayhealthy, StayHealthy, NN)</td>\n",
       "      <td>StayHealthy</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13027</th>\n",
       "      <td>32028</td>\n",
       "      <td>[(hiiiiiii, hiiiiiii, NN)]</td>\n",
       "      <td>(hiiiiiii, hiiiiiii, NN)</td>\n",
       "      <td>hiiiiiii</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13737</th>\n",
       "      <td>33214</td>\n",
       "      <td>[(1, 1, CD), (), ), )), (it, It, PRP), (is, be...</td>\n",
       "      <td>(hahahaha, hahahaha, NN)</td>\n",
       "      <td>hahahaha.</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16865</th>\n",
       "      <td>39728</td>\n",
       "      <td>[(drylegbed, Drylegbed, NNP)]</td>\n",
       "      <td>(drylegbed, Drylegbed, NNP)</td>\n",
       "      <td>Drylegbed</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16905</th>\n",
       "      <td>39870</td>\n",
       "      <td>[(deepfried, deepfried, VBN)]</td>\n",
       "      <td>(deepfried, deepfried, VBN)</td>\n",
       "      <td>deepfried</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16906</th>\n",
       "      <td>39887</td>\n",
       "      <td>[(favlorless, favlorless, NN)]</td>\n",
       "      <td>(favlorless, favlorless, NN)</td>\n",
       "      <td>favlorless</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       answer_id                                        tok_lem_POS  \\\n",
       "71           105  [(here, here, RB), (are, be, VBP), (a, a, DT),...   \n",
       "3919       10490  [(getting, get, VBG), (good, Good, NNP), (grad...   \n",
       "6832       18380  [(hi, Hi, NNP), (,, ,, ,), (adrian, Adrian, NN...   \n",
       "7851       20089                           [(uuuuuuu, uuuuuuu, NN)]   \n",
       "7852       20096                               [(yhryr, yhryr, NN)]   \n",
       "8472       21585  [(1, 1, CD), (., ., .), (individual, Individua...   \n",
       "8634       21845  [(when, when, WRB), (i, I, PRP), ('m, be, VBP)...   \n",
       "9846       24136           [(hsthethghryjyjy, hsthethghryjyjy, NN)]   \n",
       "10653      25988  [(fibromyalgia, Fibromyalgia, NNP), (?, ?, .),...   \n",
       "12481      30421                   [(stayhealthy, StayHealthy, NN)]   \n",
       "13027      32028                         [(hiiiiiii, hiiiiiii, NN)]   \n",
       "13737      33214  [(1, 1, CD), (), ), )), (it, It, PRP), (is, be...   \n",
       "16865      39728                      [(drylegbed, Drylegbed, NNP)]   \n",
       "16905      39870                      [(deepfried, deepfried, VBN)]   \n",
       "16906      39887                     [(favlorless, favlorless, NN)]   \n",
       "\n",
       "                                             misspelling  \\\n",
       "71     (asdfkjdlkfjadlfjalsdf, asdfkjdlkfjadlfjalsdf,...   \n",
       "3919                   (quintcareers, Quintcareers, NNS)   \n",
       "6832                              (yeonjea, Yeonjea, NN)   \n",
       "7851                              (uuuuuuu, uuuuuuu, NN)   \n",
       "7852                                  (yhryr, yhryr, NN)   \n",
       "8472                   (shadizubeidi, Shadizubeidi, NNP)   \n",
       "8634   (jaaaaaaaaaaaaaaaaaaajajjajajajaja, Jaaaaaaaaa...   \n",
       "9846              (hsthethghryjyjy, hsthethghryjyjy, NN)   \n",
       "10653                  (fibromyalgia, Fibromyalgia, NNP)   \n",
       "12481                     (stayhealthy, StayHealthy, NN)   \n",
       "13027                           (hiiiiiii, hiiiiiii, NN)   \n",
       "13737                           (hahahaha, hahahaha, NN)   \n",
       "16865                        (drylegbed, Drylegbed, NNP)   \n",
       "16905                        (deepfried, deepfried, VBN)   \n",
       "16906                       (favlorless, favlorless, NN)   \n",
       "\n",
       "                                   sentence bigrams unigram_suggestions  \\\n",
       "71                    asdfkjdlkfjadlfjalsdf      []                  []   \n",
       "3919                          Quintcareers.      []                  []   \n",
       "6832                             - Yeonjea.      []                  []   \n",
       "7851                                uuuuuuu      []                  []   \n",
       "7852                                  yhryr      []                  []   \n",
       "8472                          Shadizubeidi.      []                  []   \n",
       "8634   Jaaaaaaaaaaaaaaaaaaajajjajajajaja ;)      []                  []   \n",
       "9846                        hsthethghryjyjy      []                  []   \n",
       "10653                        Fibromyalgia ?      []                  []   \n",
       "12481                           StayHealthy      []                  []   \n",
       "13027                              hiiiiiii      []                  []   \n",
       "13737                             hahahaha.      []                  []   \n",
       "16865                             Drylegbed      []                  []   \n",
       "16905                             deepfried      []                  []   \n",
       "16906                            favlorless      []                  []   \n",
       "\n",
       "      bigram_suggestions  \n",
       "71                    []  \n",
       "3919                  []  \n",
       "6832                  []  \n",
       "7851                  []  \n",
       "7852                  []  \n",
       "8472                  []  \n",
       "8634                  []  \n",
       "9846                  []  \n",
       "10653                 []  \n",
       "12481                 []  \n",
       "13027                 []  \n",
       "13737                 []  \n",
       "16865                 []  \n",
       "16905                 []  \n",
       "16906                 []  "
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "non_words2.loc[(non_words2.bigram_suggestions.str.len() == 0) & (non_words2.unigram_suggestions.str.len() == 0),:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Items with no suggestions - these will be left in their original form though manual corrections could be applied if desired."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create a new column with just the most likely correction (based on frequency). Bigram suggestions are given preference before unigram suggestions. If there is no suggestion, the original word is returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new column with just the most likely correction (based on frequency)\n",
    "\n",
    "def sort_unigram_tuple(tup):  \n",
    "    tup.sort(key = lambda x: x[2], reverse=True)  \n",
    "    return tup    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keeping the unigram correction with the highest frequency\n",
    "\n",
    "non_words2['unigram_correction'] = [sort_unigram_tuple(x)[0][0] if len(x) != 0 else np.NaN for x in non_words2['unigram_suggestions']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new column with just the most likely correction (based on frequency)\n",
    "\n",
    "def sort_bigram_tuple(tup):  \n",
    "    tup.sort(key = lambda x: x[0][2], reverse=True)  \n",
    "    return tup    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keeping the bigram correction with the highest frequency\n",
    "\n",
    "non_words2['bigram_correction'] = [(sort_bigram_tuple(x)[0][0][0],x[0][1]) if len(x) != 0 else np.NaN for x in non_words2['bigram_suggestions']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting the bigram corrections into separate words\n",
    "\n",
    "mask = non_words2.loc[non_words2.bigram_correction.isnull() == False].index\n",
    "non_words2.loc[mask, 'bigram_correction'] = non_words2.loc[mask, 'bigram_correction'].apply(\n",
    "    lambda x: (x[0].split(),x[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some bigrams that were previously two words have now been corrected to one word, e.g. _paragragh, s --> paragraphs._  \n",
    "These will now be labelled Type 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelling</th>\n",
       "      <th>sentence</th>\n",
       "      <th>bigrams</th>\n",
       "      <th>unigram_suggestions</th>\n",
       "      <th>bigram_suggestions</th>\n",
       "      <th>unigram_correction</th>\n",
       "      <th>bigram_correction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>16</td>\n",
       "      <td>[(every, every, DT), (paragragh, paragragh, NN...</td>\n",
       "      <td>(paragragh, paragragh, NN)</td>\n",
       "      <td>Every paragragh's instructions depend on a mai...</td>\n",
       "      <td>[(every paragragh, 1), (paragragh s, 2)]</td>\n",
       "      <td>[(paragraph,  1,  24855495)]</td>\n",
       "      <td>[((paragraphs,  2,  5579470), 2), ((every para...</td>\n",
       "      <td>paragraph</td>\n",
       "      <td>([paragraphs], 2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>200</td>\n",
       "      <td>[(hi, Hi, NNP), (,, ,, ,), (i, I, PRP), ('m, b...</td>\n",
       "      <td>(jotard, Jotard, NNP)</td>\n",
       "      <td>Hi, I'm Jotard.</td>\n",
       "      <td>[(m jotard, 1)]</td>\n",
       "      <td>[(jotted,  2,  66054), (retard,  2,  643417), ...</td>\n",
       "      <td>[((mustard,  3,  2820441), 1)]</td>\n",
       "      <td>jotted</td>\n",
       "      <td>([mustard], 1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290</th>\n",
       "      <td>378</td>\n",
       "      <td>[(my, my, PRP$), (writing, write, VBG), (teach...</td>\n",
       "      <td>(teachet, teachet, JJ)</td>\n",
       "      <td>My writing teachet`s name is ANON_NAME_0.</td>\n",
       "      <td>[(writing teachet, 1), (teachet s, 2)]</td>\n",
       "      <td>[(teaches,  1,  5599377), (teacher,  1,  48002...</td>\n",
       "      <td>[((teachers,  2,  49832283), 2), ((writing tea...</td>\n",
       "      <td>teaches</td>\n",
       "      <td>([teachers], 2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>444</td>\n",
       "      <td>[(topic, Topic, NNP), (sentence, sentence, NN)...</td>\n",
       "      <td>(gaint, Gaint, NNP)</td>\n",
       "      <td>For example, the Rose's Gaint Boutique and Hea...</td>\n",
       "      <td>[(s gaint, 1), (gaint boutique, 2)]</td>\n",
       "      <td>[(gait,  1,  822923), (saint,  1,  38929141), ...</td>\n",
       "      <td>[((saint,  2,  38929141), 1), ((saint boutique...</td>\n",
       "      <td>gait</td>\n",
       "      <td>([saint], 1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>353</th>\n",
       "      <td>542</td>\n",
       "      <td>[(topic, Topic, NNP), (sentence, sentence, NN)...</td>\n",
       "      <td>(gaint, Gaint, NNP)</td>\n",
       "      <td>For example, the Rose's Gaint Boutique and Hea...</td>\n",
       "      <td>[(s gaint, 1), (gaint boutique, 2)]</td>\n",
       "      <td>[(gait,  1,  822923), (saint,  1,  38929141), ...</td>\n",
       "      <td>[((saint,  2,  38929141), 1), ((saint boutique...</td>\n",
       "      <td>gait</td>\n",
       "      <td>([saint], 1)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     answer_id                                        tok_lem_POS  \\\n",
       "5           16  [(every, every, DT), (paragragh, paragragh, NN...   \n",
       "142        200  [(hi, Hi, NNP), (,, ,, ,), (i, I, PRP), ('m, b...   \n",
       "290        378  [(my, my, PRP$), (writing, write, VBG), (teach...   \n",
       "315        444  [(topic, Topic, NNP), (sentence, sentence, NN)...   \n",
       "353        542  [(topic, Topic, NNP), (sentence, sentence, NN)...   \n",
       "\n",
       "                    misspelling  \\\n",
       "5    (paragragh, paragragh, NN)   \n",
       "142       (jotard, Jotard, NNP)   \n",
       "290      (teachet, teachet, JJ)   \n",
       "315         (gaint, Gaint, NNP)   \n",
       "353         (gaint, Gaint, NNP)   \n",
       "\n",
       "                                              sentence  \\\n",
       "5    Every paragragh's instructions depend on a mai...   \n",
       "142                                    Hi, I'm Jotard.   \n",
       "290          My writing teachet`s name is ANON_NAME_0.   \n",
       "315  For example, the Rose's Gaint Boutique and Hea...   \n",
       "353  For example, the Rose's Gaint Boutique and Hea...   \n",
       "\n",
       "                                      bigrams  \\\n",
       "5    [(every paragragh, 1), (paragragh s, 2)]   \n",
       "142                           [(m jotard, 1)]   \n",
       "290    [(writing teachet, 1), (teachet s, 2)]   \n",
       "315       [(s gaint, 1), (gaint boutique, 2)]   \n",
       "353       [(s gaint, 1), (gaint boutique, 2)]   \n",
       "\n",
       "                                   unigram_suggestions  \\\n",
       "5                         [(paragraph,  1,  24855495)]   \n",
       "142  [(jotted,  2,  66054), (retard,  2,  643417), ...   \n",
       "290  [(teaches,  1,  5599377), (teacher,  1,  48002...   \n",
       "315  [(gait,  1,  822923), (saint,  1,  38929141), ...   \n",
       "353  [(gait,  1,  822923), (saint,  1,  38929141), ...   \n",
       "\n",
       "                                    bigram_suggestions unigram_correction  \\\n",
       "5    [((paragraphs,  2,  5579470), 2), ((every para...          paragraph   \n",
       "142                     [((mustard,  3,  2820441), 1)]             jotted   \n",
       "290  [((teachers,  2,  49832283), 2), ((writing tea...            teaches   \n",
       "315  [((saint,  2,  38929141), 1), ((saint boutique...               gait   \n",
       "353  [((saint,  2,  38929141), 1), ((saint boutique...               gait   \n",
       "\n",
       "     bigram_correction  \n",
       "5    ([paragraphs], 2)  \n",
       "142     ([mustard], 1)  \n",
       "290    ([teachers], 2)  \n",
       "315       ([saint], 1)  \n",
       "353       ([saint], 1)  "
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask2 = non_words2.loc[mask].loc[non_words2.loc[mask].bigram_correction.apply(lambda x: len(x[0])==1)].index \n",
    "non_words2.loc[mask2].head()\n",
    "\n",
    "#142 Type 3 items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "non_words2.loc[mask2,'bigram_correction'] = non_words2.loc[mask2,'bigram_correction'].apply(lambda x: (x[0],3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keeping only the word in the bigram correction that was originally misspelled\n",
    "\n",
    "def corrected_only(bigram_tuple):\n",
    "    bigram_list = bigram_tuple[0]\n",
    "    bigram_type = bigram_tuple[1]\n",
    "    for tup in bigram_tuple:\n",
    "        if bigram_type == 1:\n",
    "            corrected_word = bigram_list[1:]\n",
    "        if bigram_type == 2:\n",
    "            corrected_word = bigram_list[:-1]\n",
    "        if bigram_type == 3:\n",
    "            corrected_word = bigram_list\n",
    "    return ' '.join(corrected_word)\n",
    "\n",
    "# Type 1 bigram = (word, misspelling), Type 2 bigram = (misspelling, word), Type 3 bigram = misspelling only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying the function\n",
    "\n",
    "non_words2.loc[mask, 'bigram_correction'] = non_words2.loc[mask, 'bigram_correction'].apply(\n",
    "    lambda x: corrected_only(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a 'final_correction' column with order of preference: bigram_correction, unigram_correction, misspelling\n",
    "\n",
    "# Changing empty strings to NaN in the bigram correction column\n",
    "non_words2.bigram_correction.replace(\"\", np.nan, inplace=True)\n",
    "\n",
    "# Creating a tuple of these three items\n",
    "non_words2['final_correction'] = list(zip(non_words2.misspelling.apply(\n",
    "    lambda x: x[1]), non_words2.unigram_correction, non_words2.bigram_correction))\n",
    "\n",
    "# Choosing which item based on if strings or not (i.e. NaNs)\n",
    "non_words2['final_correction'] = [x[2] if isinstance(x[2], str) else x[1] if isinstance(x[1], str) else x[0] for x in non_words2['final_correction']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelling</th>\n",
       "      <th>sentence</th>\n",
       "      <th>bigrams</th>\n",
       "      <th>unigram_suggestions</th>\n",
       "      <th>bigram_suggestions</th>\n",
       "      <th>unigram_correction</th>\n",
       "      <th>bigram_correction</th>\n",
       "      <th>final_correction</th>\n",
       "      <th>final_correction_POS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>(beacause, beacause, NN)</td>\n",
       "      <td>I organized the instructions by time, beacause...</td>\n",
       "      <td>[(time beacause, 1), (beacause to, 2)]</td>\n",
       "      <td>[(because,  1,  271323986)]</td>\n",
       "      <td>[((because to,  1,  3213023), 2), ((time becau...</td>\n",
       "      <td>because</td>\n",
       "      <td>because</td>\n",
       "      <td>because</td>\n",
       "      <td>(because, because, NN)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11</td>\n",
       "      <td>[(to, to, TO), (make, make, VB), (tea, tea, NN...</td>\n",
       "      <td>(wallmart, wallmart, NN)</td>\n",
       "      <td>next, you need to buy a box of tea in wallmart...</td>\n",
       "      <td>[(in wallmart, 1), (wallmart or, 2)]</td>\n",
       "      <td>[(walmart,  1,  2269839)]</td>\n",
       "      <td>[((in wall art,  1,  99805), 1), ((wall art or...</td>\n",
       "      <td>walmart</td>\n",
       "      <td>wall art</td>\n",
       "      <td>wall art</td>\n",
       "      <td>(wall art, wall art, NN)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(dovn, dovn, NN)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "      <td>[(use dovn, 1), (dovn mircowave, 2)]</td>\n",
       "      <td>[(dove,  1,  3253560), (donn,  1,  299470), (d...</td>\n",
       "      <td>[((down microwave,  2,  1960), 2), ((use down,...</td>\n",
       "      <td>dove</td>\n",
       "      <td>down</td>\n",
       "      <td>down</td>\n",
       "      <td>(down, down, NN)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>(mircowave, mircowave, VBP)</td>\n",
       "      <td>First, you should take some hot water, you can...</td>\n",
       "      <td>[(dovn mircowave, 1), (mircowave or, 2)]</td>\n",
       "      <td>[(microwave,  1,  8934594)]</td>\n",
       "      <td>[((microwave or,  1,  22584), 2), ((down micro...</td>\n",
       "      <td>microwave</td>\n",
       "      <td>microwave</td>\n",
       "      <td>microwave</td>\n",
       "      <td>(microwave, microwave, VBP)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15</td>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>(fitst, Fitst, NNP)</td>\n",
       "      <td>Fitst, boil a water in a pot.</td>\n",
       "      <td>[(fitst boil, 2)]</td>\n",
       "      <td>[(fist,  1,  7319405), (first,  1,  578161543)...</td>\n",
       "      <td>[((first boil,  1,  1366), 2)]</td>\n",
       "      <td>fist</td>\n",
       "      <td>first</td>\n",
       "      <td>first</td>\n",
       "      <td>(first, first, NNP)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>16</td>\n",
       "      <td>[(every, every, DT), (paragragh, paragragh, NN...</td>\n",
       "      <td>(paragragh, paragragh, NN)</td>\n",
       "      <td>Every paragragh's instructions depend on a mai...</td>\n",
       "      <td>[(every paragragh, 1), (paragragh s, 2)]</td>\n",
       "      <td>[(paragraph,  1,  24855495)]</td>\n",
       "      <td>[((paragraphs,  2,  5579470), 2), ((every para...</td>\n",
       "      <td>paragraph</td>\n",
       "      <td>paragraphs</td>\n",
       "      <td>paragraphs</td>\n",
       "      <td>(paragraphs, paragraphs, NN)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id                                        tok_lem_POS  \\\n",
       "0          8  [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "1         11  [(to, to, TO), (make, make, VB), (tea, tea, NN...   \n",
       "2         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "3         13  [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "4         15  [(in, in, IN), (my, my, PRP$), (country, count...   \n",
       "5         16  [(every, every, DT), (paragragh, paragragh, NN...   \n",
       "\n",
       "                   misspelling  \\\n",
       "0     (beacause, beacause, NN)   \n",
       "1     (wallmart, wallmart, NN)   \n",
       "2             (dovn, dovn, NN)   \n",
       "3  (mircowave, mircowave, VBP)   \n",
       "4          (fitst, Fitst, NNP)   \n",
       "5   (paragragh, paragragh, NN)   \n",
       "\n",
       "                                            sentence  \\\n",
       "0  I organized the instructions by time, beacause...   \n",
       "1  next, you need to buy a box of tea in wallmart...   \n",
       "2  First, you should take some hot water, you can...   \n",
       "3  First, you should take some hot water, you can...   \n",
       "4                      Fitst, boil a water in a pot.   \n",
       "5  Every paragragh's instructions depend on a mai...   \n",
       "\n",
       "                                    bigrams  \\\n",
       "0    [(time beacause, 1), (beacause to, 2)]   \n",
       "1      [(in wallmart, 1), (wallmart or, 2)]   \n",
       "2      [(use dovn, 1), (dovn mircowave, 2)]   \n",
       "3  [(dovn mircowave, 1), (mircowave or, 2)]   \n",
       "4                         [(fitst boil, 2)]   \n",
       "5  [(every paragragh, 1), (paragragh s, 2)]   \n",
       "\n",
       "                                 unigram_suggestions  \\\n",
       "0                        [(because,  1,  271323986)]   \n",
       "1                          [(walmart,  1,  2269839)]   \n",
       "2  [(dove,  1,  3253560), (donn,  1,  299470), (d...   \n",
       "3                        [(microwave,  1,  8934594)]   \n",
       "4  [(fist,  1,  7319405), (first,  1,  578161543)...   \n",
       "5                       [(paragraph,  1,  24855495)]   \n",
       "\n",
       "                                  bigram_suggestions unigram_correction  \\\n",
       "0  [((because to,  1,  3213023), 2), ((time becau...            because   \n",
       "1  [((in wall art,  1,  99805), 1), ((wall art or...            walmart   \n",
       "2  [((down microwave,  2,  1960), 2), ((use down,...               dove   \n",
       "3  [((microwave or,  1,  22584), 2), ((down micro...          microwave   \n",
       "4                     [((first boil,  1,  1366), 2)]               fist   \n",
       "5  [((paragraphs,  2,  5579470), 2), ((every para...          paragraph   \n",
       "\n",
       "  bigram_correction final_correction          final_correction_POS  \n",
       "0           because          because        (because, because, NN)  \n",
       "1          wall art         wall art      (wall art, wall art, NN)  \n",
       "2              down             down              (down, down, NN)  \n",
       "3         microwave        microwave   (microwave, microwave, VBP)  \n",
       "4             first            first           (first, first, NNP)  \n",
       "5        paragraphs       paragraphs  (paragraphs, paragraphs, NN)  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create correction_POS column\n",
    "\n",
    "non_words2['final_correction_POS'] = list(zip(non_words2.final_correction, non_words2.final_correction, non_words2.misspelling.apply(lambda x: x[2])))\n",
    "non_words2.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Incorporating corrections into `pelic_df`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46230\n",
      "9695\n"
     ]
    }
   ],
   "source": [
    "# Focusing on rows in pelif_df containing spelling mistakes to replace\n",
    "\n",
    "mask = pelic_df.index.isin(non_words2.answer_id)\n",
    "print(len(pelic_df))\n",
    "print(len(pelic_df.loc[mask]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transforming non_words2 back so that each row is a text (with a list of the misspellings and final corrections)\n",
    "\n",
    "# First create a column with tuples of misspellings and their corrections\n",
    "non_words2['misspelling_correction'] = list(zip(non_words2.misspelling, non_words2.final_correction_POS))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelling_correction</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>[((beacause, beacause, NN), (because, because,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>[(to, to, TO), (make, make, VB), (tea, tea, NN...</td>\n",
       "      <td>[((wallmart, wallmart, NN), (wall art, wall ar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>[((dovn, dovn, NN), (down, down, NN)), ((mirco...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>[((fitst, Fitst, NNP), (first, first, NNP))]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[(every, every, DT), (paragragh, paragragh, NN...</td>\n",
       "      <td>[((paragragh, paragragh, NN), (paragraphs, par...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 tok_lem_POS  \\\n",
       "answer_id                                                      \n",
       "8          [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "11         [(to, to, TO), (make, make, VB), (tea, tea, NN...   \n",
       "13         [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "15         [(in, in, IN), (my, my, PRP$), (country, count...   \n",
       "16         [(every, every, DT), (paragragh, paragragh, NN...   \n",
       "\n",
       "                                      misspelling_correction  \n",
       "answer_id                                                     \n",
       "8          [((beacause, beacause, NN), (because, because,...  \n",
       "11         [((wallmart, wallmart, NN), (wall art, wall ar...  \n",
       "13         [((dovn, dovn, NN), (down, down, NN)), ((mirco...  \n",
       "15              [((fitst, Fitst, NNP), (first, first, NNP))]  \n",
       "16         [((paragragh, paragragh, NN), (paragraphs, par...  "
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Then group by the text and combine the misspelling_corrections\n",
    "non_words3 = non_words2[['answer_id','tok_lem_POS','misspelling_correction']]\n",
    "non_words3 = non_words2.groupby('answer_id').agg({'tok_lem_POS':'first','misspelling_correction': sum}).reset_index()\n",
    "non_words3.misspelling_correction = non_words3.misspelling_correction.apply(lambda x:list(zip(x[::2], x[1::2]))) #combining items into tuples\n",
    "non_words3 = non_words3.set_index('answer_id') # use answer_id as new index since it is now unique\n",
    "non_words3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a function to find and replace misspellings\n",
    "\n",
    "def replace_misspelling(tok_lem_POS, misspelling_correction):\n",
    "    tok_lem_POS_corrected = []\n",
    "    misspellings = [x[0] for x in misspelling_correction]\n",
    "    corrections = [x[1] for x in misspelling_correction]\n",
    "    correction_dict = dict((x, y) for x, y in misspelling_correction)\n",
    "    for tok in tok_lem_POS:\n",
    "        if tok in misspellings:\n",
    "            tok = correction_dict[tok]\n",
    "        tok_lem_POS_corrected.append(tok)\n",
    "    return tok_lem_POS_corrected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>misspelling_correction</th>\n",
       "      <th>tok_lem_POS_corrected</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>[((beacause, beacause, NN), (because, because,...</td>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>[(to, to, TO), (make, make, VB), (tea, tea, NN...</td>\n",
       "      <td>[((wallmart, wallmart, NN), (wall art, wall ar...</td>\n",
       "      <td>[(to, to, TO), (make, make, VB), (tea, tea, NN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "      <td>[((dovn, dovn, NN), (down, down, NN)), ((mirco...</td>\n",
       "      <td>[(first, first, RB), (,, ,, ,), (you, you, PRP...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>[((fitst, Fitst, NNP), (first, first, NNP))]</td>\n",
       "      <td>[(in, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[(every, every, DT), (paragragh, paragragh, NN...</td>\n",
       "      <td>[((paragragh, paragragh, NN), (paragraphs, par...</td>\n",
       "      <td>[(every, every, DT), (paragraphs, paragraphs, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 tok_lem_POS  \\\n",
       "answer_id                                                      \n",
       "8          [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "11         [(to, to, TO), (make, make, VB), (tea, tea, NN...   \n",
       "13         [(first, first, RB), (,, ,, ,), (you, you, PRP...   \n",
       "15         [(in, in, IN), (my, my, PRP$), (country, count...   \n",
       "16         [(every, every, DT), (paragragh, paragragh, NN...   \n",
       "\n",
       "                                      misspelling_correction  \\\n",
       "answer_id                                                      \n",
       "8          [((beacause, beacause, NN), (because, because,...   \n",
       "11         [((wallmart, wallmart, NN), (wall art, wall ar...   \n",
       "13         [((dovn, dovn, NN), (down, down, NN)), ((mirco...   \n",
       "15              [((fitst, Fitst, NNP), (first, first, NNP))]   \n",
       "16         [((paragragh, paragragh, NN), (paragraphs, par...   \n",
       "\n",
       "                                       tok_lem_POS_corrected  \n",
       "answer_id                                                     \n",
       "8          [(i, I, PRP), (organized, organize, VBD), (the...  \n",
       "11         [(to, to, TO), (make, make, VB), (tea, tea, NN...  \n",
       "13         [(first, first, RB), (,, ,, ,), (you, you, PRP...  \n",
       "15         [(in, in, IN), (my, my, PRP$), (country, count...  \n",
       "16         [(every, every, DT), (paragraphs, paragraphs, ...  "
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Applying the above function\n",
    "\n",
    "non_words3['tok_lem_POS_corrected'] = non_words3[['tok_lem_POS','misspelling_correction']].apply(\n",
    "    lambda x: replace_misspelling(x[0],x[1]), axis=1)\n",
    "\n",
    "non_words3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a dictionary of answer_id and tok_lem_POS_corrected\n",
    "\n",
    "corrected_text_dict = pd.Series(non_words3.tok_lem_POS_corrected.values,non_words3.index).to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding the tok_lem_POS_corrected column to pelic_df\n",
    "\n",
    "pelic_df['tok_lem_POS_corrected'] = pelic_df.index.map(corrected_text_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a dictionary of answer_id and misspelling_correction\n",
    "\n",
    "misspelling_correction_dict = pd.Series(non_words3.misspelling_correction.values,non_words3.index).to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding the misspelling_correction column to pelic_df\n",
    "\n",
    "pelic_df['misspelling_correction'] = pelic_df.index.map(misspelling_correction_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a column with number of corrected errors\n",
    "\n",
    "pelic_df['len_errors'] = pelic_df.misspelling_correction.fillna('').apply(len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Texts that have not had any corrections are labelled _NaN_ in the tok_lem_POS_corrected column. However, if you want to replace these with the orignal texts, you can use the following code instead of the one in the previous cell:  \n",
    "\n",
    "```\n",
    "pelic_df['tok_lem_POS_corrected'] = pelic_df.index.map(corrected_text_dict).fillna(pelic_df['tok_lem_POS'])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('My', 'my', 'PRP$'), ('friend', 'friend', 'NN'), ('is', 'be', 'VBZ'), ('realy', 'realy', 'JJ'), ('nise', 'nise', 'RB'), ('guy', 'guy', 'NN'), ('.', '.', '.'), ('I', 'I', 'PRP'), ('like', 'like', 'VBP'), ('hem', 'hem', 'JJ'), ('becuase', 'becuase', 'NN'), ('he', 'he', 'PRP'), ('is', 'be', 'VBZ'), ('friendlly', 'friendlly', 'RB'), ('and', 'and', 'CC'), ('lovliy', 'lovliy', 'NN'), ('.', '.', '.')]\n",
      "[('my', 'my', 'PRP$'), ('friend', 'friend', 'NN'), ('is', 'be', 'VBZ'), ('real', 'real', 'JJ'), ('nice', 'nice', 'RB'), ('guy', 'guy', 'NN'), ('.', '.', '.'), ('i', 'I', 'PRP'), ('like', 'like', 'VBP'), ('hem', 'hem', 'JJ'), ('because', 'because', 'NN'), ('he', 'he', 'PRP'), ('is', 'be', 'VBZ'), ('friendly', 'friendly', 'RB'), ('and', 'and', 'CC'), ('lovely', 'lovely', 'NN'), ('.', '.', '.')]\n"
     ]
    }
   ],
   "source": [
    "# Checking with 'becuase'\n",
    "\n",
    "print(pelic_df.loc[pelic_df.text.str.contains('becuase')].iloc[1,11]) #uncorrected\n",
    "print(pelic_df.loc[pelic_df.text.str.contains('becuase')].iloc[1,12]) #corrected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see here that many appropriate corrections have been made, including _becuase_ -> _because_ , _nise_ -> _nice_ , _friendlly_ -> _friendly_ and _lovily_ -> _lovely_ .  \n",
    "Importantly, incorrect spellings that are actual words, e.g. _hem_ (should be _him_ in this case) are not corrected. In addition, as limited context is considered, there will be some inaccuracies, e.g. _realy_ (marked as an adj) -> _real_ rather than _really_ (_real nice_ is a frequent bigram)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>course_id</th>\n",
       "      <th>level_id</th>\n",
       "      <th>class_id</th>\n",
       "      <th>question_id</th>\n",
       "      <th>version</th>\n",
       "      <th>text_len</th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_lem_POS</th>\n",
       "      <th>tok_lem_POS_corrected</th>\n",
       "      <th>misspelling_correction</th>\n",
       "      <th>len_errors</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>eq0</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>149</td>\n",
       "      <td>4</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>177</td>\n",
       "      <td>I met my friend Nife while I was studying in a...</td>\n",
       "      <td>[I, met, my, friend, Nife, while, I, was, stud...</td>\n",
       "      <td>[(I, I, PRP), (met, meet, VBD), (my, my, PRP$)...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>am8</td>\n",
       "      <td>Thai</td>\n",
       "      <td>Female</td>\n",
       "      <td>149</td>\n",
       "      <td>4</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>137</td>\n",
       "      <td>Ten years ago, I met a women on the train betw...</td>\n",
       "      <td>[Ten, years, ago, ,, I, met, a, women, on, the...</td>\n",
       "      <td>[(Ten, ten, CD), (years, year, NNS), (ago, ago...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dk5</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>63</td>\n",
       "      <td>In my country we usually don't use tea bags. F...</td>\n",
       "      <td>[In, my, country, we, usually, do, n't, use, t...</td>\n",
       "      <td>[(In, in, IN), (my, my, PRP$), (country, count...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dk5</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>I organized the instructions by time.</td>\n",
       "      <td>[I, organized, the, instructions, by, time, .]</td>\n",
       "      <td>[(I, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ad1</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>59</td>\n",
       "      <td>First, prepare a port, loose tea, and cup.\\nSe...</td>\n",
       "      <td>[First, ,, prepare, a, port, ,, loose, tea, ,,...</td>\n",
       "      <td>[(First, first, RB), (,, ,, ,), (prepare, prep...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ad1</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>By time</td>\n",
       "      <td>[By, time]</td>\n",
       "      <td>[(By, by, IN), (time, time, NN)]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>eg5</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>First, prepare your cup, loose tea or bag tea,...</td>\n",
       "      <td>[First, ,, prepare, your, cup, ,, loose, tea, ...</td>\n",
       "      <td>[(First, first, RB), (,, ,, ,), (prepare, prep...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>eg5</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>I organized the instructions by time, beacause...</td>\n",
       "      <td>[I, organized, the, instructions, by, time, ,,...</td>\n",
       "      <td>[(I, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>[(i, I, PRP), (organized, organize, VBD), (the...</td>\n",
       "      <td>[((beacause, beacause, NN), (because, because,...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ad1</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>59</td>\n",
       "      <td>First, prepare a port, loose tea, and cup.\\nSe...</td>\n",
       "      <td>[First, ,, prepare, a, port, ,, loose, tea, ,,...</td>\n",
       "      <td>[(First, first, RB), (,, ,, ,), (prepare, prep...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>ad1</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>115</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>By time</td>\n",
       "      <td>[By, time]</td>\n",
       "      <td>[(By, by, IN), (time, time, NN)]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id       L1  gender course_id level_id class_id question_id  \\\n",
       "answer_id                                                                    \n",
       "1             eq0   Arabic    Male       149        4        g           5   \n",
       "2             am8     Thai  Female       149        4        g           5   \n",
       "3             dk5  Turkish  Female       115        4        w          12   \n",
       "4             dk5  Turkish  Female       115        4        w          13   \n",
       "5             ad1   Korean  Female       115        4        w          12   \n",
       "6             ad1   Korean  Female       115        4        w          13   \n",
       "7             eg5   Korean  Female       115        4        w          12   \n",
       "8             eg5   Korean  Female       115        4        w          13   \n",
       "9             ad1   Korean  Female       115        4        w          12   \n",
       "10            ad1   Korean  Female       115        4        w          13   \n",
       "\n",
       "          version  text_len  \\\n",
       "answer_id                     \n",
       "1               1       177   \n",
       "2               1       137   \n",
       "3               1        63   \n",
       "4               1         6   \n",
       "5               1        59   \n",
       "6               1         2   \n",
       "7               1        39   \n",
       "8               1        35   \n",
       "9               2        59   \n",
       "10              2         2   \n",
       "\n",
       "                                                        text  \\\n",
       "answer_id                                                      \n",
       "1          I met my friend Nife while I was studying in a...   \n",
       "2          Ten years ago, I met a women on the train betw...   \n",
       "3          In my country we usually don't use tea bags. F...   \n",
       "4                      I organized the instructions by time.   \n",
       "5          First, prepare a port, loose tea, and cup.\\nSe...   \n",
       "6                                                    By time   \n",
       "7          First, prepare your cup, loose tea or bag tea,...   \n",
       "8          I organized the instructions by time, beacause...   \n",
       "9          First, prepare a port, loose tea, and cup.\\nSe...   \n",
       "10                                                   By time   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "1          [I, met, my, friend, Nife, while, I, was, stud...   \n",
       "2          [Ten, years, ago, ,, I, met, a, women, on, the...   \n",
       "3          [In, my, country, we, usually, do, n't, use, t...   \n",
       "4             [I, organized, the, instructions, by, time, .]   \n",
       "5          [First, ,, prepare, a, port, ,, loose, tea, ,,...   \n",
       "6                                                 [By, time]   \n",
       "7          [First, ,, prepare, your, cup, ,, loose, tea, ...   \n",
       "8          [I, organized, the, instructions, by, time, ,,...   \n",
       "9          [First, ,, prepare, a, port, ,, loose, tea, ,,...   \n",
       "10                                                [By, time]   \n",
       "\n",
       "                                                 tok_lem_POS  \\\n",
       "answer_id                                                      \n",
       "1          [(I, I, PRP), (met, meet, VBD), (my, my, PRP$)...   \n",
       "2          [(Ten, ten, CD), (years, year, NNS), (ago, ago...   \n",
       "3          [(In, in, IN), (my, my, PRP$), (country, count...   \n",
       "4          [(I, I, PRP), (organized, organize, VBD), (the...   \n",
       "5          [(First, first, RB), (,, ,, ,), (prepare, prep...   \n",
       "6                           [(By, by, IN), (time, time, NN)]   \n",
       "7          [(First, first, RB), (,, ,, ,), (prepare, prep...   \n",
       "8          [(I, I, PRP), (organized, organize, VBD), (the...   \n",
       "9          [(First, first, RB), (,, ,, ,), (prepare, prep...   \n",
       "10                          [(By, by, IN), (time, time, NN)]   \n",
       "\n",
       "                                       tok_lem_POS_corrected  \\\n",
       "answer_id                                                      \n",
       "1                                                        NaN   \n",
       "2                                                        NaN   \n",
       "3                                                        NaN   \n",
       "4                                                        NaN   \n",
       "5                                                        NaN   \n",
       "6                                                        NaN   \n",
       "7                                                        NaN   \n",
       "8          [(i, I, PRP), (organized, organize, VBD), (the...   \n",
       "9                                                        NaN   \n",
       "10                                                       NaN   \n",
       "\n",
       "                                      misspelling_correction  len_errors  \n",
       "answer_id                                                                 \n",
       "1                                                        NaN           0  \n",
       "2                                                        NaN           0  \n",
       "3                                                        NaN           0  \n",
       "4                                                        NaN           0  \n",
       "5                                                        NaN           0  \n",
       "6                                                        NaN           0  \n",
       "7                                                        NaN           0  \n",
       "8          [((beacause, beacause, NN), (because, because,...           1  \n",
       "9                                                        NaN           0  \n",
       "10                                                       NaN           0  "
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pelic_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write out new PELIC_compiled.csv\n",
    "\n",
    "pelic_df.to_csv('PELIC_compiled_spellcorrected.csv', encoding='utf-8', index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pickle new pelic_df dataframe\n",
    "\n",
    "pelic_df.to_pickle('pelic_spellcorrected.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If preferred, this entire spelling correctin process can also be applied to [`answer.csv`]() instead of `PELIC_compiled`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to top](#Corrected-spelling)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}