notebooks/01_prepare_data/data_split.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data splitting is one of the most vital tasks in assessing recommendation systems. Splitting strategy greatly affects the evaluation protocol so that it should always be taken into careful consideration by practitioners.\n",
    "\n",
    "The code hereafter explains how one applies different splitting strategies for specific scenarios."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 Global settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.0 | packaged by conda-forge | (default, Feb  9 2017, 14:36:55) \n",
      "[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]\n",
      "Pyspark version: 2.2.1\n"
     ]
    }
   ],
   "source": [
    "# set the environment path to find Recommenders\n",
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "\n",
    "import pyspark\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "from reco_utils.common.spark_utils import start_or_get_spark\n",
    "from reco_utils.dataset.url_utils import maybe_download\n",
    "from reco_utils.dataset.python_splitters import (\n",
    "    python_random_split, \n",
    "    python_chrono_split, \n",
    "    python_stratified_split\n",
    ")\n",
    "from reco_utils.dataset.spark_splitters import (\n",
    "    spark_random_split, \n",
    "    spark_chrono_split, \n",
    "    spark_stratified_split,\n",
    "    spark_timestamp_split\n",
    ")\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Pyspark version: {}\".format(pyspark.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_URL = \"http://files.grouplens.org/datasets/movielens/ml-100k/u.data\"\n",
    "DATA_PATH = \"ml-100k.data\"\n",
    "\n",
    "COL_USER = \"UserId\"\n",
    "COL_ITEM = \"MovieId\"\n",
    "COL_RATING = \"Rating\"\n",
    "COL_PREDICTION = \"Rating\"\n",
    "COL_TIMESTAMP = \"Timestamp\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Data understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For illustration purpose, the data used in the examples below is the Movielens-100K dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath = maybe_download(DATA_URL, DATA_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(filepath, sep=\"\\t\", names=[COL_USER, COL_ITEM, COL_RATING, COL_TIMESTAMP])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A glimpse at the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserId  MovieId  Rating  Timestamp\n",
       "0     196      242       3  881250949\n",
       "1     186      302       3  891717742\n",
       "2      22      377       1  878887116\n",
       "3     244       51       2  880606923\n",
       "4     166      346       1  886397596"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A little more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>100000.00000</td>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.000000</td>\n",
       "      <td>1.000000e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>462.48475</td>\n",
       "      <td>425.530130</td>\n",
       "      <td>3.529860</td>\n",
       "      <td>8.835289e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>266.61442</td>\n",
       "      <td>330.798356</td>\n",
       "      <td>1.125674</td>\n",
       "      <td>5.343856e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8.747247e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>254.00000</td>\n",
       "      <td>175.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>8.794487e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>447.00000</td>\n",
       "      <td>322.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>8.828269e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>682.00000</td>\n",
       "      <td>631.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>8.882600e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>943.00000</td>\n",
       "      <td>1682.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>8.932866e+08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             UserId        MovieId         Rating     Timestamp\n",
       "count  100000.00000  100000.000000  100000.000000  1.000000e+05\n",
       "mean      462.48475     425.530130       3.529860  8.835289e+08\n",
       "std       266.61442     330.798356       1.125674  5.343856e+06\n",
       "min         1.00000       1.000000       1.000000  8.747247e+08\n",
       "25%       254.00000     175.000000       3.000000  8.794487e+08\n",
       "50%       447.00000     322.000000       4.000000  8.828269e+08\n",
       "75%       682.00000     631.000000       4.000000  8.882600e+08\n",
       "max       943.00000    1682.000000       5.000000  8.932866e+08"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of ratings are\t100000\n",
      "Total number of users are\t943\n",
      "Total number of items are\t1682\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    \"Total number of ratings are\\t{}\".format(data.shape[0]),\n",
    "    \"Total number of users are\\t{}\".format(data[COL_USER].nunique()),\n",
    "    \"Total number of items are\\t{}\".format(data[COL_ITEM].nunique()),\n",
    "    sep=\"\\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Data transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Original timestamps are converted to ISO format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[COL_TIMESTAMP]= data.apply(\n",
    "    lambda x: datetime.strftime(datetime(1970, 1, 1, 0, 0, 0) + timedelta(seconds=x[COL_TIMESTAMP].item()), \"%Y-%m-%d %H:%M:%S\"), \n",
    "    axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-12-04 15:55:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>1998-04-04 19:22:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>1997-11-07 07:18:36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>1997-11-27 05:02:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>1998-02-02 05:33:16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserId  MovieId  Rating            Timestamp\n",
       "0     196      242       3  1997-12-04 15:55:49\n",
       "1     186      302       3  1998-04-04 19:22:22\n",
       "2      22      377       1  1997-11-07 07:18:36\n",
       "3     244       51       2  1997-11-27 05:02:03\n",
       "4     166      346       1  1998-02-02 05:33:16"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Experimentation protocol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experimentation protocol is usually set up to favor a reasonable evaluation for a specific recommendation scenario. For example,\n",
    "* *Recommender-A* is to recommend movies to people by taking people's collaborative rating similarities. To make sure the evaluation is statisically sound, the same set of users for both model building and testing should be used (to avoid any cold-ness of users), and a stratified splitting strategy should be taken.\n",
    "* *Recommender-B* is to recommend fashion products to customers. It makes sense that evaluation of the recommender considers time-dependency of customer purchases, as apparently, tastes of the customers in fashion items may be drifting over time. In this case, a chronologically splitting should be used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Data split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Random split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random split simply takes in a data set and outputs the splits of the data, given the split ratios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_test = python_random_split(data, ratio=0.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000, 30000)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.shape[0], data_test.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes a multi-split is needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_validate, data_test = python_random_split(data, ratio=[0.6, 0.2, 0.2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 20000, 20000)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.shape[0], data_validate.shape[0], data_test.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ratios can be integers as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_validate, data_test = python_random_split(data, ratio=[3, 1, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For producing the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 20000, 20000)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.shape[0], data_validate.shape[0], data_test.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Chronological split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chronogically splitting method takes in a dataset and splits it on timestamp. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.1 \"Filter by\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chrono splitting can be either by \"user\" or \"item\". For example, if it is by \"user\" and the splitting ratio is 0.7, it means that first 70% ratings for each user in the data will be put into one split while the other 30% is in another. It is worth noting that a chronological split is not \"random\" because splitting is timestamp-dependent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_test = python_chrono_split(\n",
    "    data, ratio=0.7, filter_by=\"user\",\n",
    "    col_user=COL_USER, col_item=COL_ITEM, col_timestamp=COL_TIMESTAMP\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the results for one particular user:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The last 10 rows of the train data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1989</th>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>4</td>\n",
       "      <td>1997-11-03 07:31:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11807</th>\n",
       "      <td>1</td>\n",
       "      <td>219</td>\n",
       "      <td>1</td>\n",
       "      <td>1997-11-03 07:32:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50026</th>\n",
       "      <td>1</td>\n",
       "      <td>167</td>\n",
       "      <td>2</td>\n",
       "      <td>1997-11-03 07:33:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43280</th>\n",
       "      <td>1</td>\n",
       "      <td>162</td>\n",
       "      <td>4</td>\n",
       "      <td>1997-11-03 07:33:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51295</th>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>1997-11-03 07:33:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16314</th>\n",
       "      <td>1</td>\n",
       "      <td>230</td>\n",
       "      <td>4</td>\n",
       "      <td>1997-11-03 07:33:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>4</td>\n",
       "      <td>1997-11-03 07:33:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>820</th>\n",
       "      <td>1</td>\n",
       "      <td>265</td>\n",
       "      <td>4</td>\n",
       "      <td>1997-11-03 07:34:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11154</th>\n",
       "      <td>1</td>\n",
       "      <td>112</td>\n",
       "      <td>1</td>\n",
       "      <td>1997-11-03 07:34:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45732</th>\n",
       "      <td>1</td>\n",
       "      <td>57</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-11-03 07:34:19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       UserId  MovieId  Rating            Timestamp\n",
       "1989        1       90       4  1997-11-03 07:31:40\n",
       "11807       1      219       1  1997-11-03 07:32:07\n",
       "50026       1      167       2  1997-11-03 07:33:03\n",
       "43280       1      162       4  1997-11-03 07:33:40\n",
       "51295       1       35       1  1997-11-03 07:33:40\n",
       "16314       1      230       4  1997-11-03 07:33:40\n",
       "202         1       61       4  1997-11-03 07:33:40\n",
       "820         1      265       4  1997-11-03 07:34:01\n",
       "11154       1      112       1  1997-11-03 07:34:01\n",
       "45732       1       57       5  1997-11-03 07:34:19"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train[data_train[COL_USER] == 1].tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The first 10 rows of the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5682</th>\n",
       "      <td>1</td>\n",
       "      <td>49</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-11-03 07:34:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24493</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-11-03 07:35:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39865</th>\n",
       "      <td>1</td>\n",
       "      <td>131</td>\n",
       "      <td>1</td>\n",
       "      <td>1997-11-03 07:35:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6234</th>\n",
       "      <td>1</td>\n",
       "      <td>233</td>\n",
       "      <td>2</td>\n",
       "      <td>1997-11-03 07:35:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96699</th>\n",
       "      <td>1</td>\n",
       "      <td>152</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-11-03 07:36:29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4280</th>\n",
       "      <td>1</td>\n",
       "      <td>82</td>\n",
       "      <td>5</td>\n",
       "      <td>1997-11-03 07:36:29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25721</th>\n",
       "      <td>1</td>\n",
       "      <td>141</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-11-03 07:36:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5842</th>\n",
       "      <td>1</td>\n",
       "      <td>72</td>\n",
       "      <td>4</td>\n",
       "      <td>1997-11-03 07:37:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37810</th>\n",
       "      <td>1</td>\n",
       "      <td>158</td>\n",
       "      <td>3</td>\n",
       "      <td>1997-11-03 07:38:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>333</th>\n",
       "      <td>1</td>\n",
       "      <td>33</td>\n",
       "      <td>4</td>\n",
       "      <td>1997-11-03 07:38:19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       UserId  MovieId  Rating            Timestamp\n",
       "5682        1       49       3  1997-11-03 07:34:38\n",
       "24493       1       30       3  1997-11-03 07:35:15\n",
       "39865       1      131       1  1997-11-03 07:35:52\n",
       "6234        1      233       2  1997-11-03 07:35:52\n",
       "96699       1      152       5  1997-11-03 07:36:29\n",
       "4280        1       82       5  1997-11-03 07:36:29\n",
       "25721       1      141       3  1997-11-03 07:36:48\n",
       "5842        1       72       4  1997-11-03 07:37:58\n",
       "37810       1      158       3  1997-11-03 07:38:19\n",
       "333         1       33       4  1997-11-03 07:38:19"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_test[data_test[COL_USER] == 1].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Timestamps of train data are all precedent to those in test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.2 Min-rating filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A min-rating filter is applied to data before it is split by using chronological splitter. The reason of doing this is that, for multi-split, there should be sufficient number of ratings for user/item in the data.\n",
    "\n",
    "For example, the following means splitting only applies to users that have at least 10 ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_test = python_chrono_split(\n",
    "    data, filter_by=\"user\", min_rating=10, ratio=0.7,\n",
    "    col_user=COL_USER, col_item=COL_ITEM, col_timestamp=COL_TIMESTAMP\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of rows in the yielded splits of data may not sum to the original ones as users with fewer than 10 ratings are filtered out in the splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 100000)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.shape[0] + data_test.shape[0], data.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Stratified split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chronogically splitting method takes in a dataset and splits it by either user or item. The split is stratified so that the same set of users or items will appear in both training and testing data sets. \n",
    "\n",
    "Similar to chronological splitter, `filter_by` and `min_rating_filter` also apply to the stratified splitter.\n",
    "\n",
    "The following example shows the split of the sample data with a ratio of 0.7, and for each user there should be at least 10 ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_test = python_stratified_split(\n",
    "    data, filter_by=\"user\", min_rating=10, ratio=0.7,\n",
    "    col_user=COL_USER, col_item=COL_ITEM\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 100000)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.shape[0] + data_test.shape[0], data.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Data split in scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spark DataFrame is used for scalable splitting. This allows splitting operation performed on large dataset that is distributed across Spark cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the below illustrates how to do a random split on the given Spark DataFrame. For simplicity reason, the same Movielens data, which is in Pandas DataFrame, is transformed into Spark DataFrame and used for splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = start_or_get_spark()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_spark = spark.read.csv(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_spark_train, data_spark_test = spark_random_split(data_spark, ratio=0.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, it was noticed that Spark random split does not guarantee a deterministic result. This sometimes leads to issues when data is relatively small while users seek for a precision split. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(69995, 30005)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_spark_train.count(), data_spark_test.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Dimitris Paraschakis et al, \"Comparative Evaluation of Top-N Recommenders in e-Commerce: An Industrial Perspective\", IEEE ICMLA, 2015, Miami, FL, USA.\n",
    "2. Guy Shani and Asela Gunawardana, \"Evaluating Recommendation Systems\", Recommender Systems Handbook, Springer, 2015. \n",
    "3. Apache Spark, url: https://spark.apache.org/."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (recommender)",
   "language": "python",
   "name": "recommender"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}