week3/Lecture_3_churn_prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8b184384-b622-435a-8aaa-a63f6cc97561",
   "metadata": {},
   "source": [
    "# Logistic Regression for Classification\n",
    "## Churn Prediction\n",
    "\n",
    "**Objective: Identify clients that want to leave the company**\n",
    "\n",
    "* Data: https://www.kaggle.com/blastchar/telco-customer-churn\n",
    "* Target: 1, for people who leave, 0 for people who stay\n",
    "* Use Logistic Regression for chain predection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52f37462-ab3e-4f46-818e-82c6f3032a44",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a1c90dcf-c1e1-4f8e-82f3-b84e3f83c58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mutual_info_score\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbd0a7cd-9260-45fa-949a-bc0a5395404b",
   "metadata": {},
   "source": [
    "## Read Data and initial Preparation\n",
    "* Look at the data\n",
    "* Make column names and values uniform\n",
    "* Check if all columns read correctly\n",
    "* Check if \"churn\" column needs preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2b1fa6e7-4f1d-4a46-9701-e0d398a16afe",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"../data/WA_Fn-UseC_-Telco-Customer-Churn.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "335700f3-c5f8-4bf1-b085-1fb744674eab",
   "metadata": {},
   "source": [
    "**Alternative to download the Data**\n",
    "```\n",
    "data = https://www.kaggle.com/blastchar/telco-customer-churn\n",
    "!wget $data -O ../data/MA_Fn-UseC_-Telco-Customer-Churn.csv\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2b6a175b-236b-4b78-ba6e-4388ab440fe0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerID</th>\n",
       "      <th>gender</th>\n",
       "      <th>SeniorCitizen</th>\n",
       "      <th>Partner</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>tenure</th>\n",
       "      <th>PhoneService</th>\n",
       "      <th>MultipleLines</th>\n",
       "      <th>InternetService</th>\n",
       "      <th>OnlineSecurity</th>\n",
       "      <th>...</th>\n",
       "      <th>DeviceProtection</th>\n",
       "      <th>TechSupport</th>\n",
       "      <th>StreamingTV</th>\n",
       "      <th>StreamingMovies</th>\n",
       "      <th>Contract</th>\n",
       "      <th>PaperlessBilling</th>\n",
       "      <th>PaymentMethod</th>\n",
       "      <th>MonthlyCharges</th>\n",
       "      <th>TotalCharges</th>\n",
       "      <th>Churn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7590-VHVEG</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>No</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>29.85</td>\n",
       "      <td>29.85</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5575-GNVDE</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>34</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>One year</td>\n",
       "      <td>No</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>56.95</td>\n",
       "      <td>1889.5</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3668-QPYBK</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>53.85</td>\n",
       "      <td>108.15</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7795-CFOCW</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>45</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>One year</td>\n",
       "      <td>No</td>\n",
       "      <td>Bank transfer (automatic)</td>\n",
       "      <td>42.30</td>\n",
       "      <td>1840.75</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9237-HQITU</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>70.70</td>\n",
       "      <td>151.65</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerID  gender  SeniorCitizen Partner Dependents  tenure PhoneService  \\\n",
       "0  7590-VHVEG  Female              0     Yes         No       1           No   \n",
       "1  5575-GNVDE    Male              0      No         No      34          Yes   \n",
       "2  3668-QPYBK    Male              0      No         No       2          Yes   \n",
       "3  7795-CFOCW    Male              0      No         No      45           No   \n",
       "4  9237-HQITU  Female              0      No         No       2          Yes   \n",
       "\n",
       "      MultipleLines InternetService OnlineSecurity  ... DeviceProtection  \\\n",
       "0  No phone service             DSL             No  ...               No   \n",
       "1                No             DSL            Yes  ...              Yes   \n",
       "2                No             DSL            Yes  ...               No   \n",
       "3  No phone service             DSL            Yes  ...              Yes   \n",
       "4                No     Fiber optic             No  ...               No   \n",
       "\n",
       "  TechSupport StreamingTV StreamingMovies        Contract PaperlessBilling  \\\n",
       "0          No          No              No  Month-to-month              Yes   \n",
       "1          No          No              No        One year               No   \n",
       "2          No          No              No  Month-to-month              Yes   \n",
       "3         Yes          No              No        One year               No   \n",
       "4          No          No              No  Month-to-month              Yes   \n",
       "\n",
       "               PaymentMethod MonthlyCharges  TotalCharges Churn  \n",
       "0           Electronic check          29.85         29.85    No  \n",
       "1               Mailed check          56.95        1889.5    No  \n",
       "2               Mailed check          53.85        108.15   Yes  \n",
       "3  Bank transfer (automatic)          42.30       1840.75    No  \n",
       "4           Electronic check          70.70        151.65   Yes  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5c3bbee-a1c3-46e8-8106-d406755ba794",
   "metadata": {},
   "source": [
    "* The dataframe is relatively large, we can not see all variables using .head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e877a172-f42b-48ee-b33a-fa835481ea53",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>customerID</th>\n",
       "      <td>7590-VHVEG</td>\n",
       "      <td>5575-GNVDE</td>\n",
       "      <td>3668-QPYBK</td>\n",
       "      <td>7795-CFOCW</td>\n",
       "      <td>9237-HQITU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>Female</td>\n",
       "      <td>Male</td>\n",
       "      <td>Male</td>\n",
       "      <td>Male</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SeniorCitizen</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Partner</th>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dependents</th>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tenure</th>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "      <td>2</td>\n",
       "      <td>45</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PhoneService</th>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MultipleLines</th>\n",
       "      <td>No phone service</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>InternetService</th>\n",
       "      <td>DSL</td>\n",
       "      <td>DSL</td>\n",
       "      <td>DSL</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Fiber optic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OnlineSecurity</th>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OnlineBackup</th>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DeviceProtection</th>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TechSupport</th>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StreamingTV</th>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StreamingMovies</th>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Contract</th>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>One year</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>One year</td>\n",
       "      <td>Month-to-month</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PaperlessBilling</th>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PaymentMethod</th>\n",
       "      <td>Electronic check</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>Bank transfer (automatic)</td>\n",
       "      <td>Electronic check</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MonthlyCharges</th>\n",
       "      <td>29.85</td>\n",
       "      <td>56.95</td>\n",
       "      <td>53.85</td>\n",
       "      <td>42.3</td>\n",
       "      <td>70.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TotalCharges</th>\n",
       "      <td>29.85</td>\n",
       "      <td>1889.5</td>\n",
       "      <td>108.15</td>\n",
       "      <td>1840.75</td>\n",
       "      <td>151.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Churn</th>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 0             1               2  \\\n",
       "customerID              7590-VHVEG    5575-GNVDE      3668-QPYBK   \n",
       "gender                      Female          Male            Male   \n",
       "SeniorCitizen                    0             0               0   \n",
       "Partner                        Yes            No              No   \n",
       "Dependents                      No            No              No   \n",
       "tenure                           1            34               2   \n",
       "PhoneService                    No           Yes             Yes   \n",
       "MultipleLines     No phone service            No              No   \n",
       "InternetService                DSL           DSL             DSL   \n",
       "OnlineSecurity                  No           Yes             Yes   \n",
       "OnlineBackup                   Yes            No             Yes   \n",
       "DeviceProtection                No           Yes              No   \n",
       "TechSupport                     No            No              No   \n",
       "StreamingTV                     No            No              No   \n",
       "StreamingMovies                 No            No              No   \n",
       "Contract            Month-to-month      One year  Month-to-month   \n",
       "PaperlessBilling               Yes            No             Yes   \n",
       "PaymentMethod     Electronic check  Mailed check    Mailed check   \n",
       "MonthlyCharges               29.85         56.95           53.85   \n",
       "TotalCharges                 29.85        1889.5          108.15   \n",
       "Churn                           No            No             Yes   \n",
       "\n",
       "                                          3                 4  \n",
       "customerID                       7795-CFOCW        9237-HQITU  \n",
       "gender                                 Male            Female  \n",
       "SeniorCitizen                             0                 0  \n",
       "Partner                                  No                No  \n",
       "Dependents                               No                No  \n",
       "tenure                                   45                 2  \n",
       "PhoneService                             No               Yes  \n",
       "MultipleLines              No phone service                No  \n",
       "InternetService                         DSL       Fiber optic  \n",
       "OnlineSecurity                          Yes                No  \n",
       "OnlineBackup                             No                No  \n",
       "DeviceProtection                        Yes                No  \n",
       "TechSupport                             Yes                No  \n",
       "StreamingTV                              No                No  \n",
       "StreamingMovies                          No                No  \n",
       "Contract                           One year    Month-to-month  \n",
       "PaperlessBilling                         No               Yes  \n",
       "PaymentMethod     Bank transfer (automatic)  Electronic check  \n",
       "MonthlyCharges                         42.3              70.7  \n",
       "TotalCharges                        1840.75            151.65  \n",
       "Churn                                    No               Yes  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transpose dataframe to see all columns\n",
    "df.head().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fa58dde1-f003-47c2-abbe-5e348f0b0928",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['customerid', 'gender', 'seniorcitizen', 'partner', 'dependents',\n",
       "       'tenure', 'phoneservice', 'multiplelines', 'internetservice',\n",
       "       'onlinesecurity', 'onlinebackup', 'deviceprotection', 'techsupport',\n",
       "       'streamingtv', 'streamingmovies', 'contract', 'paperlessbilling',\n",
       "       'paymentmethod', 'monthlycharges', 'totalcharges', 'churn'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make columns consistent\n",
    "df.columns = df.columns.str.lower().str.replace(\" \", \"_\")\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c19deeac-085e-44e9-9c1b-7fc087d001db",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_columns = list(df.dtypes[df.dtypes == object].index)\n",
    "categorical_columns\n",
    "\n",
    "for c in categorical_columns:\n",
    "    df[c] = df[c].str.lower().str.replace(\" \", \"_\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "283e2195-afec-4301-9cad-64a7fab5ca16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>customerid</th>\n",
       "      <td>7590-vhveg</td>\n",
       "      <td>5575-gnvde</td>\n",
       "      <td>3668-qpybk</td>\n",
       "      <td>7795-cfocw</td>\n",
       "      <td>9237-hqitu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>female</td>\n",
       "      <td>male</td>\n",
       "      <td>male</td>\n",
       "      <td>male</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seniorcitizen</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>partner</th>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dependents</th>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tenure</th>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "      <td>2</td>\n",
       "      <td>45</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>phoneservice</th>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>multiplelines</th>\n",
       "      <td>no_phone_service</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no_phone_service</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>internetservice</th>\n",
       "      <td>dsl</td>\n",
       "      <td>dsl</td>\n",
       "      <td>dsl</td>\n",
       "      <td>dsl</td>\n",
       "      <td>fiber_optic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>onlinesecurity</th>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>onlinebackup</th>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>deviceprotection</th>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>techsupport</th>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>streamingtv</th>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>streamingmovies</th>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>contract</th>\n",
       "      <td>month-to-month</td>\n",
       "      <td>one_year</td>\n",
       "      <td>month-to-month</td>\n",
       "      <td>one_year</td>\n",
       "      <td>month-to-month</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>paperlessbilling</th>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>paymentmethod</th>\n",
       "      <td>electronic_check</td>\n",
       "      <td>mailed_check</td>\n",
       "      <td>mailed_check</td>\n",
       "      <td>bank_transfer_(automatic)</td>\n",
       "      <td>electronic_check</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monthlycharges</th>\n",
       "      <td>29.85</td>\n",
       "      <td>56.95</td>\n",
       "      <td>53.85</td>\n",
       "      <td>42.3</td>\n",
       "      <td>70.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>totalcharges</th>\n",
       "      <td>29.85</td>\n",
       "      <td>1889.5</td>\n",
       "      <td>108.15</td>\n",
       "      <td>1840.75</td>\n",
       "      <td>151.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>churn</th>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 0             1               2  \\\n",
       "customerid              7590-vhveg    5575-gnvde      3668-qpybk   \n",
       "gender                      female          male            male   \n",
       "seniorcitizen                    0             0               0   \n",
       "partner                        yes            no              no   \n",
       "dependents                      no            no              no   \n",
       "tenure                           1            34               2   \n",
       "phoneservice                    no           yes             yes   \n",
       "multiplelines     no_phone_service            no              no   \n",
       "internetservice                dsl           dsl             dsl   \n",
       "onlinesecurity                  no           yes             yes   \n",
       "onlinebackup                   yes            no             yes   \n",
       "deviceprotection                no           yes              no   \n",
       "techsupport                     no            no              no   \n",
       "streamingtv                     no            no              no   \n",
       "streamingmovies                 no            no              no   \n",
       "contract            month-to-month      one_year  month-to-month   \n",
       "paperlessbilling               yes            no             yes   \n",
       "paymentmethod     electronic_check  mailed_check    mailed_check   \n",
       "monthlycharges               29.85         56.95           53.85   \n",
       "totalcharges                 29.85        1889.5          108.15   \n",
       "churn                           no            no             yes   \n",
       "\n",
       "                                          3                 4  \n",
       "customerid                       7795-cfocw        9237-hqitu  \n",
       "gender                                 male            female  \n",
       "seniorcitizen                             0                 0  \n",
       "partner                                  no                no  \n",
       "dependents                               no                no  \n",
       "tenure                                   45                 2  \n",
       "phoneservice                             no               yes  \n",
       "multiplelines              no_phone_service                no  \n",
       "internetservice                         dsl       fiber_optic  \n",
       "onlinesecurity                          yes                no  \n",
       "onlinebackup                             no                no  \n",
       "deviceprotection                        yes                no  \n",
       "techsupport                             yes                no  \n",
       "streamingtv                              no                no  \n",
       "streamingmovies                          no                no  \n",
       "contract                           one_year    month-to-month  \n",
       "paperlessbilling                         no               yes  \n",
       "paymentmethod     bank_transfer_(automatic)  electronic_check  \n",
       "monthlycharges                         42.3              70.7  \n",
       "totalcharges                        1840.75            151.65  \n",
       "churn                                    no               yes  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ae842c4-0462-4f52-9a3b-f25f7d9889c3",
   "metadata": {},
   "source": [
    "* Have a look at the datatypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2b004f99-83c1-45af-a718-19c6cd685d1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "customerid           object\n",
       "gender               object\n",
       "seniorcitizen         int64\n",
       "partner              object\n",
       "dependents           object\n",
       "tenure                int64\n",
       "phoneservice         object\n",
       "multiplelines        object\n",
       "internetservice      object\n",
       "onlinesecurity       object\n",
       "onlinebackup         object\n",
       "deviceprotection     object\n",
       "techsupport          object\n",
       "streamingtv          object\n",
       "streamingmovies      object\n",
       "contract             object\n",
       "paperlessbilling     object\n",
       "paymentmethod        object\n",
       "monthlycharges      float64\n",
       "totalcharges         object\n",
       "churn                object\n",
       "dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70131e8f-7122-450e-b5e6-a561dd271df4",
   "metadata": {},
   "source": [
    "**Notes**\n",
    "* \"seniorcitizen is int\n",
    "* \"totalcharges\" is an object - should be a number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3b31e1ca-ce7f-40a3-8883-ccb84ff6ac0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         29.85\n",
       "1        1889.5\n",
       "2        108.15\n",
       "3       1840.75\n",
       "4        151.65\n",
       "         ...   \n",
       "7038     1990.5\n",
       "7039     7362.9\n",
       "7040     346.45\n",
       "7041      306.6\n",
       "7042     6844.5\n",
       "Name: totalcharges, Length: 7043, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.totalcharges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bfebea43-35ce-4bb9-8ca4-dc3c6a529154",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Unable to parse string \"_\" at position 488",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m~/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/pandas/_libs/lib.pyx\u001b[0m in \u001b[0;36mpandas._libs.lib.maybe_convert_numeric\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Unable to parse string \"_\"",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-96970bd5486f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# convert it to a number\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_numeric\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"totalcharges\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/pandas/core/tools/numeric.py\u001b[0m in \u001b[0;36mto_numeric\u001b[0;34m(arg, errors, downcast)\u001b[0m\n\u001b[1;32m    181\u001b[0m         \u001b[0mcoerce_numeric\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0merrors\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\"ignore\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"raise\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    182\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 183\u001b[0;31m             values, _ = lib.maybe_convert_numeric(\n\u001b[0m\u001b[1;32m    184\u001b[0m                 \u001b[0mvalues\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcoerce_numeric\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcoerce_numeric\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    185\u001b[0m             )\n",
      "\u001b[0;32m~/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/pandas/_libs/lib.pyx\u001b[0m in \u001b[0;36mpandas._libs.lib.maybe_convert_numeric\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Unable to parse string \"_\" at position 488"
     ]
    }
   ],
   "source": [
    "# convert it to a number\n",
    "pd.to_numeric(df[\"totalcharges\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40467db4-be51-4ed7-91ad-a3ada43cc3e0",
   "metadata": {},
   "source": [
    "**Notes**\n",
    "* This error occurs, because \"_\" are in the data and pandas does not know how to convert this into a number\n",
    "* This happened, because we replaced all \" \" with \"_\"\n",
    "* We can use ```errors=\"coerce\"``` in ```to_numeric```, then everything that can not b converted, will be converted to NaN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "73e2eb81-e8ed-4b61-b613-ea1bc39d5884",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         29.85\n",
       "1       1889.50\n",
       "2        108.15\n",
       "3       1840.75\n",
       "4        151.65\n",
       "         ...   \n",
       "7038    1990.50\n",
       "7039    7362.90\n",
       "7040     346.45\n",
       "7041     306.60\n",
       "7042    6844.50\n",
       "Name: totalcharges, Length: 7043, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert it to a number\n",
    "tc = pd.to_numeric(df[\"totalcharges\"], errors=\"coerce\")\n",
    "tc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c4d2e398-8feb-4b4a-ae9f-cf9829910a8b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tc.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1676ef45-2b8e-4044-804b-363774aeb07a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"totalcharges\"] = tc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41fd587b-56b6-4230-b89f-65f53cd8ea43",
   "metadata": {},
   "source": [
    "* Replace missing values with 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ead8fadf-d319-40b9-8d0a-fdd2133bec1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"totalcharges\"] = df[\"totalcharges\"].fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f8e8864-3908-42ca-bb71-bb66aba7e0f7",
   "metadata": {},
   "source": [
    "* Look at the \"churn\" variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "91f7dfc6-9d07-4905-9ca8-2593cd43f0c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     no\n",
       "1     no\n",
       "2    yes\n",
       "3     no\n",
       "4    yes\n",
       "Name: churn, dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"churn\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93ae415c-0f91-4c29-af46-6465ac47f90d",
   "metadata": {},
   "source": [
    "* Replace yes / no with 1 / 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "26376f90-b824-4082-9663-2fbf05a04a02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    0\n",
       "2    1\n",
       "3    0\n",
       "4    1\n",
       "Name: churn, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"churn\"] = (df[\"churn\"] == \"yes\").astype(int)\n",
    "df[\"churn\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34076988-b9f5-4ed7-9f0b-5f7cb3f6f8d0",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setting up the Validation Framework\n",
    "* Perform the train / vl /test split using scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8424e245-4579-4708-a6bf-a21b45b052d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 80% train + val = train_full, 20% test\n",
    "df_train_full, df_test = train_test_split(df, test_size=0.2, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e289d546-5919-4907-8de7-2de655cbd47c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_full length 5634, test length 1409\n"
     ]
    }
   ],
   "source": [
    "print(f\"train_full length {len(df_train_full)}, test length {len(df_test)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "986778ed-b26b-47a6-85c5-da299d719195",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 75% train, 25% val out of train_full \n",
    "# 60% train, 20% val, 20% test out of df\n",
    "df_train, df_val = train_test_split(df_train_full, test_size=0.25, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "65ff9d26-9ced-4b7c-8d9e-2cf5a43023ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train length 4225, val length 1409\n"
     ]
    }
   ],
   "source": [
    "print(f\"train length {len(df_train)}, val length {len(df_val)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5a413f3d-0c82-476d-bd6b-165665c979ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset index\n",
    "df_train = df_train.reset_index(drop=True)\n",
    "df_val = df_val.reset_index(drop=True)\n",
    "df_test = df_test.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4f77ccef-caa8-4ac1-a17a-b8ada50aa822",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = df_train[\"churn\"]\n",
    "y_val = df_val[\"churn\"]\n",
    "y_test = df_test[\"churn\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "89e50ab0-d426-4898-88d7-884ce8ee8a91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete \"churn from df_train, df_val, df_test (not from df)\n",
    "del df_train[\"churn\"]\n",
    "del df_val[\"churn\"]\n",
    "del df_test[\"churn\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb21ab5c-54cb-4516-b3f5-aad262046441",
   "metadata": {},
   "source": [
    "## EDA\n",
    "* Check missing values\n",
    "* Look at the target variable \"churn\"\n",
    "* Look at numerical and categrical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fac9dc34-cffe-4ade-92a5-6522724137ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For the EDA we will df_train_full\n",
    "df_train_full = df_train_full.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3407a146-fe9b-4bb1-b552-3628538fc40a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "customerid          0\n",
       "gender              0\n",
       "seniorcitizen       0\n",
       "partner             0\n",
       "dependents          0\n",
       "tenure              0\n",
       "phoneservice        0\n",
       "multiplelines       0\n",
       "internetservice     0\n",
       "onlinesecurity      0\n",
       "onlinebackup        0\n",
       "deviceprotection    0\n",
       "techsupport         0\n",
       "streamingtv         0\n",
       "streamingmovies     0\n",
       "contract            0\n",
       "paperlessbilling    0\n",
       "paymentmethod       0\n",
       "monthlycharges      0\n",
       "totalcharges        0\n",
       "churn               0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check missing values\n",
    "df_train_full.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50d465c-cebd-471e-84c2-7b6fcf63bd7e",
   "metadata": {},
   "source": [
    "* No missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e4f85c82-4a21-4956-ae39-1264799167d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    4113\n",
       "1    1521\n",
       "Name: churn, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the target variable\n",
    "df_train_full[\"churn\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9530461b-4b14-4254-9136-d2c6af6c4cf9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.730032\n",
       "1    0.269968\n",
       "Name: churn, dtype: float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the target variable\n",
    "df_train_full[\"churn\"].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0051530b-a87e-427e-aa4c-8111b07b6107",
   "metadata": {},
   "source": [
    "* Number of churned users is about 1/3 of the non-churned users\n",
    "* \"Churn rate\" = 0.269968: Fraction of users that churned (~27%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e595e985-5fa5-4e3a-acd4-e8622f9284ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.26996805111821087"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# churn rate can also be calculated using mean\n",
    "df_train_full[\"churn\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "672b6593-e190-4e93-9abd-add73243f4b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.27"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "global_churn_rate = df_train_full[\"churn\"].mean()\n",
    "round(global_churn_rate, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efff0bd8-32c3-42af-bf7a-f154d15339ef",
   "metadata": {},
   "source": [
    "* Look at other Variables\n",
    "* Numerical variables of interest: \"tenure\", \"monthlycharges\", \"totalcharges\"\n",
    "* Categorical variables "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "cdce154d-556c-4160-8e9e-dfdbe64af229",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "numerical = [\"tenure\", \"monthlycharges\", \"totalcharges\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "6332fa06-3f11-4988-a1c8-20100a02ca2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical = ['gender', 'seniorcitizen', 'partner', 'dependents',\n",
    "       'phoneservice', 'multiplelines', 'internetservice',\n",
    "       'onlinesecurity', 'onlinebackup', 'deviceprotection', 'techsupport',\n",
    "       'streamingtv', 'streamingmovies', 'contract', 'paperlessbilling',\n",
    "       'paymentmethod']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "2c6f5317-9483-443b-886c-e75ef90d75ba",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender              2\n",
       "seniorcitizen       2\n",
       "partner             2\n",
       "dependents          2\n",
       "phoneservice        2\n",
       "multiplelines       3\n",
       "internetservice     3\n",
       "onlinesecurity      3\n",
       "onlinebackup        3\n",
       "deviceprotection    3\n",
       "techsupport         3\n",
       "streamingtv         3\n",
       "streamingmovies     3\n",
       "contract            3\n",
       "paperlessbilling    2\n",
       "paymentmethod       4\n",
       "dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_full[categorical].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b4cde2b-bbf5-4b30-aca8-6de958c92f9f",
   "metadata": {},
   "source": [
    "**Notes**\n",
    "* A lot of categorical variables are binary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4deff5b-3de6-49e3-91d1-05d95fd08bab",
   "metadata": {},
   "source": [
    "## Feature Importance: Churn Rate and Risk Ratio\n",
    "Feature importance analysis (part of EDA) - identify which features affect our target variable\n",
    "* Churn rate\n",
    "* Risk ratio\n",
    "* Mutual information - later"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1453e9e4",
   "metadata": {},
   "source": [
    "**Churn Rate**\n",
    "* Have a look at the churn rate in different groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "ed14c6de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "churn_female: 0.27682403433476394, churn_male: 0.2632135306553911\n"
     ]
    }
   ],
   "source": [
    "churn_female = df_train_full[df_train_full[\"gender\"] == \"female\"].churn.mean()\n",
    "churn_male = df_train_full[df_train_full[\"gender\"] == \"male\"].churn.mean()\n",
    "print(f\"churn_female: {churn_female}, churn_male: {churn_male}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a978674f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "churn_partner: 0.20503330866025166, churn_no_partner: 0.3298090040927694\n"
     ]
    }
   ],
   "source": [
    "churn_partner = df_train_full[df_train_full[\"partner\"] == \"yes\"].churn.mean()\n",
    "churn_no_partner = df_train_full[df_train_full[\"partner\"] == \"no\"].churn.mean()\n",
    "print(f\"churn_partner: {churn_partner}, churn_no_partner: {churn_no_partner}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd58db3e",
   "metadata": {},
   "source": [
    "**Difference**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2426b360",
   "metadata": {},
   "source": [
    "global_churn_rate - churn_partner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f8707063",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.05984095297455855"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "global_churn_rate - churn_no_partner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54287e33",
   "metadata": {},
   "source": [
    "* For customers living with a patner the churn rate is ~6% lower than the global churn rate\n",
    "* For gender, the difference is low, it seems that the gender is not important\n",
    "* \"partner\" seems to be more important than \"gender\" for predicting the churn rate\n",
    "* Difference:\n",
    "    * global_churn_rate - churn_group > 0 => less likely to churn\n",
    "    * global_churn_rate - churn_group < 0 => more likely to churn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9f2b302",
   "metadata": {},
   "source": [
    "**Risk Ratio**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "5ee6a670",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2216593879412643"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "churn_no_partner / global_churn_rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "c12f6ff6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7594724924338315"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "churn_partner / global_churn_rate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efb9e070",
   "metadata": {},
   "source": [
    "* For people without partner, the risk to churn is 22% higher than the mean\n",
    "* For people with parther, the risk to churn is 24% lower than the mean"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d498d895",
   "metadata": {},
   "source": [
    "* Risk\n",
    "    * churn_group / global_churn_rate > 1 => more likely to churn\n",
    "    * churn_group / global_churn_rate > 1 => less likely to churn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32b63b4",
   "metadata": {},
   "source": [
    "* Both the difference and the risk ratio give the same information, but is a different way"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e435713",
   "metadata": {},
   "source": [
    "In SQL this could look like this:\n",
    "\n",
    "    ```\n",
    "    SELECT gender,\n",
    "         AVG(churn),\n",
    "         AVG(churn) - global_churn AS diff,\n",
    "         AVG(churn) / global_churn AS risk\n",
    "    FROM \n",
    "        data\n",
    "    GROUP BY \n",
    "        gender;\n",
    "    ```\n",
    "Translate this to pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "f78bffec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.276824</td>\n",
       "      <td>2796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.263214</td>\n",
       "      <td>2838</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            mean  count\n",
       "gender                 \n",
       "female  0.276824   2796\n",
       "male    0.263214   2838"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# .agg is to get a dataframe instead of a series\n",
    "df_group = df_train_full.groupby(\"gender\").churn.agg([\"mean\", \"count\"])\n",
    "df_group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "c45a48fa-6ffc-4072-a6da-52a6c4178572",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.276824</td>\n",
       "      <td>2796</td>\n",
       "      <td>0.006856</td>\n",
       "      <td>1.025396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.263214</td>\n",
       "      <td>2838</td>\n",
       "      <td>-0.006755</td>\n",
       "      <td>0.974980</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            mean  count      diff      risk\n",
       "gender                                     \n",
       "female  0.276824   2796  0.006856  1.025396\n",
       "male    0.263214   2838 -0.006755  0.974980"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_group[\"diff\"] = df_group[\"mean\"] - global_churn_rate\n",
    "df_group[\"risk\"] = df_group[\"mean\"] / global_churn_rate\n",
    "df_group"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2155467-4133-4a9d-956d-b69e195a21d8",
   "metadata": {},
   "source": [
    "* To this now for all categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "dc4035a3-7618-4192-b801-8ceca047ab95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gender\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.276824</td>\n",
       "      <td>2796</td>\n",
       "      <td>0.006856</td>\n",
       "      <td>1.025396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.263214</td>\n",
       "      <td>2838</td>\n",
       "      <td>-0.006755</td>\n",
       "      <td>0.974980</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            mean  count      diff      risk\n",
       "gender                                     \n",
       "female  0.276824   2796  0.006856  1.025396\n",
       "male    0.263214   2838 -0.006755  0.974980"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "seniorcitizen\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seniorcitizen</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.242270</td>\n",
       "      <td>4722</td>\n",
       "      <td>-0.027698</td>\n",
       "      <td>0.897403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.413377</td>\n",
       "      <td>912</td>\n",
       "      <td>0.143409</td>\n",
       "      <td>1.531208</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   mean  count      diff      risk\n",
       "seniorcitizen                                     \n",
       "0              0.242270   4722 -0.027698  0.897403\n",
       "1              0.413377    912  0.143409  1.531208"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "partner\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>partner</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.329809</td>\n",
       "      <td>2932</td>\n",
       "      <td>0.059841</td>\n",
       "      <td>1.221659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.205033</td>\n",
       "      <td>2702</td>\n",
       "      <td>-0.064935</td>\n",
       "      <td>0.759472</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             mean  count      diff      risk\n",
       "partner                                     \n",
       "no       0.329809   2932  0.059841  1.221659\n",
       "yes      0.205033   2702 -0.064935  0.759472"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "dependents\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dependents</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.313760</td>\n",
       "      <td>3968</td>\n",
       "      <td>0.043792</td>\n",
       "      <td>1.162212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.165666</td>\n",
       "      <td>1666</td>\n",
       "      <td>-0.104302</td>\n",
       "      <td>0.613651</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                mean  count      diff      risk\n",
       "dependents                                     \n",
       "no          0.313760   3968  0.043792  1.162212\n",
       "yes         0.165666   1666 -0.104302  0.613651"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "phoneservice\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>phoneservice</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.241316</td>\n",
       "      <td>547</td>\n",
       "      <td>-0.028652</td>\n",
       "      <td>0.893870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.273049</td>\n",
       "      <td>5087</td>\n",
       "      <td>0.003081</td>\n",
       "      <td>1.011412</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  mean  count      diff      risk\n",
       "phoneservice                                     \n",
       "no            0.241316    547 -0.028652  0.893870\n",
       "yes           0.273049   5087  0.003081  1.011412"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "multiplelines\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>multiplelines</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.257407</td>\n",
       "      <td>2700</td>\n",
       "      <td>-0.012561</td>\n",
       "      <td>0.953474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>no_phone_service</th>\n",
       "      <td>0.241316</td>\n",
       "      <td>547</td>\n",
       "      <td>-0.028652</td>\n",
       "      <td>0.893870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.290742</td>\n",
       "      <td>2387</td>\n",
       "      <td>0.020773</td>\n",
       "      <td>1.076948</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      mean  count      diff      risk\n",
       "multiplelines                                        \n",
       "no                0.257407   2700 -0.012561  0.953474\n",
       "no_phone_service  0.241316    547 -0.028652  0.893870\n",
       "yes               0.290742   2387  0.020773  1.076948"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "internetservice\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>internetservice</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dsl</th>\n",
       "      <td>0.192347</td>\n",
       "      <td>1934</td>\n",
       "      <td>-0.077621</td>\n",
       "      <td>0.712482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiber_optic</th>\n",
       "      <td>0.425171</td>\n",
       "      <td>2479</td>\n",
       "      <td>0.155203</td>\n",
       "      <td>1.574895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.077805</td>\n",
       "      <td>1221</td>\n",
       "      <td>-0.192163</td>\n",
       "      <td>0.288201</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     mean  count      diff      risk\n",
       "internetservice                                     \n",
       "dsl              0.192347   1934 -0.077621  0.712482\n",
       "fiber_optic      0.425171   2479  0.155203  1.574895\n",
       "no               0.077805   1221 -0.192163  0.288201"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "onlinesecurity\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>onlinesecurity</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.420921</td>\n",
       "      <td>2801</td>\n",
       "      <td>0.150953</td>\n",
       "      <td>1.559152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>no_internet_service</th>\n",
       "      <td>0.077805</td>\n",
       "      <td>1221</td>\n",
       "      <td>-0.192163</td>\n",
       "      <td>0.288201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.153226</td>\n",
       "      <td>1612</td>\n",
       "      <td>-0.116742</td>\n",
       "      <td>0.567570</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         mean  count      diff      risk\n",
       "onlinesecurity                                          \n",
       "no                   0.420921   2801  0.150953  1.559152\n",
       "no_internet_service  0.077805   1221 -0.192163  0.288201\n",
       "yes                  0.153226   1612 -0.116742  0.567570"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "onlinebackup\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>onlinebackup</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.404323</td>\n",
       "      <td>2498</td>\n",
       "      <td>0.134355</td>\n",
       "      <td>1.497672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>no_internet_service</th>\n",
       "      <td>0.077805</td>\n",
       "      <td>1221</td>\n",
       "      <td>-0.192163</td>\n",
       "      <td>0.288201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.217232</td>\n",
       "      <td>1915</td>\n",
       "      <td>-0.052736</td>\n",
       "      <td>0.804660</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         mean  count      diff      risk\n",
       "onlinebackup                                            \n",
       "no                   0.404323   2498  0.134355  1.497672\n",
       "no_internet_service  0.077805   1221 -0.192163  0.288201\n",
       "yes                  0.217232   1915 -0.052736  0.804660"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "deviceprotection\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>deviceprotection</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.395875</td>\n",
       "      <td>2473</td>\n",
       "      <td>0.125907</td>\n",
       "      <td>1.466379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>no_internet_service</th>\n",
       "      <td>0.077805</td>\n",
       "      <td>1221</td>\n",
       "      <td>-0.192163</td>\n",
       "      <td>0.288201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.230412</td>\n",
       "      <td>1940</td>\n",
       "      <td>-0.039556</td>\n",
       "      <td>0.853480</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         mean  count      diff      risk\n",
       "deviceprotection                                        \n",
       "no                   0.395875   2473  0.125907  1.466379\n",
       "no_internet_service  0.077805   1221 -0.192163  0.288201\n",
       "yes                  0.230412   1940 -0.039556  0.853480"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "techsupport\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>techsupport</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.418914</td>\n",
       "      <td>2781</td>\n",
       "      <td>0.148946</td>\n",
       "      <td>1.551717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>no_internet_service</th>\n",
       "      <td>0.077805</td>\n",
       "      <td>1221</td>\n",
       "      <td>-0.192163</td>\n",
       "      <td>0.288201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.159926</td>\n",
       "      <td>1632</td>\n",
       "      <td>-0.110042</td>\n",
       "      <td>0.592390</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         mean  count      diff      risk\n",
       "techsupport                                             \n",
       "no                   0.418914   2781  0.148946  1.551717\n",
       "no_internet_service  0.077805   1221 -0.192163  0.288201\n",
       "yes                  0.159926   1632 -0.110042  0.592390"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "streamingtv\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>streamingtv</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.342832</td>\n",
       "      <td>2246</td>\n",
       "      <td>0.072864</td>\n",
       "      <td>1.269897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>no_internet_service</th>\n",
       "      <td>0.077805</td>\n",
       "      <td>1221</td>\n",
       "      <td>-0.192163</td>\n",
       "      <td>0.288201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.302723</td>\n",
       "      <td>2167</td>\n",
       "      <td>0.032755</td>\n",
       "      <td>1.121328</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         mean  count      diff      risk\n",
       "streamingtv                                             \n",
       "no                   0.342832   2246  0.072864  1.269897\n",
       "no_internet_service  0.077805   1221 -0.192163  0.288201\n",
       "yes                  0.302723   2167  0.032755  1.121328"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "streamingmovies\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>streamingmovies</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.338906</td>\n",
       "      <td>2213</td>\n",
       "      <td>0.068938</td>\n",
       "      <td>1.255358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>no_internet_service</th>\n",
       "      <td>0.077805</td>\n",
       "      <td>1221</td>\n",
       "      <td>-0.192163</td>\n",
       "      <td>0.288201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.307273</td>\n",
       "      <td>2200</td>\n",
       "      <td>0.037305</td>\n",
       "      <td>1.138182</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         mean  count      diff      risk\n",
       "streamingmovies                                         \n",
       "no                   0.338906   2213  0.068938  1.255358\n",
       "no_internet_service  0.077805   1221 -0.192163  0.288201\n",
       "yes                  0.307273   2200  0.037305  1.138182"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "contract\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>contract</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>month-to-month</th>\n",
       "      <td>0.431701</td>\n",
       "      <td>3104</td>\n",
       "      <td>0.161733</td>\n",
       "      <td>1.599082</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>one_year</th>\n",
       "      <td>0.120573</td>\n",
       "      <td>1186</td>\n",
       "      <td>-0.149395</td>\n",
       "      <td>0.446621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two_year</th>\n",
       "      <td>0.028274</td>\n",
       "      <td>1344</td>\n",
       "      <td>-0.241694</td>\n",
       "      <td>0.104730</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    mean  count      diff      risk\n",
       "contract                                           \n",
       "month-to-month  0.431701   3104  0.161733  1.599082\n",
       "one_year        0.120573   1186 -0.149395  0.446621\n",
       "two_year        0.028274   1344 -0.241694  0.104730"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "paperlessbilling\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>paperlessbilling</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>no</th>\n",
       "      <td>0.172071</td>\n",
       "      <td>2313</td>\n",
       "      <td>-0.097897</td>\n",
       "      <td>0.637375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yes</th>\n",
       "      <td>0.338151</td>\n",
       "      <td>3321</td>\n",
       "      <td>0.068183</td>\n",
       "      <td>1.252560</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      mean  count      diff      risk\n",
       "paperlessbilling                                     \n",
       "no                0.172071   2313 -0.097897  0.637375\n",
       "yes               0.338151   3321  0.068183  1.252560"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "paymentmethod\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>diff</th>\n",
       "      <th>risk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>paymentmethod</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bank_transfer_(automatic)</th>\n",
       "      <td>0.168171</td>\n",
       "      <td>1219</td>\n",
       "      <td>-0.101797</td>\n",
       "      <td>0.622928</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>credit_card_(automatic)</th>\n",
       "      <td>0.164339</td>\n",
       "      <td>1217</td>\n",
       "      <td>-0.105630</td>\n",
       "      <td>0.608733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>electronic_check</th>\n",
       "      <td>0.455890</td>\n",
       "      <td>1893</td>\n",
       "      <td>0.185922</td>\n",
       "      <td>1.688682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mailed_check</th>\n",
       "      <td>0.193870</td>\n",
       "      <td>1305</td>\n",
       "      <td>-0.076098</td>\n",
       "      <td>0.718121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               mean  count      diff      risk\n",
       "paymentmethod                                                 \n",
       "bank_transfer_(automatic)  0.168171   1219 -0.101797  0.622928\n",
       "credit_card_(automatic)    0.164339   1217 -0.105630  0.608733\n",
       "electronic_check           0.455890   1893  0.185922  1.688682\n",
       "mailed_check               0.193870   1305 -0.076098  0.718121"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "for c in categorical:\n",
    "    print(c)\n",
    "    df_group = df_train_full.groupby(c).churn.agg([\"mean\", \"count\"])\n",
    "    df_group[\"diff\"] = df_group[\"mean\"] - global_churn_rate\n",
    "    df_group[\"risk\"] = df_group[\"mean\"] / global_churn_rate\n",
    "    display(df_group)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f170ed6-4126-4559-a50a-674e0ad2f230",
   "metadata": {},
   "source": [
    "## Feature Importance: Mutual Information\n",
    "Mutual information - concept from information theory, it tells us how much we can learn about one variable  if we want to know the value of another\n",
    "\n",
    "* https://en.wikipedia.org/wiki/Mutual_information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "351fe5fa-a53c-4bd5-a9d1-ecbe051b6215",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0983203874041556"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mutual_info_score(df_train_full[\"churn\"], df_train_full[\"contract\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e5e5bdb9-1830-47fd-b2f0-06af38233be8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0001174846211139946"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mutual_info_score(df_train_full[\"churn\"], df_train_full[\"gender\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "5d8cf549-ae5b-4045-ab14-b753ff76b719",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.009967689095399745"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mutual_info_score(df_train_full[\"churn\"], df_train_full[\"partner\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80775d85-184d-4c47-8a1d-2d1596f604da",
   "metadata": {},
   "source": [
    "* What is the relatve importance of the variables\n",
    "* e.g. \"contract\" is more important than \"gender\"\n",
    "* Do this for all categorical variables and order variables by importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "65f23ddd-c90d-4169-86c5-f340aafb1769",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mutual_info_churn_score(series):\n",
    "    return mutual_info_score(df_train_full[\"churn\"], series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "d569b35f-70da-4658-9795-330434567375",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "contract            0.098320\n",
       "onlinesecurity      0.063085\n",
       "techsupport         0.061032\n",
       "internetservice     0.055868\n",
       "onlinebackup        0.046923\n",
       "deviceprotection    0.043453\n",
       "paymentmethod       0.043210\n",
       "streamingtv         0.031853\n",
       "streamingmovies     0.031581\n",
       "paperlessbilling    0.017589\n",
       "dependents          0.012346\n",
       "partner             0.009968\n",
       "seniorcitizen       0.009410\n",
       "multiplelines       0.000857\n",
       "phoneservice        0.000229\n",
       "gender              0.000117\n",
       "dtype: float64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mi = df_train_full[categorical].apply(mutual_info_churn_score)\n",
    "mi.sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ca92fa3-8996-45ce-b3e0-3a67d0eafff4",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Feature Importance: Correlation\n",
    "How about numerical columns\n",
    "* Correlation Coefficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "d1896606-3c1c-4666-8580-5d94174a3255",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tenure           -0.351885\n",
       "monthlycharges    0.196805\n",
       "totalcharges     -0.196353\n",
       "dtype: float64"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_full[numerical].corrwith(df_train_full[\"churn\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e946ecf2-3973-4d27-a0e9-e319b348d61f",
   "metadata": {},
   "source": [
    "* When \"tenure\" and \"totalcharges\" increase, churn rate decreases\n",
    "* When \"monthlycharges\" increase, churn rate increases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "de29e966-f035-4fa8-84f1-312e5f7593b9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5953420669577875"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_full[df_train_full[\"tenure\"] <= 2].churn.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "b553359d-bfbe-4a00-9266-8c76bd31b788",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3994413407821229"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_full[(df_train_full[\"tenure\"] > 2) & (df_train_full[\"tenure\"] <= 12)].churn.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "9cb664a2-1299-4b62-a2dd-c2d28f975028",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.17634908339788277"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_full[df_train_full[\"tenure\"] > 12].churn.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f31809cb-1160-46dd-8543-ea186d998281",
   "metadata": {},
   "source": [
    "* The higher \"tenure\", the lower the churn rate - negative correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "126b9763-ff51-4b50-8deb-27f55c353411",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08795411089866156"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_full[df_train_full[\"monthlycharges\"] <= 20].churn.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "3daf2740-7c15-40ee-b767-121a610b49b8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18340943683409436"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_full[(df_train_full[\"monthlycharges\"] > 20) & (df_train_full[\"monthlycharges\"] <= 50)].churn.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "4d1ade5a-b76a-4124-b852-0e65d0fc4f50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.32499341585462205"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_full[df_train_full[\"monthlycharges\"] > 50].churn.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49c725ae-bac7-4230-b78e-0e94fed8efe7",
   "metadata": {},
   "source": [
    "* The higher \"monthlycharges\", the higher the churn rate - positive correlation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2bb525c-d6ce-411d-87be-af3352bf1bb7",
   "metadata": {},
   "source": [
    "## One-Hot Encoding\n",
    "* Use scikit learn to encode categorical features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "1add8c33-947e-4d92-abd0-c9c5f7d97b37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First convert to dictionary\n",
    "train_dicts = df_train[categorical+numerical].to_dict(orient=\"records\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "41e3af72-7cd4-413c-9aa3-03c9b1d798a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DictVectorizer(sparse=False)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dv = DictVectorizer(sparse=False) # don't use sparse matrix\n",
    "dv.fit(train_dicts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "9b25f3d4-6765-44b4-914c-f45323afe114",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['contract=month-to-month',\n",
       " 'contract=one_year',\n",
       " 'contract=two_year',\n",
       " 'dependents=no',\n",
       " 'dependents=yes',\n",
       " 'deviceprotection=no',\n",
       " 'deviceprotection=no_internet_service',\n",
       " 'deviceprotection=yes',\n",
       " 'gender=female',\n",
       " 'gender=male',\n",
       " 'internetservice=dsl',\n",
       " 'internetservice=fiber_optic',\n",
       " 'internetservice=no',\n",
       " 'monthlycharges',\n",
       " 'multiplelines=no',\n",
       " 'multiplelines=no_phone_service',\n",
       " 'multiplelines=yes',\n",
       " 'onlinebackup=no',\n",
       " 'onlinebackup=no_internet_service',\n",
       " 'onlinebackup=yes',\n",
       " 'onlinesecurity=no',\n",
       " 'onlinesecurity=no_internet_service',\n",
       " 'onlinesecurity=yes',\n",
       " 'paperlessbilling=no',\n",
       " 'paperlessbilling=yes',\n",
       " 'partner=no',\n",
       " 'partner=yes',\n",
       " 'paymentmethod=bank_transfer_(automatic)',\n",
       " 'paymentmethod=credit_card_(automatic)',\n",
       " 'paymentmethod=electronic_check',\n",
       " 'paymentmethod=mailed_check',\n",
       " 'phoneservice=no',\n",
       " 'phoneservice=yes',\n",
       " 'seniorcitizen',\n",
       " 'streamingmovies=no',\n",
       " 'streamingmovies=no_internet_service',\n",
       " 'streamingmovies=yes',\n",
       " 'streamingtv=no',\n",
       " 'streamingtv=no_internet_service',\n",
       " 'streamingtv=yes',\n",
       " 'techsupport=no',\n",
       " 'techsupport=no_internet_service',\n",
       " 'techsupport=yes',\n",
       " 'tenure',\n",
       " 'totalcharges']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dv.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "5593ef55-9f38-4f38-9c88-86f376cec8be",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = dv.transform(train_dicts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc26e2b5-9641-43e7-82c1-686bf2a71736",
   "metadata": {},
   "source": [
    "**Notes**\n",
    "* If numerical values are passed in the dictionary they recognized and stay as they are, only categorical variables are one-hot encoded.\n",
    "* Instead of first fit ```dv.fit(train_dicts)``` and then transform ```X_train = dv.transform(train_dicts)```, this can be done in one step ```X_train = dv.fit_transform(train_dicts)```\n",
    "* Now to the same transformation (without fit!) for the validaion data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "5da4e1e6-0f06-49dc-a69b-20e4fd9607ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "val_dicts = df_val[categorical+numerical].to_dict(orient=\"records\")\n",
    "X_val = dv.transform(val_dicts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de0e649-2020-4f27-b3d7-4fbd75c7bcde",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "* Binary Classification\n",
    "* Linear vs Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2670f671-b634-489e-951c-a4d4d6af6dd0",
   "metadata": {},
   "source": [
    "```\n",
    "y_i = g(x_i)\n",
    "g(x_i) in [0,1] can be treated as propability of beloonging to the positive class\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a7428f2-a6e1-46ff-b219-911635c9e6e2",
   "metadata": {},
   "source": [
    "Linear Regression: ```g(x_i) = w_0 + W.T*x_i``` can be any real number\n",
    "\n",
    "Logistic Regression: ```g(x_i) = sigmoid(w_0 + W.T*x_i)``` is between 0 and 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "fb26501e-028e-4414-9cb2-273125f12c1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "8e5a9b20-7178-4204-9208-96e540a4eef5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-7. , -6.8, -6.6, -6.4, -6.2, -6. , -5.8, -5.6, -5.4, -5.2, -5. ,\n",
       "       -4.8, -4.6, -4.4, -4.2, -4. , -3.8, -3.6, -3.4, -3.2, -3. , -2.8,\n",
       "       -2.6, -2.4, -2.2, -2. , -1.8, -1.6, -1.4, -1.2, -1. , -0.8, -0.6,\n",
       "       -0.4, -0.2,  0. ,  0.2,  0.4,  0.6,  0.8,  1. ,  1.2,  1.4,  1.6,\n",
       "        1.8,  2. ,  2.2,  2.4,  2.6,  2.8,  3. ,  3.2,  3.4,  3.6,  3.8,\n",
       "        4. ,  4.2,  4.4,  4.6,  4.8,  5. ,  5.2,  5.4,  5.6,  5.8,  6. ,\n",
       "        6.2,  6.4,  6.6,  6.8,  7. ])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# plot sigmoid\n",
    "z = np.linspace(-7,7,71)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "4ee1fe10-1186-402c-98ca-3abceeffcbaf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAf1UlEQVR4nO3deZhU9Z3v8fe3qle62Zu9m00QBAVBBPdxGRSMI5OYxSWJO9e5YnJnch+jyU0mGe+9MyaZOGY0YRBNjNEYx2gkSiQao4KKsojIbtMszdLQbN3Q9FZV3/mjekyLTXcB1ZxaPq/nqafqLN39AYoPP06dc37m7oiISPoLBR1ARESSQ4UuIpIhVOgiIhlChS4ikiFU6CIiGSInqB9cUlLiQ4cODerHi4ikpWXLlu1x9z5tbQus0IcOHcrSpUuD+vEiImnJzLYcbZsOuYiIZAgVuohIhlChi4hkCBW6iEiGUKGLiGSIDgvdzB4zs91mtuoo283MfmJm5Wa20swmJj+miIh0JJER+i+Aae1snw6MbHnMBH524rFERORYdXgeuru/aWZD29llBvBLj9+Hd7GZ9TCzAe6+M1khRUQ6g7vTFI3RFInRGIk/N0fjj6aIf/y6OepEYjEi0fi6aMyJxOLrojGIfuLZiTrEYk7Mnag77hBtWY45nD20JxeObPPaoBOSjAuLBgGVrZa3taz7VKGb2Uzio3gGDx6chB8tItmmoTnK/sNN1NQ3c+BwMzX1zdTWN3OoMcKhhgiHGiMcbIxwuDFCXVOU+qYodU0R6puiNDRHqW+O0tAco6E5SmMkFsiv4Y6/OiVlC93aWNfmrBnuPgeYAzBp0iTNrCEiQHykvK+uie0H6tlxoIHqgw3sqm1k98EGdh9sZO+hJvbVxR/1zdF2v1dBboji/ByK8nMozA1TlJ9DcX4OJcX5FOaGKcwNU5AboiA3TH5OiPyW57ycEHnh+HPux89GTihETtjIDYfICcWfwyEjN2yEQ/F1oZDFn80Ih4ywGaEQhFvWxR/xZbO2KjM5klHo24CyVsulwI4kfF8RySCxmLP9QD2b9tR9/Ni8t47KfYfZfqCehuZPjpZDBiXF+fTtlk/vonxG9i2mV1EevYrz6Nklj+6FufQozKVbYS7dC3PpWhAv8dxw9p68l4xCnwfMMrOngSlAjY6fi2S3huYoq3fUsnpHDWt3HmR9VS3rqw5S1/SX0XWXvDBDexcxsm9XLh7Vl0E9ChnUs5CB3Qvp1z1e4uFQ541mM1GHhW5mvwYuBkrMbBvwj0AugLvPBuYDVwLlwGHg5s4KKyKpaXdtA+9U7GX5lv28X3mAtTtraY7Gj6p2L8xlVP+ufP6sUk7t35VT+hQzrKSIvl3zO/XwQzZK5CyX6zrY7sCdSUskIimvrjHCovI9vF2+h7c37uWj3YcAKMoLc0Zpd269YDhnlvVgfFl3+ncrUHGfJIHdPldE0sueQ438ae0u/rh6FwvL99AUiVGYG+bsYb34/FmlnHdKCWMGdtNhkgCp0EXkqOqboixYXcV/LqvknY17iTkM6lHIl6cMYeqYfpw1pCd5Odn7IWSqUaGLyKcs37qfZ5ZU8uLKnRxqjFDWq5BZl4zgitP7M2ZANx1CSVEqdBEB4lcy/nF1FXMWVvD+1gN0yQtz5RkD+PxZpUwe2ouQDqWkPBW6SJZraI7ymyWVPLpoE1v3HWZwry58/+qxXHNWKcX5qoh0oj8tkSwVjTnPLd/GA69sYEdNAxMH9+De6aO5fGx/fbCZplToIlnG3fnT2t38YME6Nuw6xPjS7vzoC+M5b0RJ0NHkBKnQRbLIlr11fOv5D3mrfC/DS4r46Q0TmX56f33ImSFU6CJZIBKNMXfRJh54ZQN54RD/NGMs100enNX3PclEKnSRDLdqew3f/O1KVu+o5fIx/finGafTv3tB0LGkE6jQRTKUuzN34Sbuf3kdPYvymP3liUw7fUDQsaQTqdBFMlDN4Wa+8Z8f8OraXVwxth8/uGY83bvkBh1LOpkKXSTDfFB5gDufWk5VTQPfuWoMt5w/VB96ZgkVukgGeXbZNu59biV9uxbwn3ecy4TBPYOOJCeRCl0kA7g7D71Wzr++soHzR/Tm4esn0qNLXtCx5CRToYukuUg0xndeWM2v39vKZycM4v5rxukOiFlKhS6Sxg43Rfjar9/n1bW7+buLT+HuK0bpeHkWU6GLpKm6xgg3/fw9lm3Zz30zxvKVc4cGHUkCpkIXSUMNzVFue3wpy7bs5yfXTeCqcQODjiQpQIUukmYaI1FmPrGMxZv28uMvjleZy8f0yYlIGmmKxLjzyeW8uaGa+z83js9OKA06kqQQFbpImojGnL//zQpeXbub+/72dL54dlnQkSTFqNBF0sQPXl7HSx/u5NtXnsZXzhkSdBxJQSp0kTTwmyVb+Y83K/jquUO4/aLhQceRFKVCF0lxiyv28u3nV3HhyBK+e9WYoONIClOhi6SwzXvquONXyxjSuwsPXT+RHE1IIe3Qu0MkRdXUN3PL40sw4LGbzqZ7oW5/K+3TeegiKcjdufvZD9i69zBP3jaFIb2Lgo4kaUAjdJEU9KvFW1iwehffnDaaKcN7Bx1H0oQKXSTFrNlRy30vreXiUX249YJhQceRNKJCF0khdY0RZv16OT0Kc/nXL4wnFNKdEyVxCRW6mU0zs/VmVm5m97SxvbuZ/d7MPjCz1WZ2c/KjimS+f5y3mk176vi3a8+kd3F+0HEkzXRY6GYWBh4GpgNjgOvM7MiTYe8E1rj7eOBi4F/NTNOliByD372/nWeXbeOuS0Zw3iklQceRNJTICH0yUO7uFe7eBDwNzDhiHwe6WvzO+sXAPiCS1KQiGWxXbQPfeWEVk4b05GuXjQw6jqSpRAp9EFDZanlby7rWHgJOA3YAHwJfd/fYkd/IzGaa2VIzW1pdXX2ckUUyi7vzf363iqZIjB9+YbwuHpLjlsg7p61PZfyI5SuAFcBA4EzgITPr9qkvcp/j7pPcfVKfPn2OMapIZnpx5U5eWbOLf5h6KsNKdL65HL9ECn0b0Po+naXER+Kt3Qw853HlwCZgdHIiimSufXVNfG/easaVdtcpinLCEin0JcBIMxvW8kHntcC8I/bZClwGYGb9gFFARTKDimSi7/9+NbUNzfzg8+N0qEVOWIeX/rt7xMxmAQuAMPCYu682sztats8G7gN+YWYfEj9E801339OJuUXS3p/W7uKFFTv4+mUjGd3/U0coRY5ZQvdycff5wPwj1s1u9XoHcHlyo4lkrkONEb79/CpG9evKnZeMCDqOZAjdnEskAA+9Vk5VbQM//fJE8nJ0qEWSQ+8kkZNs0546Hl1UwefPKmXi4J5Bx5EMokIXOcnue3EN+Tlh7p42KugokmFU6CIn0WvrdvHaut18/bKR9O1aEHQcyTAqdJGTpDES5b4X1zK8TxE3njc06DiSgVToIifJz9/azKY9dXz3qjH6IFQ6hd5VIifBrtoG/v1PH/HXp/Xj4lF9g44jGUqFLnISPPDKBpqjzneuOi3oKJLBVOginax89yGeWVrJDecM1mTP0qlU6CKd7EcL1tMlL4dZuiJUOpkKXaQTvb91Py+vruL2C4drSjnpdCp0kU7i7tz/8jpKivO47ULdGlc6nwpdpJO8saGaxRX7uOvSkRTl67ZJ0vlU6CKdIBZz7n95PWW9Crlu8uCg40iWUKGLdILfr9zB2p21fGPqKF1EJCeN3mkiSRaJxnjglQ2M7t+Vq8cPDDqOZBEVukiSzftgB5v3Hubvp55KKNTWHOsinUOFLpJE0Zjz0GvlnDagG5eP6Rd0HMkyKnSRJPr9Bzuo2FPH1y4dgZlG53JyqdBFkiQac37y2keM6teVK8b2DzqOZCEVukiSvPThTiqq6/jaZSN17FwCoUIXSYJYzPn3P33Eqf2KmX66RucSDBW6SBLMX7WTj3Yf4q5LNTqX4KjQRU5QfHRezoi+xVx5xoCg40gWU6GLnKBX1+5i/a6D3HXpCMIanUuAVOgiJ8Dd+enrGynrVchnNDqXgKnQRU7Au5v2saLyADMvOoWcsP46SbD0DhQ5AbPf2EhJcR5fOKs06CgiKnSR47VmRy2vr6/m5vOHUZAbDjqOiApd5HjNfmMjxfk5fPmcIUFHEQESLHQzm2Zm682s3MzuOco+F5vZCjNbbWZvJDemSGrZuvcwL67cwQ1TBtO9MDfoOCIAdDgvlpmFgYeBqcA2YImZzXP3Na326QH8FJjm7lvNrG8n5RVJCY8srCAnFOKWCzRXqKSOREbok4Fyd69w9ybgaWDGEftcDzzn7lsB3H13cmOKpI7qg408s7SSz00cRL9uBUHHEflYIoU+CKhstbytZV1rpwI9zex1M1tmZl9t6xuZ2UwzW2pmS6urq48vsUjAHn97M03RGDMvGh50FJFPSKTQ27r0zY9YzgHOAj4DXAF8x8xO/dQXuc9x90nuPqlPnz7HHFYkaIebIvzq3S1MPa0fw/sUBx1H5BM6PIZOfERe1mq5FNjRxj573L0OqDOzN4HxwIakpBRJEb9dto0Dh5u5XaNzSUGJjNCXACPNbJiZ5QHXAvOO2OcF4EIzyzGzLsAUYG1yo4oEKxpzHl20ifFlPZg0pGfQcUQ+pcMRurtHzGwWsAAIA4+5+2ozu6Nl+2x3X2tmLwMrgRgw191XdWZwkZPt1bW72Lz3MA9dMUrTy0lKSuSQC+4+H5h/xLrZRyz/EPhh8qKJpJa5CysY1KOQaZpeTlKUrhQVScCKygMs2byfWy4YpptwScrSO1MkAY8srKBrQQ5fOrus451FAqJCF+lA5b7D/OHDnVw/eTDF+QkdpRQJhApdpAM/f2szITNuOn9o0FFE2qVCF2lHbUMzv1mylavGDWBA98Kg44i0S4Uu0o5nllRS1xTl1gt0IZGkPhW6yFFEojF+/tZmJg/txRml3YOOI9IhFbrIUbyyZhfbD9TrFrmSNlToIkfx6KJNlPUqZOqYfkFHEUmICl2kDR9UHmDplv3cdN4wwiFd5i/pQYUu0obH3tpEcX4OX5xUGnQUkYSp0EWOUFXTwEsrd/Kls8voWqD5QiV9qNBFjvD4O5uJuXPTeUODjiJyTFToIq3UN0V56t2tXDG2P2W9ugQdR+SYqNBFWvnt8m3U1DfrVEVJSyp0kRaxmPPYW5sYV9pdMxJJWlKhi7R4Y0M1FdV13HrBMM1IJGlJhS7S4tFFm+jXLZ8rzxgQdBSR46JCFwHWVdWyqHwPXz13KLmakUjSlN65IsDPF22mIDfEDVMGBx1F5Lip0CXr7TnUyPMrtnPNxFJ6dMkLOo7IcVOhS9Z7cvFWmiIxbj5fpypKelOhS1ZrjER5YvEWLh7VhxF9i4OOI3JCVOiS1eat2MGeQ43cqguJJAOo0CVruTuPLtrEqH5duWBESdBxRE6YCl2y1qLyPayrOsitF+pCIskMKnTJWo8s3ESfrvnMOHNg0FFEkkKFLllpfdVB3txQzY3nDiE/Jxx0HJGkUKFLVpq7sKLlQqIhQUcRSRoVumSd3QcbeGHFDr44qYyeRbqQSDKHCl2yzi/f3kJzLMYtupBIMkxChW5m08xsvZmVm9k97ex3tplFzezzyYsokjyHmyI8sXgLl4/px9CSoqDjiCRVh4VuZmHgYWA6MAa4zszGHGW/+4EFyQ4pkizPLovPSHT7hcODjiKSdImM0CcD5e5e4e5NwNPAjDb2uwv4LbA7iflEkiYai19IdGZZD87SjESSgRIp9EFAZavlbS3rPmZmg4DPArPb+0ZmNtPMlprZ0urq6mPNKnJC/rBqJ1v2HuaOvxquC4kkIyVS6G298/2I5X8Dvunu0fa+kbvPcfdJ7j6pT58+CUYUOXHuzs9e38jwkiKmjukfdByRTpGTwD7bgLJWy6XAjiP2mQQ83TLqKQGuNLOIu/8uGSFFTtSi8j2s3lHL/decQTik0blkpkQKfQkw0syGAduBa4HrW+/g7h+f/2VmvwBeVJlLKvnZ6xvp1y2fv50wqOOdRdJUh4dc3D0CzCJ+9spa4Bl3X21md5jZHZ0dUORErag8wNsb93LbBcN1mb9ktERG6Lj7fGD+Eeva/ADU3W868VgiyTP79Y10K8jhOs0XKhlOV4pKRttYfYgFa6r46rlDKc5PaPwikrZU6JLR5rxRQV44xE3nDw06ikinU6FLxqqqaeC597fxpbPLKCnODzqOSKdToUvGmv3GRtzRZf6SNVTokpF21Tbw1HtbuWZiKWW9ugQdR+SkUKFLRpr9xkaiMefOS0YEHUXkpFGhS8bZXdvAU+9u5bMTBjG4t0bnkj1U6JJx/uPNCiIxZ5ZG55JlVOiSUaoPNvLku1uYceZATWAhWUeFLhnlkYUVNEViGp1LVlKhS8bYc6iRJ97ZwtXjBzK8T3HQcUROOhW6ZIw5b1bQEIky69KRQUcRCYQKXTLCzpp6Hn97M589cxAj+mp0LtlJhS4Z4cFXP8Id/n7qqUFHEQmMCl3SXvnuQzyztJIbzhmsq0Ilq6nQJe39aMF6uuTl6MwWyXoqdElr72/dz8urq7j9wuH01h0VJcup0CVtuTv3v7yO3kV53HrhsI6/QCTDqdAlbb350R4WV+zjrktHaDYiEVTokqaiMef+P6yjrFch108ZEnQckZSgQpe09JsllazZWcvdV4wmL0dvYxFQoUsaqjnczA8XrGPysF5cNW5A0HFEUoYKXdLOA69uoKa+me/9zVjMLOg4IilDhS5pZV1VLU8s3sINU4YwZmC3oOOIpBQVuqQNd+f789bQtSCHf9Al/iKfokKXtPGHVVW8U7GXb1w+ip5FeUHHEUk5KnRJC/VNUf7fS2s5bUA3rp88OOg4IilJhS5p4YFXN7D9QD3fv3os4ZA+CBVpiwpdUt4HlQeYu7CC66cMZvKwXkHHEUlZKnRJaU2RGHc/u5K+XQu4Z/rooOOIpLSECt3MppnZejMrN7N72th+g5mtbHm8bWbjkx9VstHDfy5n/a6D/P/PnU63gtyg44iktA4L3czCwMPAdGAMcJ2ZjTlit03AX7n7OOA+YE6yg0r2Wbuzlof/XM7fnjmQS0f3CzqOSMpLZIQ+GSh39wp3bwKeBma03sHd33b3/S2Li4HS5MaUbBOJxg+1dC/M5bt/MzboOCJpIZFCHwRUtlre1rLuaG4F/tDWBjObaWZLzWxpdXV14ikl68xZWMGH22v4/oyx9NI55yIJSaTQ2zpHzNvc0ewS4oX+zba2u/scd5/k7pP69OmTeErJKsu37ufHf9zAlWf05zNn6OZbIolKZFaAbUBZq+VSYMeRO5nZOGAuMN3d9yYnnmSbmvpmvvbr9+nXrYB//tw43XxL5BgkMkJfAow0s2FmlgdcC8xrvYOZDQaeA77i7huSH1Oygbtz73Mrqapp4N+vn0D3Qp3VInIsOhyhu3vEzGYBC4Aw8Ji7rzazO1q2zwa+C/QGftoyooq4+6TOiy2Z6Kn3tjL/wyrumT6aiYN7Bh1HJO0kNBGju88H5h+xbnar17cBtyU3mmSTdVW1/NPv13DRqX2YeeHwoOOIpCVdKSqBq21o5s4nl9OtMJcff3E8Id2rReS4qNAlUJFojFlPvc+WvYf5ybUTKCnODzqSSNpK6JCLSGf5vy+t5c0N1fzL587g3FN6Bx1HJK1phC6BeWLxFn7x9mZuu2AY1+oe5yInTIUugVj4UTXfm7eay0b35d4rTws6jkhGUKHLSbe+6iD/88nljOxbzIPXTdCEFSJJokKXk6p89yFumLuYLnlh5t44ieJ8fYwjkiwqdDlptuyt44a5iwHjqdvPobRnl6AjiWQUFbqcFNv2H+b6R96lKRLjydumcEqf4qAjiWQcFbp0uqqaBq5/5F0ONjTzxK1TGNW/a9CRRDKSDmBKp9pYfYgbH3uPA4eb+dVtUzh9UPegI4lkLBW6dJqlm/dx2y+XkhMynrp9CuNKewQdSSSjqdClU7y8qoqvP/0+A3sU8vjNkxncWx+AinQ2Fboklbvz+Nub+f6La5hQ1oO5N56tKeREThIVuiRNXWOEbz//Ib9bsYOpY/rxk2snUJgXDjqWSNZQoUtSxK/+XMamPXV8Y+qp3HnJCN0GV+QkU6HLCXtmaSXffWEVXQty+dVtUzjvlJKgI4lkJRW6HLddtQ1894VVLFi9i3OH9+bB686kb9eCoGOJZC0VuhyzWMx5ekkl/zx/LU3RGHdPG8X/uOgU3WRLJGAqdDkm5bsP8q3nV/Hepn2cO7w3//y5MxhaUhR0LBFBhS4Jqqpp4ME/beA3Syopzs/hB9eM4wuTSjHTqFwkVajQpV019c3MfmMjjy3aRMydG88byqxLRtBbc3+KpBwVurSpqqaBX7y9mafe3cLBxggzxg/kG5ePoqyXrvgUSVUqdPmEtTtreWRhBb//YAfRmDPt9P7ceckIxg7UTbVEUp0KXTjY0MxLK3fy7LJtLN2yny55YW6YMoRbzh+me7CIpBEVepZqjER5Z+Nefvf+dl5eXUVDc4wRfYu5d/povnR2GT266P4rIulGhZ5FDjY08+f11fxxdRWvr6/mUGOErgU5XDOxlC9MKmN8aXedtSKSxlToGawxEmX5lgO8s3EPb23cyweVB4jEnJLiPK4aN4DLx/bjvFNKKMjVDbREMoEKPUO4OztqGlix9QArKvezovIAK7fV0BiJETIYV9qDmRcN59LRfZkwuKeu6hTJQCr0NFRT38ymPXVsqDrI2qpa1lcdZH3VQfbWNQGQlxPi9IHd+PI5Qzh3eG8mD+9Ft4LcgFOLSGdToaegpkiMqpoGth04zPb99Ww/UE/lvno2761j8566j4sboDA3zKn9irnstL6MHdidCYN7MLp/N/JyNP+3SLZJqNDNbBrwIBAG5rr7vxyx3Vq2XwkcBm5y9+VJzpq2miIxauqbqalvoqa+mQOHm9lb18S+lsfeQ03sPthA9cFGdh9sZF+rwv5vfbvmM6ykiKlj+jGspIihJUWM6teVwb266L7jIgIkUOhmFgYeBqYC24AlZjbP3de02m06MLLlMQX4WctzSnF3IjEnGnOaozEiUac5Fn+ORJ2maIzmVo/GSPzR1PLc2BylIRKjoSlKQ3OUhkiUw01R6pui1DVFqW+KUNcYpa4pwqGGCLUNEQ41NtPQHDtqpvycEL2L8ujTNZ/Snl04a0hP+nYtYED3Agb1LGRQj0IG9CggP0cfXIpI+xIZoU8Gyt29AsDMngZmAK0LfQbwS3d3YLGZ9TCzAe6+M9mBX1+/m/teXIM7xNyJuhOLtbyOOTH/y+uPH62WkykcMgpzw3TJC1OUn0OXvPjr3kV5DOldRHF+DsX5YboX5tK9MJduLc89uuTRuyiPXkV5dMkL61RBEUmKRAp9EFDZankbnx59t7XPIOAThW5mM4GZAIMHDz7WrAB0LchldP9uhEJGyCBkhhmEzQiH7OP18eUQ4RAfP+eEQuSEjJxwfDk3HCInHCI3ZC2vjfycELnhvzzyc0Pk54TIzwnHn3NDFOSGKcwNkxvWcWoRSR2JFHpbw8cjh7qJ7IO7zwHmAEyaNOm4hstnDenJWUN6Hs+XiohktESGmNuAslbLpcCO49hHREQ6USKFvgQYaWbDzCwPuBaYd8Q+84CvWtw5QE1nHD8XEZGj6/CQi7tHzGwWsID4aYuPuftqM7ujZftsYD7xUxbLiZ+2eHPnRRYRkbYkdB66u88nXtqt181u9dqBO5MbTUREjoVO0xARyRAqdBGRDKFCFxHJECp0EZEMYfHPMwP4wWbVwJZAfvjRlQB7gg5xDNIpbzplhfTKm05ZIb3ypmLWIe7ep60NgRV6KjKzpe4+KegciUqnvOmUFdIrbzplhfTKm05ZQYdcREQyhgpdRCRDqNA/aU7QAY5ROuVNp6yQXnnTKSukV950yqpj6CIimUIjdBGRDKFCFxHJECr0NpjZXWa23sxWm9kPgs7TETP732bmZlYSdJb2mNkPzWydma00s+fNrEfQmY5kZtNa/uzLzeyeoPO0x8zKzOzPZra25b369aAzdcTMwmb2vpm9GHSWjrRMpflsy3t2rZmdG3SmjqjQj2BmlxCfI3Wcu48FfhRwpHaZWRnxCby3Bp0lAa8Ap7v7OGADcG/AeT6h1YTo04ExwHVmNibYVO2KAN9w99OAc4A7UzwvwNeBtUGHSNCDwMvuPhoYTxrkVqF/2t8B/+LujQDuvjvgPB15ALibNqb8SzXu/kd3j7QsLiY+s1Uq+XhCdHdvAv57QvSU5O473X15y+uDxAtnULCpjs7MSoHPAHODztIRM+sGXAQ8CuDuTe5+INBQCVChf9qpwIVm9q6ZvWFmZwcd6GjM7Gpgu7t/EHSW43AL8IegQxzhaJOdpzwzGwpMAN4NOEp7/o344CMWcI5EDAeqgZ+3HCKaa2ZFQYfqSEITXGQaM3sV6N/Gpm8T/z3pSfy/sGcDz5jZcA/o/M4Osn4LuPzkJmpfe3nd/YWWfb5N/HDBkyczWwISmuw81ZhZMfBb4H+5e23QedpiZlcBu919mZldHHCcROQAE4G73P1dM3sQuAf4TrCx2peVhe7uf320bWb2d8BzLQX+npnFiN+gp/pk5WvtaFnN7AxgGPCBmUH88MVyM5vs7lUnMeIntPd7C2BmNwJXAZcF9Y9kO9JusnMzyyVe5k+6+3NB52nH+cDVZnYlUAB0M7NfufuXA851NNuAbe7+3//jeZZ4oac0HXL5tN8BlwKY2alAHql3tzXc/UN37+vuQ919KPE34MQgy7wjZjYN+CZwtbsfDjpPGxKZED1lWPxf8keBte7+46DztMfd73X30pb36rXAaylc5rT8Pao0s1Etqy4D1gQYKSFZOULvwGPAY2a2CmgCbkzBkWS6egjIB15p+V/FYne/I9hIf3G0CdEDjtWe84GvAB+a2YqWdd9qmQNYTtxdwJMt/7hXADcHnKdDuvRfRCRD6JCLiEiGUKGLiGQIFbqISIZQoYuIZAgVuohIhlChi4hkCBW6iEiG+C9igFr33wa3MQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, sigmoid(z));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "8c1d5e01-4944-4225-ae23-2cc2ea64a2ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_regression(xi):\n",
    "    result = w0\n",
    "    \n",
    "    for j in range(w):\n",
    "        result = result + xi[j]*w[j]\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "41408260-d8fa-4f3e-9c9e-cb43b3abdf97",
   "metadata": {},
   "outputs": [],
   "source": [
    "def logistic_regression(xi):\n",
    "    score = w0\n",
    "    \n",
    "    for j in range(w):\n",
    "        score = score + xi[j]*w[j]\n",
    "        \n",
    "    result = sigmoid(score)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9ccc3b1-175a-4083-8dcf-4a02eae3285f",
   "metadata": {},
   "source": [
    "**Notes**\n",
    "* Both logistig and linear regression are linear models\n",
    "* Advantage of linear models: fast to train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ec9e931-61ba-4a91-8cd2-bcfc10129a23",
   "metadata": {},
   "source": [
    "## Training Logistic Regression with Scikit-Learn\n",
    "* Train a Model with Scikit-Learn\n",
    "* Apply it to the Validation Dataset\n",
    "* Calculate the Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "e0e0d489-f6bd-43fd-8aa9-8f95edc004fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LogisticRegression()\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06586431-9040-48fd-984f-66a3e0be4719",
   "metadata": {},
   "source": [
    "* Look at the intersection and coefficients (weights) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "7e1007f5-5912-43d5-acb5-ac08cffc25b8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.10903663724625358"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.intercept_[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "8ef20477-679f-45fa-94e7-1d85c1edc6ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.476, -0.175, -0.408, -0.03 , -0.078,  0.063, -0.089, -0.081,\n",
       "       -0.034, -0.073, -0.336,  0.317, -0.089,  0.004, -0.258,  0.142,\n",
       "        0.009,  0.063, -0.089, -0.081,  0.266, -0.089, -0.284, -0.231,\n",
       "        0.123, -0.166,  0.059, -0.087, -0.032,  0.07 , -0.059,  0.142,\n",
       "       -0.249,  0.216, -0.121, -0.089,  0.102, -0.071, -0.089,  0.052,\n",
       "        0.213, -0.089, -0.232, -0.07 ,  0.   ])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.coef_[0].round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b750d7f-06a7-468a-8017-a953be236df0",
   "metadata": {},
   "source": [
    "* Apply the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "d11c54bd-7f96-4d84-94ad-8daaead8d4ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, ..., 1, 0, 1])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# hard predictions\n",
    "model.predict(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "662b12dc-fe18-44f9-8f8f-5b1a37085532",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.90488245, 0.09511755],\n",
       "       [0.3206462 , 0.6793538 ],\n",
       "       [0.36627403, 0.63372597],\n",
       "       ...,\n",
       "       [0.46801735, 0.53198265],\n",
       "       [0.95734485, 0.04265515],\n",
       "       [0.30113298, 0.69886702]])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# soft predictions\n",
    "model.predict_proba(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57f7ad52-9576-4c4c-8077-4eb03c546adc",
   "metadata": {},
   "source": [
    "**Notes**\n",
    "* Hard predictions, predict 0 or 1\n",
    "* Soft predictions predict a propability\n",
    "    * ```predict_proba``` gives an array with 2 columns\n",
    "    * The first column is the propabilty of the negative class\n",
    "    * The second column is the propabilty of the positive class\n",
    "* We are only interested in the propabilities of the positive class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "ab7d9895-3775-4826-ad1b-926a82001a25",
   "metadata": {},
   "outputs": [],
   "source": [
    "# soft predictions of churning\n",
    "y_pred_train = model.predict_proba(X_train)[:,1]\n",
    "y_pred = model.predict_proba(X_val)[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60086405-847c-4b02-b5e0-2a81c5b3d84a",
   "metadata": {},
   "source": [
    "* With the soft predictions we can set a threshold (e.g. 0.5) to make a hard prediction\n",
    "* We can then find the people that are likly to churn (and e.g. send a promotional e-mail / discount / ...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "5b44d5b1-7fc9-41b6-bd1c-9656d453e7a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, ..., False,  True,  True])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "churn_decision = (y_pred > 0.5)\n",
    "churn_decision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "2128b6c1-cea5-4d90-9bab-c85cf0cfa475",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3       8433-wxgna\n",
       "8       3440-jpscl\n",
       "11      2637-fkfsy\n",
       "12      7228-omtpn\n",
       "19      6711-fldfb\n",
       "           ...    \n",
       "1397    5976-jcjrh\n",
       "1398    2034-cgrhz\n",
       "1399    5276-kqwhg\n",
       "1407    6521-yytyi\n",
       "1408    3049-solay\n",
       "Name: customerid, Length: 311, dtype: object"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_val[churn_decision].customerid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56a40113-0160-44b7-a3ed-d18372ae80dd",
   "metadata": {},
   "source": [
    "* Compute the accuracy of the model\n",
    "* Compare y_val and churn_decision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "f795c8d7-4e1c-464f-abf5-a684d3693a2c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>probability</th>\n",
       "      <th>prediction</th>\n",
       "      <th>actual</th>\n",
       "      <th>correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.009014</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.204703</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.212366</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.543003</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.213884</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1404</th>\n",
       "      <td>0.314668</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1405</th>\n",
       "      <td>0.039440</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1406</th>\n",
       "      <td>0.136504</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1407</th>\n",
       "      <td>0.799891</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1408</th>\n",
       "      <td>0.837530</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1409 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      probability  prediction  actual  correct\n",
       "0        0.009014           0       0     True\n",
       "1        0.204703           0       0     True\n",
       "2        0.212366           0       0     True\n",
       "3        0.543003           1       1     True\n",
       "4        0.213884           0       0     True\n",
       "...           ...         ...     ...      ...\n",
       "1404     0.314668           0       0     True\n",
       "1405     0.039440           0       1    False\n",
       "1406     0.136504           0       0     True\n",
       "1407     0.799891           1       1     True\n",
       "1408     0.837530           1       1     True\n",
       "\n",
       "[1409 rows x 4 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pred = pd.DataFrame()\n",
    "df_pred[\"probability\"] = y_pred\n",
    "df_pred[\"prediction\"] = churn_decision.astype(int)\n",
    "df_pred[\"actual\"] = y_val\n",
    "df_pred[\"correct\"] = (df_pred.prediction == df_pred.actual)\n",
    "df_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1601768-c609-4f99-9abe-030352c793cc",
   "metadata": {},
   "source": [
    "* We need the fraction of the correct values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "1a482f44-b39d-489b-b536-8627aaf5c735",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8034066713981547"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pred.correct.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dba42df0-578e-4df7-95ea-ffa234f0e758",
   "metadata": {},
   "source": [
    "* This can also be calculated as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "41f2134d-fcce-4057-af59-8016346bd3f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8034066713981547"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(y_val == churn_decision).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee9b666e-e420-4638-82f1-3c582e05a22e",
   "metadata": {},
   "source": [
    "* The model in ~80% correct"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1589d996-fcb5-4b18-a7a9-cbf7be8516a8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Model Interpretation\n",
    "* Look at the Coefficients\n",
    "* Train a smaller Model with fewer Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "5ebca0c6-a23d-498e-8d9f-6ee47fc20cc4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'contract=month-to-month': 0.476,\n",
       " 'contract=one_year': -0.175,\n",
       " 'contract=two_year': -0.408,\n",
       " 'dependents=no': -0.03,\n",
       " 'dependents=yes': -0.078,\n",
       " 'deviceprotection=no': 0.063,\n",
       " 'deviceprotection=no_internet_service': -0.089,\n",
       " 'deviceprotection=yes': -0.081,\n",
       " 'gender=female': -0.034,\n",
       " 'gender=male': -0.073,\n",
       " 'internetservice=dsl': -0.336,\n",
       " 'internetservice=fiber_optic': 0.317,\n",
       " 'internetservice=no': -0.089,\n",
       " 'monthlycharges': 0.004,\n",
       " 'multiplelines=no': -0.258,\n",
       " 'multiplelines=no_phone_service': 0.142,\n",
       " 'multiplelines=yes': 0.009,\n",
       " 'onlinebackup=no': 0.063,\n",
       " 'onlinebackup=no_internet_service': -0.089,\n",
       " 'onlinebackup=yes': -0.081,\n",
       " 'onlinesecurity=no': 0.266,\n",
       " 'onlinesecurity=no_internet_service': -0.089,\n",
       " 'onlinesecurity=yes': -0.284,\n",
       " 'paperlessbilling=no': -0.231,\n",
       " 'paperlessbilling=yes': 0.123,\n",
       " 'partner=no': -0.166,\n",
       " 'partner=yes': 0.059,\n",
       " 'paymentmethod=bank_transfer_(automatic)': -0.087,\n",
       " 'paymentmethod=credit_card_(automatic)': -0.032,\n",
       " 'paymentmethod=electronic_check': 0.07,\n",
       " 'paymentmethod=mailed_check': -0.059,\n",
       " 'phoneservice=no': 0.142,\n",
       " 'phoneservice=yes': -0.249,\n",
       " 'seniorcitizen': 0.216,\n",
       " 'streamingmovies=no': -0.121,\n",
       " 'streamingmovies=no_internet_service': -0.089,\n",
       " 'streamingmovies=yes': 0.102,\n",
       " 'streamingtv=no': -0.071,\n",
       " 'streamingtv=no_internet_service': -0.089,\n",
       " 'streamingtv=yes': 0.052,\n",
       " 'techsupport=no': 0.213,\n",
       " 'techsupport=no_internet_service': -0.089,\n",
       " 'techsupport=yes': -0.232,\n",
       " 'tenure': -0.07,\n",
       " 'totalcharges': 0.0}"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# coefficient for each feature\n",
    "dict(zip(dv.get_feature_names(), model.coef_[0].round(3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2fc8eaf-a15d-4e17-896d-4419bd1b6f54",
   "metadata": {},
   "source": [
    "* To make it a bit easier, only use a subset of features and train a smaller model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "2f777e87-8228-4c0f-b410-623c12d89c33",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "small = [\"contract\", \"tenure\", \"monthlycharges\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "ff5de9a9-d4fa-45b7-bfa0-863a4dc1f5d0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'contract': 'two_year', 'tenure': 72, 'monthlycharges': 115.5},\n",
       " {'contract': 'month-to-month', 'tenure': 10, 'monthlycharges': 95.25},\n",
       " {'contract': 'month-to-month', 'tenure': 5, 'monthlycharges': 75.55},\n",
       " {'contract': 'month-to-month', 'tenure': 5, 'monthlycharges': 80.85},\n",
       " {'contract': 'two_year', 'tenure': 18, 'monthlycharges': 20.1},\n",
       " {'contract': 'month-to-month', 'tenure': 4, 'monthlycharges': 30.5},\n",
       " {'contract': 'month-to-month', 'tenure': 1, 'monthlycharges': 75.1},\n",
       " {'contract': 'month-to-month', 'tenure': 1, 'monthlycharges': 70.3},\n",
       " {'contract': 'two_year', 'tenure': 72, 'monthlycharges': 19.75},\n",
       " {'contract': 'month-to-month', 'tenure': 6, 'monthlycharges': 109.9}]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train[small].iloc[:10].to_dict(orient=\"records\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "7797916d-6dd0-40b4-b098-bd42adfc149a",
   "metadata": {},
   "outputs": [],
   "source": [
    "dicts_train_small = df_train[small].to_dict(orient=\"records\")\n",
    "dicts_val_small = df_val[small].to_dict(orient=\"records\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "57d90bf9-fa11-41a0-9e08-de33078a4f06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DictVectorizer(sparse='False')"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dv_small = DictVectorizer(sparse=\"False\")\n",
    "dv_small.fit(dicts_train_small)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "fb6bd334-ae36-4857-8fba-e82b6e95e019",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['contract=month-to-month',\n",
       " 'contract=one_year',\n",
       " 'contract=two_year',\n",
       " 'monthlycharges',\n",
       " 'tenure']"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dv_small.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "f9a6dcf8-2fa2-4654-bc1b-6f695ef1c519",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_small = dv_small.transform(dicts_train_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7382a012-3446-4122-955e-762b63304e81",
   "metadata": {},
   "source": [
    "* We don't need to loo at the validation datset in this example, because, we only want to havee a look at the coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "832423eb-cf28-448c-a62e-f6d222f3e709",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_small = LogisticRegression()\n",
    "model_small.fit(X_train_small, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "bd101d47-2fa3-49c2-a051-89523d388e3c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-2.4767756616486474"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# bias\n",
    "w0 = model_small.intercept_[0]\n",
    "w0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "fa4f219a-c93d-4434-9f8f-de2f0a15c7d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.97 , -0.025, -0.949,  0.027, -0.036])"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# weights\n",
    "w = model_small.coef_[0]\n",
    "w.round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "a23ce450-73ec-4e3a-bccd-830a0405f096",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'contract=month-to-month': 0.97,\n",
       " 'contract=one_year': -0.025,\n",
       " 'contract=two_year': -0.949,\n",
       " 'monthlycharges': 0.027,\n",
       " 'tenure': -0.036}"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict(zip(dv_small.get_feature_names(), model_small.coef_[0].round(3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc102f12-50c1-4c46-924e-340863b1b0e3",
   "metadata": {},
   "source": [
    "**Interpretation:**\n",
    "* Assume a customer with a monthly contract, who pays 50$ a month and has a tenure of 5 months\n",
    "* w0 = -2.48, the sigmoid of this number is the default value "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "2d2152ca-8e2e-4152-94a9-9c08b51d80e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07750241494368337"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(w0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "699eb95d-634c-4d48-9b00-dde5b51a08c4",
   "metadata": {},
   "source": [
    "* The default value is low, about 8% of the customers are assumed to churn\n",
    "* Add the other coefficient (monthly contract, monthlycharges and tenure)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "ddfddd05-8f46-48cd-8e2c-29aa38387401",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.31353374613236396"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w0 + w[0] + 50*w[3] + 5*w[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "e0944d71-33c2-4ebc-9e69-7bab543dab81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.422252426703174"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(_) # the underscore means, that the output of the previou cell is used"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec9cad00-f567-4921-bf86-c2d21bfd603d",
   "metadata": {},
   "source": [
    "* For this customer the probability of churning is 42%"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f757ca50-c1bf-47c0-82ed-e0faa9a6426a",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Using the Model\n",
    "* Train the final model: all features on train and validation set (df_train_full)\n",
    "* Convert dataframe to dictionary (in preparation for one-hot encoding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "eb4fc543-9232-4d7d-ade4-615eec208212",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'gender': 'male',\n",
       "  'seniorcitizen': 0,\n",
       "  'partner': 'yes',\n",
       "  'dependents': 'yes',\n",
       "  'phoneservice': 'yes',\n",
       "  'multiplelines': 'no',\n",
       "  'internetservice': 'no',\n",
       "  'onlinesecurity': 'no_internet_service',\n",
       "  'onlinebackup': 'no_internet_service',\n",
       "  'deviceprotection': 'no_internet_service',\n",
       "  'techsupport': 'no_internet_service',\n",
       "  'streamingtv': 'no_internet_service',\n",
       "  'streamingmovies': 'no_internet_service',\n",
       "  'contract': 'two_year',\n",
       "  'paperlessbilling': 'no',\n",
       "  'paymentmethod': 'mailed_check',\n",
       "  'tenure': 12,\n",
       "  'monthlycharges': 19.7,\n",
       "  'totalcharges': 258.35},\n",
       " {'gender': 'female',\n",
       "  'seniorcitizen': 0,\n",
       "  'partner': 'no',\n",
       "  'dependents': 'no',\n",
       "  'phoneservice': 'yes',\n",
       "  'multiplelines': 'no',\n",
       "  'internetservice': 'dsl',\n",
       "  'onlinesecurity': 'yes',\n",
       "  'onlinebackup': 'yes',\n",
       "  'deviceprotection': 'yes',\n",
       "  'techsupport': 'yes',\n",
       "  'streamingtv': 'no',\n",
       "  'streamingmovies': 'yes',\n",
       "  'contract': 'one_year',\n",
       "  'paperlessbilling': 'no',\n",
       "  'paymentmethod': 'credit_card_(automatic)',\n",
       "  'tenure': 42,\n",
       "  'monthlycharges': 73.9,\n",
       "  'totalcharges': 3160.55}]"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicts_full_train = df_train_full[categorical+numerical].to_dict(orient=\"records\")\n",
    "dicts_full_train[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36e3b790-18a9-4672-b18f-0212babd599c",
   "metadata": {},
   "source": [
    "* Create one-hot vector for categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "d0d7b204-5653-4212-aa9f-b0fe396f1b26",
   "metadata": {},
   "outputs": [],
   "source": [
    "dv = DictVectorizer(sparse=\"False\")\n",
    "X_train_full = dv.fit_transform(dicts_full_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9b42651-fc38-437f-ba1f-efc11e109dd7",
   "metadata": {},
   "source": [
    "* Create y_train_full"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "c3f6c4d0-b5c6-4b21-91b7-916f37775497",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0, ..., 1, 1, 0])"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_full = df_train_full.churn.values\n",
    "y_train_full"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f13a89-d7c4-450c-8ced-2b967c579d01",
   "metadata": {},
   "source": [
    "* Train a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "34abdc21-bf5a-4ced-b471-5c7e6c8ef9a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LogisticRegression()\n",
    "model.fit(X_train_full, y_train_full)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84e17a15-3888-4ab5-96dd-d4dd38ae00f5",
   "metadata": {},
   "source": [
    "* Repeat the process for the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "dcece7fc-67cf-429c-b81b-cd8452968bec",
   "metadata": {},
   "outputs": [],
   "source": [
    "dicts_test = df_test[categorical+numerical].to_dict(orient=\"records\")\n",
    "X_test = dv.transform(dicts_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a36d87d-883b-4b70-b8e3-48ba9c581816",
   "metadata": {},
   "source": [
    "* Make predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "id": "ff81efa5-aedc-4390-8172-96745d8f1469",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.06224296, 0.17473875, 0.37026701, 0.50936106, 0.2336794 ,\n",
       "       0.13215883, 0.36843164, 0.52992297, 0.0754754 , 0.21423481,\n",
       "       0.4056811 , 0.16754607, 0.75877652, 0.26698115, 0.06948708,\n",
       "       0.00372735, 0.43423155, 0.3780455 , 0.46376111, 0.21239477])"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict_proba(X_test)[:,1]\n",
    "y_pred[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "aaa7c137-3297-45a6-8c0e-fb3f823f7166",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False,  True, False, False, False,  True, False,\n",
       "       False, False, False,  True, False, False, False, False, False,\n",
       "       False, False])"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "churn_decision = (y_pred >= 0.5)\n",
    "churn_decision[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "437ef83c-022f-4ce8-98e5-34b3b88c3ef7",
   "metadata": {},
   "source": [
    "* Calculate Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "id": "c2a12e94-775d-4562-8d91-533d0ea2629f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.815471965933286"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(y_test == churn_decision).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e48d9b0b-97df-4dbb-8816-97aabb1ffa40",
   "metadata": {},
   "source": [
    "* The accuracy on the test is ~82%"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a77d735-3f79-4def-a99e-8f46ee2b7112",
   "metadata": {},
   "source": [
    "**Use the model**\n",
    "* Compute the score for one specific customer from the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "a85fd54f-f72d-41e6-a076-7b2ded5eb2dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'gender': 'male',\n",
       " 'seniorcitizen': 1,\n",
       " 'partner': 'yes',\n",
       " 'dependents': 'yes',\n",
       " 'phoneservice': 'yes',\n",
       " 'multiplelines': 'no',\n",
       " 'internetservice': 'fiber_optic',\n",
       " 'onlinesecurity': 'no',\n",
       " 'onlinebackup': 'yes',\n",
       " 'deviceprotection': 'no',\n",
       " 'techsupport': 'no',\n",
       " 'streamingtv': 'yes',\n",
       " 'streamingmovies': 'yes',\n",
       " 'contract': 'month-to-month',\n",
       " 'paperlessbilling': 'yes',\n",
       " 'paymentmethod': 'mailed_check',\n",
       " 'tenure': 32,\n",
       " 'monthlycharges': 93.95,\n",
       " 'totalcharges': 2861.45}"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customer = dicts_test[10]\n",
    "customer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "a7efd3dc-c1ea-4f93-a41c-8f7e10bedf88",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4056810977854366"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_small = dv.transform([customer])\n",
    "model.predict_proba(X_small)[0,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7597a7df-f960-4f2e-b659-ff704d303067",
   "metadata": {},
   "source": [
    "* This customer is not likely to churn\n",
    "* Compare with true value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "id": "59a782fd-f8db-42e9-91aa-9a93b326417c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c9fb5c5-49cb-42c1-b5c7-c7921d43982f",
   "metadata": {},
   "source": [
    "* Our decision was correct"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4ca2722-1977-4a13-8aaa-f883b620bcf8",
   "metadata": {},
   "source": [
    "## Summary\n",
    "* Feature importance - risk, mutual information, correlation\n",
    "* One-hot encoding can be implemented with ```DictVectorizer```\n",
    "* Logistic Regression - linear model like linear regression\n",
    "* Output of logreg - propability\n",
    "* Interpretation of weights is similar to linear regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "740679c9-a7a1-410e-9b3b-7ad11fb099aa",
   "metadata": {},
   "source": [
    "## Explore more"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7730285-ff5a-4240-8b7d-c2efce792bfb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml-zoomcamp",
   "language": "python",
   "name": "ml-zoomcamp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}