ml.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get started you need to install all the necessary project dependencies. You can do it by running the cell below. Exclamation mark means that terminal command shall be executed in the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we import necessary packages. Note that utils is a folder, not a package. In this folder we need \"machine_learning.py\" file, from which we import MachineLearning object. We also suppress warnings, since they make the output messy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/grigoryturchenko/tensorflow-test/env/lib/python3.8/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import pickle, os, warnings, optuna, numpy as np\n",
    "from utils.machine_learning import MachineLearning\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we cycle through the files and load only training and test datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_sets = []\n",
    "directory = 'data'\n",
    "\n",
    "files = os.listdir(directory)\n",
    "files.sort()\n",
    "files.pop(files.index('AB_NYC_2019.csv'))\n",
    "files.pop(files.index('X_cluster.pkl'))\n",
    "for file in files:\n",
    "    with open(f\"{directory}/{file}\", 'rb') as f:\n",
    "        data_sets.append(pickle.load(f))\n",
    "X_test, X_train, y_test, y_train = data_sets[0], data_sets[1], data_sets[2], data_sets[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting random seed for reproducibility and creating an instance of MachineLearning class. As an input it accepts one of:\n",
    "- [linreg](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)\n",
    "    - No sensible hyperparameters to tune\n",
    "- [knn](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html)\n",
    "    - **Andreas:** Start with ('n_neighbors', 1, 10), ('weights', ['uniform', 'distance'])\n",
    "- [tree](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html)\n",
    "    - **Waqar:** Start with ('max_depth', 3, 12), ('min_samples_split', 10, 20), ('min_samples_leaf', 1, 10), ('max_features', ['auto', 'sqrt'])\n",
    "- [rf](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html)\n",
    "- [xgb](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor)\n",
    "\n",
    "You can click an algorithm and check out what kinds of parameteres it has for tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(510)\n",
    "ml = MachineLearning('rf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the most important part of the notebook, since here we tune the hyperparameters and train the model. You should supply number of hyperparameters tuning iterations, training and test datasets (in this particular order), and hyperparameters for tuning (in any order). A few notes about hyperparameters:\n",
    "- They can be either numeric or categorical\n",
    "- If you want to supply a numeric (integer or float) parameter, you pass the following tuple: (hyperparameter_name, min, max)\n",
    "- If you want to supply a categorical parameter, you pass the following tuple: (hyperparameter_name, [val_1, val_2, ..., val_n])\n",
    "- You may supply as many hyperparameters you want, from 1 to as many as an algo has\n",
    "\n",
    "Sometimes the training might take quite a while if you pass many hyperparameters with wide values range. However, the algorithm does not really try all possible combinations of hyperparameters, as it gets rid of not promising combinations quite early, so do not be afraid of experimenting!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml.fit(\n",
    "    100, X_train, y_train, X_test, y_test,\n",
    "    ('n_estimators', [100, 250, 500]),\n",
    "    ('max_depth', 3, 10),\n",
    "    ('min_samples_split', 10, 20),\n",
    "    ('min_samples_leaf', 1, 10),\n",
    "    ('max_features', ['auto', 'sqrt'])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the training is done, we can check out the hyperparameters the model was trained with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'criterion': 'squared_error',\n",
       " 'max_depth': 10,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 2,\n",
       " 'min_samples_split': 11,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 500,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can make predictions and calculate RMSE metric. The lower it is, the better the model is able of predicting the target variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.4270844628644313\n",
      "Test RMSE:  0.4555017324289047\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You shouldn't stop at this point, as our goal is to try and improve the performance as much as we can. For this case, you can see that `n_neighbors` parameter is equal to 10, which is the upper limit of what we supplied to the `fit` method. That means that we can try a higher range of neighbors and see if it makes RMSE even lower. Let us create an instance of the class once again and fit the model with different params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MachineLearning('rf')\n",
    "ml.fit(\n",
    "    100, X_train, y_train, X_test, y_test,\n",
    "    ('n_estimators', [500, 600, 700]),\n",
    "    ('max_depth', 10, 15),\n",
    "    ('min_samples_split', 10, 20),\n",
    "    ('min_samples_leaf', 1, 10),\n",
    "    ('max_features', ['auto', 'sqrt'])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out the hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'criterion': 'squared_error',\n",
       " 'max_depth': 15,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 10,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 700,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out the metrics. Now as we increased the number of neighbors the model is overfitted, because Train RMSE is 0 (the model perfectly predicted all the training examples), but Test RMSE is not much less. The model once again used the upper limit of `n_neighbors`, meaning that increasing this number further does not make much sense, and we can stick to the previous version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.36742415342131707\n",
      "Test RMSE:  0.4522771323623893\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MachineLearning('rf')\n",
    "ml.fit(\n",
    "    100, X_train, y_train, X_test, y_test,\n",
    "    ('n_estimators', [700, 800, 900, 1000]),\n",
    "    ('max_depth', 15, 20),\n",
    "    ('min_samples_split', 10, 20),\n",
    "    ('min_samples_leaf', 1, 10),\n",
    "    ('max_features', ['auto', 'sqrt'])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'criterion': 'squared_error',\n",
       " 'max_depth': 15,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 10,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 700,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.3677849636549292\n",
      "Test RMSE:  0.4518659789411234\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MachineLearning('rf')\n",
    "ml.fit(\n",
    "    100, X_train, y_train, X_test, y_test,\n",
    "    ('n_estimators', [700]),\n",
    "    ('max_depth', [15]),\n",
    "    ('min_samples_split', 2, 10),\n",
    "    ('min_samples_leaf', 1, 10),\n",
    "    ('max_features', ['sqrt'])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'criterion': 'squared_error',\n",
       " 'max_depth': 15,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 2,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 700,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.3278128179884352\n",
      "Test RMSE:  0.4514605491246648\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MachineLearning('rf')\n",
    "ml.fit(\n",
    "    100, X_train, y_train, X_test, y_test,\n",
    "    ('n_estimators', [700, 800, 900, 1000]),\n",
    "    ('max_depth', [15]),\n",
    "    ('min_samples_split', 2, 10),\n",
    "    ('min_samples_leaf', 1, 10),\n",
    "    ('max_features', ['sqrt'])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'criterion': 'squared_error',\n",
       " 'max_depth': 15,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 3,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 900,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.33458745014242597\n",
      "Test RMSE:  0.451524173294318\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we made up our mind with the model, we should save it by running this cell. Make sure you are saving the right model! I reran the first model in the cell above to save this particular version of KNN regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MachineLearning('xgb')\n",
    "ml.fit(\n",
    "    100, X_train, y_train, X_test, y_test,\n",
    "    ('loss', ['squared_error', 'absolute_error']),\n",
    "    ('learning_rate', [0.001, 0.01, 0.1]),\n",
    "    ('n_estimators', [100, 350, 500]),\n",
    "    ('max_depth', [3, 15]),\n",
    "    ('min_samples_split', 2, 10),\n",
    "    ('min_samples_leaf', 1, 10),\n",
    "    ('max_features', ['sqrt', 'log2']),\n",
    "    ('n_iter_no_change', [10]),\n",
    "    ('tol', [0.0001]),\n",
    "    ('validation_fraction', [0.25])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'alpha': 0.9,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'criterion': 'friedman_mse',\n",
       " 'init': None,\n",
       " 'learning_rate': 0.1,\n",
       " 'loss': 'squared_error',\n",
       " 'max_depth': 3,\n",
       " 'max_features': 'log2',\n",
       " 'max_leaf_nodes': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 8,\n",
       " 'min_samples_split': 8,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 350,\n",
       " 'n_iter_no_change': 10,\n",
       " 'random_state': None,\n",
       " 'subsample': 1.0,\n",
       " 'tol': 0.0001,\n",
       " 'validation_fraction': 0.25,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.4488394429037245\n",
      "Test RMSE:  0.4575576353763742\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MachineLearning('xgb')\n",
    "ml.fit(\n",
    "    100, X_train, y_train, X_test, y_test,\n",
    "    ('loss', ['squared_error', 'absolute_error']),\n",
    "    ('learning_rate', [0.001, 0.01, 0.1]),\n",
    "    ('n_estimators', [500, 750, 1000]),\n",
    "    ('max_depth', [6, 15]),\n",
    "    ('min_samples_split', 2, 10),\n",
    "    ('min_samples_leaf', 1, 10),\n",
    "    ('max_features', ['sqrt', 'log2']),\n",
    "    ('n_iter_no_change', [10]),\n",
    "    ('tol', [0.0001]),\n",
    "    ('validation_fraction', [0.25])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'alpha': 0.9,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'criterion': 'friedman_mse',\n",
       " 'init': None,\n",
       " 'learning_rate': 0.1,\n",
       " 'loss': 'squared_error',\n",
       " 'max_depth': 6,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 9,\n",
       " 'min_samples_split': 9,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 500,\n",
       " 'n_iter_no_change': 10,\n",
       " 'random_state': None,\n",
       " 'subsample': 1.0,\n",
       " 'tol': 0.0001,\n",
       " 'validation_fraction': 0.25,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.4321055649810131\n",
      "Test RMSE:  0.45437897778949027\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MachineLearning('xgb')\n",
    "ml.fit(\n",
    "    100, X_train, y_train, X_test, y_test,\n",
    "    ('loss', ['squared_error', 'absolute_error']),\n",
    "    ('learning_rate', [0.001, 0.01]),\n",
    "    ('n_estimators', [1000]),\n",
    "    ('max_depth', [6, 15]),\n",
    "    ('min_samples_split', 2, 10),\n",
    "    ('min_samples_leaf', 1, 10),\n",
    "    ('max_features', ['sqrt', 'log2']),\n",
    "    ('n_iter_no_change', [10]),\n",
    "    ('tol', [0.0001]),\n",
    "    ('validation_fraction', [0.25])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'alpha': 0.9,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'criterion': 'friedman_mse',\n",
       " 'init': None,\n",
       " 'learning_rate': 0.01,\n",
       " 'loss': 'squared_error',\n",
       " 'max_depth': 6,\n",
       " 'max_features': 'log2',\n",
       " 'max_leaf_nodes': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 8,\n",
       " 'min_samples_split': 10,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'n_estimators': 1000,\n",
       " 'n_iter_no_change': 10,\n",
       " 'random_state': None,\n",
       " 'subsample': 1.0,\n",
       " 'tol': 0.0001,\n",
       " 'validation_fraction': 0.25,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.4423452114147251\n",
      "Test RMSE:  0.45711374253112563\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MachineLearning('linreg')\n",
    "ml.fit(1, X_train, y_train, X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'copy_X': True,\n",
       " 'fit_intercept': True,\n",
       " 'n_jobs': None,\n",
       " 'normalize': 'deprecated',\n",
       " 'positive': False}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml.model.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.47465352491425084\n",
      "Test RMSE:  0.47768408774812077\n"
     ]
    }
   ],
   "source": [
    "pred_train = ml.predict(X_train)\n",
    "pred_test = ml.predict(X_test)\n",
    "print(f'Train RMSE: {ml.get_rmse(y_train, pred_train)}\\nTest RMSE:  {ml.get_rmse(y_test, pred_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you still have some issues with how the code works, you can use the help function to read the documentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on method fit in module utils.machine_learning:\n",
      "\n",
      "fit(n_trials: int, X_train: numpy.ndarray, y_train: numpy.ndarray, X_test: numpy.ndarray, y_test: numpy.ndarray, *args) -> None method of utils.machine_learning.MachineLearning instance\n",
      "    Find optimal hyperparameters and fit the model\n",
      "    \n",
      "    ## Parameters\n",
      "    `n_trials` : int\n",
      "        Number of optimization iterations\n",
      "    \n",
      "    `X_train`, `y_train`, `X_test`, `y_test` : np.ndarray\n",
      "        Training and test datasets\n",
      "    \n",
      "    `*args` : tuple\n",
      "        Tuples containing hyperparameters and their values. If a hyperparameter is:\n",
      "        - Numeric, then the tuple passed is (hyperparameter_name, min, max)\n",
      "        - Categorical, then the tuple passed is (hyperparameter_name, [val_1, val_2, ..., val_n])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(ml.fit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "203e9959ca7ebbf0117ecf3e0dfde7f27540c9b63f49411d7f46e608c40da6fe"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}