gtc_solaiman_logistic-regression_tfidf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dcad0cdf",
   "metadata": {},
   "source": [
    "# Benchmark of existing approaches for detecting machine-generated text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84e279fd-6245-41b4-9fd0-10c8fa904382",
   "metadata": {},
   "source": [
    "This notebook enables to evaluate the logistic regression-based text classifier by Solaiman et al."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c32dde55",
   "metadata": {},
   "source": [
    "## Solaiman: Logistic Regression\n",
    "\n",
    "_Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford,\n",
    "Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. Release strategies and the social impacts of language models._\n",
    "\n",
    "Implementation of the text classifier based on logistic regression.\n",
    "\n",
    "Logic extracted form `baseline.py` by Solaiman et al.: [https://github.com/openai/gpt-2-output-dataset/tree/master](https://github.com/openai/gpt-2-output-dataset/tree/master)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b2f3110-6571-4d5b-8ecd-12efd8b09fc2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Install dependencies\n",
    "To run the code in this notebook, you must install the required dependencies ([https://github.com/HendrikStrobelt/detecting-fake-text/blob/master/requirements.txt](detecting-fake-text/requirements.txt)). \n",
    "\n",
    "See the `gtc_solaiman_RoBERTa.ipynb` notebook for further information.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52ec015b-28d8-4d57-bdc3-fe523b19c920",
   "metadata": {},
   "source": [
    "## Evaluation (with downsampling)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf0768a4-2253-407c-9dc5-3275cc70d299",
   "metadata": {},
   "source": [
    "This section is devoted to evaluating the Solaiman et al. model under different datasets. \n",
    "\n",
    "Here, we consider **pre-processed** dataset, with downsampling. We aim to evalute the Solaiman et al. technique on a dataset with limited noise. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83bf3979-485c-4b10-b1f3-9edb10897533",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef7992bf-0a25-4382-8037-91feb38450ff",
   "metadata": {},
   "source": [
    "Definition of imports and auxiliary functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "73268589",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import numpy as np\n",
    "from scipy import sparse\n",
    "\n",
    "from sklearn.model_selection import PredefinedSplit, GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4d8f939a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def _load_split(data_dir, source, n=np.inf):\n",
    "    path = os.path.join(data_dir, source)\n",
    "    texts = []\n",
    "    for i, line in enumerate(open(path)):\n",
    "        if i >= n:\n",
    "            break\n",
    "        texts.append(json.loads(line)['text'])\n",
    "    return texts\n",
    "\n",
    "def load_split(data_dir, source, webtext_source, n=np.inf):\n",
    "    webtext = _load_split(data_dir, webtext_source, n=n//2)\n",
    "    gen = _load_split(data_dir, source, n=n//2)\n",
    "    texts = webtext+gen\n",
    "    labels = [0]*len(webtext)+[1]*len(gen)\n",
    "    return texts, labels\n",
    "\n",
    "\n",
    "def load_single_split(data_dir, source, machine_generated=True, n=np.inf):\n",
    "    data = _load_split(data_dir, source, n)\n",
    "    if machine_generated:\n",
    "        labels = [1]*len(data)\n",
    "    else:\n",
    "        labels = [0]*len(data)\n",
    "    return data, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5235d2e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression(solver='liblinear')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25555009-899a-404a-9dc6-e1d46a8fda0b",
   "metadata": {},
   "source": [
    "Train the classifier using subsets of the GPT2vsWebText dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0080a085",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_train=200000\n",
    "n_valid=10000\n",
    "n_jobs=-1\n",
    "verbose=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "30ef5582",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_texts, train_labels = load_split('./data/training/', 'GPT2-xl-1542M-k40.train.filtered.jsonl', 'webtext.train.filtered.jsonl', n=n_train)\n",
    "valid_texts, valid_labels = load_split('./data/validation/', 'GPT2-xl-1542M-k40.valid.machine.jsonl', 'webtext.valid.human.jsonl', n=n_valid)\n",
    "test_texts, test_labels = load_split('./data/test/', 'GPT2-xl-1542M-k40.test.machine.jsonl', 'webtext.test.human.jsonl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0290c121",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200000"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e5d2dc1b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a2f32f40",
   "metadata": {},
   "outputs": [],
   "source": [
    "vect = TfidfVectorizer(ngram_range=(1, 2), min_df=5, max_features=2**21)\n",
    "train_features = vect.fit_transform(train_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4294f8ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_features = vect.transform(valid_texts)\n",
    "test_features = vect.transform(test_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e116925",
   "metadata": {},
   "source": [
    "skip the following grid search grid and use the default params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f30d7264",
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'C': [1/64, 1/32, 1/16, 1/8, 1/4, 1/2, 1, 2, 4, 8, 16, 32, 64]}\n",
    "# params = {'C': [8, 16]}\n",
    "split = PredefinedSplit([-1]*n_train+[0]*n_valid)\n",
    "search = GridSearchCV(model, params, cv=split, n_jobs=n_jobs, verbose=verbose, refit=False)\n",
    "search.fit(sparse.vstack([train_features, valid_features]), train_labels+valid_labels)\n",
    "model = model.set_params(**search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0750984c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(C=16, solver=&#x27;liblinear&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(C=16, solver=&#x27;liblinear&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression(C=16, solver='liblinear')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_features, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "41afe8f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 16}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5785a002",
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_accuracy = model.score(valid_features, valid_labels)*100.\n",
    "test_accuracy = model.score(test_features, test_labels)*100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0745fcf5-23bf-4afd-a43d-1f5de2612863",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91.27"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "26679b52-dc1d-438f-8b89-a7c7f436c5b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33deeec0",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on WebText + GPT2-xl-1542M data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1a8f1683",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = './data/test/'\n",
    "source='GPT2-xl-1542M-k40.test.machine.jsonl'\n",
    "webtext_source='webtext.test.human.jsonl'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "05b58a34",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_mg_texts, test_mg_labels = load_single_split(data_dir, source, machine_generated=True)\n",
    "test_mg_features = vect.transform(test_mg_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3de399dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_accuracy = model.score(test_mg_features, test_mg_labels)*100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "855d3c8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "89.74"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fa8fccd9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'webtext.test.human.jsonl'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "webtext_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "53ceb54a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91.58"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_hw_texts, test_hw_labels = load_single_split(data_dir, webtext_source, machine_generated=False)\n",
    "test_hw_features = vect.transform(test_hw_texts)\n",
    "test_accuracy = model.score(test_hw_features, test_hw_labels)*100.\n",
    "test_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82392e97-62c5-4059-9cc9-0205da5e0d30",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on GPT3-175b Machine generated text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "44941fdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "def create_dataset(raw_data, is_machine_generated=True):\n",
    "    X = np.array(raw_data)\n",
    "    if is_machine_generated:\n",
    "        y = np.ones_like(X, dtype=int)\n",
    "    else:\n",
    "        y = np.zeros_like(X, dtype=int)\n",
    "    return pd.DataFrame(data={'X':X, 'y':y})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "e98ce3b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt3_data_path = \"./data/test/\"\n",
    "gpt3_ds_mg_filename = 'GPT3-175b.test.machine.jsonl'\n",
    "with open(os.path.join(gpt3_data_path, gpt3_ds_mg_filename), 'r') as file:\n",
    "    gpt3_ds_mg = [x for x in file.readlines() if x is not None and x != \"\\n\"]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "id": "4fb299c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt3_mg_pd = create_dataset(gpt3_ds_mg, is_machine_generated=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "id": "37361109",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_features_and_labels_for_gpt3(dataset, vectorizer):\n",
    "    gpt3_test_text = [i[1:-2] for i in dataset['X']]\n",
    "    gpt3_test_labels = dataset['y'].tolist()\n",
    "    gpt3_test_features = vectorizer.transform(gpt3_test_text)\n",
    "    return (gpt3_test_features, gpt3_test_labels, gpt3_test_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "f999c244",
   "metadata": {},
   "outputs": [],
   "source": [
    "g3_mg_feat, g3_mg_labels, _ = get_features_and_labels_for_gpt3(\n",
    "    gpt3_mg_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "id": "c1a39c35",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt3_mg_accuracy = model.score(g3_mg_feat, g3_mg_labels)*100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "id": "67645eb5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15.257731958762887"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpt3_mg_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "id": "c7b707fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"Glacier Ridge Christian School\\n\\nGlacier Rid...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"His father was a professor and his mother was...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Image caption Bailiffs removed protesters fro...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"Contents\\n\\n1. The Meaning of Love\\n\\n2. Our ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"Interviews\\n\\nInterview with Ariel Efraim (So...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>480</th>\n",
       "      <td>\"Get Rangers updates directly to your inbox Su...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>481</th>\n",
       "      <td>\"Posted 01 September 2010 - 07:51 PM\\n\\nUsing ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>482</th>\n",
       "      <td>\"Jeff Chapman\\n\\nJeffrey Scott Chapman (born M...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>483</th>\n",
       "      <td>\"You can look into the Issue Tracker and repor...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>484</th>\n",
       "      <td>\"Airport attack: What happened at Zaventem Air...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>485 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     X  y\n",
       "0    \"Glacier Ridge Christian School\\n\\nGlacier Rid...  1\n",
       "1    \"His father was a professor and his mother was...  1\n",
       "2    \"Image caption Bailiffs removed protesters fro...  1\n",
       "3    \"Contents\\n\\n1. The Meaning of Love\\n\\n2. Our ...  1\n",
       "4    \"Interviews\\n\\nInterview with Ariel Efraim (So...  1\n",
       "..                                                 ... ..\n",
       "480  \"Get Rangers updates directly to your inbox Su...  1\n",
       "481  \"Posted 01 September 2010 - 07:51 PM\\n\\nUsing ...  1\n",
       "482  \"Jeff Chapman\\n\\nJeffrey Scott Chapman (born M...  1\n",
       "483  \"You can look into the Issue Tracker and repor...  1\n",
       "484  \"Airport attack: What happened at Zaventem Air...  1\n",
       "\n",
       "[485 rows x 2 columns]"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpt3_mg_pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "id": "aec7b14c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-5 {color: black;background-color: white;}#sk-container-id-5 pre{padding: 0;}#sk-container-id-5 div.sk-toggleable {background-color: white;}#sk-container-id-5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-5 div.sk-item {position: relative;z-index: 1;}#sk-container-id-5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-5 div.sk-item::before, #sk-container-id-5 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-5 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-5 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-5 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-5 div.sk-label-container {text-align: center;}#sk-container-id-5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(C=8, solver=&#x27;liblinear&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" checked><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(C=8, solver=&#x27;liblinear&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression(C=8, solver='liblinear')"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c008815d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on GROVER dataset p=0.94"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "id": "f16c7331",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_data_path = \"./data/test/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "id": "2969b88f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_features_and_labels_for_grover(dataset, vectorizer):\n",
    "    test_text = [i for i in dataset['X']]\n",
    "    test_labels = dataset['y'].tolist()\n",
    "    test_features = vectorizer.transform(test_text)\n",
    "    return (test_features, test_labels, test_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "551618ed",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### dataset mega"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "id": "efe92ff2",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_ds_mg_filename = 'Grover-mega-p0.94.test.machine.jsonl'\n",
    "grover_ds_hw_filename = 'Grover.human.jsonl'\n",
    "grover_ds_mg = pd.read_json(os.path.join(grover_data_path, grover_ds_mg_filename), lines = True)\n",
    "grover_ds_hw = pd.read_json(os.path.join(grover_data_path, grover_ds_hw_filename), lines = True)\n",
    "grover_mg_pd = create_dataset(grover_ds_mg[\"text\"], is_machine_generated=True)\n",
    "grover_hw_pd = create_dataset(grover_ds_hw[\"text\"], is_machine_generated=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47fd5204",
   "metadata": {},
   "source": [
    "Test on GROVER human-written sentences\n",
    "Same dataset as p=0.94; same accuracy value expected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "id": "7934329b",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_hw094_feat, grover_hw094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_hw_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "id": "7927410c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "74.66666666666667"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_hw094_accuracy = model.score(grover_hw094_feat, grover_hw094_labels)*100.\n",
    "grover_hw094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6189b3b9",
   "metadata": {},
   "source": [
    "Machine-generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "id": "2e1bb0c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_mg094_feat, grover_mg094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_mg_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "id": "f8f462c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18.39"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_mg094_accuracy = model.score(grover_mg094_feat, grover_mg094_labels)*100.\n",
    "grover_mg094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69c1c21f",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### dataset medium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "id": "20bd3130",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_ds_mg_filename = 'Grover-medium-p0.94.test.machine.jsonl'\n",
    "grover_ds_hw_filename = 'Grover.human.jsonl'\n",
    "grover_ds_mg = pd.read_json(os.path.join(grover_data_path, grover_ds_mg_filename), lines = True)\n",
    "grover_ds_hw = pd.read_json(os.path.join(grover_data_path, grover_ds_hw_filename), lines = True)\n",
    "grover_mg_pd = create_dataset(grover_ds_mg[\"text\"], is_machine_generated=True)\n",
    "grover_hw_pd = create_dataset(grover_ds_hw[\"text\"], is_machine_generated=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ae9e1c8",
   "metadata": {},
   "source": [
    "human-written"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "id": "44168136",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_hw094_feat, grover_hw094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_hw_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "id": "df2c2d40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "74.66666666666667"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_hw094_accuracy = model.score(grover_hw094_feat, grover_hw094_labels)*100.\n",
    "grover_hw094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "007ffce3",
   "metadata": {},
   "source": [
    "machine-generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "id": "85c122b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_mg094_feat, grover_mg094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_mg_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "id": "ac1116e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.88"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_mg094_accuracy = model.score(grover_mg094_feat, grover_mg094_labels)*100.\n",
    "grover_mg094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65417d9c",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### dataset base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "id": "3224b5a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_ds_mg_filename = 'Grover-base-p0.94.test.machine.jsonl'\n",
    "grover_ds_hw_filename = 'Grover.human.jsonl'\n",
    "grover_ds_mg = pd.read_json(os.path.join(grover_data_path, grover_ds_mg_filename), lines = True)\n",
    "grover_ds_hw = pd.read_json(os.path.join(grover_data_path, grover_ds_hw_filename), lines = True)\n",
    "grover_mg_pd = create_dataset(grover_ds_mg[\"text\"], is_machine_generated=True)\n",
    "grover_hw_pd = create_dataset(grover_ds_hw[\"text\"], is_machine_generated=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69976829",
   "metadata": {},
   "source": [
    "human-written"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "id": "b2db7bda",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_hw094_feat, grover_hw094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_hw_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "id": "a6629565",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "74.66666666666667"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_hw094_accuracy = model.score(grover_hw094_feat, grover_hw094_labels)*100.\n",
    "grover_hw094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a4f2b4d",
   "metadata": {},
   "source": [
    "machine-generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "id": "0275a434",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_mg094_feat, grover_mg094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_mg_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "87aaa6c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21.65"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_mg094_accuracy = model.score(grover_mg094_feat, grover_mg094_labels)*100.\n",
    "grover_mg094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f9e5cdf",
   "metadata": {},
   "source": [
    "## Evaluation (no downsampling)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5386798a-3586-42f1-9ddc-976ce0707cab",
   "metadata": {},
   "source": [
    "This section is devoted to evaluating the Solaiman et al. model under different datasets. \n",
    "\n",
    "Here, we consider **raw** dataset, with no downsampling."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8a643c9-47fb-4a4b-a7e9-5b9d80101fc2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecefb66e",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_texts, train_labels = load_split('./data/training/', 'GPT2-xl-1542M.train.filtered.jsonl', 'webtext.train.filtered.jsonl', n=n_train)\n",
    "valid_texts, valid_labels = load_split('./data/validation/', 'GPT2-xl-1542M.valid.machine.jsonl', 'webtext.valid.machine.jsonl', n=n_valid)\n",
    "test_texts, test_labels = load_split('./data/test/', 'GPT2-xl-1542M.test.machine.jsonl', 'webtext.test.human.jsonl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "b9791b20",
   "metadata": {},
   "outputs": [],
   "source": [
    "vect = TfidfVectorizer(ngram_range=(1, 2), min_df=5, max_features=2**21)\n",
    "train_features = vect.fit_transform(train_texts)\n",
    "valid_features = vect.transform(valid_texts)\n",
    "test_features = vect.transform(test_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "be896407",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-4 {color: black;background-color: white;}#sk-container-id-4 pre{padding: 0;}#sk-container-id-4 div.sk-toggleable {background-color: white;}#sk-container-id-4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-4 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-4 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-4 div.sk-item {position: relative;z-index: 1;}#sk-container-id-4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-4 div.sk-item::before, #sk-container-id-4 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-4 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-4 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-4 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-4 div.sk-label-container {text-align: center;}#sk-container-id-4 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-4 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(C=8, solver=&#x27;liblinear&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" checked><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(C=8, solver=&#x27;liblinear&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression(C=8, solver='liblinear')"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_features, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "ad573228",
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_accuracy = model.score(valid_features, valid_labels)*100.\n",
    "test_accuracy = model.score(test_features, test_labels)*100."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fb0cc37-4ee9-4d70-a6f8-ec62487f3446",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on WebText + GPT2-xl-1542M data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "08730314",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = './data/test/'\n",
    "source='GPT2-xl-1542M.test.machine.jsonl'\n",
    "webtext_source='webtext.test.human.jsonl'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "65cc3e4b-e792-48fa-a2e8-6019a760f64a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'GPT2-xl-1542M.test.machine.jsonl'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "271f3aaf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "68.60000000000001"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_mg_texts, test_mg_labels = load_single_split(data_dir, source, machine_generated=True)\n",
    "test_mg_features = vect.transform(test_mg_texts)\n",
    "test_accuracy = model.score(test_mg_features, test_mg_labels)*100.\n",
    "test_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "c2dfe538",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'webtext'"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "webtext_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "7b4b8682",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "73.48"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_hw_texts, test_hw_labels = load_single_split(data_dir, webtext_source, machine_generated=False)\n",
    "test_hw_features = vect.transform(test_hw_texts)\n",
    "test_accuracy = model.score(test_hw_features, test_hw_labels)*100.\n",
    "test_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e3b8383-9fca-434b-b096-c1b978f9c2fd",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on GPT3-175b Machine generated text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c64cf4c9-7293-4483-ac27-c63fe273936e",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt3_data_path = \"./data/test/\"\n",
    "gpt3_ds_mg_filename = 'GPT3-175b.test.machine.jsonl'\n",
    "with open(os.path.join(gpt3_data_path, gpt3_ds_mg_filename), 'r') as file:\n",
    "    gpt3_ds_mg = [x for x in file.readlines() if x is not None and x != \"\\n\"]\n",
    "gpt3_mg_pd = create_dataset(gpt3_ds_mg, is_machine_generated=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5885d2b-b51c-41be-b9cf-f71bf5c99497",
   "metadata": {},
   "outputs": [],
   "source": [
    "g3_mg_feat, g3_mg_labels, _ = get_features_and_labels_for_gpt3(gpt3_mg_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bf2ac72-080b-440c-bab2-037f489c1273",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt3_mg_accuracy = model.score(g3_mg_feat, g3_mg_labels)*100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "300feac0-90ff-4d1c-ad60-44ad69e97009",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt3_mg_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50226c8d-fedd-4659-a81b-7faac65712d9",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on GROVER dataset p=0.94"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a8466a6-e6a5-44a7-8f74-77806b1b3d11",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### dataset mega"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f75b302-8b7d-4d4f-88ba-f4b68c1648c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_data_path = \"./data/test/\"\n",
    "\n",
    "grover_ds_mg_filename = 'Grover-mega-p0.94.test.machine.jsonl'\n",
    "grover_ds_hw_filename = 'Grover.human.jsonl'\n",
    "grover_ds_mg = pd.read_json(os.path.join(grover_data_path, grover_ds_mg_filename), lines = True)\n",
    "grover_ds_hw = pd.read_json(os.path.join(grover_data_path, grover_ds_hw_filename), lines = True)\n",
    "grover_mg_pd = create_dataset(grover_ds_mg[\"text\"], is_machine_generated=True)\n",
    "grover_hw_pd = create_dataset(grover_ds_hw[\"text\"], is_machine_generated=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed66c426-3c7c-408c-a693-f5b1be322350",
   "metadata": {},
   "source": [
    "Test on GROVER human-written sentences\n",
    "Same dataset as p=0.94; same accuracy value expected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73516617-c40a-4b57-ab85-c4b9b760787e",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_hw094_feat, grover_hw094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_hw_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "id": "a4dc420d-0fc2-471c-be85-509e599de363",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "74.66666666666667"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_hw094_accuracy = model.score(grover_hw094_feat, grover_hw094_labels)*100.\n",
    "grover_hw094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00c89b43-43fe-4dd0-9652-b7a87e326acd",
   "metadata": {},
   "source": [
    "Machine-generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "id": "b064c69d-8f47-44db-a463-a6fa424f6c9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_mg094_feat, grover_mg094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_mg_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "id": "20d72767-80ea-45b4-911d-5e98b0135cee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18.39"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_mg094_accuracy = model.score(grover_mg094_feat, grover_mg094_labels)*100.\n",
    "grover_mg094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4366949c-5225-43f3-a4ec-c84d5c9a60b9",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### dataset medium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed18af6a-3789-43c1-98ad-fe842446acc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_ds_mg_filename = 'Grover-medium-p0.94.test.machine.jsonl'\n",
    "grover_ds_hw_filename = 'Grover.human.jsonl'\n",
    "grover_ds_mg = pd.read_json(os.path.join(grover_data_path, grover_ds_mg_filename), lines = True)\n",
    "grover_ds_hw = pd.read_json(os.path.join(grover_data_path, grover_ds_hw_filename), lines = True)\n",
    "grover_mg_pd = create_dataset(grover_ds_mg[\"text\"], is_machine_generated=True)\n",
    "grover_hw_pd = create_dataset(grover_ds_hw[\"text\"], is_machine_generated=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f276b7-7f8a-4911-9322-b76f2b30b01b",
   "metadata": {},
   "source": [
    "human-written"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0fbfc48-9b32-4952-aca0-28fe2f68b193",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_hw094_feat, grover_hw094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_hw_pd, vect)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "id": "f1fa1703-3f5b-4212-8a6d-3650d43855b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "74.66666666666667"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_hw094_accuracy = model.score(grover_hw094_feat, grover_hw094_labels)*100.\n",
    "grover_hw094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdac9270-d854-4b9e-8fbb-23e5f186a7c0",
   "metadata": {},
   "source": [
    "machine-generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1116b07f-b77f-4778-bcc6-cacb5dae0873",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_mg094_feat, grover_mg094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_mg_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "id": "7c38cc97-7218-45bd-b199-c7922f7979eb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.88"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_mg094_accuracy = model.score(grover_mg094_feat, grover_mg094_labels)*100.\n",
    "grover_mg094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "748a7fd9-418d-47ce-8813-9c47963d5e44",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### dataset base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a94d7266-db95-42c7-b6cd-153840825e13",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_ds_mg_filename = 'Grover-base-p0.94.test.machine.jsonl'\n",
    "grover_ds_hw_filename = 'Grover.human.jsonl'\n",
    "grover_ds_mg = pd.read_json(os.path.join(grover_data_path, grover_ds_mg_filename), lines = True)\n",
    "grover_ds_hw = pd.read_json(os.path.join(grover_data_path, grover_ds_hw_filename), lines = True)\n",
    "grover_mg_pd = create_dataset(grover_ds_mg[\"text\"], is_machine_generated=True)\n",
    "grover_hw_pd = create_dataset(grover_ds_hw[\"text\"], is_machine_generated=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dcd9d61-37bf-42d9-a970-e52ab4e207db",
   "metadata": {},
   "source": [
    "human-written"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc54938c-4552-4382-bd08-02757a0f9bbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_hw094_feat, grover_hw094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_hw_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "id": "06557561-2482-4707-a3aa-250caabfe2ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "74.66666666666667"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_hw094_accuracy = model.score(grover_hw094_feat, grover_hw094_labels)*100.\n",
    "grover_hw094_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e10e58bc-9e91-450f-96dc-8f28c4392ecb",
   "metadata": {},
   "source": [
    "machine-generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbaa2933-3ee0-401e-9bff-5970dcbf7d0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_mg094_feat, grover_mg094_labels, _ = \\\n",
    "    get_features_and_labels_for_grover(\n",
    "    grover_mg_pd, vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "f18caa3c-5467-4c5e-954d-a80c9fbdc8fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21.65"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_mg094_accuracy = model.score(grover_mg094_feat, grover_mg094_labels)*100.\n",
    "grover_mg094_accuracy"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}