examples/auto_annotations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "839ee94d-5ead-46da-bd60-5b11e4aaebc4",
   "metadata": {},
   "source": [
    "# Auto Annotations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "521a7025-81f7-45e0-a8ee-8454f2e5e01a",
   "metadata": {},
   "source": [
    "Note: AlpacaFarm now uses [AlpacaEval 1](https://github.com/tatsu-lab/alpaca_eval/tree/main). For annotations and evaluations, since `text-davinci-003` has been depreciated by OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f9ee20f0-007c-4f6a-af1c-ad39b959984b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/yanndubois/Desktop/GitHub/alpaca_farm\n"
     ]
    }
   ],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "445bf8b0-90b2-4691-b6d8-cef65cbde034",
   "metadata": {},
   "source": [
    "## Setting up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b4f2875-9f29-4662-b63e-f96a8887aa14",
   "metadata": {},
   "source": [
    "All of our annotators currently use OpenAI API. So the first step is to setup your OpenAI key (and potentially your organization). This can be done by either running setting your environment variable like \n",
    "- `export OPENAI_API_KEY=\"sk...\"` "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9b8222f-c017-4aae-9b29-cc8210e946fe",
   "metadata": {},
   "source": [
    "The key object that you need for automatic annotations (both for training or for eval) is the `PairwiseAutoAnnotator`, which is a minor wrapper around [AlapcaEval's `PairwiseAnnotator`](https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/annotators/pairwise_evaluator.py#L20) (wrapper to deal with separate instruction and input). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "32d6dcf4-b0fb-4c56-a1fd-1db509af05e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Creating the annotator from `alpaca_eval_gpt4`.\n",
      "INFO:root:Saving annotations to `/Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json`.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n"
     ]
    }
   ],
   "source": [
    "from alpaca_farm.utils import jload\n",
    "from alpaca_farm.auto_annotations import PairwiseAutoAnnotator\n",
    "\n",
    "annotator = PairwiseAutoAnnotator(annotators_config=\"alpaca_eval_gpt4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b99ac04-0d48-410e-95fd-cf3b1360fc4e",
   "metadata": {},
   "source": [
    "## Annotating paired outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ef99a2c-77ef-41e6-9db8-eae091afd952",
   "metadata": {},
   "source": [
    "Now let's annotate some pairwise preference. All we need is some data. The annotator takes in either list of dictionaries, or pandas dataframes. The keys of dictionaries (or columns in dataframe) need to always contain `instruction` and `input` which defines the instruction. The annotator also needs a pair of outputs to compare, if you have a sequence of such pairs under the keys `output_1` and `output_2` then you can directly use `annotate_pairs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7ab42c1d-3bda-4797-a1bd-877b958e8657",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of paired output:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'instruction': 'If you could help me write an email to my friends inviting them to dinner on Friday, it would be greatly appreciated.',\n",
       "  'input': '',\n",
       "  'output_1': \"Dear Friends, \\r\\n\\r\\nI hope this message finds you well. I'm excited to invite you to dinner on Friday. We'll meet at 7:00 PM at [location]. I look forward to seeing you there. \\r\\n\\r\\nBest,\\r\\n[Name]\",\n",
       "  'output_2': \"Hey everyone! \\n\\nI'm hosting a dinner party this Friday night and I'd love for all of you to come over. We'll have a delicious spread of food and some great conversations. \\n\\nLet me know if you can make it - I'd love to see you all there!\\n\\nCheers,\\n[Your Name]\"}]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs_pairs = jload(\"examples/data/outputs_pairs.json\")[:6]\n",
    "print(\"Example of paired output:\\n\")\n",
    "outputs_pairs[-1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "67caa934-5a37-429a-affc-dbd182be7c40",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Annotation chunk:   0%|                                                       | 0/1 [00:00<?, ?it/s]INFO:root:Annotating 6 examples with alpaca_eval_gpt4\n",
      "INFO:root:Using `openai_completions` on 6 prompts using gpt-4.\n",
      "INFO:root:Kwargs to completion: {'model': 'gpt-4', 'is_chat': True, 'temperature': 0}. num_procs=5\n",
      "\n",
      "prompt_batches:   0%|                                                         | 0/6 [00:00<?, ?it/s]\u001b[AINFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\n",
      "prompt_batches:  17%|████████▏                                        | 1/6 [00:03<00:16,  3.38s/it]\u001b[AINFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\n",
      "prompt_batches:  33%|████████████████▎                                | 2/6 [00:04<00:07,  1.78s/it]\u001b[AINFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\n",
      "prompt_batches: 100%|█████████████████████████████████████████████████| 6/6 [00:04<00:00,  1.29it/s]\u001b[A\n",
      "INFO:root:Completed 6 examples in 4.8 seconds.\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk: 100%|███████████████████████████████████████████████| 1/1 [00:06<00:00,  6.59s/it]\n"
     ]
    }
   ],
   "source": [
    "annotated = annotator.annotate_pairs(outputs_pairs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1dd953df-7eb1-4446-86a9-f3b3fbfba665",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'instruction': 'If you could help me write an email to my friends inviting them to dinner on Friday, it would be greatly appreciated.',\n",
       "  'input': '',\n",
       "  'output_1': \"Dear Friends, \\r\\n\\r\\nI hope this message finds you well. I'm excited to invite you to dinner on Friday. We'll meet at 7:00 PM at [location]. I look forward to seeing you there. \\r\\n\\r\\nBest,\\r\\n[Name]\",\n",
       "  'output_2': \"Hey everyone! \\n\\nI'm hosting a dinner party this Friday night and I'd love for all of you to come over. We'll have a delicious spread of food and some great conversations. \\n\\nLet me know if you can make it - I'd love to see you all there!\\n\\nCheers,\\n[Your Name]\",\n",
       "  'annotator': 'alpaca_eval_gpt4',\n",
       "  'preference': 2.0,\n",
       "  'price_per_example': 0.012989999999999998,\n",
       "  'time_per_example': 0.8035085201263428,\n",
       "  'raw_completion': \"[\\n    {'model': 'model_1', 'rank': 1},\\n    {'model': 'model_2', 'rank': 2}\\n]\"}]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotated[-1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e648a4b0-5d19-4bc7-b63e-d4cb4ab4fd51",
   "metadata": {},
   "source": [
    "Here we see that the annotations adds two keys:\n",
    "- `'preference'`: the index of the preferred output, here `preference=1` so `output_1` is preferred. In the case where both outputs are the same we give `preference=0`\n",
    "- `'annotator'`: the name of the simulated annotator as found in the `annotators_config`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa8ee339-cf40-4a3c-a861-19c265f962bc",
   "metadata": {},
   "source": [
    "`annotate_pairs` is the main function and should be used if you have paired outputs to annotate. In many usecases, however, you  will outputs in different formats. In the following we discuss two additional helper function `annotate_head2head` and `annotate_samples` which are paticularly well suited for the typical format during evaluation and training respectively. Both functions call `annotate_pairs` under the hood after a reformatting step."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80b41af4-731c-4d12-83b4-9b69f4b2bc3f",
   "metadata": {},
   "source": [
    "## Evaluation through pairwise comparison\n",
    "For evaluation we need two components:\n",
    "- outputs from the model we want to compare\n",
    "- outputs on the same examples from the baseline model\n",
    "\n",
    "Often case both of those components will be in a different list of dictionary (one list for each model). In this case all dictionaries need to contain an `output` column. Let us load such data from our simulated RLHF model and text-davinci-003 baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e2a9cce3-0fe9-4dc2-9b49-de59f641e170",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of baseline output:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'instruction': 'If you could help me write an email to my friends inviting them to dinner on Friday, it would be greatly appreciated.',\n",
       "  'input': '',\n",
       "  'output': \"Dear Friends, \\r\\n\\r\\nI hope this message finds you well. I'm excited to invite you to dinner on Friday. We'll meet at 7:00 PM at [location]. I look forward to seeing you there. \\r\\n\\r\\nBest,\\r\\n[Name]\"}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs_baseline = jload(\"examples/data/outputs_baseline.json\")[:6]\n",
    "print(\"Example of baseline output:\\n\")\n",
    "outputs_baseline[-1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1f808315-da5c-4af3-a026-64e3d07801f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of rlhf output:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'instruction': 'If you could help me write an email to my friends inviting them to dinner on Friday, it would be greatly appreciated.',\n",
       "  'input': '',\n",
       "  'output': 'Dear Friends, \\n\\nI am writing to invite you all to a dinner on Friday evening. It is a casual affair, and I am looking forward to a fun evening catching up with you all. I am planning to make a selection of delicious dishes, ranging from appetizers to mains and desserts. There will be something for everyone to enjoy, and I am sure it will be a night to remember.\\n\\nThe dinner will be held at my place on Friday, April 17th at 7pm. If you are interested in joining me, please RSVP to this email by Thursday, April 16th. I am looking forward to seeing you all there! \\n\\nThank you, \\n\\n[Name]'}]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs_rlhf = jload(\"examples/data/outputs_rlhf.json\")[:6]\n",
    "print(\"Example of rlhf output:\\n\")\n",
    "outputs_rlhf[-1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d6ff9ed-e67b-41df-b706-f2633ec9b909",
   "metadata": {},
   "source": [
    "The annotator's function of interest when we have two sequences of outputs is `annotate_head2head`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e613e0ca-2db8-436c-a4c7-bf71de116ae0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Annotation chunk:   0%|                                                       | 0/1 [00:00<?, ?it/s]INFO:root:Annotating 6 examples with alpaca_eval_gpt4\n",
      "INFO:root:Using `openai_completions` on 6 prompts using gpt-4.\n",
      "INFO:root:Kwargs to completion: {'model': 'gpt-4', 'is_chat': True, 'temperature': 0}. num_procs=5\n",
      "\n",
      "prompt_batches:   0%|                                                         | 0/6 [00:00<?, ?it/s]\u001b[AINFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\n",
      "prompt_batches:  17%|████████▏                                        | 1/6 [00:03<00:18,  3.66s/it]\u001b[AINFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\n",
      "prompt_batches:  67%|████████████████████████████████▋                | 4/6 [00:03<00:01,  1.37it/s]\u001b[AINFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\n",
      "prompt_batches: 100%|█████████████████████████████████████████████████| 6/6 [00:04<00:00,  1.43it/s]\u001b[A\n",
      "INFO:root:Completed 6 examples in 4.4 seconds.\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk: 100%|███████████████████████████████████████████████| 1/1 [00:06<00:00,  6.07s/it]\n"
     ]
    }
   ],
   "source": [
    "annotated = annotator.annotate_head2head(outputs_1=outputs_baseline, outputs_2=outputs_rlhf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a72e4e7a-cd40-41c3-87b7-995e323e35d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'instruction': 'The sentence you are given might be too wordy, complicated, or unclear. Rewrite the sentence and make your writing clearer by keeping it concise. Whenever possible, break complex sentences into multiple sentences and eliminate unnecessary words.\\n\\nIf you have any questions about my rate or if you find it necessary to increase or decrease the scope for this project, please let me know.',\n",
       "  'input': 'If you have any questions about my rate or if you find it necessary to increase or decrease the scope for this project, please let me know.',\n",
       "  'output_1': 'If you have questions about my rate or need to modify the scope of this project, please let me know.',\n",
       "  'output_2': 'I am available to answer any questions you may have about my rate or if you need to alter the scope of this project. Feel free to contact me if you have any queries or require any additional information.',\n",
       "  'annotator': 'alpaca_eval_gpt4',\n",
       "  'preference': 1.0,\n",
       "  'price_per_example': 0.01287,\n",
       "  'time_per_example': 0.7309945424397787,\n",
       "  'raw_completion': \"[\\n    {'model': 'model_2', 'rank': 1},\\n    {'model': 'model_1', 'rank': 2}\\n]\"}]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotated[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ff1699e-eef3-4c98-bc83-512f5f401c6f",
   "metadata": {},
   "source": [
    "We see that the format of the output is the same as before. Here `preference` indicates that the simulator prefered the output `output_1`, which corresponds to the baseline model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcd6a404-4fc1-4f0c-9858-02f24a189b95",
   "metadata": {},
   "source": [
    "## Training\n",
    "For training we typically have multiple outputs for each instruction which are sampled from the same (SFT) model as seen in the following data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "50354b51-12c5-4833-81af-549d78ef3dc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of different sampled outputs from SFT:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'instruction': 'Describe a time when you had to make a difficult decision.',\n",
       "  'input': '',\n",
       "  'output': [\"I had to make a difficult decision a few years ago when I was offered a job that would require me to move to a different city. I had to weigh the pros and cons of this job offer, and consider what it would mean for me to leave my friends and family in order to take it. In the end, I decided to turn down the job offer since I wasn't ready to quit my current job, move to a different city, and leave my loved ones behind.\",\n",
       "   'I had to make a difficult decision last year when I was faced with a life-changing opportunity. I had to decide whether or not to leave my current job and move to a different city for the chance to further my career. After much consideration, I chose to take the risk and make the move. Thankfully, it paid off and I am now enjoying my new job and exploring the city.',\n",
       "   \"I had to make a difficult decision when I was offered a job in a city that was significantly farther away from my friends and family. I weighed the pros and cons and consulted with people I trusted, but in the end I had to make the choice that was best for my future and my well-being. It was a difficult decision, but I'm glad I made the choice that allowed me to further my career and start my journey towards independence.\",\n",
       "   'I remember a time when I had to decide whether to stay in my hometown or move to a new city. It was a difficult decision because I had grown up in the town, and I was used to the familiarity of my childhood home. On the other hand, the new city had more job opportunities and a better environment that I thought would be beneficial for my future. After weighing the pros and cons, I made the decision to move to the new city. It was the right choice, as I was able to find a job I enjoyed and make some great friends in the new place.']}]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs_samples = jload(\"examples/data/multisamples_sft.json\")[:1]\n",
    "print(\"Example of different sampled outputs from SFT:\\n\")\n",
    "outputs_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0df755cb-8ff5-478d-b397-682ab3d745e1",
   "metadata": {},
   "source": [
    "In this case, you can use the annotator's `annotate_samples`, which first samples pairs of outputs that have the same instruction/input and then annotate those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cc8557da-46fd-4906-a209-a02cc12eef9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Filtered unique instruction/input pairs: 4 -> 1\n",
      "Annotation chunk:   0%|                                                       | 0/1 [00:00<?, ?it/s]INFO:root:Annotating 1 examples with alpaca_eval_gpt4\n",
      "INFO:root:Using `openai_completions` on 1 prompts using gpt-4.\n",
      "INFO:root:Kwargs to completion: {'model': 'gpt-4', 'is_chat': True, 'temperature': 0}. num_procs=5\n",
      "\n",
      "prompt_batches:   0%|                                                         | 0/1 [00:00<?, ?it/s]\u001b[AINFO:root:Using OAI client number 1 out of 1.\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\n",
      "prompt_batches: 100%|█████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.78s/it]\u001b[A\n",
      "INFO:root:Completed 1 examples in 3.0 seconds.\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk: 100%|███████████████████████████████████████████████| 1/1 [00:04<00:00,  4.54s/it]\n"
     ]
    }
   ],
   "source": [
    "annotated = annotator.annotate_samples(outputs_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6a55348a-a0ca-4fe1-b6d1-b9fa5d1c339b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'sample_id': 0,\n",
       "  'instruction': 'Describe a time when you had to make a difficult decision.',\n",
       "  'input': '',\n",
       "  'output_1': \"I had to make a difficult decision a few years ago when I was offered a job that would require me to move to a different city. I had to weigh the pros and cons of this job offer, and consider what it would mean for me to leave my friends and family in order to take it. In the end, I decided to turn down the job offer since I wasn't ready to quit my current job, move to a different city, and leave my loved ones behind.\",\n",
       "  'output_2': \"I had to make a difficult decision when I was offered a job in a city that was significantly farther away from my friends and family. I weighed the pros and cons and consulted with people I trusted, but in the end I had to make the choice that was best for my future and my well-being. It was a difficult decision, but I'm glad I made the choice that allowed me to further my career and start my journey towards independence.\",\n",
       "  'annotator': 'alpaca_eval_gpt4',\n",
       "  'preference': 2.0,\n",
       "  'price_per_example': 0.014759999999999999,\n",
       "  'time_per_example': 2.95092511177063,\n",
       "  'raw_completion': \"[\\n    {'model': 'model_2', 'rank': 1},\\n    {'model': 'model_1', 'rank': 2}\\n]\"}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotated"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da77b705-1845-4994-bd98-17ecfbff3d7f",
   "metadata": {},
   "source": [
    "By default there will only be one pair per instruction,input. If you use `is_unique_instructions=False` then you will get as many pairs as outputs.    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "515669f8-622c-47f9-9997-c0dedeb797c0",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Going further"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec356bc8-82cc-4ab0-aa05-b2f3bbce3408",
   "metadata": {},
   "source": [
    "### Adding noise"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0403b9a8-b444-4861-a837-68fc869dfde9",
   "metadata": {},
   "source": [
    "During training we typically flip the the label with probability 0.25 to emulate the variability of human annotations. To do so you can either:\n",
    "- initialize the annotator with `PairwiseAutoAnnotator(p_label_flip=0.25)`\n",
    "- set the noise of an initialized annotator `annotator.set_noise(p_label_flip=0.25)` \n",
    "- give the noise to annotate_samples `annotate_samples(..., p_label_flip=0.25)` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "68702d25-3a12-4de6-bef8-e21e79c8d113",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Filtered unique instruction/input pairs: 4 -> 1\n",
      "Annotation chunk:   0%|                                                       | 0/1 [00:00<?, ?it/s]INFO:root:Adding random noise to the labels p_label_flip=0.25.\n",
      "INFO:root:Annotating 0 examples with alpaca_eval_gpt4\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk: 100%|███████████████████████████████████████████████| 1/1 [00:01<00:00,  1.46s/it]\n"
     ]
    }
   ],
   "source": [
    "annotated = annotator.annotate_samples(outputs_samples, p_label_flip=0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1d663d65-5da5-40b5-a14b-f2f925649132",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'sample_id': 0,\n",
       "  'instruction': 'Describe a time when you had to make a difficult decision.',\n",
       "  'input': '',\n",
       "  'output_1': \"I had to make a difficult decision a few years ago when I was offered a job that would require me to move to a different city. I had to weigh the pros and cons of this job offer, and consider what it would mean for me to leave my friends and family in order to take it. In the end, I decided to turn down the job offer since I wasn't ready to quit my current job, move to a different city, and leave my loved ones behind.\",\n",
       "  'output_2': \"I had to make a difficult decision when I was offered a job in a city that was significantly farther away from my friends and family. I weighed the pros and cons and consulted with people I trusted, but in the end I had to make the choice that was best for my future and my well-being. It was a difficult decision, but I'm glad I made the choice that allowed me to further my career and start my journey towards independence.\",\n",
       "  'annotator': 'alpaca_eval_gpt4',\n",
       "  'preference': 2.0,\n",
       "  'price_per_example': 0.014759999999999999,\n",
       "  'time_per_example': 2.95092511177063,\n",
       "  'raw_completion': \"[\\n    {'model': 'model_2', 'rank': 1},\\n    {'model': 'model_1', 'rank': 2}\\n]\"}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotated"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5a596a8-0941-4e19-b19a-ee03dad7c1e1",
   "metadata": {},
   "source": [
    "### Cost & time efficiency: avoiding duplicate annotation\n",
    "Often time you will find yourself reannotating examples that you already annotated. This is particularly true if you sample many times from the same model (eg to get pairwise preferences for training or to evaluate best-of-n) or if the instructions you are considering require short outputs => many models might give the same exact answer.\n",
    "\n",
    "This means that you have to spend unecessary money and time. Thankfully `PairwiseAnnotator` stores previous annotations and will reuse those. For example, let us reannotate the previous evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "58d43c91-0ef4-417e-bbb2-ce585f6c35ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Annotation chunk:   0%|                                                       | 0/1 [00:00<?, ?it/s]INFO:root:Annotating 0 examples with alpaca_eval_gpt4\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk: 100%|███████████████████████████████████████████████| 1/1 [00:01<00:00,  1.45s/it]\n"
     ]
    }
   ],
   "source": [
    "annotated = annotator.annotate_head2head(outputs_1=outputs_baseline, outputs_2=outputs_rlhf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "cd5e9051-23fe-4294-9993-0c7afbc74495",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'instruction': 'If you could help me write an email to my friends inviting them to dinner on Friday, it would be greatly appreciated.',\n",
       "  'input': '',\n",
       "  'output_1': \"Dear Friends, \\r\\n\\r\\nI hope this message finds you well. I'm excited to invite you to dinner on Friday. We'll meet at 7:00 PM at [location]. I look forward to seeing you there. \\r\\n\\r\\nBest,\\r\\n[Name]\",\n",
       "  'output_2': 'Dear Friends, \\n\\nI am writing to invite you all to a dinner on Friday evening. It is a casual affair, and I am looking forward to a fun evening catching up with you all. I am planning to make a selection of delicious dishes, ranging from appetizers to mains and desserts. There will be something for everyone to enjoy, and I am sure it will be a night to remember.\\n\\nThe dinner will be held at my place on Friday, April 17th at 7pm. If you are interested in joining me, please RSVP to this email by Thursday, April 16th. I am looking forward to seeing you all there! \\n\\nThank you, \\n\\n[Name]',\n",
       "  'annotator': 'alpaca_eval_gpt4',\n",
       "  'preference': 2.0,\n",
       "  'price_per_example': 0.015269999999999999,\n",
       "  'time_per_example': 0.7309945424397787,\n",
       "  'raw_completion': \"[\\n    {'model': 'model_1', 'rank': 1},\\n    {'model': 'model_2', 'rank': 2}\\n]\"}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotated[-1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84836c18-9f26-4b1f-8660-7f2fa39817ee",
   "metadata": {},
   "source": [
    "We now get the annotations without actually having to reannotate any example.\n",
    "By default, the annotations are saved on disk in the directory that contains the annotators configs. You can remove this caching by giving `caching_path=None` to `PairwiseAutoAnnotator`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58bb89a3-7200-4f58-a7f5-9175f36d7cad",
   "metadata": {},
   "source": [
    "### Evaluating win rates "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78922f60-a9aa-4383-b18d-c95a93c6546c",
   "metadata": {},
   "source": [
    "Let's show how to get win rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "826421f1-e644-42c9-9375-d8b08fb958ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "from alpaca_eval.metrics import pairwise_to_winrate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "93ffaa6c-9570-4384-b157-d6f95b6d90ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# all the following is selfinstruct eval\n",
    "outputs_baseline = jload(\"examples/data/outputs_baseline.json\")\n",
    "outputs_rlhf = jload(\"examples/data/outputs_rlhf.json\")\n",
    "outputs_sft = jload(\"examples/data/outputs_sft.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2e000d42-6c37-40c6-81ac-eef3d34a4816",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Creating the annotator from `alpaca_eval_gpt4`.\n",
      "INFO:root:Saving annotations to `/Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json`.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n"
     ]
    }
   ],
   "source": [
    "annotator = PairwiseAutoAnnotator()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "74b79ba1-aeb3-4dd0-8669-a171ba4a2c37",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Annotation chunk:   0%|                                                       | 0/2 [00:00<?, ?it/s]INFO:root:Annotating 0 examples with alpaca_eval_gpt4\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk:  50%|███████████████████████▌                       | 1/2 [00:01<00:01,  1.51s/it]INFO:root:Annotating 0 examples with alpaca_eval_gpt4\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk: 100%|███████████████████████████████████████████████| 2/2 [00:03<00:00,  1.53s/it]\n",
      "INFO:root:drop 2 outputs that are not preferences\n",
      "INFO:root:drop 2 outputs that are not preferences\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'win_rate': 34.0,\n",
       " 'standard_error': 2.9343554261657316,\n",
       " 'n_wins': 80,\n",
       " 'n_wins_base': 160,\n",
       " 'n_draws': 10,\n",
       " 'n_total': 250,\n",
       " 'discrete_win_rate': 34.0}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotated_sft = annotator.annotate_head2head(outputs_1=outputs_baseline, outputs_2=outputs_sft)\n",
    "pairwise_to_winrate(preferences=[a[\"preference\"] for a in annotated_sft])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9c05eb80-0f24-4570-86ea-40da1a2a95a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Annotation chunk:   0%|                                                       | 0/2 [00:00<?, ?it/s]INFO:root:Annotating 0 examples with alpaca_eval_gpt4\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk:  50%|███████████████████████▌                       | 1/2 [00:01<00:01,  1.70s/it]INFO:root:Annotating 0 examples with alpaca_eval_gpt4\n",
      "INFO:root:Saving all annotations to /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "INFO:root:Loading all annotations from /Users/yanndubois/Desktop/GitHub/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4/annotations_seed0_configs.json.\n",
      "Annotation chunk: 100%|███████████████████████████████████████████████| 2/2 [00:03<00:00,  1.60s/it]\n",
      "INFO:root:drop 1 outputs that are not preferences\n",
      "INFO:root:drop 1 outputs that are not preferences\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'win_rate': 53.98406374501992,\n",
       " 'standard_error': 3.145897059163296,\n",
       " 'n_wins': 135,\n",
       " 'n_wins_base': 115,\n",
       " 'n_draws': 1,\n",
       " 'n_total': 251,\n",
       " 'discrete_win_rate': 53.98406374501992}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annotated_rlhf = annotator.annotate_head2head(outputs_1=outputs_baseline, outputs_2=outputs_rlhf)\n",
    "pairwise_to_winrate(preferences=[a[\"preference\"] for a in annotated_rlhf])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e703c1ea-8406-4599-8a43-d5a5de63459e",
   "metadata": {},
   "source": [
    "### Configuring annotators"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4927636c-96f5-4125-a553-a5be011c2a42",
   "metadata": {},
   "source": [
    "The most important argument to `PairwiseAnnotator` is `annotators_config` which defines the pool of annotators (the API provider, the model, the prompt, and the decoding parameters) . We provide many options out of the box.\n",
    "\n",
    "Here's the desciption of `annotators_config` from the docstring:\n",
    "```\n",
    "A dictionary or path to a yaml file containing the configuration for the pool of annotators. The keys in the first dictionary should be the annotator's name, and the value should be a dictionary of the annotator's configuration which should have the following keys:\n",
    "\n",
    "- prompt_templates (dict): a dictionary of prompt templates or path to the prompts. The keys should be \"without_inputs\" and \"with_inputs\". Each template should contain placeholders for keys in the example dictionary, typically {instruction} and {output_1} {output_2}.\n",
    "- fn_decoder (str): function in `alpaca_farm.auto_annotations.pairwise_annotators.decoders.py` for completions.\n",
    "- decoder_kwargs (dict): kwargs for fn_decode. E.g. model_name, max_tokens, temperature, tokens_to_avoid, tokens_to_favor\n",
    "- outputs_to_match (dict): Kwargs for fn_completion_parser. With the default regex parser it needs `outputs_to_match` which is a dictionary of outputs to match from the completions. The values should be a regex pattern that should be matched, the keys should be the corresponding preference value. For example {1: 'Output \\(a\\)'} will match the output \"Output (a)\" and set the preference to 1.\n",
    "- other kwargs to `SinglePairwiseAutoAnnotator` such as batch_size\n",
    "```\n",
    "\n",
    "And here is config `\"annotators/test/configs.yaml\"` of the annotator we used above. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0233e875-ae40-4b53-bc6b-05cf0201bbf0",
   "metadata": {},
   "source": [
    "For more information, please refer to [AlpacaEval](https://github.com/tatsu-lab/alpaca_eval/tree/main). In our code, we directly use AlpacaEval, with a minor modification to separate instructions with and without inputs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}