diff --git a/examples/README.md b/examples/README.md index 62b22dea5..c55f22565 100644 --- a/examples/README.md +++ b/examples/README.md @@ -24,5 +24,9 @@ Folders * [Prompting examples](https://github.com/google-gemini/cookbook/tree/main/examples/prompting): A directory with examples of various prompting techniques. * [JSON Capabilities](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Tuning.ipynb): A directory with guides containing different types of tasks you can do with JSON schemas. * [Automate Google Workspace tasks with the Gemini API](https://github.com/google-gemini/cookbook/tree/main/examples/Apps_script_and_Workspace_codelab): This codelabs shows you how to connect to the Gemini API using Apps Script, and uses the function calling, vision and text capabilities to automate Google Workspace tasks - summarizing a document, analyzing a chart, sending an email and generating some slides directly. All of this is done from a free text input. +* [ChromaDB](https://github.com/google-gemini/cookbook/tree/main/examples/chromadb): A directory with guides with to using Gemini with ChromaDB integrations. +* [LangChain](https://github.com/google-gemini/cookbook/tree/main/examples/langchain): A directory with guides with to using Gemini with LangChain integrations. +* [LlamaIndex](https://github.com/google-gemini/cookbook/tree/main/examples/llamaindex): A directory with guides with to using Gemini with LlamaIndex integrations. +* [QDrant](https://github.com/google-gemini/cookbook/tree/main/examples/qdrant): A directory with guides with to using Gemini with QDrant integrations. There are even more examples in the [quickstarts](https://github.com/google-gemini/cookbook/tree/main/quickstarts) folder and in the [Awesome Gemini page](../Awesome_gemini.md). diff --git a/examples/llamaindex/Evaluate_RAG_with_LlamaIndex.ipynb b/examples/llamaindex/Evaluate_RAG_with_LlamaIndex.ipynb new file mode 100644 index 000000000..908b2bf69 --- /dev/null +++ b/examples/llamaindex/Evaluate_RAG_with_LlamaIndex.ipynb @@ -0,0 +1,550 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "9NdTPlcnHXXe" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "cellView": "form", + "id": "ryJhxbEjHh0o" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6xrMnk6fHLec" + }, + "source": [ + "# Gemini API: RAG Evaluation with LlamaIndex\n", + "This notebook demonstrates how to evaluate Retrieval-Augmented Generation (RAG) with LlamaIndex. You will learn how to:\n", + "\n", + "1. Read the data\n", + "2. Create a vector store with Gemini embeddings\n", + "3. Run a query engine with Gemini models\n", + "4. Evaluate results using LlamaIndex components" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FzoekDciM_pM" + }, + "source": [ + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "id": "BFGzsGq46wYu" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.5/15.5 MB\u001b[0m \u001b[31m42.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m40.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m169.4/169.4 kB\u001b[0m \u001b[31m9.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m29.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m361.3/361.3 kB\u001b[0m \u001b[31m18.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m295.8/295.8 kB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m34.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m318.9/318.9 kB\u001b[0m \u001b[31m13.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.2/49.2 kB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.5/4.5 MB\u001b[0m \u001b[31m12.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m150.7/150.7 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m679.1/679.1 kB\u001b[0m \u001b[31m27.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h" + ] + } + ], + "source": [ + "!pip install -q llama-index\n", + "!pip install -q llama-index-multi-modal-llms-gemini\n", + "!pip install -q llama-index-embeddings-gemini" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "e3CLKz257sIF" + }, + "outputs": [], + "source": [ + "# Import core components\n", + "from llama_index.core import VectorStoreIndex, StorageContext, Settings\n", + "from llama_index.core import SimpleDirectoryReader, SimpleKeywordTableIndex\n", + "\n", + "# Import components for parsing and evaluating nodes\n", + "from llama_index.core.node_parser import SentenceSplitter\n", + "from llama_index.core.evaluation import RelevancyEvaluator\n", + "\n", + "# Import embeddings and gemini models from llama_index\n", + "from llama_index.embeddings.gemini import GeminiEmbedding\n", + "from llama_index.llms.gemini import Gemini\n", + "\n", + "# Import standard libraries\n", + "import os\n", + "import pandas as pd\n", + "from IPython.display import Markdown\n", + "\n", + "# Apply nest_asyncio to enable nesting of async functions in an already running async loop\n", + "import nest_asyncio\n", + "nest_asyncio.apply()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Vi2nxDWz3zEy" + }, + "source": [ + "## Configure your API key\n", + "\n", + "To run the following cell, your API key must be stored in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) for an example.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "UJRVFAMa29aQ" + }, + "outputs": [], + "source": [ + "from google.colab import userdata\n", + "GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n", + "\n", + "os.environ[\"GOOGLE_API_KEY\"] = GOOGLE_API_KEY" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "s5K8Hd3_9xKj" + }, + "source": [ + "# Define Model and Embeddings" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "id": "bjUh1vEX39A-" + }, + "outputs": [], + "source": [ + "llm = Gemini(model=\"models/gemini-1.5-flash-latest\")" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "IAmZ-at8vEYf" + }, + "outputs": [], + "source": [ + "Settings.embed_model = GeminiEmbedding(\n", + " model_name=\"models/text-embedding-004\", api_key=GOOGLE_API_KEY\n", + ")\n", + "Settings.llm = llm" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "q_bqu_sm38Xh" + }, + "source": [ + "# Data preparation\n", + "Responses to queries about the USA Constitution will be evaluated. Start by downloading and reading the data." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "d1okB8Hp37pK" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " % Total % Received % Xferd Average Speed Time Time Time Current\n", + " Dload Upload Total Spent Left Speed\n", + "100 48557 100 48557 0 0 131k 0 --:--:-- --:--:-- --:--:-- 132k\n" + ] + } + ], + "source": [ + "!mkdir -p 'data/constitution/'\n", + "!curl 'https://www.gutenberg.org/cache/epub/5/pg5.txt' -o 'data/constitution/usa_constitution.txt'" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "w51UTjhj4BaT" + }, + "outputs": [], + "source": [ + "documents = SimpleDirectoryReader('./data/constitution/').load_data()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WWbWumAiHkEC" + }, + "source": [ + "Break documents into nodes and store them." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "id": "YOCl-wQP8wNy" + }, + "outputs": [], + "source": [ + "node_parser = SentenceSplitter.from_defaults(chunk_size=200, chunk_overlap=75)\n", + "nodes = node_parser.get_nodes_from_documents(documents)" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "id": "aTiVQsgU8mMA" + }, + "outputs": [], + "source": [ + "storage_context = StorageContext.from_defaults()\n", + "storage_context.docstore.add_documents(nodes)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hS1xgKh3I16d" + }, + "source": [ + "Create vector store and define query engine." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "id": "n7oZV6Yx6XUO" + }, + "outputs": [], + "source": [ + "vector_index = VectorStoreIndex(nodes, storage_context=storage_context)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "id": "clFkGAukfjh0" + }, + "outputs": [], + "source": [ + "query_engine = vector_index.as_query_engine()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XccH8Icj3x4U" + }, + "source": [ + "## RAG Evaluation\n", + "\n", + "Set up the evaluation pipeline." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "id": "xpv2YEvU4NBg" + }, + "outputs": [], + "source": [ + "eval_questions = [\n", + " 'In what situations can impeachment occur?',\n", + " 'From what age can someone become a Representative?']" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "id": "lP3s7X-27Y1B" + }, + "outputs": [], + "source": [ + "evaluator = RelevancyEvaluator(llm=llm)" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "id": "GxQQ4HiJAvRW" + }, + "outputs": [], + "source": [ + "# define jupyter display function\n", + "def display_eval_df(query: str, response: str, eval_result: str) -> None:\n", + " eval_df = pd.DataFrame(\n", + " {\n", + " \"Query\": [query],\n", + " \"Response\": [response.response],\n", + " \"Source\": \"\\n\".join(source_node.node.get_content() for source_node in response.source_nodes),\n", + " \"Evaluation Result\": [eval_result.feedback],\n", + " },\n", + " index=[0],\n", + " )\n", + " eval_df = eval_df.style.set_properties(\n", + " **{\n", + " \"inline-size\": \"600px\",\n", + " \"overflow-wrap\": \"break-word\",\n", + " \"vertical-align\": \"top\"\n", + " }\n", + " )\n", + " display(eval_df)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mz2HC2HTGffi" + }, + "source": [ + "Test your evaluation pipeline with `eval_questions`. You will see a query, the response generated by the LLM, the source from which the LLM derived its response, and an evaluation decision indicating the accuracy and appropriateness of the response (YES/NO)." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "id": "hBhsqaXYJiJY" + }, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
 QueryResponseSourceEvaluation Result
0In what situations can impeachment occur?Impeachment can occur for treason, bribery, or other high crimes and misdemeanors. \n", + "The Senate shall choose their other Officers, and also a President\r\n", + "pro tempore, in the Absence of the Vice-President, or when he shall\r\n", + "exercise the Office of President of the United States.\r\n", + "\r\n", + "The Senate shall have the sole Power to try all Impeachments.\r\n", + "When sitting for that Purpose, they shall be on Oath or Affirmation.\r\n", + "When the President of the United States is tried, the Chief Justice\r\n", + "shall preside: And no Person shall be convicted without the Concurrence\r\n", + "of two thirds of the Members present.\r\n", + "\r\n", + "Judgment in cases of Impeachment shall not extend further than to removal\r\n", + "from Office, and disqualification to hold and enjoy any Office of honor,\r\n", + "Trust or Profit under the United States: but the Party convicted shall\r\n", + "nevertheless be liable and subject to Indictment, Trial, Judgment and\r\n", + "Punishment, according to Law.\r\n", + "\r\n", + "Section 4.\n", + "Section 4. The President, Vice President and all civil Officers of the\r\n", + "United States, shall be removed from Office on Impeachment for,\r\n", + "and Conviction of, Treason, Bribery, or other high Crimes and Misdemeanors.\r\n", + "\r\n", + "ARTICLE THREE\r\n", + "\r\n", + "Section 1. The judicial Power of the United States, shall be vested\r\n", + "in one supreme Court, and in such inferior Courts as the Congress may\r\n", + "from time to time ordain and establish. The Judges, both of the supreme\r\n", + "and inferior Courts, shall hold their Offices during good behavior,\r\n", + "and shall, at stated Times, receive for their Services, a Compensation,\r\n", + "which shall not be diminished during their Continuance in Office.\r\n", + "\r\n", + "Section 2.YES\n", + "
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
 QueryResponseSourceEvaluation Result
0From what age can someone become a Representative?Someone must be at least 25 years old to become a Representative. \n", + "No Person shall be a Representative who shall not have attained to the\r\n", + "Age of twenty five Years, and been seven Years a citizen of the United States,\r\n", + "and who shall not, when elected, be an Inhabitant of that State in which\r\n", + "he shall be chosen.\r\n", + "\r\n", + "Representatives and direct Taxes shall be apportioned among\r\n", + "the several States which may be included within this Union,\r\n", + "according to their respective Numbers, which shall be determined\r\n", + "by adding to the whole number of free Persons, including those\r\n", + "bound to Service for a Term of Years, and excluding Indians not taxed,\r\n", + "three fifths of all other Persons. The actual Enumeration shall be made\r\n", + "within three Years after the first Meeting of the Congress of the\r\n", + "United States, and within every subsequent Term of ten Years,\r\n", + "in such Manner as they shall by law Direct.\n", + "Article 1\r\n", + "\r\n", + "Section 1. All legislative Powers herein granted shall be vested in a\r\n", + "Congress of the United States, which shall consist of a Senate and\r\n", + "House of Representatives.\r\n", + "\r\n", + "Section 2. The House of Representatives shall be composed of Members\r\n", + "chosen every second Year by the People of the several States,\r\n", + "and the electors in each State shall have the qualifications requisite\r\n", + "for electors of the most numerous branch of the State legislature.\r\n", + "\r\n", + "No Person shall be a Representative who shall not have attained to the\r\n", + "Age of twenty five Years, and been seven Years a citizen of the United States,\r\n", + "and who shall not, when elected, be an Inhabitant of that State in which\r\n", + "he shall be chosen.YES \n", + "
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "for question in eval_questions:\n", + " # Query the engine for the current question\n", + " llm_response = query_engine.query(question)\n", + "\n", + " # Evaluate the response for the current question\n", + " eval_result = evaluator.evaluate_response(query=question, response=llm_response)\n", + "\n", + " # Display the results for the current question\n", + " display_eval_df(question, llm_response, eval_result)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7ifaTQz0HhN9" + }, + "source": [ + "## Summary\n", + "This notebook shows how to integrate the Gemini API with LlamaIndex to create a Retrieval-Augmented Generation (RAG) pipeline with built-in evaluation.\n", + "\n", + "For further exploration, please check LlamaIndex [documentation](https://docs.llamaindex.ai/en/latest/) to see more functions that you might want to try with Gemini API." + ] + } + ], + "metadata": { + "colab": { + "name": "Evaluate_RAG_with_LlamaIndex.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}