01b_Visualize_Reviews_Dataset_Glue_Spark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualize full Amazon Customer Reviews Dataset with Spark and AWS Glue Interactive Sessions\n",
    "\n",
    "In this notebook, we will query the full reviews data using AWS Glue interactive sessions with PySpark. With [AWS Glue](https://aws.amazon.com/glue/) interactive sessions, you can rapidly build, test, and run data preparation and analytics applications. Interactive sessions provides a programmatic interface for building and testing extract, transform, and load (ETL) scripts for data preparation. This is a powerful tool for data engineers and scientists who are doing interactive development on large datasets which require distributed computing because they do not need to manage infrastructure for Spark clusters.\n",
    "\n",
    "In the data science lifecycle, exploratory data analysis (EDA) is an important step for data scientists and engineers to gain an intuitive understanding of their datasets. This intuitive understanding then guides decisions which come further in the ML lifecycle like model training, deployment, and monitoring. In this notebook, we will use PySpark and Spark SQL to do EDA and associated visualizations of the Amazon reviews dataset.\n",
    "\n",
    "**Please Note: if you are running this workshop on your own AWS Account (not an AWS lead workshop), you will need to update your SageMaker execution role IAM permissions via the documentation [here](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-notebooks-glue.html).**\n",
    "\n",
    "![glue-interactive-sessions](./img/gis.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='1'></a>\n",
    "## Set up Kernel and Required Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "First, check that the correct kernel is chosen.\n",
    "\n",
    "<img src=\"img/gis_kernel_set_up.png\" width=\"300\"/>\n",
    "\n",
    "You can click on that to see and check the details of the image, kernel, and instance type.\n",
    "\n",
    "<img src=\"img/gis_kernel_and_instance_type.png\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Dataset Column Descriptions\n",
    "\n",
    "- `marketplace`: 2-letter country code (in this case all \"US\").\n",
    "- `customer_id`: Random identifier that can be used to aggregate reviews written by a single author.\n",
    "- `review_id`: A unique ID for the review.\n",
    "- `product_id`: The Amazon Standard Identification Number (ASIN).  `http://www.amazon.com/dp/<ASIN>` links to the product's detail page.\n",
    "- `product_parent`: The parent of that ASIN.  Multiple ASINs (color or format variations of the same product) can roll up into a single parent.\n",
    "- `product_title`: Title description of the product.\n",
    "- `product_category`: Broad product category that can be used to group reviews (in this case digital videos).\n",
    "- `star_rating`: The review's rating (1 to 5 stars).\n",
    "- `helpful_votes`: Number of helpful votes for the review.\n",
    "- `total_votes`: Number of total votes the review received.\n",
    "- `vine`: Was the review written as part of the [Vine](https://www.amazon.com/gp/vine/help) program?\n",
    "- `verified_purchase`: Was the review from a verified purchase?\n",
    "- `review_headline`: The title of the review itself.\n",
    "- `review_body`: The text of the review.\n",
    "- `review_date`: The date the review was written.\n",
    "- `year`: The year derived from the review date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "%stop_session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "%additional_python_modules seaborn,psutil\n",
    "%number_of_workers 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set Seaborn Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "sns.set_style = \"seaborn-whitegrid\"\n",
    "\n",
    "sns.set(\n",
    "    rc={\n",
    "        \"font.style\": \"normal\",\n",
    "        \"axes.facecolor\": \"white\",\n",
    "        \"grid.color\": \".8\",\n",
    "        \"grid.linestyle\": \"-\",\n",
    "        \"figure.facecolor\": \"white\",\n",
    "        \"figure.titlesize\": 20,\n",
    "        \"text.color\": \"black\",\n",
    "        \"xtick.color\": \"black\",\n",
    "        \"ytick.color\": \"black\",\n",
    "        \"axes.labelcolor\": \"black\",\n",
    "        \"axes.grid\": True,\n",
    "        \"axes.labelsize\": 10,\n",
    "        \"xtick.labelsize\": 10,\n",
    "        \"font.size\": 10,\n",
    "        \"ytick.labelsize\": 10,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helper Code to Display Values on Bars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "def show_values_barplot(axs, space):\n",
    "    def _show_on_plot(ax):\n",
    "        for p in ax.patches:\n",
    "            _x = p.get_x() + p.get_width() + float(space)\n",
    "            _y = p.get_y() + p.get_height()\n",
    "            value = round(float(p.get_width()), 2)\n",
    "            ax.text(_x, _y, value, ha=\"left\")\n",
    "\n",
    "    if isinstance(axs, np.ndarray):\n",
    "        for idx, ax in np.ndenumerate(axs):\n",
    "            _show_on_plot(ax)\n",
    "    else:\n",
    "        _show_on_plot(axs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Set Glue Database and Table names to query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "database_name = \"default\"\n",
    "table_name = \"amazon_reviews_parquet\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 1. Which Product Categories are Highest Rated by Average Rating?\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df = spark.sql(\"\"\"SELECT product_category, AVG(star_rating) AS avg_star_rating \n",
    "    FROM {}.{} \n",
    "    GROUP BY product_category \n",
    "    ORDER BY avg_star_rating DESC\"\"\".format(database_name, table_name)\n",
    ")\n",
    "df.show(50, truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# Store number of categories for later\n",
    "num_categories = df.toPandas().shape[0]\n",
    "\n",
    "# Store average star ratings for later\n",
    "average_star_ratings = df.toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "\n",
    "# Create plot\n",
    "barplot = sns.barplot(y=\"product_category\", x=\"avg_star_rating\", data=df.toPandas(), saturation=1)\n",
    "\n",
    "if num_categories < 10:\n",
    "    sns.set(rc={\"figure.figsize\": (10.0, 5.0)})\n",
    "\n",
    "# Set title and x-axis ticks\n",
    "plt.title(\"Average Rating by Product Category\")\n",
    "plt.xticks([1, 2, 3, 4, 5], [\"1-Star\", \"2-Star\", \"3-Star\", \"4-Star\", \"5-Star\"])\n",
    "\n",
    "# Helper code to show actual values afters bars\n",
    "show_values_barplot(barplot, 0.1)\n",
    "\n",
    "plt.xlabel(\"Average Rating\")\n",
    "plt.ylabel(\"Product Category\")\n",
    "\n",
    "# Export plot if needed\n",
    "plt.tight_layout()\n",
    "# plt.savefig('avg_ratings_per_category.png', dpi=300)\n",
    "\n",
    "# Show graphic\n",
    "plt.show()\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Which Product Categories Have the Most Reviews?\n",
    "## _This query takes a minute or two.  Please be patient._\n",
    "\n",
    "Note that in this cell, you are using PySpark syntax to process the data in your Glue table while the previous section was using Spark SQL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "from pyspark.sql import functions as f\n",
    "df_raw = spark.sql(\"SELECT * FROM {}.{}\".format(database_name, table_name))\n",
    "df = (\n",
    "    df_raw\n",
    "    .select(\"product_category\", \"star_rating\")\n",
    "    .groupBy(\"product_category\")\n",
    "    .agg(f.count(\"star_rating\").alias(\"count_star_rating\"))\n",
    "    .orderBy(\"count_star_rating\", ascending=False)\n",
    ")\n",
    "\n",
    "df.show(50, truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# Store counts for later\n",
    "count_ratings = df.toPandas()[\"count_star_rating\"]\n",
    "\n",
    "# Store max ratings for later\n",
    "max_ratings = df.toPandas()[\"count_star_rating\"].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "\n",
    "# Create Seaborn barplot\n",
    "barplot = sns.barplot(y=\"product_category\", x=\"count_star_rating\", data=df.toPandas(), saturation=1)\n",
    "\n",
    "if num_categories < 10:\n",
    "    sns.set(rc={\"figure.figsize\": (10.0, 5.0)})\n",
    "\n",
    "# Set title\n",
    "plt.title(\"Number of Ratings per Product Category for Subset of Product Categories\")\n",
    "\n",
    "# Set x-axis ticks to match scale\n",
    "if max_ratings > 200000:\n",
    "    plt.xticks([100000, 1000000, 5000000, 10000000, 15000000, 20000000], [\"100K\", \"1m\", \"5m\", \"10m\", \"15m\", \"20m\"])\n",
    "    plt.xlim(0, 20000000)\n",
    "elif max_ratings <= 200000:\n",
    "    plt.xticks([50000, 100000, 150000, 200000], [\"50K\", \"100K\", \"150K\", \"200K\"])\n",
    "    plt.xlim(0, 200000)\n",
    "\n",
    "plt.xlabel(\"Number of Ratings\")\n",
    "plt.ylabel(\"Product Category\")\n",
    "\n",
    "plt.tight_layout()\n",
    "\n",
    "# Export plot if needed\n",
    "# plt.savefig('ratings_per_category.png', dpi=300)\n",
    "\n",
    "# Show the barplot\n",
    "plt.show()\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. When did each product category become available in the Amazon catalog based on the date of the first review?\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT product_category, MIN(year) AS first_review_year\n",
    "    FROM {}.{}\n",
    "    WHERE year >= 1995\n",
    "    GROUP BY product_category\n",
    "    ORDER BY first_review_year \n",
    "\"\"\".format(database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show(50, truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "def get_x_y(df):\n",
    "    \"\"\" Get X and Y coordinates; return tuple \"\"\"\n",
    "    series = df[\"first_review_year\"].value_counts().sort_index()\n",
    "    # new_series = series.reindex(range(1,21)).fillna(0).astype(int)\n",
    "    return series.index, series.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "X, Y = get_x_y(df.toPandas())\n",
    "print(X)\n",
    "print(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "\n",
    "fig = plt.figure(figsize=(12, 5))\n",
    "ax = plt.gca()\n",
    "\n",
    "ax.set_title(\"Number Of First Product Category Reviews Per Year for Subset of Categories\")\n",
    "ax.set_xlabel(\"Year\")\n",
    "ax.set_ylabel(\"Count\")\n",
    "\n",
    "\n",
    "ax.plot(X, Y, color=\"black\", linewidth=2, marker=\"o\")\n",
    "ax.fill_between(X, [0] * len(X), Y, facecolor=\"lightblue\")\n",
    "\n",
    "ax.locator_params(integer=True)\n",
    "\n",
    "ax.set_xticks(range(1995, 2016, 1))\n",
    "ax.set_yticks(range(0, max(Y) + 2, 1))\n",
    "\n",
    "plt.xticks(rotation=45)\n",
    "\n",
    "# fig.savefig('first_reviews_per_year.png', dpi=300)\n",
    "plt.show()\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. What is the breakdown of ratings (1-5) per product category?\n",
    "## _This query takes a minute or two.  Please be patient._\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT product_category, star_rating, COUNT(*) AS count_reviews\n",
    "    FROM {}.{}\n",
    "    GROUP BY  product_category, star_rating\n",
    "    ORDER BY  product_category ASC, star_rating DESC, count_reviews\n",
    "\"\"\".format(database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show(50, truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare for Stacked Percentage Horizontal Bar Plot Showing Proportion of Star Ratings per Product Category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# Create grouped DataFrames by category and by star rating\n",
    "grouped_category = df.toPandas().groupby(\"product_category\")\n",
    "grouped_star = df.toPandas().groupby(\"star_rating\")\n",
    "\n",
    "# Create sum of ratings per star rating\n",
    "df_sum = df.toPandas().groupby([\"star_rating\"]).sum()\n",
    "\n",
    "# Calculate total number of star ratings\n",
    "total = df_sum[\"count_reviews\"].sum()\n",
    "print(total)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# Create dictionary of product categories and array of star rating distribution per category\n",
    "distribution = {}\n",
    "count_reviews_per_star = []\n",
    "i = 0\n",
    "\n",
    "for category, ratings in grouped_category:\n",
    "    count_reviews_per_star = []\n",
    "    for star in ratings[\"star_rating\"]:\n",
    "        count_reviews_per_star.append(ratings.at[i, \"count_reviews\"])\n",
    "        i = i + 1\n",
    "    distribution[category] = count_reviews_per_star\n",
    "\n",
    "# Check if distribution has been created succesfully\n",
    "print(distribution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# Sort distribution by average rating per category\n",
    "sorted_distribution = {}\n",
    "\n",
    "average_star_ratings.iloc[:, 0]\n",
    "for index, value in average_star_ratings.iloc[:, 0].items():\n",
    "    sorted_distribution[value] = distribution[value]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_sorted_distribution_pct = pd.DataFrame(sorted_distribution).transpose().apply(\n",
    "    lambda num_ratings: num_ratings/sum(num_ratings)*100, axis=1\n",
    ")\n",
    "df_sorted_distribution_pct.columns=['5', '4', '3', '2', '1']\n",
    "df_sorted_distribution_pct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "\n",
    "categories = df_sorted_distribution_pct.index\n",
    "\n",
    "# Plot bars\n",
    "if len(categories) > 10:\n",
    "    plt.figure(figsize=(10,10))\n",
    "else: \n",
    "    plt.figure(figsize=(10,5))\n",
    "\n",
    "df_sorted_distribution_pct.plot(kind=\"barh\", \n",
    "                                stacked=True, \n",
    "                                edgecolor='white',\n",
    "                                width=1.0,\n",
    "                                color=['green', \n",
    "                                       'orange', \n",
    "                                       'blue', \n",
    "                                       'purple', \n",
    "                                       'red'])\n",
    "\n",
    "plt.title(\"Distribution of Reviews Per Rating Per Category\", \n",
    "          fontsize='16')\n",
    "\n",
    "plt.legend(bbox_to_anchor=(1.04,1), \n",
    "           loc=\"upper left\",\n",
    "           labels=['5-Star Ratings', \n",
    "                   '4-Star Ratings', \n",
    "                   '3-Star Ratings', \n",
    "                   '2-Star Ratings', \n",
    "                   '1-Star Ratings'])\n",
    "\n",
    "plt.xlabel(\"% Breakdown of Star Ratings\", fontsize='14')\n",
    "plt.gca().invert_yaxis()\n",
    "plt.tight_layout()\n",
    "\n",
    "plt.show()\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. How Many Reviews per Star Rating? (5, 4, 3, 2, 1)\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT star_rating, COUNT(*) AS count_reviews\n",
    "    FROM {}.{}\n",
    "    GROUP BY star_rating\n",
    "    ORDER BY star_rating DESC, count_reviews \n",
    "\"\"\".format(database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "\n",
    "chart = df.toPandas().plot.bar(\n",
    "    x=\"star_rating\", y=\"count_reviews\", rot=\"0\", figsize=(10, 5), title=\"Review Count by Star Ratings\", legend=False\n",
    ")\n",
    "\n",
    "plt.xlabel(\"Star Rating\")\n",
    "plt.ylabel(\"Review Count\")\n",
    "\n",
    "plt.show()\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. How Did Star Ratings Change Over Time?\n",
    "Is there a drop-off point for certain product categories throughout the year?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Average Star Rating Across All Product Categories\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT year, ROUND(AVG(star_rating),4) AS avg_rating\n",
    "    FROM {}.{}\n",
    "    WHERE year >= 1995\n",
    "    GROUP BY year\n",
    "    ORDER BY year\n",
    "\"\"\".format(database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show(50, truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df.toPandas()[\"year\"] = pd.to_datetime(df.toPandas()[\"year\"], format=\"%Y\").dt.year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "\n",
    "fig = plt.gcf()\n",
    "fig.set_size_inches(12, 5)\n",
    "\n",
    "fig.suptitle(\"Average Star Rating Over Time (Across Subset of Product Categories)\")\n",
    "\n",
    "ax = plt.gca()\n",
    "# ax = plt.gca().set_xticks(df['year'])\n",
    "ax.locator_params(integer=True)\n",
    "ax.set_xticks(df.toPandas()[\"year\"].unique())\n",
    "\n",
    "df.toPandas().plot(kind=\"line\", x=\"year\", y=\"avg_rating\", color=\"red\", ax=ax)\n",
    "\n",
    "# plt.xticks(range(1995, 2016, 1))\n",
    "# plt.yticks(range(0,6,1))\n",
    "plt.xlabel(\"Years\")\n",
    "plt.ylabel(\"Average Star Rating\")\n",
    "plt.xticks(rotation=45)\n",
    "\n",
    "# fig.savefig('average-rating.png', dpi=300)\n",
    "plt.show()\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Average Star Rating Per Product Categories Across Time\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT product_category, year, ROUND(AVG(star_rating), 4) AS avg_rating_category\n",
    "    FROM {}.{}\n",
    "    WHERE year >= 1995\n",
    "    GROUP BY product_category, year\n",
    "    ORDER BY year \n",
    "\"\"\".format(database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show(50, truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "def plot_categories(df):\n",
    "    df_categories = df[\"product_category\"].unique()\n",
    "    for category in df_categories:\n",
    "        # print(category)\n",
    "        df_plot = df.loc[df[\"product_category\"] == category]\n",
    "        df_plot.plot(\n",
    "            kind=\"line\",\n",
    "            x=\"year\",\n",
    "            y=\"avg_rating_category\",\n",
    "            c=np.random.rand(\n",
    "                3,\n",
    "            ),\n",
    "            ax=ax,\n",
    "            label=category,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "\n",
    "fig = plt.gcf()\n",
    "fig.set_size_inches(12, 5)\n",
    "\n",
    "fig.suptitle(\"Average Star Rating Over Time Across Subset Of Categories\")\n",
    "\n",
    "ax = plt.gca()\n",
    "\n",
    "ax.locator_params(integer=True)\n",
    "ax.set_xticks(df.toPandas()[\"year\"].unique())\n",
    "\n",
    "plot_categories(df.toPandas())\n",
    "\n",
    "plt.xlabel(\"Year\")\n",
    "plt.ylabel(\"Average Star Rating\")\n",
    "plt.legend(bbox_to_anchor=(0, -0.15, 1, 0), loc=2, ncol=2, mode=\"expand\", borderaxespad=0)\n",
    "\n",
    "# fig.savefig('average_rating_category_all_data.png', dpi=300)\n",
    "plt.show()\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Which Star Ratings (1-5) are Most Helpful?\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT star_rating, AVG(helpful_votes) AS avg_helpful_votes\n",
    "    FROM {}.{}\n",
    "    GROUP BY  star_rating\n",
    "    ORDER BY  star_rating ASC\n",
    "\"\"\".format(database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "plt.clf()\n",
    "\n",
    "chart = df.toPandas().plot.bar(\n",
    "    x=\"star_rating\", y=\"avg_helpful_votes\", rot=\"0\", figsize=(10, 5), title=\"Helpfulness Of Star Ratings\", legend=False\n",
    ")\n",
    "\n",
    "plt.xlabel(\"Star Rating\")\n",
    "plt.ylabel(\"Average Helpful Votes\")\n",
    "\n",
    "plt.show()\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Which Products have Most Helpful Reviews?  How Long are the Most Helpful Reviews?\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT product_title, helpful_votes, star_rating,\n",
    "           LENGTH(review_body) AS review_body_length,\n",
    "           SUBSTR(review_body, 1, 100) AS review_body_substr\n",
    "    FROM {}.{}\n",
    "    ORDER BY helpful_votes DESC LIMIT 10 \n",
    "\"\"\".format(database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show(50, truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9. What is the Ratio of Positive (5, 4) to Negative (3, 2 ,1) Reviews?\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT (CAST(positive_review_count AS DOUBLE) / CAST(negative_review_count AS DOUBLE)) AS positive_to_negative_sentiment_ratio\n",
    "    FROM (\n",
    "      SELECT count(*) AS positive_review_count\n",
    "      FROM {}.{}\n",
    "      WHERE star_rating >= 4\n",
    "\n",
    "    ), (\n",
    "      SELECT count(*) AS negative_review_count\n",
    "      FROM {}.{}\n",
    "      WHERE star_rating < 4 \n",
    "    )\n",
    "\"\"\".format(database_name, table_name, database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10. Which Customers are Abusing the Review System by Repeatedly Reviewing the Same Product?  What Was Their Average Star Rating for Each Product?\n",
    "## _This query takes a minute or two.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# SQL statement\n",
    "df = spark.sql(\"\"\"\n",
    "    SELECT customer_id, product_category, product_title, \n",
    "    ROUND(AVG(star_rating),4) AS avg_star_rating, COUNT(*) AS review_count \n",
    "    FROM {}.{} \n",
    "    GROUP BY customer_id, product_category, product_title \n",
    "    HAVING COUNT(*) > 1 \n",
    "    ORDER BY review_count DESC\n",
    "    LIMIT 50\n",
    "\"\"\".format(database_name, table_name)\n",
    ")\n",
    "\n",
    "df.show(50, truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "%stop_session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "source": [
    "# Conclusions\n",
    "\n",
    "Through this notebook, we have seen how you can use AWS Glue interactive sessions in SageMaker Studio to process large datasets at scale for ETL and EDA tasks. Benefits of this development paradigm include...\n",
    "\n",
    "1. natively distributed processing which can scale with your datasets \n",
    "2. no need to manage cluster infrastructure\n",
    "3. integration with SageMaker studio for a one stop shop for the ML lifecycle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Release Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "\n",
    "<p><b>Shutting down your kernel for this notebook to release resources.</b></p>\n",
    "<button class=\"sm-command-button\" data-commandlinker-command=\"kernelmenu:shutdown\" style=\"display:none;\">Shutdown Kernel</button>\n",
    "        \n",
    "<script>\n",
    "try {\n",
    "    els = document.getElementsByClassName(\"sm-command-button\");\n",
    "    els[0].click();\n",
    "}\n",
    "catch(err) {\n",
    "    // NoOp\n",
    "}    \n",
    "</script>"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.m5.2xlarge",
  "kernelspec": {
   "display_name": "Glue Python [PySpark and Ray] (SparkAnalytics 1.0)",
   "language": "python",
   "name": "conda-env-sm_glue_is-glue_pyspark__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-sparkanalytics-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "python",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "Python_Glue_Session",
   "pygments_lexer": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}