Learning_Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lxml.html #Scraping\n",
    "import requests #Get HTTP requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's scrape Steam, and open the html page using the request library\n",
    "\n",
    "Now let’s open up the web page using requests and pass that response to lxml.html.fromstring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "html = requests.get('https://store.steampowered.com/explore/new/')\n",
    "\n",
    "doc = lxml.html.fromstring(html.content) #Provides an object of htmlElement type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_releases = doc.xpath('//div[@id=\"tab_newreleases_content\"]')[0]\n",
    "\n",
    "#returns all the divs in the HTML page which have an id of tab_newrelases_content\n",
    "#//div[@id=\"tab_newreleases_cotent\"]\n",
    "#Take first element since only one div on the page has the id."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "// these double forward slashes tell lxml that we want to search for all tags in the HTML document which match our requirements/filters. Another option was to use / (a single forward slash). The single forward slash returns only the immediate child tags/nodes which match our requirements/filters\n",
    "\n",
    "div tells lxml that we are searching for divs in the HTML page\n",
    "\n",
    "\n",
    "[@id=\"tab_newreleases_content\"] tells lxml that we are only interested in those divs which have an id of tab_newreleases_content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Extract the titles and prices**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Onimusha: Warlords / 鬼武者',\n",
       " 'Trickster VR: Co-op Dungeon Crawler',\n",
       " 'My Time At Portia',\n",
       " 'RESIDENT EVIL 2 / BIOHAZARD RE:2 \"1-Shot Demo\"',\n",
       " 'Love Language Japanese',\n",
       " 'What Never Was',\n",
       " 'Tales of Vesperia: Definitive Edition',\n",
       " 'Catherine Classic',\n",
       " 'Medieval Kingdom Wars',\n",
       " 'FrostRunner',\n",
       " 'ATOM RPG: Post-apocalyptic indie game',\n",
       " 'Garden Paws',\n",
       " 'GRIS',\n",
       " 'Insurgency: Sandstorm',\n",
       " \"Tom Clancy's Rainbow Six® Siege - Year 4 Pass\",\n",
       " 'Katamari Damacy REROLL',\n",
       " 'Kenshi',\n",
       " 'Mutant Year Zero: Road to Eden',\n",
       " 'Parkitect',\n",
       " 'Equilinox',\n",
       " 'Farming Simulator 19',\n",
       " 'Wallpaper Engine',\n",
       " 'Warhammer 40,000: Mechanicus',\n",
       " 'HITMAN™ 2',\n",
       " 'Thronebreaker: The Witcher Tales',\n",
       " 'Thief Simulator',\n",
       " 'Supply Chain Idle',\n",
       " 'One Hour One Life',\n",
       " 'Football Manager 2019',\n",
       " 'Return of the Obra Dinn',\n",
       " 'RimWorld',\n",
       " 'The Jackbox Party Pack 5',\n",
       " \"Assassin's Creed® Odyssey\",\n",
       " 'Creed: Rise to Glory™',\n",
       " 'CrossCode',\n",
       " 'Pummel Party']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles = new_releases.xpath('.//div[@class=\"tab_item_name\"]/text()')\n",
    "titles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". tells lxml that we are only interested in the tags which are the children of the new_releases tag\n",
    "\n",
    "[@class=\"tab_item_name\"] is pretty similar to how we were filtering divs based on id. The only difference is that here we are filtering based on the class name\n",
    "\n",
    "/text() tells lxml that we want the text contained within the tag we just extracted. In this case, it returns the title contained in the div with the tab_item_name class nam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['$19.99',\n",
       " '$11.04',\n",
       " '$29.99',\n",
       " 'Free',\n",
       " '$15.99',\n",
       " 'Free',\n",
       " '$49.99',\n",
       " '$19.99',\n",
       " '$15.99',\n",
       " 'Free To Play',\n",
       " '$14.99',\n",
       " '$19.99',\n",
       " '$16.99',\n",
       " '$29.99',\n",
       " '$29.99',\n",
       " '$29.99',\n",
       " '$29.99',\n",
       " '$34.99',\n",
       " '$29.99',\n",
       " '$9.99',\n",
       " '$34.99',\n",
       " '$3.99',\n",
       " '$29.99',\n",
       " '$59.99',\n",
       " '$29.99',\n",
       " '$14.99',\n",
       " 'Free To Play',\n",
       " '$19.99',\n",
       " '$49.99',\n",
       " '$19.99',\n",
       " '$34.99',\n",
       " '$29.99',\n",
       " '$59.99',\n",
       " '$29.99',\n",
       " '$19.99',\n",
       " '$9.89']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prices = new_releases.xpath('.//div[@class=\"discount_final_price\"]/text()')\n",
    "prices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "tags = new_releases.xpath('.//div[@class=\"tab_item_top_tags\"]')\n",
    "total_tags = []\n",
    "for tag in tags:\n",
    "    total_tags.append(tag.text_content()) #text_content returns the text in HTML tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Action', ' Violent', ' Hack and Slash', ' Classic'],\n",
       " ['Adventure', ' Co-op', ' Action', ' VR'],\n",
       " ['Open World', ' Crafting', ' Simulation', ' RPG'],\n",
       " ['Gore', ' Action', ' Violent', ' Zombies'],\n",
       " ['Nudity', ' Sexual Content', ' Casual', ' Indie'],\n",
       " ['Free to Play', ' Adventure', ' Indie', ' Puzzle'],\n",
       " ['RPG', ' Anime', ' JRPG', ' Action RPG'],\n",
       " ['Sexual Content', ' Anime', ' Puzzle', ' Nudity'],\n",
       " ['Strategy', ' Medieval', ' RTS', ' Simulation'],\n",
       " ['Free to Play', ' Indie', ' Action', ' Parkour'],\n",
       " ['RPG', ' Post-apocalyptic', ' Turn-Based Combat', ' Isometric'],\n",
       " ['Adventure', ' Simulation', ' Indie', ' RPG'],\n",
       " ['Atmospheric', ' Indie', ' Great Soundtrack', ' Adventure'],\n",
       " ['FPS', ' Realistic', ' Shooter', ' Multiplayer'],\n",
       " ['Action', ' FPS', ' Multiplayer', ' Tactical'],\n",
       " ['Casual', ' Action', ' Great Soundtrack', ' Puzzle'],\n",
       " ['Open World', ' Sandbox', ' RPG', ' Survival'],\n",
       " ['Strategy', ' RPG', ' Adventure', ' Turn-Based Combat'],\n",
       " ['Management', ' Simulation', ' Building', ' Sandbox'],\n",
       " ['Simulation', ' Indie', ' Casual', ' Sandbox'],\n",
       " ['Simulation', ' Multiplayer', ' Realistic', ' Agriculture'],\n",
       " ['Utilities', ' Software', ' Design & Illustration', ' Animation & Modeling'],\n",
       " ['Strategy', ' Warhammer 40K', ' Turn-Based Tactics', ' Turn-Based'],\n",
       " ['Stealth', ' Action', ' Singleplayer', ' Multiplayer'],\n",
       " ['Adventure', ' RPG', ' Card Game', ' Story Rich'],\n",
       " ['Simulation', ' Stealth', ' Adventure', ' Crime'],\n",
       " ['Free to Play', ' Casual'],\n",
       " ['Massively Multiplayer', ' Indie', ' Survival', ' Simulation'],\n",
       " ['Simulation', ' Sports', ' Management', ' Soccer'],\n",
       " ['Mystery', ' Detective', ' Singleplayer', ' Atmospheric'],\n",
       " ['Base Building', ' Survival', ' Strategy', ' Sandbox'],\n",
       " ['Casual', ' Local Multiplayer', ' Multiplayer', ' Comedy'],\n",
       " ['Open World', ' Action', ' RPG', ' Singleplayer'],\n",
       " ['Sports', ' Action', ' Simulation', ' Indie'],\n",
       " ['Action RPG', ' Pixel Graphics', ' RPG', ' Female Protagonist'],\n",
       " ['Multiplayer', ' Casual', ' Indie', ' Board Game']]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags = [tag.split(\",\") for tag in total_tags]\n",
    "tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Extract the Platform: How to scrape spans**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "platforms_div = new_releases.xpath('.//div[@class=\"tab_item_details\"]')\n",
    "total_platforms = []\n",
    "\n",
    "for game in platforms_div:\n",
    "    temp = game.xpath('.//span[contains(@class, \"platform_img\")]')\n",
    "    platforms = [t.get('class').split(' ')[-1] for t in temp]\n",
    "    if 'hmd_separator' in platforms:\n",
    "        platforms.remove('hmd_separator')\n",
    "    total_platforms.append(platforms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = []\n",
    "for info in zip(titles,prices, tags, total_platforms):\n",
    "    resp = {}\n",
    "    resp['title'] = info[0]\n",
    "    resp['price'] = info[1]\n",
    "    resp['tags'] = info[2]\n",
    "    resp['platforms'] = info[3]\n",
    "    output.append(resp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'title': 'Onimusha: Warlords / 鬼武者',\n",
       "  'price': '$19.99',\n",
       "  'tags': ['Action', ' Violent', ' Hack and Slash', ' Classic'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Trickster VR: Co-op Dungeon Crawler',\n",
       "  'price': '$11.04',\n",
       "  'tags': ['Adventure', ' Co-op', ' Action', ' VR'],\n",
       "  'platforms': ['win', 'htcvive', 'oculusrift', 'windowsmr']},\n",
       " {'title': 'My Time At Portia',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Open World', ' Crafting', ' Simulation', ' RPG'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'RESIDENT EVIL 2 / BIOHAZARD RE:2 \"1-Shot Demo\"',\n",
       "  'price': 'Free',\n",
       "  'tags': ['Gore', ' Action', ' Violent', ' Zombies'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Love Language Japanese',\n",
       "  'price': '$15.99',\n",
       "  'tags': ['Nudity', ' Sexual Content', ' Casual', ' Indie'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'What Never Was',\n",
       "  'price': 'Free',\n",
       "  'tags': ['Free to Play', ' Adventure', ' Indie', ' Puzzle'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Tales of Vesperia: Definitive Edition',\n",
       "  'price': '$49.99',\n",
       "  'tags': ['RPG', ' Anime', ' JRPG', ' Action RPG'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Catherine Classic',\n",
       "  'price': '$19.99',\n",
       "  'tags': ['Sexual Content', ' Anime', ' Puzzle', ' Nudity'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Medieval Kingdom Wars',\n",
       "  'price': '$15.99',\n",
       "  'tags': ['Strategy', ' Medieval', ' RTS', ' Simulation'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'FrostRunner',\n",
       "  'price': 'Free To Play',\n",
       "  'tags': ['Free to Play', ' Indie', ' Action', ' Parkour'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'ATOM RPG: Post-apocalyptic indie game',\n",
       "  'price': '$14.99',\n",
       "  'tags': ['RPG', ' Post-apocalyptic', ' Turn-Based Combat', ' Isometric'],\n",
       "  'platforms': ['win', 'mac', 'linux']},\n",
       " {'title': 'Garden Paws',\n",
       "  'price': '$19.99',\n",
       "  'tags': ['Adventure', ' Simulation', ' Indie', ' RPG'],\n",
       "  'platforms': ['win', 'mac']},\n",
       " {'title': 'GRIS',\n",
       "  'price': '$16.99',\n",
       "  'tags': ['Atmospheric', ' Indie', ' Great Soundtrack', ' Adventure'],\n",
       "  'platforms': ['win', 'mac']},\n",
       " {'title': 'Insurgency: Sandstorm',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['FPS', ' Realistic', ' Shooter', ' Multiplayer'],\n",
       "  'platforms': ['win']},\n",
       " {'title': \"Tom Clancy's Rainbow Six® Siege - Year 4 Pass\",\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Action', ' FPS', ' Multiplayer', ' Tactical'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Katamari Damacy REROLL',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Casual', ' Action', ' Great Soundtrack', ' Puzzle'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Kenshi',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Open World', ' Sandbox', ' RPG', ' Survival'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Mutant Year Zero: Road to Eden',\n",
       "  'price': '$34.99',\n",
       "  'tags': ['Strategy', ' RPG', ' Adventure', ' Turn-Based Combat'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Parkitect',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Management', ' Simulation', ' Building', ' Sandbox'],\n",
       "  'platforms': ['win', 'mac', 'linux']},\n",
       " {'title': 'Equilinox',\n",
       "  'price': '$9.99',\n",
       "  'tags': ['Simulation', ' Indie', ' Casual', ' Sandbox'],\n",
       "  'platforms': ['win', 'mac', 'linux']},\n",
       " {'title': 'Farming Simulator 19',\n",
       "  'price': '$34.99',\n",
       "  'tags': ['Simulation', ' Multiplayer', ' Realistic', ' Agriculture'],\n",
       "  'platforms': ['win', 'mac']},\n",
       " {'title': 'Wallpaper Engine',\n",
       "  'price': '$3.99',\n",
       "  'tags': ['Utilities',\n",
       "   ' Software',\n",
       "   ' Design & Illustration',\n",
       "   ' Animation & Modeling'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Warhammer 40,000: Mechanicus',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Strategy', ' Warhammer 40K', ' Turn-Based Tactics', ' Turn-Based'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'HITMAN™ 2',\n",
       "  'price': '$59.99',\n",
       "  'tags': ['Stealth', ' Action', ' Singleplayer', ' Multiplayer'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Thronebreaker: The Witcher Tales',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Adventure', ' RPG', ' Card Game', ' Story Rich'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Thief Simulator',\n",
       "  'price': '$14.99',\n",
       "  'tags': ['Simulation', ' Stealth', ' Adventure', ' Crime'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Supply Chain Idle',\n",
       "  'price': 'Free To Play',\n",
       "  'tags': ['Free to Play', ' Casual'],\n",
       "  'platforms': ['win', 'mac', 'linux']},\n",
       " {'title': 'One Hour One Life',\n",
       "  'price': '$19.99',\n",
       "  'tags': ['Massively Multiplayer', ' Indie', ' Survival', ' Simulation'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Football Manager 2019',\n",
       "  'price': '$49.99',\n",
       "  'tags': ['Simulation', ' Sports', ' Management', ' Soccer'],\n",
       "  'platforms': ['win', 'mac']},\n",
       " {'title': 'Return of the Obra Dinn',\n",
       "  'price': '$19.99',\n",
       "  'tags': ['Mystery', ' Detective', ' Singleplayer', ' Atmospheric'],\n",
       "  'platforms': ['win', 'mac']},\n",
       " {'title': 'RimWorld',\n",
       "  'price': '$34.99',\n",
       "  'tags': ['Base Building', ' Survival', ' Strategy', ' Sandbox'],\n",
       "  'platforms': ['win', 'mac', 'linux']},\n",
       " {'title': 'The Jackbox Party Pack 5',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Casual', ' Local Multiplayer', ' Multiplayer', ' Comedy'],\n",
       "  'platforms': ['win', 'mac', 'linux']},\n",
       " {'title': \"Assassin's Creed® Odyssey\",\n",
       "  'price': '$59.99',\n",
       "  'tags': ['Open World', ' Action', ' RPG', ' Singleplayer'],\n",
       "  'platforms': ['win']},\n",
       " {'title': 'Creed: Rise to Glory™',\n",
       "  'price': '$29.99',\n",
       "  'tags': ['Sports', ' Action', ' Simulation', ' Indie'],\n",
       "  'platforms': ['win', 'htcvive', 'oculusrift']},\n",
       " {'title': 'CrossCode',\n",
       "  'price': '$19.99',\n",
       "  'tags': ['Action RPG', ' Pixel Graphics', ' RPG', ' Female Protagonist'],\n",
       "  'platforms': ['win', 'mac', 'linux']},\n",
       " {'title': 'Pummel Party',\n",
       "  'price': '$9.89',\n",
       "  'tags': ['Multiplayer', ' Casual', ' Indie', ' Board Game'],\n",
       "  'platforms': ['win']}]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}