How to scrape Google results in three lines of Python code
Here's a quick and easy way to scrape Google search engine results into a Pandas dataframe in just three lines of Python code...
EcommerceTools makes it really quick and easy to scrape Google search engine results in Python. In this simple project, we’ll use EcommerceTools to search Google for your chosen keywords, use web scraping to scrape the search engine results, and display the output in a Pandas dataframe.
Install EcommerceTools
First, open up a Jupyter notebook or a terminal and install the EcommerceTools Python package using the Pip Python package manager. You can do this in Jupyter by entering the command below and then pressing shift and enter to execute the cell.
!pip3 install --upgrade ecommercetools
Import packages
Next, import the seo module from ecommercetools by entering from ecommercetools import seo. This will import
only the SEO module we need for this task.
from ecommercetools import seo
Scrape the Google search results
Finally, pass your search query to the seo.get_serps() function and assign the output to a variable and then print
the results. The function will go to Google, search for your chosen search term, and scrape the first page of
results and return them in a Pandas dataframe.
results = seo.get_serps("scrape google results")
results
| title | link | text | |
|---|---|---|---|
| 0 | Is it ok to scrape data from Google results? -... | https://stackoverflow.com/questions/22657548/i... | |
| 1 | Free Google Search scraper and SERP API · Apify | https://apify.com/apify/google-search-scraper | Unofficial API to scrape and export data from ... |
| 2 | How to scrape Google Search results | Apify Blog | https://blog.apify.com/unofficial-google-searc... | 25 Feb 2019 — Unofficial Google Search API fro... |
| 3 | How To Scrape Google With Python | Hacker Noon | https://hackernoon.com/how-to-scrape-google-wi... | 28 Dec 2019 — So I made my own, and here is a ... |
| 4 | Scrape 100 Google search results in 30 seconds... | https://samuelschmitt.com/scrape-google-search... | 30 Sept 2019 — Scrape Google Search and get th... |
| 5 | How to Scrape Google Results for Free Using Py... | https://predictivehacks.com/how-to-scrape-goog... | 24 Oct 2020 — scrape google results in pyhton.... |
| 6 | Here's how you can scrape Google search result... | https://ajaypanthagani.medium.com/heres-how-yo... | 10 Sept 2019 — Here I would like to tell you a... |
| 7 | Scrape Google Search Results using Python Beau... | https://www.journaldev.com/44755/scrape-google... | 1 What is Web Scraping? 2 How Does Web Scrapin... |
| 8 | How to scrape Google search results using Python | https://practicaldatascience.co.uk/data-scienc... | 13 Mar 2021 — Scraping Google search results u... |
| 9 | Google Scraper 101 | Easy Data Technology | Sc... | https://scrapingrobot.com/blog/google-scraper/ | Why Scrape Google Results? — In the case of th... |
Matt Clarke, Sunday, May 02, 2021
Other posts you might like
How to use the Pandas truncate() function
Have you ever needed to chop the top or bottom off a Pandas dataframe, or extract a specific section from the middle? If so, there’s a Pandas function called truncate()...
How to use Spacy for noun phrase extraction
Noun phrase extraction is a Natural Language Processing technique that can be used to identify and extract noun phrases from text. Noun phrases are phrases that function grammatically as nouns...
