How to create a Google rank checker tool using Python

Most off-the-shelf SEO tools come with a rank checker that allows you to monitor your position for given phrases in the Google search engine results. If you want to create your rank checker tool, it’s fairly easy to do using some Python web scraping skills and a little of your time. Here’s a really quick and easy way to do it.

Import the packages

First, open a Jupyter notebook and install my EcommerceTools Python package. EcommerceTools includes a range of modules for performing a range of ecommerce and marketing tasks, including various technical SEO tasks, such as web scraping and accessing data from the Google Search Console API. You can install it by entering pip3 install --upgrade ecommercetools in your terminal.

!pip3 install --upgrade ecommercetools

You’ll also need to include the date and time modules, so we can create and manipulate dates in Python, as well as the Pandas library for creating dataframes, and the seo module from EcommerceTools, which includes a tool to scrape Google search engine results in a single line of code.

import time
import datetime
import pandas as pd
from ecommercetools import seo

Scrape the Google search engine results

Rather than building a Google search engine results scraper from scratch, we’ll use the get_serps() function from EcommerceTools. To use this, all you need to do is pass the search term to get_serps() and define the number of pages of search engine results to return. The function will return a Pandas dataframe containing the title, snippet, URL, and position of each page in the SERPs.

df = seo.get_serps("contractual churn model", pages=5)

df.head(10)

Get the top ranking page for a domain

Next, we’ll create a function to identify the top ranking page for a given domain from the dataframe returned by get_serps(). This takes the dataframe above, plus a string containing the domain of your site and returns the data.

def get_top_rank(df, domain):
    """Return the rank of the domain's top ranking page for a search term."""
    
    data = df[df['link'].str.startswith(domain)].head(1).to_dict('records')
    
    if len(data) > 0:
        return data[0]
    else:
        return

Create the Google rank checker function

Now we’ll create a function called rank_checker() to bulk check the rankings of a Python list of different keywords you want to monitor. This will first create an empty Pandas dataframe, then loop over each of the keywords in the list, run the keyword through the get_serps() function and return the highest ranking page using get_top_rank().

It outputs the dataframe in a dataframe containing the keyword, date, position or rank, title, link and snippet from the Google SERPs, so you can store it, write it to a database or analyse the data as you wish.

def rank_checker(keywords, domain, sleep=2, verbose=False):
    """Scrapes Google Search Engine Results Pages (SERPs) and returns the highest
    ranking page for each keyword phrase on a specified domain. Note that larger
    volumes of keywords may result in the application being temporarily blocked. 
    
    Args:
        keywords (list): List of keyword phrases for which to check Google rankings.
        domain (string): Fully qualified domain name i.e. https://practicaldatascience.co.uk
        sleep (integer, optional): Optional sleep time to avoid blocking.
    
    Returns:
        df (dataframe): Pandas dataframe containing highest rank for each keyword.
    """
    
    df = pd.DataFrame(columns=['keyword', 'date', 'position', 'title', 'link', 'text'])

    for keyword in keywords:

        if verbose:
            print('Fetching:', keyword)
            
        data = get_top_rank(seo.get_serps(keyword, pages=5), domain)
        time.sleep(2)

        if data:
            data['keyword'] = keyword
        else:
            data = {'keyword': keyword, 'position': '', 'title': '', 'link': '', 'text': ''}

        date = datetime.datetime.now()
        data['date'] = date.strftime('%Y-%m-%d')

        df = df.append(data, ignore_index=True)
        
    return df

Run the rank checker on your target keywords

Finally, we can run the Google rank checker on a Python list containing your target keywords. This will work fine for relatively small volumes of keywords, but you are likely to get temporarily blocked by Google if you want to examine larger volumes.

keywords = [
    'contractual churn model', 
    'non contractual churn model', 
    'product matching model',
    'marketing mix model',
    'purchase intention model',
    'inventory classification model',
    'rfm model',
    'causal impact model',
    'product classification model', 
    'learning to rank model',
    'customer segmentation model',
    'bg/nbd model',
    'abc xyz inventory classification model', 
    'abc inventory classification model',
    'isolation forest model',
    'response model',
    'product attribute extraction model',
    'next product to buy model',
    'xgboost classification model',
    'scikit learn linear regression model'
]

To run the code, we’ll pass in the list above containing the keywords whose rank we wish to check, the domain of the site you want to monitor, a sleep parameter to avoid battering Google with numerous consecutive requests, and a boolean argument to turn on verbose mode so you can watch the SERPs being scraped.

domain = 'https://practicaldatascience.co.uk'
df = rank_checker(keywords, domain, sleep=10, verbose=True)

Fetching: contractual churn model
Fetching: non contractual churn model
Fetching: product matching model
Fetching: marketing mix model
Fetching: purchase intention model
Fetching: inventory classification model
Fetching: rfm model
Fetching: causal impact model
Fetching: product classification model
Fetching: learning to rank model
Fetching: customer segmentation model
Fetching: bg/nbd model
Fetching: abc xyz inventory classification model
Fetching: abc inventory classification model
Fetching: isolation forest model
Fetching: response model
Fetching: product attribute extraction model
Fetching: next product to buy model
Fetching: xgboost classification model
Fetching: scikit learn linear regression model

I’ve been using this to periodically fetch rankings for a bunch of keywords I want to monitor and it works fine. However, overdo the use of it and Google will temporarily block you leaving it out of action for a short period. This is a common problem when scraping the SERPs and one that can be complex to resolve without resorting to proxies or paid scraping APIs.

df

Matt Clarke, Monday, October 04, 2021