How to identify striking distance keywords with Python

Striking distance keywords are those which appear just off the bottom of the first page of search engine results. Keywords that appear on the first page have the greatest visibility and the highest search volume, making them more likely to be clicked, so a key objective in digital marketing is to get keywords ranking higher through search engine optimisation or SEO.

One popular SEO strategy involves identifying keywords that are within striking distance of the first page to optimise them, helping to focus and prioritise optimisation efforts, so results are generated faster and with the least amount of effort.

Since striking distance keywords are already ranking fairly close to the bottom of the first page, it often only takes a little more optimisation to give them a boost and move them to page one, making the technique a great way to get quick improvements in SEO.

In this project, I’ll show you how you can use Python to extract keywords and their average rankings from Google Search Console, and use Pandas to identify those which are within striking distance.

Install the packages

First, open a Jupyter notebook and install my EcommerceTools if you don’t already have it installed. You can install EcommerceTools by entering the command !pip3 install ecommercetools in a cell in your Jupyter notebook and then executing it. It will install all the dependencies you need for the project.

!pip3 install ecommercetools

We’ll be using the SEO module from EcommerceTools to fetch data from the Google Search Console account and will then display and manipulate the GSC data using Pandas, so import the pandas package and import seo from ecommercetools.

import pandas as pd
from ecommercetools import seo

Configure your Google Search Console connection

In order to authenticate with Google Search Console and extract the data from your account you will need to create a client secrets JSON keyfile. You can do this using the Google API Console. Once that’s created, download the JSON keyfile and create a variable called key to tell EcommerceTools its path.

You’ll also benefit from creating a few other variables to hold the URL of your site, and the start and end date for the period for which you want to retrieve GSC data.

key = "client-secrets.json"
site_url = "http://flyandlure.org"
start_date = '2021-01-01'
end_date = '2021-06-31'

Fetch your Google Search Console data

To fetch data from GSC using EcommerceTools you need to create a Python dictionary called a payload that contains the API parameters you want to run through the API. The payload defines the start and end date and fetches the page and query dimensions containing the URL and the search keywords the page ranks for.

By default, the GSC API will return metrics for clicks, impressions, ctr, and position. We then pass that payload to the query_google_search_console() function, along with the site_url variable and the key.

payload = {
    'startDate': start_date, 
    'endDate': end_date,
    'dimensions': ['page', 'query'],  
    'rowLimit': 10000,
    'startRow': 0
}

df = seo.query_google_search_console(key, site_url, payload)

The query_google_search_console() function will return a Pandas dataframe containing the clicks, impressions, click-through rate (CTR), and average position (or rank) for each of the queries within your selected time period. To sort these and show the biggest potential wins, I’d recommend you use the sort_values() function to show the highest volume queries at the top of your dataframe.

df.sort_values(by=['impressions', 'position'], ascending=False).head(10)

Identify striking distance keywords

Now we’ll create a little helper function to filter the above dataframe and show only the site’s striking distance keywords. There’s no formal definition of what a striking distance keyword really is.

Some people view it as any keyword that ranks on page two or three of search engine results, but you might want to define tighter criteria, such as 10 to 14 on larger sites, or those where search engine optimisation is constant.

We’ll pass in the dataframe of GSC query metrics and first filter it, so it only shows those queries that exceed a minimum volume of impressions, as this will help us focus on the keywords likely to generate the biggest search traffic improvements. Then, we’ll filter the dataframe again, so it shows only the keywords in the ranking range we want to examine.

def striking_distance_keywords(df, min_imps=1000, min_rank=10, max_rank=30):
    """Return a dataframe containing only striking distance keywords.
    
    Args:
        df (dataframe): Pandas dataframe containing query, impressions and position
        min_imps (int, optional, default=1000): Optional minimum volume of impressions
        min_rank (int, optional, default=10): Minimum search rank
        max_rank (int, optional, default=10): Maximum search rank
        
    Returns:
        df (dataframe): Filtered dataframe containing only the keywords within striking distance
    """
    
    df = df[df['impressions'] >= min_imps]
    df = df[(df['position'] > min_rank) & (df['position'] < max_rank)]
    return df.sort_values(by='position', ascending=True)

Now we can run our striking_distance_keywords() function using the dataframe of page and query data we retrieved from the Google Search Console API. I’ve set the function to require a minimum of 10 impressions, and have defined a striking distance keyword as one that appears between position 10 and 30.

df_striking_distance = striking_distance_keywords(df, min_imps=10, min_rank=10, max_rank=30)

df_striking_distance.head(10)

Obviously, since site rankings change all the time, you need to repeat this process regularly and obtain the current striking distance keywords, make your optimisations, and then repeat the process. Doing so should see your average search positions and your traffic improve.

Improving rankings for striking distance keywords

There are several things you can try to improve the search engine rankings for striking distance keywords, but one of the most popular methods is to improve internal links for these phrases by identifying other pages on your site that include these keywords or related ones, and then adding internal links to the page shown in the striking distance keywords report we created above.

A quick way to do this is to take each of your target keywords returned in the striking distance keyword report and perform a site: search for your domain on Google with the keyword afterwards in double quotes, for example: site:flyandlure.org "simms waders".

This will search pages Google has indexed and return any pages where an exact match is found for the chosen keyword phrase. All you need to do then is go to the page returned, edit the content, and add internal links to the target page from each page identified.

seo.get_serps("site:flyandlure.org \"simms waders\"")

seo.get_serps("site:flyandlure.org \"orvis battenkill\"")

Another effective technique is to identify keyword opportunities present in pages for which you are already ranking.

For example, if your site is already receiving traffic for the organic keywords such as “fly reels”, but that keyword phrase is missing from both the title and meta description, then adding it can increase visibility in search engine results, generating extra clicks.

Matt Clarke, Saturday, August 07, 2021