How to use EcommerceTools for technical SEO

There’s often a lot of faffing around required to get marketing and ecommerce data from various systems into Pandas so you can analyse it, or use it within more complex models. I built the EcommerceTools Python package to take the hassle out of this process and make it quick and easy to fetch and analyse data.

At the moment, it only does the basics, but it’s still useful for my daily ecommerce and marketing work. The aim is to eventually create a package that includes all the tools I need to analyse ecommerce, marketing, and SEO data and create models. In this article I’ll explain how you can use it to analyse technical SEO data.

Install the packages

To get started, open a Jupyter notebook and install the EcommerceTools Python package from PyPi by entering !pip3 install ecommercetools in a code cell and then executing it. Then import pandas and the seo module from ecommercetools.

from ecommercetools import seo
import pandas as pd

1. Discover XML sitemap locations

First, we’ll take a look at the XML sitemaps features. The get_sitemaps() function takes the location of a robots. txt file (always stored at the root of a domain), and returns the URLs of any XML sitemaps listed within. This returns a Python list containing the URL of each sitemap.

from ecommercetools import seo

sitemaps = seo.get_sitemaps("http://www.bbc.co.uk/robots.txt")
sitemaps

['http://www.bbc.co.uk/sitemaps/index-uk-archive.xml',
 'http://www.bbc.co.uk/sitemaps/index-uk-news.xml',
 'http://www.bbc.co.uk/video_sitemap.xml',
 'http://www.bbc.co.uk/sitemap.xml',
 'https://www.bbc.co.uk/food/sitemap.xml',
 'http://www.bbc.co.uk/sitemap.xml',
 'http://www.bbc.co.uk/mobile_sitemap.xml',
 'http://www.bbc.co.uk/sitemap.xml',
 'https://www.bbc.co.uk/ideas/sitemap.xml']

2. Read an XML sitemap into Pandas

The get_dataframe() function allows you to download the URLs in an XML sitemap to a Pandas dataframe. If the sitemap contains child sitemaps, each of these will be retrieved. You can save the Pandas dataframe to CSV in the usual way.

from ecommercetools import seo

df = seo.get_sitemap("http://flyandlure.org/sitemap.xml")
print(df.head())

3. Get Core Web Vitals from PageSpeed Insights

You can also obtain site performance data. The get_core_web_vitals() function retrieves the Core Web Vitals metrics for a list of sites from the Google PageSpeed Insights API and returns results in a Pandas dataframe. The function requires a Google PageSpeed Insights API key.

from ecommercetools import seo

pagespeed_insights_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
urls = ['https://www.bbc.co.uk', 'https://www.bbc.co.uk/iplayer']
df = seo.get_core_web_vitals(pagespeed_insights_key, urls)
print(df.head())

4. Get Google Knowledge Graph data

The get_knowledge_graph() function returns the Google Knowledge Graph data for a given search term. This requires the use of a Google Knowledge Graph API key. By default, the function returns output in a Pandas dataframe, but you can pass the output="json" argument if you wish to receive the JSON data back.

from ecommercetools import seo

knowledge_graph_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
knowledge_graph = seo.get_knowledge_graph(knowledge_graph_key, "tesla", output="dataframe")
print(knowledge_graph)

5. Get Google Search Console API data

The query_google_search_console() function runs a search query on the Google Search Console API and returns data in a Pandas dataframe. This function requires a JSON client secrets key with access to the Google Search Console API.

from ecommercetools import seo

key = "google-search-console.json"
site_url = "http://flyandlure.org"
payload = {
    'startDate': "2019-01-01",
    'endDate': "2019-12-31",
    'dimensions': ["page", "device", "query"],
    'rowLimit': 100,
    'startRow': 0
}

df = seo.query_google_search_console(key, site_url, payload)
print(df.head())

6. Get the number of “indexed” pages

The get_indexed_pages() function uses the “site:” prefix to search Google for the number of pages “indexed”. This is very approximate and may not be a perfect representation, but it’s usually a good guide of site “size” in the absence of other data.

from ecommercetools import seo

urls = ['https://www.bbc.co.uk', 'https://www.bbc.co.uk/iplayer', 'http://flyandlure.org']
df = seo.get_indexed_pages(urls)
print(df.head())

7. Scrape keyword suggestions from Google Autocomplete

The google_autocomplete() function returns a set of keyword suggestions from Google Autocomplete. The include_expanded=True argument allows you to expand the number of suggestions shown by appending prefixes and suffixes to the search terms.

from ecommercetools import seo

suggestions = seo.google_autocomplete("data science", include_expanded=False)
print(suggestions)

suggestions = seo.google_autocomplete("data science", include_expanded=True)
print(suggestions)

8. Retrieve robots.txt content

The get_robots() function returns the contents of a robots.txt file in a Pandas dataframe so it can be parsed and analysed.

from ecommercetools import seo

robots = seo.get_robots("http://www.flyandlure.org/robots.txt")
print(robots)

9. Scrape Google search engine results

The get_serps() function is one of the quickest and easiest ways to scrape Google search results using Python. This simple function takes a keyword phrase and returns a Pandas dataframe containing the Google search engine results for a given search term.

This is only designed for infrequent use and doesn’t include any features to prevent it from being blocked. If you want to perform large-scale web scraping of Google SERPs then you’ll need a much more sophisticated solution.

from ecommercetools import seo

serps = seo.get_serps("data science blog")
print(serps)

Matt Clarke, Saturday, March 20, 2021