A quick guide to search intent classification for SEO

Scales by Sigmund, Unsplash.

10 minutes to read

Search intent classification has been around for almost 20 years, but has only recently started to move into the mainstream in ecommerce and technical SEO. Here’s a quick guide to search intent classification to help you understand what it is, why it matters to ecommerce and SEO, and how you can use it to help improve business performance.

What is search intent classification?

When we use the internet, we all have different objectives. For example, we might want to navigate to a specific site, we might want to find the answer to a question, or we might want to purchase a specific item. Our search intent (also called user intent) is often identifiable from the terms we enter when browsing.

Search intent classification is a machine learning technique that allows data scientists to identify the probable search intent of a user from the words they entered into their search engine. Modern machine learning techniques mean it’s now easier to generate accurate results.

What are the main types of search intent?

Search intent has been covered in various research papers over the past 20 years, and researchers have proposed several ways of classifying it, with their proposals coming from a mixture of web analytics or log data, as well as from qualitative user surveys. The classifications typically fall into two main types: browsing behaviour or search intent.

Browsing behaviour

Marchionini (1997) was one of the first scientists to propose a classification. He used browsing behaviour and classified it into three types: directed, semi-directed, and undirected browsing.

Browsing	Summary
Directed	Directed browsing is focused, systematic, and directed by a specific target or object. For example, looking to purchase a specific item.
Semi-directed	Semi-directed browsing is predictive or "generally purposeful", but the target is less definite and browsing less systematic.
Undirected	There is no real goal in undirected browsing and there's little focus. It's the online equivalent of flipping through a magazine or channel hopping from the sofa.

Search intent

Both Broder (2002) and Jansen, Booth, and Spink (2007) (some of the best known researchers in the search intent field), propose three different hierarchical levels of search intent: informational, navigational, and transactional. They’ve become the standard definitions for this problem.

Intent	Summary
Informational	Informational searching aims to locate content relating to the information need of the searcher. These can range from vague and broad search terms, to highly specific terms and make up 60-80% of web traffic. Example: "who is ceo of tesla?"
Navigational	Navigational intent aims to get the user to a specific website, so tends to be mostly branded search, but also includes generic searches for sites a user expects to exist. Example: "tesla"
Transactional	The goal of transactional item is obviously to find a specific item to purchase. Again, this can be a very broad term (such as "laptops") or a very specific term (i.e. Dell Precision 7750). Example: "buy tesla roadster"

Others have added an extra classification called “commercial investigation”. This is effectively transactional intent that didn’t lead to a purchase at the time, perhaps because the customer is still researching, or because the price was wrong, or the content didn’t answer their questions. I’m not entirely convinced that it’s really any different to “transactional” intent.

The internet Picture by Sigmund, Unsplash.

How is intent classification a business benefit?

Intent classification was originally developed to improve search engines, and it’s only in the past decade or so that it’s been adopted by ecommerce and SEO experts. It has several benefits, but it remains debatable whether these couldn’t be figured out using a more easily obtained alternative metric. It’s undeniably a cool technology, but does it really offer anything unique?

Real time classification	By understanding search intent in real time, search engines can provide results more likely to satisfy the searcher. Google holds several patents for intent based search result interactions and modifies search results depending on the search intent.
Learning to Rank	The on-site equivalent of the real time intent classification used by Google is the incorporation of search intent data within on-site Learning to Rank models. These clever systems control the order of search results to maximise clicks and conversions and are starting to become more common on ecommerce sites.
Content alignment	From an SEO perspective, a key benefit of classifying terms by search intent is that it allows you to align content to intent, so customers land on a page which better meets their search requirements.
Content strategy	On a related level to content alignment, intent classification can also reveal whether your content team is generating "low-value" informational traffic instead of transactional content that helps boost your sales.
Ad strategy	Understanding search intent can also help you avoid running ads on paid search keywords that may not deliver that much return. Though you don't need search intent models to see this.

What data sources are used for intent classification?

To classify content by intent you need two things: search keywords and (depending on the modeling approach you use) labeled training data.

Keyword data

Most historic studies have used web server log data, but these days, it this no longer contains search keywords, as most get stripped out and anonymised. In their absence, you can use keywords extracted from Google Analytics, Google Search Console, or other SEO tools, such as Ahrefs.

Training data

In addition to the keywords, features identifying potential user intent can be obtained from web scraping Google search results. Since Google personalises the results according to search intent, you can glean search intent by scraping the features from Google. Hand-crafted features from on-site behaviour (such as add-to-carts, transactions, and other searches) can also be used to infer search intent and help you construct training data.

How can I construct a model?

There are two main approaches to creating the model. You can use a supervised learning classification model, trained on a pre-labeled dataset, or you can use an unsupervised learning approach, such as clustering, to identify intent from behavioural data.

Supervised learning	Supervised learning models, such as the XGBoost, require labeled training datasets and hand-crafted features that will allow the model to identify correlations between features and classes. While this is a perfectly acceptable approach, the training data needs to be site-specific and isn't trivial to create.
Unsupervised learning	Unsupervised learning approaches, such as K-means clustering, don't require data to be pre-classified, so are usually easier to work with. They can also use more than the three standard classes identified, which may have some advantages.

How can I create a training dataset?

Two main approaches are used, and both of them rely on the fact that Google SERPs are already modified based on the search intent it has identified.

Mohammadi et al. 2020, scraped Google search results for each search term (using a scraper that utilised proxies to avoid hitting a Captcha and getting blocked) and then extracted features from the results, such as featured snippets, rich snippets, People Also Ask (PAA) boxes, and Knowledge Graph boxes. They manually labeled their data and compared it to K-means clusters.

Benjamin Burkholder used a paid service to scrape Google search engine results and extract the same features, and used the features themselves to dictate how intent was classified. Searches showing a People Also Ask (PAA) box were labeled informational. Those with a Knowledge Graph or site links were deemed navigational, while those with ads were labeled transactional.

Does it work?

The Mohammadi et al. (2020) study shows that there’s a link to the features of the Google SERPs and the search intent of users, so there’s certainly some value in scraping these features and using them to identify potential search intent. However, they may not be clear cut.

The K-means clusters below show the spread of classifications across the optimum number of clusters identified using the elbow method, which emphasise the overlap between the classes and the clusters.