How to scrape People Also Ask data using Python

People Also Ask or PAA boxes have been becoming increasingly common in Google’s search results over the past few years. They show a range of questions and answers related to the user’s query which are scraped from the sites in the results.

Most SEOs and content writers work hard to try to get their content to appear within the PAAs, which requires researching the related questions and providing succinct answers to them within the copy. Answering users questions through copywriting has become even more important.

In this project, I’ll show you a really quick and easy way to fetch People Also Ask data computationally, so you can speed up your research process. If you’re looking for a way to check whether deep learning models, like those used by Google, can extract the answers from your copy, check out my post on Extractive Question Answering.

Load the packages

We’re not going to reinvent the wheel in this project and will be using a great little Python package called people_also_ask which will do all the hard work of web scraping for us. Install the package by entering pip3 install people_also_ask in your terminal and import the package.

import people_also_ask

Get People Also Ask questions related to a search term

Let’s say you’re writing an article on ferns and want to know what questions people interested in ferns ask when researching them. Simply pass your search term to the get_related_questions() function and it will return a list.

people_also_ask.get_related_questions("ferns")

['Do ferns like shade or sun?',
 'What Ferns take full sun UK?',
 'Where do ferns grow best?',
 'Do ferns grow back every year?']

If you want more than the default five questions, just pass in an integer as the second argument and you can get a longer list of terms to consider covering.

people_also_ask.get_related_questions("tree ferns", 10)

['How do you take care of a tree fern?',
 'Can tree ferns grow in full sun?',
 'Should I cut the fronds off my tree fern?',
 'Are tree ferns Hardy?',
 'How often do you water a tree fern?',
 'Do tree ferns lose their fronds in winter?',
 'How long does it take for tree fern fronds to grow?',
 'How fast do Australian tree ferns grow?',
 'Why are my tree fern leaves turning brown?',
 'What ferns grow on trees?',
 'How do you look after a tree fern?']

Fetching the answers to People Also Ask questions

Besides looking at questions people ask, it also helps to look at the answers present to see if you can better them and improve your chances of appearing in the PAA box. To find the answer for a question, pass the question to the get_answer() function.

people_also_ask.get_answer("What is a tree fern?")

{'has_answer': True,
 'question': 'What is a tree fern?',
 'related_questions': ['How do you look after a tree fern?',
  'Is a tree fern a tree?',
  'Can you cut the top off a tree fern?',
  'Can you over water a tree fern?'],
 'response': 'Tree fern\nPlants\nThe tree ferns are the ferns that grow with a trunk elevating the fronds above ground level. Most tree ferns are members of the "core tree ferns", belonging to the families Dicksoniaceae, Metaxyaceae, and Cibotiaceae in the order Cyatheales.\n Wikipedia\n \nSize\nAge\nRoot\nView 45+ more',
 'heading': None,
 'title': None,
 'link': None,
 'displayed_link': None,
 'snippet_str': None,
 'snippet_data': None,
 'date': None,
 'snippet_type': 'Whole Page Tab Container',
 'snippet_str_body': None,
 'raw_text': 'Tree fern\nPlants\nThe tree ferns are the ferns that grow with a trunk elevating the fronds above ground level. Most tree ferns are members of the "core tree ferns", belonging to the families Dicksoniaceae, Metaxyaceae, and Cibotiaceae in the order Cyatheales.\n Wikipedia\n \nSize\nAge\nRoot\nView 45+ more'}

Fetching simple PAA answers

Finally, if you only need a short answer, without the detailed background information shown above, you can instead use the get_simple_answer() function. It’s as easy as that.

people_also_ask.get_simple_answer("Are ferns easy to grow?")

'Most ferns are easy to grow and will thrive in any moist, well-drained, shady site in well-dug, humus-rich, neutral to alkaline soil.'

Matt Clarke, Saturday, March 13, 2021