How to assess product copy using EQA models

In ecommerce, writing good product copy is both an art and a science. Not only does product copy need to be written in the correct tone and style for your brand and audience, it also needs to clearly convey the product features and benefits to encourage site visitors to purchase.

Many less experienced ecommerce copywriters often focus more on style of their copy, and less on thinking about what customers will want to know before they buy. As a result, they write lovely, readable copy that fails to communicate product features and benefits.

When customers can’t find the answers to their pre-purchase questions, not only is the product conversion rate lower, but customers will also be much more likely to contact your customer service or sales team for the answers, which increases your costs. Or, they just go elsewhere…

The rise of questions and answers in SERPs

These days, one increasingly import element of product copy is not just how well it answers the questions customers have, but also how well search engines can interpret the copy to allow it to answer the questions their users have entered into their search engine.

Search engines have increasingly adopted Extractive Question Answering or EQA to handle this. This deep learning technique uses Natural Language Understanding (NLU) models, such as BERT, to read and understand product content and extract the answers to the users’ questions.

Product copy now needs to include the right keywords to ensure customers can find it via on-site and off-site search, include the answers to any pre-purchase questions, sell the features and benefits, and format the information to allow Extractive Question Answering to do its stuff.

In this project…

In this project, we’re going to test a hunch I have. One of the things I always get my ecommerce copywriters to do is clearly explain: what the product does, how it benefits the customer, and why they should buy that particular product. These are the key things that sell technical products, and reduce the likelihood for customers to abandon their order and call customer services for help.

If we use EQA to find the answers to these common questions, the answers provided should give an indication of how well the copy is written to tackle these key issues. If the EQA model can extract the answers, then I suspect customers will also be able to do the same, and search engines also!

Load the packages

To get started, open up a Jupyter notebook and import pandas and the pipeline module from transformers. You’ll likely want to turn off warnings, as the pipeline currently throws a warning as one of the features used is soon to be deprecated.

import pandas as pd
from transformers import pipeline
import warnings
warnings.simplefilter("ignore")

Load the pipeline

Hugging Face has already created a pre-built model pipeline to handle question answering. This has been trained on the Stanford Question Answering Dataset (SQuAD). It’s a pretty massive model that was trained on an even bigger dataset, and it can handle Extractive Question Answering out-of-the-box. Define the pipeline and then wait for it to download.

nlp = pipeline("question-answering")

Load your dataset

Next, load up your dataset. I’ve used the GoNutrition dataset I created for my post on How to auto-generate product summaries using deep learning. This includes a few product descriptions I wrote when I worked on the GoNutrition launch many years ago.

df = pd.read_csv('gonutrition.csv')
df.head()

Define your questions

To see if EQA can find the key points in the product copy I’ve devised a series of questions. If you’re trying this out on your own products, you’ll likely want to make them specific to your product range and perhaps modify them based on the product category you’re examining.

questions = [
    "What is this product for?",
    "Why will it benefit me?",
    "What is it made from?",
    "What is special about this product?"
]

Run the Extractive Question Answering model

Now that’s set up, we can run the EQA model. First, we’ll create a Pandas dataframe in which to store the results. Then we’ll use iterrows() to iterate over each row in our product dataset and truncate the description to the first 512 characters, which is a limitation of this model. Arguably, not a bad thing for us, as it means the copywriters need to ensure they put the key points at the start.

In each iteration we’ll loop over the questions in our list and run them through the transformer model. We’ll then create a dictionary of the results, including the product name, question, answer, and confidence score, and append this to our dataframe of results. It only takes a second per product, so is fairly quick when you run it on a decent GPU.

df_items = pd.DataFrame(columns = ['product_name', 'question', 'answer', 'score'])

for index, row in df.iterrows(): 

    text = row['product_description'][:512]

    for question in questions: 
        result = nlp(question=question, context=text)

        item = {
            'product_name': row['product_name'],
            'question': question,
            'answer': result['answer'],
            'score': round(result['score'], 4)
        }

        df_items = df_items.append(item, ignore_index=True)

df_items.head(10)

Inspect the results

Finally, we can inspect the results of the EQA. I suspect, from a sports nutrition perspective, the answers could probably be better. However, this could imply that the copy could be further improved to generate better scores. My completely untested hypothesis is that if EQA can find the answer easily, then humans and search engines using EQA and Product Knowledge Graphs probably can too.

The verdict

I’m not sure the approach is perfect, but the prototype seems to work. Aggregating mean scores for the questions asked on each product gives a breakdown of performance, which could be used to prioritise which products have copy most in need of improvement.

When I rewrote the copy to optimise it for EQA, I was able to generate text that covered the key points much more succinctly, which should work for humans too. I’d imagine there may be a correlation between better scores from the EQA and stronger product conversion. If you try it on your own product copy, I’d be really interested to hear if you think it works.

df_items.groupby('product_name').agg(
    avg_score=('score', 'mean')
)

Further reading

• Chaybouti, S., 2021. EfficientQA: a RoBERTa Based Phrase-Indexed Question-Answering System. arXiv preprint arXiv:2101.02157.

• Fajcik, M., Jon, J., Kesiraju, S. and Smrz, P., 2020. Rethinking the objectives of extractive question answering. arXiv preprint arXiv:2008.12804.

• Seo, M., Kwiatkowski, T., Parikh, A.P., Farhadi, A. and Hajishirzi, H., 2018. Phrase-indexed question answering: A new challenge for scalable document comprehension. arXiv preprint arXiv:1804.07726.

Matt Clarke, Sunday, March 14, 2021