How to create content recommendations using TF IDF

After work, when I’m not learning about data science, practising data science, or writing about data science, I like to browse classic car auction sites looking for cars I can’t afford to buy, don’t have enough room to house, and whose purchase would lead to divorce and bankruptcy.

One of my favourite of such sites is The Market, as it includes well-written product copy that other car auction sites don’t have. However, while its inventory is small, it currently lacks a recommendation engine that serves up other cars I might like to imagine I could afford to buy.

I totally get why The Market doesn’t have recommendations. The number of cars sold is very low, there are a limited number of concurrent auctions, and most people make a single purchase, so a regular “customers who bought this also bought” model would be useless.

Content-based recommendations

However, despite the lack of sales data normally required to generate product recommendations, there’s still a way that these could be added. We could generate recommendations based on content similarity instead.

For example, if you’re looking at a listing for a Ferrari 308 GTB, you might also be interested in checking out the 308 GTS. We can do this via two Natural Language Processing (NLP) techniques: Term-Frequency Inverse Document Frequency or TF-IDF, and cosine similarity.

Term Frequency Inverse Document Frequency (TF-IDF)

TF-IDF is a statistic which show the importance of specific words in a document versus the other documents in collection of documents, or “corpus”. Basically, TF-IDF counts up the number of times a given phrase occurs within a document and compares it to other documents.

If a page contains the words “Ferrari 308” numerous times, and other documents in the corpus do not, then it’s probable that the document is about the “Ferrari 308”. Simply find all the documents where the scores for a phrase are high and you’ve got your matches.

Cosine similarity

Cosine similarity measures the similarity between two vectors. Since TF-IDF returns vectors showing the score a document gets versus the corpus, we can use cosine similarity to identify the closest matches after we’ve used TF-IDF to generate the vectors.

I’ll skip the complicated maths, but basically we first generate the TF-IDF vectors containing the raw numbers, and then use cosine similarity to check these across all documents. We can then sort the output and identify the closest matches based on their text similarity.

Picture by Sid Ramirez, Unsplash.

Import the packages

To get started, open up a Jupyter notebook and import pandas, numpy, the TfidfVectorizer, cosine_similarity and linear_kernel modules from scikit-learn.

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.metrics.pairwise import linear_kernel

Load the data

Next, load up your dataset. I’m using some product descriptions I scraped from The Market, but you can use product page content, blog posts, or anything else you have which is similar.

df = pd.read_csv('themarket_pages.csv')
df.sample(10)

df['text'].sample(5)

598     1972 MERCEDES-BENZ 250CE W114\nBackground\nThe...
1026    1965 SUNBEAM ALPINE Series V\nBackground\nFoll...
265     1988 DAIMLER Double Six\nBackground\nJaguar's ...
766     1961 MGA Roadster 1600 Mk 1\nBackground\nThe M...
521     1955 LAND ROVER Series 1 Soft Top. 86 Inch\nBa...
Name: text, dtype: object

Prepare the data

Next, we’ll tidy up the data a little. There are some duplicate page titles in here, so we’ll drop these from the dataframe and return a list of the indices, so we can use them for looking up values. We’ll also fill in some NaN values with blanks to avoid TF-IDF complaining.

indices = pd.Series(df.index, index=df['title']).drop_duplicates()
content = df['text'].fillna('')

Create TF-IDF model

First, we’ll set up TfidfVectorizer and tell it to use English stop words. This will remove common words like “the” and “of” to leave the more important ones. TF-IDF will additionally down-weight common words that appear across documents.

tfidf = TfidfVectorizer(stop_words='english')

Next, we’ll create a TF-IDF matrix by passing the text column to the fit_transform() function. That will give us the numbers from which we can calculate similarities.

tfidf_matrix = tfidf.fit_transform(content)

Now we have our matrix of TF-IDF vectors, we can use linear_kernel() to calculate a cosine similarity matrix for the vectors. There are several ways to do this, but the below approach worked for me.

cosine_similarities = linear_kernel(tfidf_matrix, tfidf_matrix)

Get recommendations based on text similarity

Now the model is built, and we have our TF-IDF matrix and a cosine similarity matrix covering all the documents, we can create a helper function to generate content recommendations. The code in this is a bit fiddly, so I’ve annotated it at each step.

Basically, it takes the dataframe of text, the name of the column being used to search from, the value to search for, the cosine similarity matrix, and the number of recommendations to return. It then looks up the title and returns the documents with the closest cosine similarity.

def get_recommendations(df, column, value, cosine_similarities, limit=10):
    """Return a dataframe of content recommendations based on TF-IDF cosine similarity.
    
    Args:
        df (object): Pandas dataframe containing the text data. 
        column (string): Name of column used, i.e. 'title'. 
        value (string): Name of title to get recommendations for, i.e. 1982 Ferrari 308 GTSi For Sale by Auction
        cosine_similarities (array): Cosine similarities matrix from linear_kernel
        limit (int, optional): Optional limit on number of recommendations to return. 
        
    Returns: 
        Pandas dataframe. 
    """
    
    # Return indices for the target dataframe column and drop any duplicates
    indices = pd.Series(df.index, index=df[column]).drop_duplicates()

    # Get the index for the target value
    target_index = indices[value]

    # Get the cosine similarity scores for the target value
    cosine_similarity_scores = list(enumerate(cosine_similarities[target_index]))

    # Sort the cosine similarities in order of closest similarity
    cosine_similarity_scores = sorted(cosine_similarity_scores, key=lambda x: x[1], reverse=True)

    # Return tuple of the requested closest scores excluding the target item and index
    cosine_similarity_scores = cosine_similarity_scores[1:limit+1]

    # Extract the tuple values
    index = (x[0] for x in cosine_similarity_scores)
    scores = (x[1] for x in cosine_similarity_scores)    

    # Get the indices for the closest items
    recommendation_indices = [i[0] for i in cosine_similarity_scores]

    # Get the actutal recommendations
    recommendations = df[column].iloc[recommendation_indices]

    # Return a dataframe
    df = pd.DataFrame(list(zip(index, recommendations, scores)), 
                      columns=['index','recommendation', 'cosine_similarity_score']) 

    return df

Generate the recommendations

Finally, we can put it in action and see how it works. First, we’ll take the title of the “1982 Ferrari 308 GTSi For Sale by Auction” auction and see what we get back. It works perfectly. The closest matches are the 308 GTB, the 308 GTS, and another 308 GTB, followed by more Ferraris.

recommendations = get_recommendations(df, 
                                      'title', 
                                      '1982 Ferrari 308 GTSi For Sale by Auction', 
                                      cosine_similarities)

recommendations.head(10)

The “1959 LAND ROVER SERIES II LWB For Sale by Auction” search was a bit tougher, but all the Series II Land Rovers do appear at the top, along with a Range Rover, which seems fair enough. The approach seems to work really well on this content.

recommendations = get_recommendations(df, 
                                      'title', 
                                      '1959 LAND ROVER SERIES II LWB For Sale by Auction', 
                                      cosine_similarities)

recommendations.head(10)

Matt Clarke, Saturday, August 14, 2021