How to create a collaborative filtering recommender system

Recommender systems, or recommendation engines as they’re also known, are everywhere these days. Whether you’re looking for books on Amazon, tracks on Spotify, movies on Netflix or a date on Tinder/Grinder, you’ll be served up the recommendations using this method.

As with everything in data science, there are many different ways you can generate recommendations, but probably the most widely used method is collaborative filtering. It can be done in any language, including SQL, which makes it straightforward to implement.

This fairly simple system was popularised by Amazon and has become widespread on content and ecommerce sites across the internet. Collaborative filtering actually comes in two different forms - item-based collaborative filtering and user-based collaborative filtering. They work in the same way, but their source data differs, allowing them to serve a slightly different style of date, movie, or product recommendation.

Some commonly seen examples of this are:

• Customers who bought this also bought…
• Customers who liked this also liked…
• Customers who read this also read…
• Customers who engaged with this also engaged with…
• Customers who watched this also watched…
• Customers who swiped right on him/her also swiped right on…

User-based collaborative filtering

In user-based collaborative filtering you build up a matrix of every user and record all of their interactions, like the tracks they’ve listened to, the movies they’ve watched, the scores they’ve given or the articles they’ve read. This gives you one row per user with an item (like a track or movie) per column with the metric inside.

Using this matrix, we can compute the similarity between users based on what they liked, watched or rated, and identify users who have similar tastes. If a user has similar tastes to another and has read, watched or rated some of the same items, you can recommend them other things they might like.

User-based collaborative filtering works, but it does have some issues. Firstly, people’s tastes change over time, so someone who liked Justin Bieber when they were 13 may now have moved on in life. Secondly, as you need to create one row per user, you end up with a truly enormous matrix on larger sites making this approach harder to scale. For example, if you’ve got 200K users on your platform and only 50K active, you’ll still need a row for each one.

The other issue with user-based collaborative filtering is that it can be rigged by users. Create a bot to generate fake user accounts and then give your item a glowing rating using a shilling attack and this method could be gamed.

Item-based collaborative filtering

Item-based collaborative filtering is much the same as user-based collaborative filtering, but instead of looking at relationships between users it looks at relationships between the items, so it’s less sensitive to shilling attacks (particularly as it’s often based on items purchased, making it expensive to rig.) Amazon has used this technique for years and it works well. As you only need one row per item, instead of per user, it’s also much more scalable, and it’s less sensitive to time constraints.

You can perform item-based collaborative filtering in various ways, depending on your source data and the recommendation you wish to make. We’ll be using item-based collaborative filtering to create a movie recommender system so we’ll be making a matrix of every film, identifying which movies are similar to other movies and then making recommendations to users.

1. Load the data

We are going to make our movie recommendations using the item-based collaborative filtering technique on the excellent MovieLens dataset, which is available in various sizes. The “ml-latest-small” dataset is a subset of the full data based on 100,000 ratings by 600 users across 9000 different movies, and comes in at around 1MB, so it’s ideal for practicing.

Download a copy of the zip file and place the files into a folder called data, then load up Pandas and read the CSV files into dataframes. We don’t actually need all the files, so just load up the movies.csv and ratings.csv files.

import pandas as pd
import numpy as np
import warnings

warnings.filterwarnings('ignore')
pd.set_option('max_columns', 10)
pd.set_option('max_rows', 1000)

df_movies = pd.read_csv('data/movies.csv')
df_ratings = pd.read_csv('data/ratings.csv')

Next, we’ll view the head() of each dataframe to check out the content within. As you can see from below, df_movies contains the movieId, title and a list of genres. The df_ratings dataframe contains the userId, the movieId, their rating and a timestamp.

df_movies.sample(3)

df_ratings.sample(3)

2. Re-shape the data and create a matrix

As the first step we need to merge the movies dataframe to the ratings dataframe. This gives us a massive dataframe containing one row per movie review, along with the userId and rating.

df_movies_ratings = df_movies.merge(df_ratings, on='movieId', how='left')
df_movies_ratings.head(3)

Next, we need to take df_movie_ratings and reshape the dataframe using Pandas so that each row in the dataframe represents a unique user and each column represents a review and its rating. This changes our dataframe from long format data to wide format data. With one column created for every movie in the dataset, this gives us loads of columns (9724 in fact) and most of them will contain NaN (not a number) values because no person is ever going to review every movie in the dataset.

df_user_ratings = df_movies_ratings.pivot_table(index='userId', columns=['title'], values='rating')
df_user_ratings.head(3)

3: Identify users who rated a given film

Next we’re first going to fetch all the ratings by userId on “Star Wars: Episode IV - A New Hope (1977)” from the df_user_ratings dataframe. Although I’ve never seen this film, I’m fairly sure that people who watch it will also have watched and like other movies in the franchise, so the results will be easily to interpret. This gives us a series contain the userId and rating for every person in the dataset, if they’ve rated the film or not. The users who did not rate the film will have a NaN value in the rating column.

rated_movie = df_user_ratings['Star Wars: Episode IV - A New Hope (1977)']
rated_movie.head(3)

userId
1.0    5.0
2.0    NaN
3.0    NaN
Name: Star Wars: Episode IV - A New Hope (1977), dtype: float64

4: Identify correlations

Next, we’re going to find the correlations of all movies against our target film using the Pandas corrwith() function. This computes a pairwise correlation of the the rated movie series (or vector) with all of the other movies and returns the movies which have the most ratings in common. By dropping the NaN values we get to see only the movies with ratings.

similar_movies = df_user_ratings.corrwith(rated_movie)
similar_movies.dropna(inplace=True)
similar_movies = pd.DataFrame(similar_movies, columns=['correlation'])
similar_movies.head(5)

If we sort the movies in descending order of correlation with our target film, we get back a list of the ones which are highly correlated. However, as you’ll see from the list below, although they are highly correlated, they actually look like pretty weak recommendations, which shows us that we still have work to do.

similar_movies.sort_values(by='correlation', ascending=False).head(5)

5: Engineer features to improve performance

The problem with the above approach is that it doesn’t take the number of ratings into consideration. People who like Star Wars, might not want to watch Shrek the Halls, even if the correlation score seems perfect. This has happened because small numbers of people who liked Star Wars also rated other films. To overcome this we need to engineer some additional features to let us filter the volume of reviews so that they’re only considered similar if there’s a reasonable number of ratings in common. We first need to calculate the total number of ratings and the mean rating for each movie.

df_movies_ratings['total_ratings'] = df_movies_ratings.groupby('movieId')['rating'].transform('count')
df_movies_ratings['mean_rating'] = df_movies_ratings.groupby('movieId')['rating'].transform('mean')
df_movies_ratings.head(3)

Now we’ve calculated the total ratings and mean rating for each movie, we’ll drop the duplicate rows and create a new dataframe called df_movie_statistics which includes the core information we need.

df_movie_statistics = df_movies_ratings[['movieId', 'title', 'total_ratings', 'mean_rating']]
df_movie_statistics.drop_duplicates('movieId', keep='first', inplace=True)

df_movie_statistics.head(3)

Now we have a count of total ratings, we can filter our original df_movie_statistics dataframe to create a new df_popular_movies dataframe which contains only those with 50 or more ratings. If we use this, we should find that the quality of the results go up, because they’re less impacted by noise and should provide a better indication of consistency. Then, if we print the top five results we can see the five mostly highly rated films where there are at least 50 ratings. This looks spot on.

df_popular_movies = df_movie_statistics['total_ratings'] >= 50
df_popular_movies = df_movie_statistics[df_popular_movies].sort_values(['total_ratings', 
                                                    'mean_rating'], ascending=False)
df_popular_movies.head()

If we sort the df_popular_movies dataframe in ascending order of the total_ratings we see that they start from 50, which is exactly what we need.

df_popular_movies.sort_values(by='total_ratings', ascending=True).head()

By merging the df_popular_movies dataframe above containing the mean ratings on films where more than 50 people have voted with our similar_movies dataframe we get a better picture. If we then drop the NaN values on all of the films that do not have ratings, and sort in descending order of correlation, we get back a list of films that were also rated by people who watched our target film.

Now the results look pretty good. We’ve got other Star Wars films in the top results, Star Trek and some other films that are highly rated in general. That gives us a “people who watched this also watched” style recommendation.

similar_movies = similar_movies.reset_index()
popular_similar_movies = similar_movies.merge(df_popular_movies, on='title', how='left')
popular_similar_movies = popular_similar_movies.dropna()
popular_similar_movies.sort_values(by='correlation', ascending=False).head(10)

We can modify this very easily to give results for “people who liked this film also liked” simply by filtering against the mean rating. If we assume that any score over 4 is a good indication of a film someone really liked, by adding this additional filter we get a tweaked set of results with “better” films.

popular_similar_liked_movies = popular_similar_movies[popular_similar_movies['mean_rating'] >= 4]
popular_similar_liked_movies.sort_values(by='correlation', ascending=False).head(10)

The collaborative filtering approach is just one of a number of different algorithms you can use for recommender systems. In the next part, we’ll try another approach which makes use of some other data within the MovieLens dataset.

Matt Clarke, Tuesday, March 02, 2021