# Building Powerful Recommendation Systems with LensKit in Python

- Canonical: https://33rdsquare.com/how-to-build-a-recommendation-system-using-lenskit-and-evaluate-it-using-ndcg-in-python/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

## Introduction

In today‘s age of information overload, recommendation systems have become an indispensable tool for helping users navigate the vast expanse of online content. By learning user preferences and item similarities, these systems act as personalized guides, surfacing relevant items that users are likely to enjoy. For businesses, recommenders are a powerful engine for driving engagement and revenue. Studies show that [35% of Amazon‘s revenue](https://www.mckinsey.com/industries/retail/our-insights/how-retailers-can-keep-up-with-consumers) and [75% of Netflix‘s viewer activity](https://www.wired.co.uk/article/how-do-netflixs-algorithms-work-machine-learning-helps-to-predict-what-viewers-will-like) come from recommendations.

In this post, we‘ll dive deep into the world of recommendation systems, focusing on collaborative filtering (CF) techniques. We‘ll explore the theory behind popular algorithms, evaluate them using ranking metrics, and see how Python‘s LensKit library makes it easy to build powerful recommenders. Whether you‘re a data scientist looking to boost conversion rates or a developer fascinated by the magic of personalized recommendations, this guide will equip you with the knowledge and tools to build state-of-the-art systems. Let‘s get started!

## A Brief History of Recommender Systems

The concept of recommendation systems dates back to the early 1990s, with the rise of collaborative filtering. CF is based on the simple idea that users who agreed in the past are likely to agree in the future. If Alice and Bob both liked movies X and Y, and Alice also liked movie Z, then Bob is probably going to enjoy Z as well. Early CF systems like [GroupLens](https://dl.acm.org/doi/10.1145/192844.192905) and [Ringo](https://www.media.mit.edu/publications/ringo-an-experimental-interface-for-integrating-personal-information-spaces/) used nearest-neighbor algorithms to find similar users and make recommendations based on their ratings.

As datasets grew larger and sparser, memory-based CF ran into scalability issues. This led to the development of model-based approaches like matrix factorization (MF) in the late 2000s. MF algorithms learn latent user and item factors to predict unobserved ratings, allowing them to handle massive datasets. [Netflix‘s $1M Prize](https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429) in 2009 spawned widespread interest in MF techniques.

Recent years have seen the rise of deep learning approaches, which can learn complex non-linear interactions and integrate side information like item metadata and user demographics. Techniques like [neural collaborative filtering](https://arxiv.org/abs/1708.05031), [autoencoders](https://dl.acm.org/doi/10.1145/3178876.3185989), and [graph neural networks](https://arxiv.org/abs/1907.08259) have pushed the state-of-the-art on benchmark datasets. However, traditional MF-based approaches remain widely used in industry for their simplicity and interpretability.

## Evaluating Recommenders: Ranking Metrics

Before we jump into building recommenders, let‘s discuss evaluation. The choice of metric depends on the goal of the system. For example, Netflix might care about maximizing total viewing time, while Spotify aims to increase listening diversity. In general, ranking metrics are preferred over error metrics like RMSE, since users only see the top recommendations and aren‘t shown predicted ratings.

Normalized Discounted Cumulative Gain (nDCG) has become a gold standard for evaluating recommender systems. nDCG measures the quality of a ranking by assigning higher weights to items at the top of the list. The intuition is that users are more likely to interact with items ranked higher. Given a list of recommended items for user $u$, the Discounted Cumulative Gain (DCG) at position $p$ is calculated as:

$$DCG_p = \sum_{i=1}^p \frac{2^{rel_i} – 1}{\log_2(i+1)}$$

where $rel_i$ is the relevance score (e.g. user rating) of the item at position $i$. The ideal DCG for user $u$ is the maximum possible DCG given their known ratings:

$$IDCG_p = \sum_{i=1}^{|REL_u|} \frac{2^{rel_i} – 1}{\log_2(i+1)}$$

where $REL_u$ is the list of user $u$‘s known relevant items, ordered by descending relevance. Finally, the normalized DCG for user $u$ is:

$$nDCG_p = \frac{DCG_p}{IDCG_p}$$

To get the overall nDCG for a set of recommendations, we average the nDCG scores across all users.

Other common ranking metrics for recommenders include:

- Mean Average Precision (MAP): Measures the average precision of a ranking across all possible truncation levels. Precision is the fraction of recommended items that are relevant.
- Expected Reciprocal Rank (ERR): Similar to nDCG, but uses a cascade model of user behavior. The assumption is that users view items from top to bottom and stop at the first relevant item.
- Brier Score: Measures the calibration of predicted ratings. A well-calibrated model assigns higher probabilities to items the user is more likely to interact with.

With metrics in mind, let‘s see how LensKit can help us build and evaluate recommendation models.

## Building Recommenders with LensKit

[LensKit](https://lkpy.lenskit.org/) is a powerful Python toolkit for building and evaluating recommender systems. It provides a unified interface for training, generating recommendations, and evaluating different algorithms. LensKit supports both collaborative filtering (user-user, item-item) and matrix factorization approaches out of the box, with options for tuning model hyperparameters.

We‘ll walk through the process of building a movie recommender using the MovieLens 100K dataset, containing 100,000 ratings from 1000 users on 1700 movies.

### Step 1: Load and Inspect Data

First, let‘s load the ratings data using LensKit‘s data module:

```
from lenskit.datasets import ML100K

data = ML100K(‘data/ml-100k‘)
print(‘Number of ratings:‘, data.ratings.shape[0])
print(‘Number of unique users:‘, data.ratings.user.nunique())
print(‘Number of unique movies:‘, data.ratings.item.nunique())

data.ratings.head()
```

```
Number of ratings: 100000
Number of unique users: 943
Number of unique movies: 1682
```

We see that the `ratings` DataFrame contains 100,000 movie ratings from 943 unique users on 1682 films. The raw data is typically very sparse, with users only rating a small fraction of all possible movies.

### Step 2: Create Train/Test Splits

To evaluate our models, we need to split the data into training and testing sets. LensKit‘s `crossfold` module offers several splitting strategies:

```
from lenskit import crossfold as xf

train, test = xf.partition_users(data, 5, xf.SampleN(5))
```

Here, we use the `partition_users` function to create 5 folds of the data. For each user, a random sample of 5 ratings is withheld for testing, with the remaining ratings used for training. This ensures that every user is represented in both the train and test sets – important for evaluating personalized recommendations.

### Step 3: Build and Evaluate Models

Now for the fun part – building models! We‘ll compare two popular CF algorithms:

1. Item-based K-Nearest Neighbors (KNN)
2. Alternating Least Squares (ALS) Matrix Factorization

```
from lenskit.algorithms import Recommender, als, item_knn as knn

algo_knn = knn.ItemItem(20)
algo_als = als.BiasedMF(50)

all_recs = []

for algo_name, algo in [(‘KNN‘, algo_knn), (‘ALS‘, algo_als)]:
    fittable = util.clone(algo)
    fittable = Recommender.adapt(fittable)
    fittable.fit(train)

    users = test.user.unique()
    recs = batch.recommend(fittable, users, 100)
    recs[‘Algorithm‘] = algo_name
    all_recs.append(recs)

all_recs = pd.concat(all_recs, ignore_index=True)
```

Our `algo_knn` model uses item-based collaborative filtering to find movies similar to a user‘s highly-rated items. We set the number of neighbors to 20, so each movie‘s 20 most similar films (based on cosine similarity) are considered for recommendations.

`algo_als` uses matrix factorization to learn 50-dimensional latent factors for each user and movie. The factors are optimized to reconstruct the observed ratings using alternating least squares. We can then multipy the user and movie factor matrices to estimate unseen ratings.

To generate recommendations, we fit each algorithm to the training data using LensKit‘s `Recommender.adapt()` interface. This handles the boilerplate of creating a new instance for each fold. We then produce 100 recommendations per user with the `batch.recommend()` function, and store them along with the algorithm name for comparison.

Finally, let‘s evaluate our models using nDCG:

```
from lenskit.metrics import topn

ndcg_metric = topn.ndcg
eval_results = topn.evaluate(test, all_recs, metric=ndcg_metric)

print(‘Mean nDCG@10 by Algorithm:‘)
print(eval_results.groupby(‘Algorithm‘).ndcg.mean())
```

```
Mean nDCG@10 by Algorithm:
Algorithm
ALS    0.388
KNN    0.256
Name: ndcg, dtype: float64
```

Looking at the results, we see that ALS outperforms KNN with a mean nDCG@10 of 0.388 vs 0.256. This suggests that the latent factors approach is better able to capture user preferences and rank relevant movies higher. However, both scores leave significant room for improvement compared to a perfect ranking (nDCG=1).

We can easily swap in other ranking metrics using the same `topn.evaluate()` function. For example, to calculate mean average precision (MAP):

```
map_metric = topn.precision
eval_results = topn.evaluate(test, all_recs, metric=map_metric)

print(‘Mean MAP@10 by Algorithm:‘)
print(eval_results.groupby(‘Algorithm‘).precision.mean())
```

It‘s always a good idea to compare models across multiple metrics to get a holistic view of performance. We might find that certain algorithms perform better for different objectives (e.g. precision vs recall).

## Comparison with Other Libraries

While LensKit is a powerful toolkit for building recommenders in Python, it‘s not the only game in town. Other popular libraries include:

- [Surprise](http://surpriselib.com/): A Python scikit for building and evaluating recommender systems. Surprise supports a wide range of algorithms, from baseline models to state-of-the-art techniques like SVD++ and NMF. It also integrates with pandas DataFrames for easy data manipulation.
- [TensorRec](https://github.com/jfkirk/tensorrec): A TensorFlow recommendation system that focuses on implicit feedback datasets. TensorRec supports a variety of representation learning models, including matrix factorization, weighted matrix factorization, and generalized matrix factorization.
- [Microsoft Recommenders](https://github.com/microsoft/recommenders): A repository of best practices for building recommendation systems, including utility functions, state-of-the-art algorithms, and tools for serving models in production. The library integrates with Azure Machine Learning for easy experimentation and deployment.

So why choose LensKit? Some key advantages include:

1. Unified interface for training, generating recs, and evaluation
2. Support for both explicit and implicit feedback datasets
3. Wide range of built-in metrics for evaluating ranking quality
4. Modular design that allows for easy extension with custom algorithms and components
5. Detailed documentation and active community support

Ultimately, the best library depends on your specific use case and technology stack. It‘s worth experimenting with multiple options to see which one fits your needs.

## The Future of Recommenders: Challenges and Opportunities

As impressive as modern recommendation systems are, there‘s still plenty of room for improvement. Some key challenges and opportunities include:

1. Incorporating diversity and novelty: Recommenders can sometimes fall into "filter bubbles", only exposing users to narrow slices of content. Striking a balance between relevance and serendipity is an open challenge.
2. Handling cold-start users and items: When new users or items enter the system, there is little data to learn their preferences. Techniques like meta-learning and transfer learning can help bootstrap recommendations for cold-start scenarios.
3. Combating bias and discrimination: Recommenders can unintentionally amplify societal biases present in training data. Developing algorithms that are fair and inclusive is a major challenge, requiring interdisciplinary collaboration between technologists and domain experts.
4. Ensuring privacy and security: As recommenders handle sensitive user data, it‘s critical to develop systems that protect individual privacy. Techniques like differential privacy and federated learning allow for personalization while minimizing data sharing.
5. Explaining recommendations: Blackbox models can be difficult to interpret, leading to lack of transparency. Developing methods for generating human-understandable explanations can increase user trust and acceptance of recommendations.

As AI continues to advance, we can expect recommender systems to become even more sophisticated in understanding user preferences and needs. With the rise of new modalities like voice, video and virtual reality, recommenders will play a crucial role in helping users navigate ever-expanding digital worlds.

Ultimately, the goal of recommendation systems is to enrich people‘s lives by connecting them with personally meaningful content. By combining cutting-edge AI with human-centered design, we can build systems that don‘t just predict what users will like, but genuinely understand their aspirations, identities and values. The future of recommendations is not just about teaching computers to know us, but to help us better understand ourselves.

---

Source: [Building Powerful Recommendation Systems with LensKit in Python](https://33rdsquare.com/how-to-build-a-recommendation-system-using-lenskit-and-evaluate-it-using-ndcg-in-python/)