Interpreting Machine Learning Models using Shapley Values and Game Theory

Introduction

As machine learning models become more complex and are deployed in high-stakes applications, there is an increasing need to be able to interpret and explain the predictions of these "black box" models. Machine learning interpretability aims to shed light on how models make decisions and which features are most influential in driving predictions.

One promising approach for machine learning interpretability borrows concepts from cooperative game theory, namely Shapley values. In this article, we‘ll take a deep dive into Shapley values and demonstrate how they can be used to interpret machine learning models. We‘ll cover the theoretical foundations, walk through intuitive examples, and show how to use Shapley values in practice with Python code. By the end, you‘ll appreciate the power and flexibility of this game theory-based approach for explaining complex ML models.

Game Theory Basics

To understand Shapley values, it‘s helpful to first review some key concepts from game theory. At its core, game theory is a branch of mathematics that studies strategic interactions between rational decision-makers. It provides a framework for modeling how individuals or groups make decisions when the outcome depends on the choices of multiple agents.

Some key concepts in game theory include:

• Players: The decision-makers in the game
• Actions: The choices available to each player
• Payoffs: The rewards or penalties each player receives based on the actions of all players
• Strategies: A player‘s approach for making decisions, such as always choosing a certain action or randomizing between actions
• Equilibrium: A stable state where each player is making the best decision they can, given the decisions of the other players

Game theory has many applications, from economics and political science to computer science and psychology. By formalizing multi-agent interactions, it provides valuable insights into real-world strategic behaviors.

Cooperative Game Theory and Shapley Values

Within the realm of game theory, there are several distinct branches. One of these is cooperative (or coalitional) game theory, which models how groups of players can work together to achieve mutually beneficial outcomes. The focus is on what groups of players can achieve by cooperating, rather than on the specific actions of individual players.

A central solution concept in cooperative game theory is the Shapley value, named after mathematician Lloyd Shapley. In a cooperative game, the Shapley value assigns a unique distribution of payoffs to each player based on their average marginal contribution to all possible coalitions.

Intuitively, the Shapley value measures how much each player contributes to the overall cooperation, taking into account all the different ways the players can cooperate. Players who are more critical to achieving good outcomes (i.e. have higher marginal contributions) are assigned higher payoffs.

Formally, for a cooperative game with n players, the Shapley value for player i is calculated as:

Where:

• N is the total number of players
• S is a coalition (subset) of players not including player i
• v(S) is the value (or payoff) that coalition S can obtain on its own
• v(S ∪ {i}) is the value that coalition S can obtain if player i joins the coalition

This formula considers all possible coalitions, calculates player i‘s marginal contribution to each coalition, and averages these marginal contributions to get player i‘s overall Shapley value. It has several nice mathematical properties, such as efficiency (Shapley values sum to the total value of the grand coalition containing all players) and symmetry (players with equivalent marginal contributions receive the same Shapley value).

Shapley Values for Machine Learning Interpretability

While Shapley values originated in cooperative game theory, it turns out they are also highly useful for machine learning interpretability. The key idea is to view the features in a machine learning model as "players" in a cooperative game, with the model‘s prediction as the "payoff."

Each subset of features can be thought of as a coalition, and the Shapley value of a feature measures its average marginal contribution to the model‘s prediction across all possible coalitions. In other words, it quantifies how much the feature influences the prediction on average, considering all possible subsets of the other features.

Here‘s a step-by-step breakdown of how Shapley values can be used to interpret a machine learning model:

1. Define the "game" as the model‘s prediction task for a single data point. The "players" are the feature values for that data point.

2. Consider all possible subsets (coalitions) of features. For a model with n features, there are 2^n possible subsets.

3. For each subset of features, calculate the model‘s prediction. This corresponds to the value or payoff of that coalition.

4. For each feature, calculate its marginal contribution to the prediction for every possible coalition. This is the difference between the model‘s prediction with vs. without the feature, given the other features in the coalition.

5. Average the marginal contributions of each feature across all coalitions to get its Shapley value.

6. The Shapley values of all the features sum up to the difference between the model‘s prediction for the data point and the average prediction. This property, known as local accuracy, means Shapley values provide an additive explanation of the model‘s prediction.

7. Repeat this process for each data point to get feature attributions for the entire dataset. The global importance of each feature can be summarized by averaging absolute Shapley values across all data points.

Shapley values offer several advantages as an approach to machine learning interpretability:

• They provide both local explanations (for individual predictions) and global explanations (average feature importance across the dataset)
• They are theoretically grounded in game theory and have desirable mathematical properties
• They can be calculated for any type of machine learning model (model-agnostic)
• They consider interaction effects between features and conditional expectations

However, exact computation of Shapley values is computationally expensive, as it requires considering all possible feature subsets. In practice, approximation methods are used to efficiently estimate Shapley values.

Interpreting Machine Learning Models with SHAP

Shapley Additive Explanations (SHAP) is a popular Python package that implements Shapley values for machine learning interpretability. It provides a unified framework for interpreting predictions from any machine learning model and includes optimized implementations for different model types.

Let‘s walk through an example of how to use SHAP to interpret a machine learning model. We‘ll use a dataset of house prices and build a gradient boosting model to predict the price based on features like the number of rooms, square footage, and location.

First, we train the model:

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=3, random_state=42)
model.fit(X_train, y_train)

Next, we use SHAP to calculate feature attributions for the model‘s predictions on the test set:

import shap

explainer = shap.Explainer(model)
shap_values = explainer(X_test)

The shap_values object contains the Shapley values for each feature for each data point in the test set. We can visualize the feature attributions for a single prediction:

shap.plots.waterfall(shap_values[0])

This waterfall plot shows how each feature contributes to the model‘s prediction for the first data point in the test set, relative to the model‘s base value (average prediction). Red bars indicate features that push the prediction higher, while blue bars indicate features that push the prediction lower. The sum of all the feature attributions equals the difference between the model‘s prediction and the base value.

We can also get a global view of feature importance by averaging the absolute Shapley values for each feature across the entire dataset:

shap.plots.bar(shap_values)

The bar plot shows the average absolute Shapley values for each feature, ranking them from most to least important in driving the model‘s predictions.

SHAP also provides a variety of other visualizations, such as dependence plots to show how feature values relate to Shapley values, and interaction plots to highlight feature interactions. It is a powerful tool for both data scientists and business stakeholders to gain insights into complex machine learning models.

Conclusion

Shapley values offer a principled and flexible approach for interpreting machine learning models, by framing the prediction task as a cooperative game and assigning each feature a payoff based on its average marginal contribution. While the theory behind Shapley values is rooted in game theory, recent advances have made them practical to compute for real-world machine learning applications.

Tools like SHAP have made Shapley values accessible to data science practitioners and have been widely adopted in industry and academia. Some interesting applications of SHAP and Shapley values include:

• Identifying important risk factors in credit scoring models in finance
• Analyzing key biomarkers for disease diagnosis in healthcare
• Auditing fairness and bias in models used for high-stakes decisions like lending or hiring
• Understanding customer behaviors in marketing and recommender systems

As machine learning continues to expand into all domains, interpretability will only become more critical. Regulators, consumers, and other stakeholders are increasingly demanding transparency and explanations for algorithmic decisions. At the same time, research into interpretable machine learning is rapidly evolving, with new methods and refinements being developed.

Shapley values are a key pillar of this emerging field and are helping to bridge the gap between the predictive power of black box models and the interpretability needs of real-world applications. By combining rigorous theory with practical tools, they are enabling data scientists to build more transparent, accountable, and trustworthy machine learning systems.