Shedding Light on Black Box Models: Global Model Interpretability Techniques

Machine learning models have become increasingly complex in recent years, with many state-of-the-art models like deep neural networks and boosted trees being essentially "black boxes." While these models can achieve impressive predictive performance, their lack of transparency is a major drawback, especially in high-stakes domains like healthcare, finance, and criminal justice. When a model is making critical decisions that impact people‘s lives, we need to be able to understand how it arrives at its predictions.

This is where model interpretability comes in. Interpretability refers to the degree to which a human can understand the cause of a model‘s decision. We can break this down further into global interpretability and local interpretability. Global interpretability aims to explain the model‘s behavior as a whole, across all possible inputs. Local interpretability focuses on explaining an individual prediction.

In this post, we‘ll dive into some powerful techniques for achieving global interpretability of black box machine learning models. While this is still an active area of research with many open challenges, these techniques provide a good starting point for peeking inside the black box. Let‘s get started!

Feature Importance

One of the most fundamental questions we might ask about a model is: which features have the biggest impact on the predictions? Feature importance methods aim to quantify the contribution of each input feature to the model‘s output.

Permutation Importance

Permutation feature importance measures how much the model‘s predictive accuracy decreases when a single feature is randomly shuffled. The idea is that if the model heavily relies on a particular feature, then permuting its values should cause a big drop in performance.

To compute permutation importance, we follow these steps:

1. Get a trained model
2. Shuffle the values of a single feature, leaving the other features unchanged
3. Make predictions using the shuffled data and calculate the error
4. Repeat steps 2-3 for each feature
5. Features with higher permutation importance scores are more important to the model

Here‘s an example of calculating permutation importance with Python‘s scikit-learn library:

import numpy as np
from sklearn.inspection import permutation_importance

perm_importance = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=0)

features = X_test.columns
importances = perm_importance.importances_mean
std = perm_importance.importances_std

indices = np.argsort(importances)[::-1]

for f in range(X_test.shape[1]):
    print("%d. feature %s (%f)" % (f + 1, features[indices[f]], importances[indices[f]]))

Permutation importance has some nice properties: it‘s model-agnostic (you can apply it to any model), it captures the impact of each feature in the context of all the other features, and it‘s easy to interpret. However, it does have some limitations. Permutation doesn‘t account for feature interactions, and it can be computationally expensive, especially if you increase the number of repeats.

SHAP Values

SHAP (SHapley Additive exPlanations) is a game-theoretic approach to explaining the output of any machine learning model. The goal of SHAP is to explain the prediction of an instance x by computing the contribution of each feature to the prediction. The SHAP explanation method computes Shapley values from coalitional game theory. The feature values of a data instance act as players in a coalition. Shapley values tell us how to fairly distribute the "payout" (the prediction) among the features.

A key benefit of SHAP is that the Shapley values are the only possible locally accurate and globally consistent feature attribution values. This means:

• Local accuracy: the sum of the feature attributions is equal to the output of the function we are seeking to explain
• Global consistency: lets the SHAP values provide global interpretability by summing the attributions across the entire dataset

Here‘s how to compute SHAP values with the shap Python library:

import shap

# compute SHAP values
explainer = shap.Explainer(model)
shap_values = explainer(X_test)

# plot global SHAP values
shap.plots.bar(shap_values) 

SHAP values provide a theoretically justified, model-agnostic approach to global interpretability, but there are some challenges. Computing exact Shapley values is NP-hard in general, so the shap library uses approximation methods that can still be quite slow. The plots can also be more difficult to understand compared to permutation importance.

Gain and Split Importances

For tree-based models like random forests and gradient boosted trees, there are some special feature importance measures available. Gain importance (also called gini importance or mean decrease impurity) measures the average gain of the feature when it is used in trees. The gain of a feature is the improvement in the split criterion (e.g. gini impurity) when the feature is used to split a node, weighted by the number of samples that reach the node. A feature is more important if it frequently appears in trees with high gain.

Split importance simply counts the number of times a feature is used to split the data across all trees. The more often a feature is used, the more important it is assumed to be.

In Python, you can easily extract these importance scores from trained scikit-learn random forest or gradient boosting models:

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier()
model.fit(X_train, y_train)

for i, feature in enumerate(X_train.columns):
    print(f‘{feature}: {model.feature_importances_[i]}‘)

Both gain importance and split importance are fast to compute and directly measure a feature‘s contribution to the model structure. However, they are biased towards high cardinality features and those with many missing values. They may not reflect a feature‘s true importance when strong interactions are present.

Visualizing Feature Effects

While feature importances tell us what variables most affect predictions, they don‘t tell us how the features affect the predictions. Partial dependence plots (PDPs) and accumulated local effects (ALEs) are popular techniques for visualizing a feature‘s effect on the predicted outcome.

Partial Dependence Plots

A partial dependence plot shows the marginal effect one or two features have on the predicted outcome of a machine learning model. A partial dependence plot can show whether the relationship between the target and a feature is linear, monotonic or more complex.

To calculate the partial dependence of a feature f_i, we:

1. Create a mesh grid containing all possible values for f_i
2. For each unique value of f_i, replace all instances of f_i in the test set with that value
3. Get the model‘s prediction for each manipulated instance
4. Calculate the average prediction across all instances for each unique value of f_i
5. Plot f_i vs the average predictions

Here‘s how to create a PDP using scikit-learn:

from sklearn.inspection import PartialDependenceDisplay

PartialDependenceDisplay.from_estimator(model, X_test, [i])

PDPs are intuitive to understand and fast to compute for a single feature. However, they can be computationally expensive for multiple features and may be biased by unrealistic feature combinations. They also assume that the features for which the partial dependence is computed are not correlated with other features, which is often not true in real-world datasets.

Accumulated Local Effects

Accumulated local effects (ALE) plots were proposed to address some of the shortcomings of PDPs. Rather than averaging the predictions over the feature distribution, ALE plots average and accumulate the changes in the predictions. This makes ALE plots faster to compute and unbiased even when features are correlated.

To calculate the accumulated local effect of a feature f_i, we:

1. Divide the range of f_i into intervals
2. Within each interval, compute the difference in predictions when f_i changes from the upper to the lower limit of the interval, while keeping other features fixed
3. Average these differences over all instances in the dataset
4. Accumulate these average differences across all intervals
5. Center the accumulated differences so they sum to zero
6. Plot f_i vs the centered accumulated differences

Currently, there is no stable Python library for computing ALE plots, but there are some in development.

ALEs provide a faster, unbiased alternative to PDPs for visualizing feature effects, but they can still be difficult to interpret, especially for non-technical stakeholders.

Identifying Feature Interactions

Features in a machine learning model often interact with each other, meaning the effect of one feature depends on the value of another feature. Identifying and quantifying these interactions is crucial for a complete understanding of the model‘s behavior.

Friedman‘s H-Statistic

Friedman‘s H-statistic measures the interaction strength between two features. It‘s based on the idea of partial dependence functions. If two features interact, the partial dependence function of one feature will be different depending on the value of the other feature.

To compute the H-statistic for features x_i and x_j:

1. Estimate the partial dependence functions for x_i at different values of x_j
2. Calculate the variance of the partial dependence function for x_i across the different values of x_j
3. Normalize this variance by the variance of the partial dependence function for x_i not conditioned on x_j

An H-statistic of 0 indicates no interaction, while an H-statistic of 1 indicates a complete interaction (the effect of x_i completely depends on x_j). The H-statistic is symmetric, so H(x_i, x_j) = H(x_j, x_i).

Friedman also proposed a test statistic to evaluate whether the H-statistic differs significantly from zero, allowing us to test the null hypothesis of no interaction.

The H-statistic provides a simple, interpretable measure of pairwise feature interactions, but it can be computationally intensive to compute for all pairs of features in a high-dimensional dataset.

Limitations and Future Directions

While the techniques we‘ve covered provide valuable insights into the global behavior of black box models, they do have some limitations:

• They can be computationally expensive, especially for large datasets and complex models.
• They often assume feature independence, which may not hold in practice.
• They can be difficult to interpret for non-technical stakeholders.
• They may not capture all types of feature interactions or higher-order effects.

Moreover, as machine learning models continue to grow in size and complexity, manual interpretation of these models may become infeasible. This has led to a growing interest in automated machine learning (AutoML) systems that can automatically train, tune, and interpret models.

Some exciting future directions in model interpretability include:

• Developing more efficient, scalable algorithms for computing global interpretation measures
• Creating user-friendly visual interfaces for exploring model behavior
• Integrating interpretability techniques into the model training process itself (e.g. through regularization or constraints)
• Automating the selection and application of appropriate interpretability techniques based on the model type, data characteristics, and user requirements

Conclusion

Black box machine learning models are increasingly being used to make high-stakes decisions, making model interpretability a critical concern. Global interpretation techniques like feature importance, partial dependence plots, accumulated local effects, and interaction detection provide valuable insights into a model‘s overall behavior.

However, interpretation of complex models remains a challenging, multifaceted problem. As we continue to push the boundaries of machine learning performance, we must also strive to develop more sophisticated, automated, and user-friendly methods for understanding these powerful but opaque models. Only then can we truly harness the potential of machine learning while ensuring transparency, fairness, and accountability.