A Beginner‘s Guide to Machine Learning Explainability

Machine learning models have become increasingly complex and are often considered "black boxes", meaning it‘s difficult to understand how they make predictions. However, being able to interpret and explain these models is critical, especially in high-stakes domains like healthcare, finance, and criminal justice. This has led to a growing field called machine learning interpretability or explainable AI (XAI).

In this beginner‘s guide, we‘ll provide an overview of machine learning explainability, discuss why it matters, and dive into some of the most popular techniques. We‘ll focus on intuitive explanations and provide Python code snippets to illustrate key concepts. By the end, you‘ll have a solid foundation for interpreting even the most complex models.

Why Interpretability Matters

Black box models like deep neural networks, random forests, and gradient boosted trees are immensely powerful but can be difficult to interpret. A complex neural network might have hundreds of layers and millions of parameters. While these models can achieve state-of-the-art performance, debugging them or understanding their predictions is challenging.

There are several reasons why interpretability is important:

1. Debugging and improving models: Understanding how models make predictions is key to finding and fixing errors, as well as identifying potential improvements.

2. Complying with regulations: In regulated industries like finance and healthcare, being able to explain model predictions is often a legal requirement.

3. Avoiding bias: Black box models can inadvertently learn societal biases present in training data. Interpretability techniques can uncover these biases.

4. Building trust with users: End users are more likely to trust and adopt models that provide clear explanations for their predictions.

5. Advancing scientific understanding: Interpretable models can lead to novel insights and scientific discoveries in fields like medicine and biology.

With the importance of interpretability in mind, let‘s look at some popular techniques.

Permutation Feature Importance

One of the simplest and most widely used interpretability techniques is permutation feature importance. The key idea is to measure how much the model‘s performance decreases when a single feature is randomly shuffled. This breaks the relationship between the feature and the target, so a large drop in performance indicates that the feature is important.

Here are the steps to calculate permutation feature importance:

1. Train a model on the original dataset and measure its performance
2. For each feature:
   a. Shuffle the feature values to break the relationship with the target
   b. Make predictions using the shuffled data and calculate performance
   c. Record the difference in performance between the shuffled and original data
3. Sort features by their importance scores

This process can be repeated multiple times to get more stable importance estimates.

Here‘s how you can implement permutation importance in Python using the sklearn library:

from sklearn.inspection import permutation_importance

# Train model
model = xgboost.XGBRegressor().fit(X_train, y_train) 

# Calculate permutation importances
importances = permutation_importance(model, X_test, y_test,
                                     n_repeats=10,
                                     random_state=0)

# Print feature importances
for i in importances.importances_mean.argsort()[::-1]:
    print(f"{X.columns[i]:<20}" 
          f"{importances.importances_mean[i]:.3f}"
          f" +/- {importances.importances_std[i]:.3f}")

The main advantage of permutation importance is that it‘s model-agnostic, meaning it can be applied to any model that makes predictions. You can use it with neural networks, random forests, or any other black box model.

One downside is that it doesn‘t provide detailed insights into how each feature affects the predictions. It only gives you a high-level view of what‘s important. Additionally, permutation importance can be biased by unrealistic data instances if a feature is shuffled to a value that would not occur naturally.

Global Surrogate Models

Another popular interpretability technique is to train a simple, interpretable model to mimic the predictions of a black box model. This is known as a surrogate model or oracle model.

The steps are:

1. Train a complex model on the original dataset
2. Generate predictions from the complex model
3. Train a simple model (e.g. linear regression, decision tree) to predict the outputs of the complex model

The surrogate model is an approximation of the original model but is much easier to interpret. You can examine the coefficients of a linear model or visualize the decision tree to get insights into how the black box model makes predictions.

Here‘s an example of training a decision tree surrogate to interpret a neural network:

from sklearn.neural_network import MLPClassifier
from sklearn.tree import DecisionTreeClassifier

# Train neural network
nn = MLPClassifier().fit(X_train, y_train)

# Generate predictions
nn_preds = nn.predict(X_test)

# Train surrogate decision tree
surrogate = DecisionTreeClassifier(max_depth=3).fit(X_test, nn_preds) 

# Visualize surrogate tree  
plot_tree(surrogate)

Surrogate models provide a global, approximate interpretation of the original model. The main advantages are that they are simple to understand and can be applied to any black box model.

However, there are a few caveats. Surrogate models are an approximation and may not capture all the nuances and nonlinearities of the original model. There‘s also no perfect metric for assessing how well the surrogate mimics the original model. A surrogate that‘s 90% accurate might be "good enough" in some cases but insufficient in others.

Local Interpretable Model-Agnostic Explanations (LIME)

Unlike surrogate models which provide global interpretations, LIME focuses on explaining individual predictions. The key idea is to fit a simple, interpretable model around a single data point to explain how the black box model made that specific prediction.

Here are the steps:

1. Select a data point to explain
2. Perturb the data point to create many slight variations
3. Get predictions for the variations using the black box model
4. Train a simple model (e.g. linear regression) to predict the black box model predictions using the perturbed data
5. Extract feature importances from the simple model to explain the original prediction

The simple model acts as a local approximation of how the black box model behaves in the vicinity of the data point being explained. The feature importances tell us what features the black box model focused on for this particular example.

Here‘s how to use LIME for explaining a single prediction:

import lime
import lime.lime_tabular

explainer = lime.lime_tabular.LimeTabularExplainer(
    training_data=X_train, 
    feature_names=X.columns,
    class_names=[‘price‘],  
    mode=‘regression‘
)

exp = explainer.explain_instance(
    data_row=X_test.iloc[0], 
    predict_fn=model.predict
)

exp.show_in_notebook(show_table=True)

The explain_instance method perturbs a single data point, gets black box model predictions, fits a local surrogate model, and extracts feature importances. The show_in_notebook method displays the results.

LIME is a powerful technique because it provides explanations for individual predictions in a model-agnostic way. The perturbations can uncover surprising insights into how complex models behave.

One limitation of LIME is instability, meaning slightly different perturbations can sometimes lead to very different explanations. LIME also assumes that features are independent, so it can struggle with datasets that have strongly correlated features.

Shapley Values

Shapley values are a technique from cooperative game theory that have been adapted to explain machine learning models. The key idea is to fairly distribute the total gain (or loss) among the features based on their contributions.

To calculate Shapley values:

1. Consider all possible subsets of features
2. For each subset, calculate the model‘s prediction with those features included vs excluded
3. The Shapley value for a feature is its average marginal contribution across all subsets

Shapley values have several desirable properties. They always sum up to the total gain (or loss), meaning they provide a complete accounting of the model‘s prediction. They also satisfy other fairness properties like symmetry (features that contribute equally get the same value) and linearity.

Here‘s how to calculate Shapley values using the shap library:

import shap

explainer = shap.Explainer(model)
shap_values = explainer(X_test)

# Visualize Shapley values
shap.plots.waterfall(shap_values[0])

The shap library provides an efficient implementation to approximate Shapley values for machine learning models. The waterfall plot shows how each feature pushes the model‘s prediction higher or lower.

While Shapley values have solid theoretical foundations, they can be computationally expensive since the number of possible subsets grows exponentially with the number of features. The shap library uses approximations to handle this, but the results are not always identical to the true Shapley values.

Partial Dependence Plots

Partial dependence plots (PDPs) show how a feature affects the model‘s predictions on average. They help visualize the relationship between a feature and the predictions while accounting for the effects of all the other features.

To calculate partial dependence:

1. Select a feature
2. For each unique value of the feature:
   a. Make a copy of the data and set the feature to that value for all examples
   b. Get the model‘s predictions for the modified data
   c. Average the predictions
3. Plot the feature values vs. the average predictions

PDPs are a global method, showing the average effect of a feature across the entire dataset. They are useful for detecting whether a feature has a linear, monotonic, or more complex relationship with the predictions.

Here‘s how to create a PDP using sklearn:

from sklearn.inspection import partial_dependence

# Calculate partial dependence
pdp, axes = partial_dependence(model, X, features=[0, 1], 
                               percentiles=(0.05, 0.95))

# Plot partial dependence
fig, ax = plt.subplots()
ax.plot(axes[0], pdp[0].ravel())
ax.set_xlabel(‘Feature 0‘)
ax.set_ylabel(‘Partial dependence‘)

This code calculates the partial dependence of the model on the first two features of X and plots the PDP for the first feature.

One limitation of PDPs is that they assume features are independent. If there are interactions between features, PDPs can be misleading. They also don‘t show the distribution of the actual data, so it‘s possible to interpret PDPs in regions where there is little or no data.

Other Techniques and Considerations

We‘ve covered some of the most widely used explainability techniques, but there are many others worth exploring:

• Individual Conditional Expectation (ICE) plots: Like PDPs but show individual curves for each data point
• Accumulated Local Effects (ALE) plots: Similar to PDPs but handle correlated features better
• Feature interaction plots: Show how the effect of one feature depends on the value of another
• Feature ablation: Measuring model performance after removing features
• Integrated Gradients: Attributing predictions to input features by integrating gradients

An important distinction is between model-agnostic and model-specific techniques. Model-agnostic methods like LIME, Shapley values, and permutation importance can be applied to any model, while model-specific techniques are tailored to a particular algorithm (e.g. interpreting the coefficients of a linear model or extracting decision rules from a tree).

Another key consideration is the scope of the explanations. Some techniques provide global insights about the model‘s behavior across the entire dataset, while others focus on explaining individual predictions (local explanations). Both are valuable and the appropriate choice depends on the use case.

It‘s also worth noting that while interpretability techniques are powerful, they are not a silver bullet. Most techniques come with assumptions and limitations. Some can be sensitive to small perturbations in the data or model while others only provide approximate explanations. It‘s important to understand these limitations and not over-interpret the results.

In real-world applications, explainability is often used in conjunction with other tools like model validation, error analysis, and human oversight. Interpretability techniques can surface potential issues and build trust, but they are not a substitute for rigorous testing and monitoring.

Conclusion

We‘ve covered a lot of ground in this guide to machine learning interpretability. We discussed the importance of explainability, explored techniques like permutation importance, LIME, and Shapley values, and delved into key considerations like local vs. global and model-agnostic vs. model-specific approaches.

The field of interpretable machine learning is rapidly evolving, with new techniques and applications emerging all the time. As models become more complex and impactful, the ability to explain their predictions will only become more critical.

To continue your learning journey, we recommend diving deeper into the techniques we covered, exploring additional methods, and most importantly, applying these techniques to real datasets and models. Hands-on experience is invaluable for building intuition and uncovering insights.

We hope this guide has provided a solid foundation and piqued your interest in this important and exciting field. Happy explaining!

Additional Resources

• Book: "Interpretable Machine Learning" by Christoph Molnar
• Book: "Explanatory Model Analysis" by Przemyslaw Biecek and Tomasz Burzykowski
• Python libraries: lime, shap, interpret, eli5
• Course: "Interpretable Machine Learning" on Coursera
• Paper: "A Unified Approach to Interpreting Model Predictions" (introduces SHAP values)
• Paper: ""Why Should I Trust You?": Explaining the Predictions of Any Classifier" (introduces LIME)