Demystifying Model Interpretation using ELI5 in Python
Machine learning models have become increasingly complex in recent years, achieving impressive performance on challenging tasks in fields like computer vision, natural language processing, and predictive analytics. However, this complexity often comes at the cost of interpretability – many state-of-the-art models are essentially "black boxes", producing outputs based on complex internal logic that is difficult for humans to understand or explain.
This lack of interpretability can be problematic in many real-world applications of machine learning. In high-stakes domains like healthcare, finance, and criminal justice, it‘s important to be able to understand and explain how models arrive at their predictions in order to build trust, ensure fairness, and guard against unintended biases. Even in lower-stakes applications, the ability to interpret models can be valuable for debugging, model selection, feature engineering, and deriving insights.
Fortunately, the field of interpretable machine learning has developed a variety of techniques to "open up the black box" and shed light on how models work under the hood. In this post, we‘ll take a deep dive into model interpretation using the popular Python package ELI5. We‘ll cover what ELI5 is, how it works, and walk through detailed examples of using it to interpret different types of models.
What is Model Interpretation?
Before we jump into ELI5 specifically, let‘s first take a step back and define what we mean by model interpretation. At a high level, model interpretation refers to the process of explaining how a machine learning model works in a way that is understandable to humans. This can take a few different forms:
-
Global interpretation aims to explain the overall behavior of a model across its entire domain. This could involve analyzing the weights, parameters, or structure of a trained model to identify the most important features, understand the high-level patterns it has learned, and characterize its strengths and weaknesses.
-
Local interpretation focuses on explaining individual predictions made by a model. Rather than trying to understand the model as a whole, local interpretation techniques aim to clarify why the model made a particular prediction for a given input, often by identifying the specific features or aspects of the input that were most relevant.
Another important distinction is between model-agnostic and model-specific interpretation techniques:
-
Model-agnostic approaches aim to be generic and flexible enough to work with any type of machine learning model. They treat the model as a black box and interpret it based solely on its inputs and outputs, without relying on any knowledge of the model‘s internal structure or parameters. Model-agnostic techniques are very versatile, but may not be able to provide as much detail as model-specific ones.
-
Model-specific interpretation techniques are tailored to work with particular model architectures, such as linear models, decision trees, or neural networks. By exploiting knowledge of how the model is structured and trained, model-specific approaches can often extract more detailed and semantically meaningful information. However, they are less flexible than model-agnostic techniques.
With this background in mind, let‘s now turn our attention to ELI5 and how it fits into the model interpretation landscape.
Introducing ELI5
ELI5 (which stands for "Explain Like I‘m 5") is an open source Python package that provides a set of tools for visualizing and debugging various machine learning models. It aims to implement best practices for model interpretation in a simple, unified API that can work with many different types of models, including those in scikit-learn, Keras, XGBoost, and more.
Some of the key features and capabilities of ELI5 include:
- Inspecting model weights and feature importances
- Explaining individual predictions by showing how each feature contributed
- Examining decision paths and rules for tree-based models
- Computing permutation importances to measure feature relevance
- Visualizing results with built-in plotting utilities
- Supporting many model types (linear models, decision trees, ensembles, neural nets, etc.)
ELI5 provides both model-agnostic and model-specific functionality. It can extract global information like feature importances for any black-box model, while also providing more detailed explanations for individual predictions of specific model types like linear models and decision trees.
The core philosophy behind ELI5 is to provide a simple, intuitive, and visual approach to model interpretation that puts the human in the loop. Rather than just generating static tables of numbers, ELI5 aims to produce output that is easy to explore and interact with, even for non-experts.
Now that we have a high-level understanding of what ELI5 is and how it works, let‘s dive into some concrete examples of using it to interpret models in Python.
Inspecting Feature Importances with ELI5
One of the most basic but useful things ELI5 can do is compute and visualize global feature importances for a trained model. This tells us which features the model relies on most heavily in making its predictions, which is useful both for gaining insights into the problem domain and for identifying potential issues like irrelevant or redundant features.
ELI5 provides a simple function called explain_weights() that will inspect a trained model and return an explanation of its feature importances. Here‘s a basic example of how to use it with a scikit-learn Random Forest model:
from sklearn.ensemble import RandomForestClassifier
from eli5 import explain_weights
model = RandomForestClassifier()
model.fit(X_train, y_train)
explanation = explain_weights(model)
print(explanation)This will print out a text report showing the total weight assigned to each feature, along with a normalized relative importance score. The exact details of how the weights are computed depends on the model type – for random forests and other tree ensembles, ELI5 uses the feature importance scores derived from how often each feature is split on.
We can also generate a visual plot of the feature importances using ELI5‘s format_as_dataframe() function to convert the explanation to a Pandas DataFrame, which we can then plot with Matplotlib:
import matplotlib.pyplot as plt
weights_df = explain_weights(model).format_as_dataframe()
weights_df.plot.barh(x=‘feature‘, y=‘weight‘)
plt.xlabel(‘Weight‘)
plt.ylabel(‘Feature‘)
plt.show()Techniques like this provide a simple way to get an overall sense of what a model has learned, and are a good starting point for model interpretation. However, to really understand why a model makes the specific predictions it does, we need to dig deeper and look at local explanations.
Explaining Individual Predictions with ELI5
In addition to global model interpretation, ELI5 provides powerful tools for generating local explanations of individual model predictions. These explanations aim to show how each input feature contributed to a particular prediction, giving insight into why the model made the decision it did.
To explain a prediction, we can use the explain_prediction() function, passing in the trained model along with the input data for the instance we want to analyze. Here‘s a simple example:
from eli5 import explain_prediction
model = RandomForestClassifier()
model.fit(X_train, y_train)
instance = X_test[0] # choose an instance to explain
explanation = explain_prediction(model, instance)
print(explanation)The explanation shows a list of the input features and their corresponding contributions to the model‘s prediction. For tree ensembles like a random forest, this breaks down the decision path into the individual splits made at each node.
We can also generate a visual version of the explanation using the show_prediction() function:
from eli5.sklearn import explain_prediction_sklearn
show_prediction = explain_prediction_sklearn.show_prediction
show_prediction(model, instance)This will display an interactive HTML visualization of the prediction explanation that allows us to see exactly how each feature contributed to the output.
Local explanations like this are incredibly useful for understanding and debugging model behavior. They allow us to see not just what the model predicted, but why it made that prediction, which is crucial for building trust and identifying potential issues.
Visualizing Decision Trees with ELI5
For decision trees and tree ensembles, ELI5 provides an additional set of tools for visualizing and inspecting the internal structure of the trees themselves. This allows us to see the specific decision rules the model learned and gain insight into its overall logic.
To visualize a decision tree, we can use the explain_decision_tree() function:
from eli5.sklearn import explain_decision_tree
model = DecisionTreeClassifier()
model.fit(X_train, y_train)
explanation = explain_decision_tree(model)
print(explanation)This will print out a text-based representation of the decision tree, showing each split and the corresponding feature threshold. We can also generate a graphical version using the format_as_svg() method:
svg = explanation.format_as_svg()
with open(‘tree.svg‘, ‘wb‘) as f:
f.write(svg.encode(‘utf-8‘))The resulting SVG file will contain a visual depiction of the tree that clearly shows the decision path from the root downwards.
Being able to inspect the actual decision logic of a tree-based model in this way is incredibly powerful, especially in domains where interpretability is paramount. We can quite literally see the exact rules the model used to make its predictions.
Measuring Permutation Importances with ELI5
Another useful model interpretation technique supported by ELI5 is permutation importance. Permutation importance is a model-agnostic approach that measures how much the model‘s performance degrades when a given feature is randomly shuffled.
The idea is that if shuffling a feature causes a big drop in performance, that feature must be very important to the model. Conversely, features that can be shuffled without affecting performance are less relevant.
To compute permutation importances with ELI5, we can use the PermutationImportance class:
from eli5.sklearn import PermutationImportance
model = RandomForestClassifier()
model.fit(X_train, y_train)
perm_imp = PermutationImportance(model, random_state=123).fit(X_test, y_test)
explanation = explain_weights(perm_imp)
print(explanation)This will print out the feature importances derived from permutation, showing how much each feature contributes to the model‘s performance on the test set. Like the other explanations, we can also visualize these results graphically.
Permutation importance provides a different perspective on feature relevance compared to the built-in importance scores used by explain_weights(). It is especially useful for black-box models where we don‘t have access to internal model parameters.
Comparing Multiple Models with ELI5
One final useful feature of ELI5 is the ability to easily compare explanations across multiple models. This allows us to see how different models interpret the same inputs, which can be very useful for model selection and debugging.
To compare models, we can simply pass a list of models to the explain_weights() or explain_prediction() functions:
models = [
RandomForestClassifier(),
LogisticRegression(),
GradientBoostingClassifier()
]
for model in models:
model.fit(X_train, y_train)
explanations = [explain_weights(model) for model in models]
print(format_as_text(explanations))This will print out the global feature importances for each model side-by-side, allowing us to easily compare them. We can do the same thing for local prediction explanations:
instance = X_test[0]
explanations = [explain_prediction(model, instance) for model in models]
print(format_as_text(explanations))Being able to compare explanations like this is a powerful tool for understanding how different models see the problem, and for identifying potential issues or inconsistencies.
Conclusion
Model interpretation is a crucial part of the machine learning workflow, especially as models become more complex and are applied to more high-stakes problems. ELI5 provides a powerful and flexible toolkit for interpreting models in Python, with support for both global and local explanations across a variety of model types.
In this post, we covered some of the key features and use cases of ELI5, including:
- Inspecting feature importances
- Explaining individual predictions
- Visualizing decision trees
- Measuring permutation importances
- Comparing multiple models
We walked through concrete examples of using ELI5 to interpret different types of models, and discussed the various insights and debugging capabilities it provides.
Of course, ELI5 is just one of many tools available for model interpretation, and it has its own strengths and limitations. In particular, the level of detail and specificity it can provide is inherently limited by the black-box nature of many models.
For richer, more granular interpretation capabilities, tools that make use of model-specific information (e.g. gradients and activations in neural networks) are required. Techniques like LIME, SHAP, and Integrated Gradients can be used to generate more sophisticated local explanations, while activation visualizations and concept-based explanations can shed light on the internal representations learned by deep neural networks.
That said, ELI5 is a great starting point for model interpretation that is simple to use, flexible to work with a variety of models, and produces human-friendly visualizations and explanations. It‘s a valuable tool in the toolkit of any data scientist or ML practitioner.
To learn more about ELI5 and model interpretation, check out the following resources:
- The ELI5 documentation: https://eli5.readthedocs.io/en/latest/
- The Interpretable Machine Learning Book: https://christophm.github.io/interpretable-ml-book/
- The InterpretML repo: https://github.com/interpretml/interpret
I hope this post has helped demystify model interpretation and showed you how ELI5 can be used to better understand and debug machine learning models! Happy interpreting!
