Shapash: Illuminating the Black Box of Machine Learning Models

Machine learning (ML) has become an indispensable tool across industries, automating decisions in everything from loan approvals to medical diagnoses to content recommendations. However, the predictive power of ML models often comes at the cost of interpretability. Many state-of-the-art approaches, such as deep learning and ensemble methods, are essentially "black boxes" – while they can achieve impressive accuracy, it‘s difficult to explain how they arrive at their outputs.

This lack of transparency can be a major barrier to adopting ML, especially in regulated domains like finance and healthcare. According to a recent survey by H2O.ai, nearly 90% of organizations report that explainability and interpretability are critical factors when selecting an ML platform^1. Another study by IBM found that 84% of C-suite executives believe being able to explain and show the fairness of AI decisions is important to their business^2.

Responding to this need, researchers and practitioners have developed a variety of techniques and tools to crack open the black box of ML models. One promising solution is Shapash, an open source Python library that provides a unified interface for explaining models from popular frameworks like scikit-learn, XGBoost, LightGBM, and others. Developed by data scientists at the French insurance company MAIF, Shapash aims to make interpretable machine learning more accessible with an intuitive API and interactive visualizations.

How Shapash Works

At its core, Shapash uses a combination of local and global explanation methods to help users understand model behavior at both the individual prediction and overall dataset levels.

The key techniques Shapash employs include:

• SHAP (SHapley Additive exPlanations): SHAP is a game theoretic approach that assigns each feature an importance value for a particular prediction. It considers all possible combinations of features and how the model output changes when a feature is included vs excluded[^3]. Shapash uses SHAP to break down a single data point and see the positive and negative contributions of each attribute.

• Permutation Feature Importance: This method measures the increase in model error after randomly shuffling each feature. Features are considered "important" if model performance decreases substantially when their values are permuted[^4]. Shapash applies permutation importance to identify the most influential variables across the entire dataset.

• Partial Dependence Plots (PDP): PDP shows the marginal effect of a feature on the model output, averaging out the effects of all other features. It helps uncover non-linear relationships and interactions between variables[^5]. Shapash uses PDP to visualize how changing the value of a single feature impacts predictions.

• Sensitivity Analysis: Also known as "what-if" analysis, sensitivity analysis involves varying the values of input features and observing how model outputs change in response. It‘s useful for understanding the stability and robustness of a model. Shapash allows users to perform sensitivity checks through an interactive web app.

Shapash unifies these diverse explainability techniques into a single, user-friendly package. With just a few lines of code, users can generate a wealth of visualizations and metrics to debug their models, validate their performance, and communicate insights to stakeholders.

Comparing Shapash to Other Interpretability Libraries

Shapash is not the only tool aiming to make ML models more interpretable. Other open source frameworks include:

Library	Key Features	Use Cases
LIME	Local interpretable model-agnostic explanations	Explaining individual predictions of any black box model
InterpretML	Glassbox and blackbox explainers from Microsoft	Interpretability for enterprise customers, integrates with Azure ML
Alibi	Black box and white box model inspection from Seldon	Explaining models deployed on the Seldon platform for MLOps
Skater	Model agnostic interpretation library from Oracle	General purpose explainability, similar scope as Shapash
tf-explain	Interpretability methods for TensorFlow models	Targeted at deep learning use cases

While all of these tools share the goal of demystifying ML models, Shapash differentiates itself in a few key ways:

1. Unified interface: Shapash provides a consistent API for multiple explanation methods, making it easy to experiment with different approaches.

2. Integration with common ML frameworks: Shapash works out of the box with models built using popular libraries like scikit-learn and XGBoost.

3. Interactive web app: Shapash includes a customizable GUI for exploring model explanations, facilitating collaboration between data scientists and business stakeholders.

4. Feature labeling: Shapash allows users to attach custom, human-readable descriptions to raw feature names for clearer communication.

Interpreting Models in the Real World with Shapash

To illustrate the value of Shapash in practice, let‘s consider a few case studies across different industries.

Healthcare

ML is increasingly being applied to medical imaging, disease diagnosis, and treatment planning. However, the high stakes nature of healthcare means that models must be carefully vetted before being put into production.

One area where interpretability is critical is in predicting patient outcomes. Suppose a hospital system trained a model to forecast which patients are at risk of being readmitted within 30 days of discharge. With Shapash, doctors and hospital administrators could analyze the key factors driving a particular patient‘s risk score, such as their age, vital signs, and medical history. This information could help suggest interventions to prevent an unnecessary readmission.

Finance

In the banking sector, ML models are used for everything from fraud detection to credit scoring. However, regulations like the Fair Credit Reporting Act and the Equal Credit Opportunity Act require that lenders are able to explain the reasons for denying a loan.

Shapash could help banks and fintechs build more transparent credit decisioning systems. For each applicant, they could generate a report showing how factors like income, employment status, and credit utilization influenced the model‘s output. This would help back offices handle customer inquiries and facilitate fairer lending practices.

Human Resources

Many companies are turning to ML to streamline hiring and promotion decisions. Models can be trained to score resumes, analyze job interviews, and predict employee performance and turnover.

Without careful auditing, however, these systems run the risk of perpetuating historical biases and discrimination. With Shapash, HR teams could inspect models for fairness, seeing if they put too much weight on demographic characteristics like gender or race. They could also compare different modeling approaches to find the ones that are both accurate and unbiased.

The Future of Interpretable Machine Learning

While libraries like Shapash have made significant progress in promoting model transparency, there is still much work to be done to achieve truly interpretable AI systems. Some active areas of research include:

• Causal interpretability: Going beyond correlations to uncover the causal relationships driving model predictions. Techniques like causal Shapley values and counterfactual explanations aim to answer "what if" questions and reason about interventions[^6].

• Example-based explanations: Generating human-understandable explanations by selecting representative examples from the training data. This includes methods like prototype selection and criticisms that illustrate a model‘s decision boundary[^7].

• Interpretability for deep learning: Developing new architectures and training procedures to make deep neural networks more interpretable. Approaches include attention mechanisms, sparse activations, and interpretable layer-wise relevance propagation[^8].

• Natural language explanations: Automatically generating textual explanations of model reasoning, similar to how a human would justify a decision. Work in this area involves combining natural language processing with visualization and rule extraction techniques[^9].

As interpretable ML techniques become more sophisticated, we can expect to see them integrated into end-to-end model development and deployment workflows. Just as MLOps platforms have sprung up to streamline the machine learning lifecycle, we may see the emergence of "XAIOps" tools focused on continuously auditing models for explainability, fairness, and robustness.

In the near term, libraries like Shapash are making interpretability more accessible to data scientists and ML engineers. By providing a unified interface to various explanation methods, Shapash allows practitioners to easily debug their models, build trust with stakeholders, and promote more responsible AI development.

While there is no one-size-fits-all approach to interpretability, Shapash is a valuable addition to any machine learning toolkit. As the field continues to evolve, it will be exciting to see how Shapash and other projects shape the future of transparent, accountable, and understandable AI.

References

[^3]: Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems (pp. 4765-4774).
[^4]: Breiman, L. (2001). Random forests. Machine learning, 45(1), 5-32.
[^5]: Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of statistics, 1189-1232.
[^6]: Heskes, T., Sijben, E., Bucur, I. G., & Claassen, T. (2020). Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. Advances in Neural Information Processing Systems, 33.
[^7]: Kim, B., Khanna, R., & Koyejo, O. O. (2016, December). Examples are not enough, learn to criticize! Criticism for Interpretability. In NIPS (pp. 2288-2296).
[^8]: Montavon, G., Samek, W., & Müller, K. R. (2018). Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73, 1-15.
[^9]: Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016). Generating visual explanations. In European Conference on Computer Vision (pp. 3-19). Springer, Cham.