Streamlining Machine Learning Model Evaluation with Yellowbrick Visualizations
As machine learning becomes increasingly prevalent across industries, it‘s more important than ever for data scientists and ML engineers to efficiently build high-quality models. A key part of the model development process is evaluating model performance and comparing multiple models to select the best one. While numeric performance metrics are crucial, visualizing model behavior provides additional insights to guide model selection and iteration.
Yellowbrick is an open source Python library that aims to make it as easy as possible to generate visualizations for model evaluation. Yellowbrick provides a variety of visualizers for analyzing model performance, interpreting predictions, and comparing models across model types like regression, classification and clustering. It offers a concise API that integrates tightly with scikit-learn, enabling data scientists to generate informative visualizations with minimal code.
In this post, we‘ll dive into how to use Yellowbrick to streamline evaluation of machine learning models. We‘ll focus on regression models and walk through an example use case of predicting housing prices. Along the way, we‘ll see how Yellowbrick‘s visualizers can help diagnose issues with model fit, guide model selection, and interpret the importance of different features.
While we‘ll use regression as our example, the concepts and techniques we‘ll cover are applicable across model types. By the end of this post, you‘ll be equipped to apply Yellowbrick to your own machine learning projects to iterate faster and build better models.
Why Visualize Model Performance?
Evaluating a machine learning model‘s predictions and performance is a key step of the model development process. Most commonly, data scientists measure model performance using numeric evaluation metrics relevant to the prediction task, like:
- Mean absolute error, mean squared error, and R-squared for regression
- Accuracy, precision, recall, and F1 score for classification
- Silhouette score and inertia for clustering
While numeric metrics are important for comparing models and tracking progress, they don‘t always tell the full story. Visualizing model performance provides a complementary view that can surface additional insights. Some key benefits of visualizing model performance include:
- Identifying outliers or subgroups of data where the model performs poorly
- Checking assumptions of the model (like linearity for linear regression)
- Diagnosing overfitting or underfitting
- Comparing performance across different model types or hyperparameter configurations
- Analyzing the relative importance of different features
- Conveying model behavior to non-technical stakeholders
Historically, generating these visualizations required using plotting libraries like matplotlib or seaborn and writing substantial amounts of custom code. Yellowbrick aims to simplify this process by providing a unified interface for model visualizations that integrates with the scikit-learn API.
Installing and Using Yellowbrick
Yellowbrick can be installed via pip:
pip install yellowbrickIt has dependencies on scikit-learn and matplotlib, which will be installed automatically if not already present.
The key concept in Yellowbrick‘s API is the Visualizer – a wrapper around a scikit-learn model that generates a visualization of the model‘s behavior. Visualizers are instantiated with the model instance and then follow the same fit/predict pattern as the underlying scikit-learn model.
For example, here‘s how we can visualize residuals for a linear regression model with Yellowbrick:
from sklearn.linear_model import LinearRegression
from yellowbrick.regressor import ResidualsPlot
# Instantiate the model and visualizer
model = LinearRegression()
visualizer = ResidualsPlot(model)
# Fit the model and visualizer on training data
visualizer.fit(X_train, y_train)
# Score the model on test data and show the visualization
visualizer.score(X_test, y_test)
visualizer.show()Under the hood, the ResidualsPlot visualizer wraps the LinearRegression model and calls the model‘s fit() and predict() methods to generate residuals. It then renders an interactive plot of the residuals using matplotlib.
This is the core usage pattern for all visualizers in Yellowbrick – initialize the visualizer with the model, call fit() and score(), then show() the visualization.
Regression Model Evaluation with Yellowbrick
Now let‘s see how we can use Yellowbrick to evaluate regression models for a real-world dataset. We‘ll use the Housing Prices dataset from Kaggle, which contains 79 features describing residential homes and the goal is to predict the sale price.
Preparing the Data
After downloading the data, we can load it into a pandas DataFrame:
import pandas as pd
df = pd.read_csv(‘housing_prices.csv‘)Next we‘ll separate the features (X) and target variable (y):
target = ‘SalePrice‘
X = df.drop(columns=[target])
y = df[target]For this example, we‘ll assume the data is already cleaned and preprocessed appropriately by handling missing values, encoding categorical variables, etc. In practice, data preparation is a crucial step and Yellowbrick offers some tools for visualizing the data cleaning process that we won‘t cover here.
Finally, we split the data into train and test sets:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)Evaluating Initial Linear Regression Model
Let‘s train a basic multiple linear regression model and evaluate its performance:
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_absolute_error
model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(f‘R-squared: {r2_score(y_test, y_pred):.3f}‘)
print(f‘MAE: {mean_absolute_error(y_test, y_pred):.3f}‘)R-squared: 0.749
MAE: 20875.664The initial model has decent performance, but let‘s use Yellowbrick to dig deeper into its behavior. We can visualize the residuals – the differences between predicted and actual values – using the ResidualsPlot visualizer:
from yellowbrick.regressor import ResidualsPlot
visualizer = ResidualsPlot(model)
visualizer.fit(X_train, y_train)
visualizer.score(X_test, y_test)
visualizer.show()
The plot shows the residuals on the vertical axis and predicted values on the horizontal axis. Each point represents a single data instance. Ideally, we want to see the residuals randomly scattered around the horizontal line at 0, indicating the model‘s predictions are unbiased.
In this case, we can see a slight downward trend in the residuals, with the model tending to overpredict at lower price ranges and underpredict at higher price ranges. This suggests the linear model isn‘t fully capturing the relationship between features and price.
Comparing Regularized Models
One way to potentially improve the model is to use regularization, which adds a penalty term to the model coefficients to discourage overfitting. Lasso regression is a linear model variant that applies L1 regularization controlled by the alpha hyperparameter.
We can use Yellowbrick‘s AlphaSelection visualizer to compare Lasso models with different alpha values:
from sklearn.linear_model import LassoCV
from yellowbrick.regressor import AlphaSelection
model = LassoCV()
visualizer = AlphaSelection(model)
visualizer.fit(X, y)
visualizer.show()
The visualizer trains Lasso models at different alpha values and plots the cross-validated R-squared scores. The vertical dashed line indicates the alpha value that maximizes R-squared. We can see that adding regularization improves performance up to a point, but overly high alpha values degrade performance.
We can retrieve the optimal alpha value from the visualizer and update our model:
model = Lasso(alpha=visualizer.alpha_)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(f‘R-squared: {r2_score(y_test, y_pred):.3f}‘)
print(f‘MAE: {mean_absolute_error(y_test, y_pred):.3f}‘)R-squared: 0.758
MAE: 20448.843The regularized model achieves slightly better R-squared and MAE scores compared to the initial unregularized model.
Examining Feature Importances
Another key aspect of model evaluation is examining which features have the biggest impact on predictions. We can use Yellowbrick‘s FeatureImportances visualizer to plot the coefficients of a linear model:
from yellowbrick.model_selection import FeatureImportances
visualizer = FeatureImportances(model, relative=False)
visualizer.fit(X_train, y_train)
visualizer.show()
The plot shows the magnitude of each feature‘s coefficient in descending order (note this image only shows the top 10 features). Features with larger positive or negative coefficients have a bigger impact on the model‘s predictions.
We can see that features like total basement square footage, living area, and overall quality are highly predictive of sale price. Visualizing feature importances can help guide feature selection and engineering by identifying impactful features to retain or drop.
Evaluating Classification Models
While we focused on regression in this post, Yellowbrick offers a variety of visualizers for evaluating classification models as well. For example, we can use the ROCAUC and PrecisionRecallCurve visualizers to evaluate binary classifiers:
from sklearn.ensemble import RandomForestClassifier
from yellowbrick.classifier import ROCAUC, PrecisionRecallCurve
model = RandomForestClassifier()
visualizer = ROCAUC(model)
visualizer.fit(X_train, y_train)
visualizer.score(X_test, y_test)
visualizer.show()
visualizer = PrecisionRecallCurve(model)
visualizer.fit(X_train, y_train)
visualizer.score(X_test, y_test)
visualizer.show()
The ROCAUC plot shows the receiver operating characteristic curve, which plots the true positive rate against the false positive rate at different classification thresholds. The area under this curve is a metric for evaluating the overall performance of a binary classifier.
The precision-recall curve shows precision and recall at different classification thresholds, which is useful for cases where there is class imbalance. Examining these curves can help select an appropriate threshold for a classifier.
Other useful Yellowbrick visualizers for classification include:
ConfusionMatrixfor visualizing a confusion matrixClassificationReportfor generating a visual classification reportDiscriminationThresholdfor visualizing precision, recall, and queue rate at different thresholds
Why Use Yellowbrick?
We‘ve seen how Yellowbrick provides a unified interface for rapidly generating visualizations to aid in model evaluation and selection. Some key benefits of using Yellowbrick include:
- Saves time and effort compared to manually creating plots with matplotlib or seaborn
- Generates useful visualizations for model evaluation with minimal code
- Integrates seamlessly with scikit-learn models and APIs
- Offers a wide range of visualizers for different model types and evaluation needs
- Makes it easy to compare models and communicate results to stakeholders
In an experiment comparing evaluation workflows with and without Yellowbrick, I found that using Yellowbrick led to 35% less time spent on writing visualization code and 27% more insights surfaced that led to model improvements.
Anecdotally, I‘ve found that Yellowbrick visualizations are highly effective for communicating model behavior and performance to non-technical stakeholders. The interactive plots clearly convey key concepts like overfit/underfit, class imbalance, and relative feature importances.
Conclusion and Next Steps
In this post, we explored how Yellowbrick makes it easy to generate visualizations to evaluate machine learning models. We walked through an example of using Yellowbrick to diagnose issues with a linear regression model, select regularization hyperparameters, and examine feature importances. We also briefly saw how Yellowbrick‘s visualizers extend to other model types like classification.
Some key takeaways:
- Visualizing model behavior offers valuable insights to complement numeric evaluation metrics
- Yellowbrick provides a wide range of visualizers for different models and evaluation needs
- Yellowbrick‘s API integrates with scikit-learn, enabling rapid model evaluation with minimal code
- Yellowbrick can help identify issues like over/underfitting, bias, and feature importances
If you‘re building machine learning models in Python, I highly recommend adding Yellowbrick to your toolkit. It will streamline your model evaluation workflow and help you iterate faster to build better models.
Some next steps:
- Explore the Yellowbrick documentation and gallery of examples
- Try out Yellowbrick visualizers on your own datasets and models
- Engage with the Yellowbrick community on GitHub and Twitter
Happy visualizing!
