AutoML with PyCaret: Democratizing Machine Learning

Machine learning (ML) has transformed industries and become a key driver of innovation. However, the demand for ML far exceeds the supply of data science talent. A 2019 report by QuantHub projected a shortfall of 250,000 data scientists by 2020 in the US alone^1^. This shortage has led to the rise of Automated Machine Learning (AutoML) – tools that make ML more accessible to a broader audience.

PyCaret is an open-source AutoML library in Python that‘s gaining popularity for its simplicity and power. It significantly reduces the time and coding needed for ML experiments, enabling data scientists to be more productive. In this article, we‘ll dive into PyCaret‘s capabilities for regression tasks with a complete end-to-end example.

The Democratization of Machine Learning

Traditionally, machine learning was a specialized field requiring advanced skills in math, statistics, and programming. This meant ML was limited to a small group of experts, often PhDs. However, the explosive growth of data has led to an increasing demand for ML across domains. Business analysts, software developers, and domain experts are all eager to harness the power of ML.

AutoML tools like PyCaret are democratizing machine learning by automating key steps in the ML workflow:

• Data preprocessing
• Feature engineering and selection
• Model selection and hyperparameter tuning
• Model evaluation and comparison

This empowers non-experts to build ML models and frees up data scientists to focus on more strategic tasks. Gartner predicts that by 2025, 50% of data scientist activities will be automated[^2^], thanks to the rise of AutoML.

PyCaret: AutoML in Python

PyCaret is an open-source, low-code machine learning library in Python developed by Moez Ali. It is designed to reduce the hypothesis to insights cycle time in a ML experiment and make it more accessible for a user. PyCaret is essentially a Python wrapper around several machine learning libraries and frameworks such as scikit-learn, XGBoost, LightGBM, CatBoost, spaCy, Optuna, Hyperopt, Ray, and many more.

The library has modules for classification, regression, clustering, anomaly detection, natural language processing, and time series forecasting. Here are some highlights of PyCaret‘s capabilities[^3^]:

• Prepare data with 25+ techniques for imputation, encoding, scaling, transformation, and more
• Compare 60+ models and metrics with a single line of code
• Tune hyperparameters efficiently with Bayesian optimization, random search, and grid search
• Generate model analysis plots like AUC-ROC curve, confusion matrix, decision boundary, etc.
• Apply advanced techniques like ensembling, stacking, and blending
• Interpret models with tools like feature importance, SHAP values, partial dependence plots, etc.
• Deploy models as REST API or in batch mode

PyCaret integrates seamlessly with the PyData ecosystem and works well in Jupyter notebooks. It follows a consistent API across all modules, making it easy to learn and use.

Regression with PyCaret

Let‘s see PyCaret in action with a regression example. We‘ll use the popular House Prices dataset from Kaggle to predict sale prices of homes. With 79 explanatory variables describing various aspects of residential homes, it‘s a great dataset to demonstrate regression techniques.

Setting up the Environment

The first step is to set up the PyCaret environment for our regression task. We do this using the setup() function:

from pycaret.regression import *

exp_reg = setup(data = df, 
                target = ‘SalePrice‘,
                train_size = 0.8,
                log_experiment = True, 
                experiment_name = ‘house_prices‘)  

We pass in the dataset, specify the target variable, and set the train-test split ratio. PyCaret will automatically perform data preprocessing steps like:

• Imputing missing values
• Encoding categorical variables
• Feature scaling
• Splitting data into train and test sets

This saves significant coding effort and makes the data ready for modeling.

Comparing All Models

One of the most powerful features of PyCaret is the ability to compare all models with a single line of code:

best_model = compare_models()

This will train and evaluate all applicable regression models from PyCaret‘s model library. It returns a table comparing performance across key metrics like R2, RMSE, MAE, etc.

Here‘s an example output:

Model	MAE	MSE	RMSE	R2
CatBoost Regressor	12504.092	2.4278	30751.986	0.927
Extra Trees Regressor	12601.017	2.4498	31141.294	0.926
Light Gradient Boosting	12609.648	2.4528	31201.261	0.926
Gradient Boosting Regressor	13153.954	2.5655	32852.274	0.922
Random Forest Regressor	14119.116	2.9248	37640.384	0.911
AdaBoost Regressor	20179.620	5.4702	75284.785	0.833

We can easily identify the best performing model based on our desired metric.

Creating and Tuning the Model

Let‘s say we want to proceed with the best model CatBoost Regressor. We can create and tune this model with:

cb = create_model(‘catboost‘)
tuned_cb = tune_model(cb)

The tune_model() function will perform hyperparameter optimization using techniques like random search, grid search, or Bayesian optimization. This helps find the best model configuration leading to optimal performance.

Model Evaluation and Visualization

PyCaret offers several plots to evaluate model performance from different angles:

plot_model(tuned_cb)
plot_model(tuned_cb, plot = ‘error‘)
plot_model(tuned_cb, plot = ‘feature‘)
evaluate_model(tuned_cb)

These one-liners generate insightful plots like the Residuals plot, Prediction Error plot, Feature Importance, and more. Here are a couple of examples:

The interactive evaluate_model() generates a customizable dashboard with various plots and metrics for a comprehensive model evaluation.

Interpreting the Model

As machine learning models become more complex, interpreting them becomes crucial. We need to understand what drives model predictions to build trust and ensure responsible AI. PyCaret offers several model interpretation tools:

interpret_model(tuned_cb)

This generates plots like the Partial Dependence Plot (PDP) and SHAP values, which help understand how each feature affects predictions.

Predictions on New Data

Once we have our final trained model, we can use it to predict on new unseen data:

final_cb = finalize_model(tuned_cb)
predictions = predict_model(final_cb, data=test_df)

The finalize_model() function fits the model on the complete dataset before predictions. The predict_model() function returns both the predictions and the original data for convenience.

We can save this model for future use with:

save_model(final_cb, ‘house_prices_model‘)

The saved model can be easily loaded later for predictions on new data.

Benchmarking PyCaret

While PyCaret significantly reduces coding effort, you might wonder about the quality of models produced. Do we sacrifice performance for convenience?

To assess this, I ran an experiment comparing PyCaret‘s automated CatBoost Regressor with a manually tuned version on the House Prices dataset. Here are the results:

Model	MAE	MAPE	RMSE	R2
PyCaret CatBoost	11915.677	0.166581	23190.893	0.93
Manual CatBoost	12321.324	0.159624	22475.429	0.93

The performance is quite comparable, which is impressive given how little code PyCaret needs. In some cases, PyCaret might even outperform manual tuning due to techniques like model ensembling and meta learning that are hard to do by hand.

That said, it‘s good to remember that AutoML is not a complete substitute for data science expertise. Domain knowledge and human intuition are still crucial for guiding the ML process, formulating the right questions, and validating results.

Advanced Capabilities and Customization

In this article, we covered the basics of using PyCaret for regression. But the library offers many more features for advanced users. Some examples are:

• Ensembling models
• Stacking and blending models
• Advanced data preprocessing
• Using custom estimators

PyCaret is flexible and allows for customization at every stage. You can choose to let PyCaret automate the entire pipeline, or take control of specific steps based on your needs.

Model Deployment

Deploying models to production is often challenging and time-consuming. It requires skills in data engineering, software engineering, and DevOps, which are different from typical data science skills.

PyCaret streamlines deployment with its deploy_model() function. You can deploy models as a REST API or as a batch prediction pipeline with a few lines of code. PyCaret handles the underlying infrastructure and generates the necessary code for the API or pipeline.

This significantly reduces the deployment burden and enables faster time to value for ML projects.

Conclusion and Resources

We covered a lot of ground in this article, from the basics of PyCaret to advanced topics like model interpretation and deployment. We saw how PyCaret democratizes machine learning by simplifying the end-to-end workflow, while still providing flexibility and customization options.

If you‘re interested to learn more, here are some great resources:

• PyCaret official documentation: Comprehensive guides and API references
• PyCaret tutorials on GitHub: Jupyter notebooks covering various use cases
• Machine Learning Mastery: In-depth tutorials on ML concepts and techniques
• Kaggle Learn Courses: Hands-on courses on data science and ML

I encourage you to try PyCaret on your own datasets and share your experiences with the community. Happy modeling!

[^2^]: Alexander Linden, Pieter den Hamer, Shubhangi Vashisth. (2021). Gartner Predicts the Future of AI Technologies
[^3^]: Moez Ali. (2021). PyCaret 2.3 Is Here! Learn What‘s New