Optimized model

In a previous post, we introduced PyCaret as a powerful Python library for automating machine learning tasks and demonstrated its capabilities on a beginner-friendly regression problem. If you haven‘t checked that out yet, I highly recommend starting there to learn the basics of using PyCaret for regression.

In today‘s post, we‘ll build upon those fundamentals and explore more advanced techniques in PyCaret to further improve our regression models. Specifically, we‘ll cover:

• Customizing the data preprocessing pipeline with normalization, transformation, and more
• Boosting performance with ensembling methods like bagging, boosting, blending, and stacking
• Tuning the hyperparameters of ensemble models
• Putting it all together with a real-world dataset

Whether you‘re participating in the Data Science Blogathon or looking to take your automated ML skills to the next level, this post will equip you with valuable tools and techniques to optimize your regression models. Let‘s dive in!

Recap: PyCaret Basics for Regression

Before we get into the advanced techniques, let‘s quickly review the key steps for building a regression model in PyCaret:

1. Set up the PyCaret environment with the setup() function, specifying the data, target variable, and other optional configurations.
2. Compare all available models with compare_models() to get a quick performance baseline.
3. Create individual models with create_model() for further analysis and optimization.
4. Tune hyperparameters of a model with tune_model() to find the optimal configuration.
5. Evaluate the final model on a holdout set and generate predictions with predict_model().

With these foundational steps in mind, we‘re ready to explore how to customize and enhance each part of the pipeline. We‘ll be using the "Sarah Gets a Diamond" dataset, which contains information on nearly 6,000 diamonds to predict their price based on carat weight, cut, color, clarity, and other attributes.

Customizing the Data Preprocessing Pipeline

One of PyCaret‘s strengths is the ability to easily customize the data preprocessing steps that occur automatically when you initialize the setup() function. With a few lines of code, you can normalize, transform, handle missing data and outliers, bin numeric features, and more. Let‘s look at a few important preprocessing techniques:

Normalization and Scaling

Many machine learning algorithms perform better when the input features are on a similar scale. Two common approaches are normalization (rescaling to a 0-1 range) and standardization (transforming to zero mean and unit variance). In PyCaret, simply set the normalize and normalize_method parameters in setup():

from pycaret.datasets import get_data
data = get_data("diamond")
exp1 = setup(data, target = ‘Price‘, session_id=123,

normalize = True, normalize_method = "zscore")

Target Transformation

In addition to normalizing the input features, you may want to transform the target variable if it has a skewed distribution. Common transformations include log, square root, and box-cox. To apply target transformation in PyCaret, use the transform_target and transform_target_method parameters in setup():

  
exp2 = setup(data, target = ‘Price‘, session_id=123,
             transform_target = True, transform_target_method = "yeo-johnson")

Handling Rare Categorical Levels

Categorical features may contain levels that appear very rarely in the dataset. These rare levels can add noise and instability to the model. One way to handle them is to group the rare levels into a new "rare" category. In PyCaret, you can do this with the combine_rare_levels and rare_level_threshold parameters:

exp3 = setup(data, target = ‘Price‘, session_id=123,
             combine_rare_levels = True, rare_level_threshold = 0.05)  

This will combine all categorical levels that make up less than 5% of the data into a single "Rare" level.

Binning Numeric Features

Another useful preprocessing technique is binning (or discretizing) numeric features into categorical bins. This can help capture non-linear relationships and reduce the impact of outliers. PyCaret makes it easy to automatically bin numeric columns with the bin_numeric_features parameter:

exp4 = setup(data, target = ‘Price‘, session_id=123,
             bin_numeric_features = ["Carat Weight"])

This will use the Sturges rule to determine the optimal number of bins for the specified columns.

With our custom preprocessing pipeline set up, we can proceed to train and compare models just as before. The key difference is that PyCaret will apply our specified preprocessing steps under the hood first.

Ensembling for Extra Performance Gains

Now that we‘ve optimized our data preprocessing, let‘s explore some powerful ensembling techniques to squeeze out even better performance from our models. Ensembling combines multiple base models to produce a stronger final model. PyCaret supports several popular ensembling methods with a single line of code.

Bagging

Bagging (short for Bootstrap Aggregating) trains multiple models on different subsets of the training data, then combines their individual predictions. This reduces model variance and overfitting. To bag any model in PyCaret, just use the ensemble_model() function:

dt = create_model("dt")
bagged_dt = ensemble_model(dt, method="Bagging")

You can also control the number of base estimators with the n_estimators parameter.

Boosting

Boosting is another popular ensembling technique that trains models sequentially, with each model learning to correct the errors of the previous ones. Boosted models often achieve higher performance than individual models. To boost in PyCaret:

boosted_dt = ensemble_model(dt, method="Boosting")  

The default boosting method is Adaptive Boosting (AdaBoost), but you can also specify other methods like Gradient Boosting.

Blending

Blending combines multiple base models by training them on the same data, then averaging their predictions. Unlike bagging, the base models are typically different algorithms. To blend in PyCaret, first create the base models, then pass them to blend_models():

lr = create_model("lr")
rf = create_model("rf")
knn = create_model("knn")
blender = blend_models(estimator_list = [lr, rf, knn])

Stacking

Stacking is a more sophisticated ensembling method that trains a new "meta-model" on the outputs of the base models. The meta-model learns to optimally combine the base model predictions. To stack in PyCaret:

stacker = stack_models(estimator_list = [lr, rf, knn], 
                       meta_model = create_model("ridge"))

By default, the meta-model is a simple linear regression, but you can specify any PyCaret regression model.

Tuning Ensemble Hyperparameters

Just like individual models, ensembles have hyperparameters that can be tuned for optimal performance. After creating an ensemble with ensemble_model(), blend_models(), or stack_models(), simply pass it to the standard tune_model() function:

tuned_bagged_dt = tune_model(bagged_dt)

PyCaret will search for the best ensemble-specific hyperparameters like the number of estimators, boosting learning rate, and blending weights.

Putting it All Together

Let‘s see how our customized preprocessing pipeline and ensembling techniques can boost performance on the diamond pricing dataset. We‘ll compare a default PyCaret model to one with our advanced optimizations:

# Default model
exp_default = setup(data, target = ‘Price‘, session_id=123)
lr_default = create_model("lr")

exp_optimized = setup(data, target = ‘Price‘, session_id=123, 
normalize = True, normalize_method = "zscore",
transform_target = True, transform_target_method = "yeo-johnson",
combine_rare_levels = True, rare_level_threshold = 0.05,
bin_numeric_features = ["Carat Weight"])
lr_optimized = create_model("lr")
boosted_optimized = ensemble_model(lr_optimized, method="Boosting")
tuned_boosted_optimized = tune_model(boosted_optimized)

After training, we can compare the performance metrics:

print(lr_default)
print(tuned_boosted_optimized)

On this dataset, our optimized pipeline achieved an R2 score of 0.93 compared to 0.89 for the default linear regression – a significant improvement!

Of course, every dataset is different, and not all techniques will help in every case. The key is to experiment with different preprocessing and ensembling configurations to find what works best for your specific problem.

Next Steps

We‘ve covered a lot of ground in this post, but there‘s still much more to explore with PyCaret. In future posts, we‘ll dive into:

• Advanced techniques for classification problems
• Interpreting model predictions and feature importances
• Saving and deploying trained models
• Integrating PyCaret with other data science tools and workflows

In the meantime, I encourage you to check out the official PyCaret tutorials and experiment with the techniques we‘ve learned on your own datasets. Feel free to connect with me on LinkedIn or GitHub to share your experiences and insights.

Thanks for reading, and happy automated machine learning!