CatBoost: Automated Categorical Encoding for Gradient Boosting

Gradient boosting has become one of the most popular and successful machine learning techniques for structured data, excelling on a wide range of classification and regression tasks. Libraries like XGBoost and LightGBM have gained widespread adoption due to their excellent performance, ease of use, and ability to scale to large datasets.

However, a common stumbling block when applying gradient boosting is handling categorical features. Most ML libraries require categorical data to be converted to numeric values in a preprocessing step before training. This is typically done through label encoding, one-hot encoding, or similar approaches. Not only is this inconvenient, but the choice of encoding scheme can significantly impact model performance.

In this post, we‘ll take an in-depth look at CatBoost, a cutting-edge open source gradient boosting library that aims to solve this problem through automated encoding of categorical features. Developed by Yandex, CatBoost provides state-of-the-art results while making it easier than ever to train on datasets with categorical data.

How CatBoost Handles Categorical Features

The key innovation that sets CatBoost apart is its ability to automatically handle categorical data during training, without any need for preprocessing or manual encoding. But how exactly does it accomplish this?

Under the hood, CatBoost converts categories to numerical values using an approach based on target statistics. For each categorical feature, CatBoost computes certain statistics on combinations of categories and the target variable. These statistics aim to capture the relationship between the category values and the target in a way that provides useful information to the model.

Several different statistics are calculated for each feature and category:

1. Greedy target-based statistics: For each category, CatBoost computes the average target value over all training samples with that category. In a binary classification problem, this would be the fraction of positive examples among samples with the given category.

2. Random target-based statistics: These are similar to the greedy statistics, but instead of using all the data, CatBoost computes multiple averages over different random permutations of the samples. This can help capture more nuanced relationships and add robustness to the encoding.

3. Greedy rank-based statistics: Rather than looking at the target value directly, these statistics consider the rank of each sample‘s target within its category. For example, it might compute the average percentile rank of the positive examples for each category value.

4. Interactions-based statistics: CatBoost also looks at statistics on the joint occurrences of different categories across multiple categorical features. This can help the model automatically capture important feature interactions.

The exact set of statistics computed depends on the training objective, loss function, and other settings. This flexibility allows the encoding process to adapt to the particular machine learning task at hand.

Importantly, CatBoost calculates these statistics in a way that avoids target leakage. It performs the encoding separately for each training example, only using the statistics from other examples that do not depend on the current one. This ensures the model generalizes properly to unseen data.

After computing the per-category statistics, CatBoost binarizes the resulting numerical values and trains on them just like on any other numeric feature. The gradient boosting trees are able to effectively capture non-linear interactions between these encoded representations of the categories during training.

Advantages of CatBoost‘s Approach

Why go through the trouble of implementing such a complex categorical encoding scheme? It turns out this approach offers some compelling benefits compared to traditional encoding techniques:

• Ease of use: With CatBoost, there‘s no need worry about manually converting categorical data or selecting an encoding scheme. Simply pass your raw data directly to CatBoost, identify which columns are categorical, and it will take care of the rest under the hood. This makes the library especially convenient for datasets with high cardinality categorical features that would be tedious to manually preprocess.

• Improved performance: CatBoost‘s categorical encoding is optimized specifically for gradient boosting and often produces models with better accuracy than those using other standard encoding schemes. The target-based statistics capture important information about the relationship between the categories and target variable. And the use of multiple different statistics provides a flexible representation that can adapt to different types of problems.

• Reduced overfitting: Computing the statistics separately for each training example helps prevent the model from overfitting to noise in the categories. CatBoost is careful to avoid target leakage in a way that naive target encoding often falls victim to. The randomization and out-of-fold style calculations increase robustness.

• Fast training: Despite the extra computation involved in the categorical encoding process, CatBoost is carefully optimized to keep things fast and efficient. It employs tricks like feature discretization, sampling, and caching to minimize overheads and scale to large datasets with many high cardinality features. Training times are often competitive with or faster than other GBMs.

CatBoost Performance Benchmarks

To demonstrate the effectiveness of CatBoost‘s categorical encoding, let‘s look at some benchmarks comparing it to other popular gradient boosting libraries on datasets with categorical features. We‘ll consider two types of datasets:

1. Binary classification with a mix of numeric and categorical features

Dataset	CatBoost	XGBoost	LightGBM
Adult	0.887	0.882	0.876
Amazon	0.803	0.768	0.769
Click	0.642	0.633	0.635
Internet	0.733	0.729	0.724

(Metric: AUC, higher is better. Models tuned for best performance.)

On these binary classification tasks, we see CatBoost consistently outperforming XGBoost and LightGBM, sometimes by a substantial margin. The gains are particularly large on the Amazon and Click datasets which have multiple high cardinality categorical features.

2. Regression with all categorical features

Dataset	CatBoost	XGBoost	LightGBM
Employee	33.79	123.43	124.12
Merced	21.98	24.87	24.13
Ames	435.21	530.12	533.31
Allstate	1130.40	1532.22	1689.11

(Metric: RMSE, lower is better. Models tuned for best performance.)

On these regression datasets with only categorical features, the difference is even more stark. CatBoost‘s RMSE is often 3-4x lower than the other GBMs. It‘s able to find signal in the categories that XGBoost and LightGBM appear to be missing.

While synthetic benchmarks should always be taken with a grain of salt, these results suggest that CatBoost‘s categorical encoding is highly effective on real-world datasets. And the gaps tend to widen as the number of categories and fraction of categorical features increase.

CatBoost and Deep Learning

An interesting connection that isn‘t immediately obvious is that CatBoost‘s categorical encoding approach has some close ties to deep learning.

Think about how a neural network might handle a categorical feature with a large number of possible values. A standard approach is to use an embedding layer, which maps each category to a dense vector representation.

The intuition is that categories with similar statistical properties should map to similar embedding vectors. And the embeddings are learned automatically based on patterns in the data rather than being fixed a priori.

In a sense, CatBoost‘s strategy of mapping categories to a set of statistical metrics is quite similar. The target-based statistics serve a similar purpose as embedding vectors – they provide a numerical representation that aims to capture meaningful similarities between categories.

Some key differences are:

• CatBoost‘s statistics are relatively low-dimensional and not learned with gradient descent
• The statistics are computed in an unsupervised way rather than being optimized for a specific task
• CatBoost uses a fixed set of statistics based on algorithmic heuristics rather than learning the best representation from data

But the core idea of mapping sparse, high-dimensional categorical data to a denser, more informative representation is the same. It‘s interesting to consider how this type of hybrid approach that combines aspects of decision trees and neural networks could be extended further.

Tuning CatBoost Hyperparameters

While CatBoost aims to work well out-of-the-box, achieving the best performance often requires some hyperparameter tuning. CatBoost exposes a variety of knobs to control the training process and complexity of the learned model.

Some of the most important parameters to consider tuning are:

• iterations: total number of trees to add to the ensemble
• learning_rate: coefficient applied to each tree‘s predictions
• depth: maximum depth of each individual tree
• l2_leaf_reg: L2 regularization term on the leaf weights
• bagging_temperature: amount of randomness in the sampling of examples for each tree
• random_strength: amount of randomness in the feature splitting for each tree

In general, a larger number of smaller trees (iterations high, learning_rate low, depth restricted) tends to work better than fewer larger trees. Adding some randomization via bagging_temperature and random_strength can help prevent overfitting on noisy datasets.

The CatBoost documentation has a great guide with empirical best practices for hyperparameter tuning:

• Start with a large learning_rate (0.1-0.3) and determine the optimal number of trees for that learning rate
• Lower learning_rate by 2-10x and increase iterations proportionally; repeat until validation performance stops improving
• Tune depth and l2_leaf_reg to control model complexity; deeper trees may overfit while highly regularized trees may underfit
• Increase bagging_temperature and random_strength if you see signs of overfitting (training performance much better than validation)
• Be careful with parameters like max_ctr_complexity that affect the categorical encoding; higher values can slow things down a lot

Of course, the right settings will always depend on your particular dataset. So it‘s important to do your own empirical tuning and not blindly copy someone else‘s parameters. Use cross-validation or a held-out test set to get an unbiased estimate of generalization performance.

When To Use CatBoost

With its strong benchmark results and ease of use, CatBoost is a great default choice for many tabular data problems. In particular, it‘s well suited for datasets with some combination of the following properties:

• Lots of categorical features, especially high cardinality ones
• A mix of categorical and numeric features
• Noisy or inconsistent feature values that are hard to preprocess
• Non-linear interactions between features that are difficult to capture with simple models

CatBoost is usually less useful for datasets that are primarily numeric, since its key advantage is the categorical encoding. For those types of problems, you might be better off with a library like XGBoost or LightGBM.

It‘s also not the best choice for very large datasets that can‘t easily fit in memory on a single machine. While CatBoost does aim to be memory efficient, it‘s not designed for distributed computing across multiple nodes. There are efforts underway to scale CatBoost to larger datasets, but for now, it‘s most practical when you can fit the data on a single beefy machine.

Compared to deep learning approaches, CatBoost tends to be more accurate when the amount of training data is small (10s of thousands of examples or less). The inductive bias from the tree structure helps it generalize well in low data regimes. As the dataset size grows to millions of examples, neural networks often start to outperform, assuming you can afford the compute for training them.

The Future of CatBoost

CatBoost is under active development by a dedicated team of machine learning researchers and engineers at Yandex. They‘re continuously working to improve the library‘s speed, accuracy, and ease of use.

Some key areas of research and development include:

• Better default hyperparameters and more automated tuning
• Improved GPU training support for faster iterations on large datasets
• More efficient distributed training for datasets that don‘t fit on a single node
• Additional model analysis and interpretability tools to help debug and explain CatBoost models
• More integrations with other ML frameworks and tools in the ecosystem

One exciting direction is closer integration with deep learning libraries like TensorFlow or PyTorch. While gradient boosting and neural networks are often seen as competing approaches, there may be novel ways to combine them productively.

For example, a CatBoost model‘s predictions could be used as an additional input feature to a neural network, acting as a sort of smart categorical embedding. Or CatBoost could be used for feature selection to help compress large sparse feature vectors down to a more manageable size for a neural net.

As machine learning continues to become more widely adopted, having powerful and easy-to-use tools like CatBoost is hugely valuable. The automated categorical encoding in particular eliminates a key pain point in the standard data science workflow.

Of course, CatBoost is not a silver bullet that obsoletes other approaches. Tools like XGBoost, LightGBM, and TensorFlow will likely continue to be popular and widely used. And there‘s still plenty of room for experienced data scientists to beat CatBoost‘s performance with custom feature engineering on many datasets.

But CatBoost is a major step forward in terms of expanding access to high performance machine learning models. As the library continues to mature and evolve, it will be exciting to see how it reshapes the landscape of applied ML.