A Complete Guide to Encoding Numerical Features in Machine Learning

Properly encoding and transforming features is a critical step in developing effective machine learning models. While categorical features often get the most attention when it comes to encoding, numerical features frequently require preprocessing as well.

There are a number of reasons you may want or need to encode numerical features before training a model:

• Many models, like neural networks, assume all input features are on a similar scale. Features on dramatically different scales can slow down or prevent convergence.
• Some models, like decision trees, can be sensitive to the specific values taken by a numerical feature. Grouping values together via binning can be helpful.
• Numerical features with a skewed distribution often benefit from transformation to enable models to more easily learn from them.
• Extremely large or small values could cause numerical overflow/underflow issues for some algorithms.

In this post, we‘ll dive into some of the most common and useful techniques for encoding numerical features. We‘ll discuss the tradeoffs and share coding examples for each method. By the end, you‘ll be well-equipped to preprocess numerical data for your machine learning projects.

Discretization / Binning

One of the simplest approaches for encoding numerical features is discretization or binning. The idea is to divide the range of possible values into a set of intervals, or bins, and then assign each datapoint to a bin. The original numerical value is replaced by the bin number.

There are a few different common techniques for determining the boundaries of the bins:

Equal-width Binning

Equal-width binning divides the range of values into N bins of equal size. The width of each bin is:

w = (max - min) / N

Where min and max are the minimum and maximum values of the feature. The boundaries of each bin are then:

[min, min+w), [min+w, min+2w), ..., [min+(N-1)w, max] 

Here‘s how we would implement equal-width binning in Python using numpy:

import numpy as np

def equal_width_bins(data, n_bins):
    a = min(data)
    b = max(data)
    w = (b - a) / n_bins 
    bins = [(a + w*i, a + w*(i+1)) for i in range(n_bins)]
    return np.digitize(data, [b[0] for b in bins])

The main advantage of equal-width binning is simplicity. However, it does not handle outliers well and can lead to some bins having many more datapoints than others if the distribution is not uniform.

Equal-frequency Binning

Equal-frequency binning, also known as quantile binning, divides the data into N bins, each containing approximately the same number of datapoints. With equal-frequency binning, the width of each bin can vary.

Here‘s an implementation using numpy:

def equal_freq_bins(data, n_bins):
    percentiles = np.linspace(0, 100, n_bins + 1)
    bins = np.percentile(data, percentiles)
    return np.digitize(data, bins)

Equal-frequency binning handles skewed distributions and outliers better than equal-width. The downside is that it is more computationally expensive.

K-means Binning

K-means binning uses the k-means clustering algorithm to determine the bin boundaries. K-means attempts to divide the data into K clusters such that the inertia, or sum of squared distances from each point to its cluster center, is minimized.

We can use the KBinsDiscretizer from scikit-learn to perform k-means binning:

from sklearn.preprocessing import KBinsDiscretizer

est = KBinsDiscretizer(n_bins=10, encode=‘ordinal‘, strategy=‘kmeans‘)
est.fit_transform(X)

K-means binning is a good option if the data has a clustered structure. Note that k-means requires specifying the number of clusters/bins in advance.

Custom Binning

Finally, we may want to specify custom bin boundaries based on domain knowledge or business rules. For example, if the feature represents ages, we might want to have separate bins for children, teenagers, adults, and seniors.

Custom binning can be done using pandas:

def custom_bins(data, bins):
    labels = [‘Bin‘+str(i) for i in range(1, len(bins))]
    return pd.cut(data, bins=bins, labels=labels)

bins = [0, 12, 20, 65, 100]
labels = custom_bins(ages, bins)

Transformations

Another set of techniques for encoding numerical features involves applying mathematical transformations.

Categorical Encoding

For numerical features with low cardinality (few unique values), we can treat them like categorical variables. This is done by first mapping the numerical values to strings, then using a standard categorical encoding like one-hot or label encoding.

For example:

mapping = {0:‘zero‘, 1:‘one‘, 2:‘two‘}
data[‘num_as_cat‘] = data[‘num_feature‘].map(mapping)

data = pd.get_dummies(data, columns=[‘num_as_cat‘])

This approach is straightforward but only works for features with a small number of distinct values.

Power Transforms

Power transforms are a family of parametric, monotonic transformations that aim to map data from any distribution to as close to a Gaussian distribution as possible.

The most common power transform is the Box-Cox transform, which has a single parameter lambda that is determined from the data. The Box-Cox transform includes the log transform as a special case.

Scikit-learn provides an implementation of the Box-Cox transform:

from sklearn.preprocessing import PowerTransformer

pt = PowerTransformer(method=‘box-cox‘)
transformed = pt.fit_transform(X)

Power transforms are useful for reducing skewness and making the data more normally distributed. This is important for linear models and neural networks. The downside is that they require strictly positive data.

Normalization and Scaling

Normalization is the process of scaling individual features to have a unit norm. In other words, normalization scales the values so that they range from 0 to 1. This is also known as Min-Max scaling.

Scikit-learn‘s MinMaxScaler performs normalization:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
scaled = scaler.fit_transform(X)

Standardization is another scaling technique where the features are centered around 0 with a standard deviation of 1. This is done by subtracting the mean and then dividing by the standard deviation for each feature. Standardization is helpful for optimization algorithms in machine learning.

Scikit-learn provides a StandardScaler for standardization:

from sklearn.preprocessing import StandardScaler  

scaler = StandardScaler()
scaled = scaler.fit_transform(X)

It‘s important to fit scalers only to the training data and then transform the validation and test data to avoid data leakage.

Choosing an Encoding Method

With all these different options for encoding numerical features, how do we decide which one to use? Here are a few factors to consider:

• Number of unique values: If the feature has a small number of unique values, consider mapping to categories and one-hot encoding. For features with many unique values, the other encoding techniques will be more appropriate.

• Distribution: If the feature has a skewed distribution, consider applying a power transform like Box-Cox first. For normally distributed data, standardization often works well.

• Range of values: If the feature has a bounded range, min-max scaling will preserve that property. Standardization is a better choice for unbounded data.

• Presence of outliers: Equal frequency binning and power transforms can help reduce the impact of outliers.

• Interpretability: Discretization/binning maintains some level of interpretability, since the original values are just grouped together. The transforms make the data less interpretable and the original scale is lost.

In general, it‘s a good idea to start with simple methods like equal-width binning or standardization and iterate from there. Plotting histograms of the feature before and after encoding can help you assess if the encoding is having the desired effect.

Conclusion

In this post, we covered several important techniques for encoding numerical features: discretization/binning, power transforms, and normalization/scaling. We discussed the tradeoffs and showed how to implement each method using Python and scikit-learn.

Here are a few key takeaways and best practices to keep in mind:

• Don‘t just feed raw numerical data into your models without considering if encoding would be helpful. Unencoded features can cause poor performance.
• The best encoding technique depends on the characteristics of your data – consider the distribution, range, and number of unique values when making a choice.
• Start simple and iterate. You likely won‘t find the optimal encoding on the first attempt.
• Don‘t forget to apply any transformations to your validation and test data using statistics from the training data to avoid leakage.
• Use visualizations like histograms and scatter plots to understand the effects of encodings on your data.

Proper feature encoding can make the difference between a weak model and a strong one. Equipped with the techniques covered here, you‘re well on your way to creating robust, high-performing machine learning models. Happy encoding!