Feature Transformation | Feature Transformation Cheatsheet

Feature Transformation Cheat Sheet: The Ultimate Guide

Introduction

Feature transformation is a crucial step in the machine learning pipeline that involves converting raw data into a representation that is more suitable for modeling. The goal is to transform features in a way that captures the underlying patterns and relationships while avoiding issues like outliers, skewed distributions, or high cardinality.

Applying the right feature transformations can significantly improve the performance and interpretability of your models. Conversely, failing to properly transform your features can lead to poor results or even cause algorithms to fail entirely. As such, it‘s important for data scientists and machine learning engineers to have a strong grasp of the various feature transformation techniques at their disposal.

In this comprehensive guide, we‘ll dive deep into the most effective methods for transforming features based on their data types. We‘ll also provide a handy cheat sheet that you can reference when deciding how to preprocess your features. Finally, we‘ll walk through some concrete code examples to demonstrate how these transformations can be applied in practice using Python libraries like scikit-learn and pandas.

Whether you‘re a beginner just getting started with machine learning or an experienced practitioner looking to hone your skills, this guide will equip you with the knowledge you need to master the art of feature transformation. Let‘s get started!

Numerical Features

We‘ll begin by discussing how to transform numerical features, which are one of the most common data types you‘ll encounter. Numerical features can be either discrete (e.g. counts) or continuous (e.g. measurements). Some popular techniques for numerical features include:

Scaling and normalization: This involves changing the range of the feature values, typically to be between 0 and 1 or to have a mean of 0 and standard deviation of 1. Scaling puts features on a common scale, which is important when combining multiple features in a model. Common scaling methods include min-max scaling and standardization.

Log transform: Applying a log transform to right-skewed distributions can help make the data more normally distributed. This is useful for features with a wide range of values or exponential relationships.

Binning: Converting a continuous feature into discrete bins or categories can be a way to handle outliers or simplify the relationship with the target variable. Equal-width or equal-frequency binning are two common approaches.

Polynomial features: Adding polynomial terms (e.g. square, cube, etc.) as additional features can capture non-linear relationships between the feature and target. Be cautious about overfitting though.

Power transforms: The Box-Cox and Yeo-Johnson transforms aim to stabilize variance and minimize skew by applying a power function to the feature values.

Tips:

• Visualize the distribution of numerical features using histograms or density plots
• Scale to a common range when features are in different units
• For linear models, ensure features are on a similar scale to avoid regularization bias
• Apply log transforms to reduce right skew and make relationship more linear
• Experiment with binning for complex non-linear relationships

Code Example:

from sklearn.preprocessing import MinMaxScaler, StandardScaler, PolynomialFeatures
from sklearn.compose import TransformedTargetRegressor
import numpy as np

X = df[[‘age‘, ‘num_children‘, ‘income‘]]

# Min-max scaling
scaler = MinMaxScaler()
X_minmax = scaler.fit_transform(X)

# Log transform 
X_log = np.log1p(X)

# Polynomial features
poly = PolynomialFeatures(2) 
X_poly = poly.fit_transform(X)

Categorical Features

Categorical features represent discrete classes or labels rather than numeric quantities. They are very common in real-world datasets and can be nominal (unordered categories) or ordinal (ordered categories). Since most machine learning algorithms cannot handle raw categorical values, encoding them as numbers is an essential transformation. Here are some of the most effective techniques:

One-hot encoding: This creates a binary feature for each unique category value. A value of 1 indicates the presence of that category and 0 otherwise. This is the most common encoding for nominal features, but can be problematic for high-cardinality features.

Label encoding: Assigns an integer value to each unique category. Can be used for ordinal features to maintain the order of the categories. For nominal features, the arbitrary integer assignment may mislead tree-based algorithms.

Count/frequency encoding: Replaces each category with the number of times it appears in the dataset (count) or the fraction of rows it appears in (frequency). Useful for reducing cardinality while still capturing some information.

Target encoding: Replaces each category with the average value of the target variable for that category. Powerful technique but must be done carefully to avoid overfitting/leakage. Use cross-validation.

Embedding: Creates lower-dimensional numerical representations that capture relationships between categories. Useful for very high cardinality features.

Tips:

• Use bar plots to visualize category frequencies
• Combine rare categories into an "Other" category to reduce cardinality
• Ordinal features can be label encoded to integers
• One-hot encode nominal features, especially for linear models
• Mean target encoding is powerful but prone to overfitting
• Apply count/frequency encoding to handle rare categories while avoiding too many features

Code Example:

from sklearn.preprocessing import OneHotEncoder, LabelEncoder
from category_encoders import TargetEncoder

X = df[[‘color‘, ‘size‘, ‘material‘]]

# One-hot encoding
onehot = OneHotEncoder()
X_onehot = onehot.fit_transform(X)

# Label encoding
label = LabelEncoder()
X_label = label.fit_transform(X)

# Target encoding
target = TargetEncoder(cols=[‘color‘, ‘material‘])
X_target = target.fit_transform(X, y)

Text Features

Transforming raw text into a structured numerical representation is a common task in natural language processing (NLP) and machine learning applications involving text data. The goal is to convert the unstructured text into a high-dimensional vector that captures the relevant information. Some widely used techniques for representing text include:

Bag-of-Words: Represents text as a vector of word frequencies. Each unique word in the vocabulary becomes a feature, with the value being the count of that word in the document. Ignores word order.

TF-IDF: Similar to bag-of-words, but weights word frequencies by their inverse document frequency across the corpus. This gives more importance to words that are unique to a particular document.

N-grams: Bag-of-words and TF-IDF consider only single words (unigrams). N-grams capture sequences of N consecutive words, which can provide more context. Bigrams and trigrams are commonly used.

Word embeddings: Maps words to dense vector representations that capture semantic similarity between words. Popular techniques include Word2Vec, GloVe, and FastText. Vectors can be averaged or concatenated to represent full documents.

Tips:

• Preprocess text by lowercasing, removing punctuation/stopwords, stemming/lemmatizing
• Visualize word frequencies using word clouds
• Combine N-grams of different lengths
• Use character-level N-grams for text with many misspellings
• Apply dimensionality reduction (e.g. PCA, T-SNE) to visualize/cluster document vectors
• Fine-tune pre-trained embeddings on your specific corpus

Code Example:

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

docs = [‘This is the first document.‘,
        ‘This document is the second document.‘,
        ‘And this is the third one.‘,
        ‘Is this the first document?‘]

# Bag-of-words
bow = CountVectorizer()
X_bow = bow.fit_transform(docs)

# TF-IDF
tfidf = TfidfVectorizer(ngram_range=(1,2))
X_tfidf = tfidf.fit_transform(docs)

Datetime Features

Temporal data is another very common data type that often requires special handling. Raw timestamps can be difficult for algorithms to extract meaningful patterns from. Instead, we can engineer more informative features from datetime columns. Here are some examples:

Extract components: Separate timestamp into individual component features like year, month, day, hour, minute, etc. Can capture seasonality or time-of-day effects.

Differences between dates: Calculate the time since another relevant date, such as the first purchase date for a customer. Represents the "age" or elapsed time.

Cyclical encoding: Encode sequential features like hour, day, month as points on a unit circle using sine and cosine functions. Captures the cyclical nature of these features.

Is_weekend: Create a binary feature indicating if a day is a weekend or not. Similarly, is_holiday for relevant holidays. Captures important calendar effects.

Aggregate statistics: Group by a time period and calculate aggregate statistics like count, mean, or mode. For example, number of sales per month, or average price per weekday.

Tips:

• Extract both numerical (e.g. year) and categorical (e.g. month name) time features
• Consider both absolute (e.g. hour of day) and relative (e.g. hours since last visit) times
• Use domain knowledge to create relevant time-based features
• Be mindful of time zones and use consistent formatting
• Plot time series to visualize trends, seasonality, and outliers

Code Example:

import pandas as pd

def transform_datetime(df):
    df[‘date‘] = pd.to_datetime(df[‘date‘])

    df[‘year‘] = df[‘date‘].dt.year
    df[‘month‘] = df[‘date‘].dt.month
    df[‘day‘] = df[‘date‘].dt.day
    df[‘dayofweek‘] = df[‘date‘].dt.dayofweek
    df[‘hour‘] = df[‘date‘].dt.hour

    df[‘is_weekend‘] = (df[‘dayofweek‘] >= 5).astype(int)

    df[‘hour_sin‘] = np.sin(2*np.pi*df.hour/24)
    df[‘hour_cos‘] = np.cos(2*np.pi*df.hour/24)

    return df

Geospatial Features

With the increasing prevalence of location-based data, being able to work with geospatial features is an important skill for data scientists. Raw location data like latitude and longitude pairs, addresses, or geohashes need to be transformed into more useful representations for machine learning. Some common techniques include:

Distance: Calculate the distance between two locations, such as the distance from a user‘s home to a store. Can be done using Euclidean distance or more sophisticated formulas like Haversine that account for Earth‘s curvature.

Clustering: Group nearby locations into clusters. Useful for finding geographic regions or neighborhoods. Density-based clustering like DBSCAN is well-suited for spatial data.

Geographic features: Create features based on geographic attributes like elevation, climate, land use, population density, etc. Can provide important context about a location.

Spatial join: Combine information from different geospatial datasets based on location. For example, joining census data with store locations.

Tips:

• Ensure consistent coordinate system (e.g. WGS84) for all location data
• Standardize address formats using geocoding services
• Use spatial visualizations like choropleths or heatmaps
• Consider both distance and travel time
• Be aware of edge cases like international date line or poles

Code Example:

from sklearn.cluster import DBSCAN
from geopy.distance import great_circle

coords = df[[‘latitude‘, ‘longitude‘]].values

# Calculate pairwise distances
dist_matrix = pd.DataFrame(data=[[great_circle(c1,c2).miles for c2 in coords] for c1 in coords])

# DBSCAN clustering
dbscan = DBSCAN(eps=1, min_samples=5, metric=‘precomputed‘)
clusters = dbscan.fit_predict(dist_matrix.values)

Cheat Sheet

To summarize, here is a quick reference guide for how to transform common feature types:

Numerical

• Scale to common range (0-1 or standardized)
• Log transform skewed distributions
• Discretize/bin for non-linear relationships
• Create polynomial features for higher-order interactions

Categorical

• Use one-hot encoding for nominal features
• Label encoding for ordinal features
• Target encoding for high-cardinality features
• Combine rare categories into "Other"

Text

• Bag-of-words with counts or TF-IDF weighting
• Include N-grams (bigrams, trigrams)
• Use word embeddings (Word2Vec, GloVe) and average/concatenate
• Apply NLP preprocessing (lowercase, remove stopwords, stemming)

Datetime

• Extract components (year, month, day, etc.)
• Calculate diffs from reference points
• Encode hour/day/month cyclically as sin/cos
• Create binary features for weekends, holidays

Geospatial

• Calculate distances between points
• Use DBSCAN or other clustering algorithms
• Create geographic context features
• Spatially join with other datasets

Of course, the specific transformations you use will depend on your particular dataset and modeling task. It‘s important to let your data guide your decisions, while also using your domain knowledge to create meaningful features.

Conclusion

We‘ve covered a lot of ground in this guide to feature transformation. Hopefully you now have a solid understanding of how to approach transforming different types of raw data into useful input for machine learning models.

Some key takeaways:

1. Feature transformation is a critical step for both improving model performance and extracting insights from your data.

2. The appropriate transformation depends on the data type – numerical, categorical, text, datetime, and geospatial data all require different techniques.

3. Experimentation is key – try different transformations and compare their impact on your model‘s performance. Use visualizations to guide your decisions.

4. Domain knowledge is also important for creating relevant, meaningful features. Think about what patterns or relationships could be important for your particular problem.

5. Pay attention to data issues like outliers, missing values, and inconsistent formatting. Addressing these upfront will make the transformation process much smoother.

6. Don‘t forget to apply transformations to any new data points at inference time using the same pipeline you used during training to avoid data leakage.

With practice and experience, you‘ll develop an intuition for what transformations are likely to be effective. This cheat sheet should give you a solid starting point, but don‘t be afraid to experiment and try new techniques. The art of feature engineering is all about extracting maximum value from your data!

Thanks for reading! Hopefully this guide has been helpful in demystifying the process of feature transformation. Go forth and transform your data into powerful input for your machine learning models!