The Complete Guide to Feature Engineering: From Zero to Hero

Feature engineering is one of the most important yet often overlooked skills in data science and machine learning. At its core, feature engineering is the process of creating new input features for ML models using domain knowledge of the data. The quality of the features you use to train your models has a huge impact on their ultimate predictive performance. In fact, feature engineering is so critical that data scientists typically spend the bulk of their time on data preparation and feature engineering, rather than actual modeling.

According to various surveys, data scientists spend 70-80% of their time on collecting, cleaning and organizing data, with feature engineering being a key component. Only the remaining 20-30% is spent on model building, evaluation and deployment. This underscores the vital importance of becoming skilled at feature engineering.

In this comprehensive guide, we‘ll dive deep into feature engineering. You‘ll learn the essential concepts and techniques, and see detailed code examples walking through their application on real-world datasets. By the end, you‘ll have a solid grasp of feature engineering and be well on your way from zero to hero. Let‘s get started!

Missing Data Imputation Techniques

In the real world, missing data is extremely common in the datasets used for machine learning. Missing data can occur for many reasons – survey questions left blank, data entry errors, problems in data collection or processing, etc. Regardless of the cause, missing data poses challenges for ML modeling. Most models cannot handle missing values natively, so the data must be "cleaned" through a process called imputation.

There are several techniques for dealing with missing data through imputation:

Complete Case Analysis

The simplest approach is to simply remove any observations that have missing values. This is known as complete case analysis – we only keep observations (rows) that have complete data with no missing values. The downside is that this can remove a large fraction of the original data if many observations have some missing values, leaving a much smaller dataset to work with.

Mean/Median/Mode Imputation

Instead of deleting observations with missing data, we can fill in or "impute" the missing values. The most basic way to do this is to calculate the mean, median or mode of the non-missing values and use that to fill in the missing ones.

For continuous numeric features, we typically use the mean or median. For categorical features, we use the mode (most frequently occurring) value. Let‘s see an example imputing missing Age values in the Titanic dataset using the median:

import pandas as pd

# Load data
df = pd.read_csv(‘titanic.csv‘)

# Impute missing Age values with median
df[‘Age‘].fillna(df[‘Age‘].median(), inplace=True) 

Imputation with a Missing Value Indicator

One potential issue with mean/median/mode imputation is that it doesn‘t tell the model that the value was originally missing. The model doesn‘t know the difference between an imputed value and one that was actually observed.

To avoid this, we can combine imputation with an additional binary "missing value indicator" variable. This new feature will be 1 for observations where the value was originally missing and 0 otherwise. In this way, the model has information on missingness explicitly provided to it.

import numpy as np

# Create missing value indicator for Age
df[‘Age_missing‘] = np.where(df[‘Age‘].isnull(), 1, 0)

# Impute missing values 
df[‘Age‘].fillna(df[‘Age‘].median(), inplace=True)

Categorical Encoding Methods

Machine learning models require all input features to be numeric. So if we have categorical variables (values chosen from a discrete set of possibilities), we need to encode them numerically before we can feed them to a model. There are several ways to perform this encoding:

One-Hot Encoding

One-hot encoding creates new binary dummy variables to represent the different categories. If a categorical variable has $n$ distinct values, one-hot encoding generates $n$ new binary features, with a 1 indicating the presence of that category and 0 otherwise.

For example, let‘s one-hot encode the Sex feature in the Titanic data:

pd.get_dummies(df[‘Sex‘], prefix=‘Sex‘).head()

This will create new Sex_male and Sex_female binary dummy variables. One-hot encoding is very widely used, but it does have the downside of increasing the dimensionality of the feature space, especially if some categorical variables have many distinct values.

Ordinal Encoding

For categorical variables that have a natural ordering or hierarchy to their values, we can use ordinal encoding. This simply replaces the categories with integers reflecting their ordered position.

Example:

ordering = [‘Never‘, ‘Rarely‘, ‘Sometimes‘, ‘Often‘, ‘Always‘]

df[‘Frequency‘] = df[‘Frequency‘].apply(lambda x: ordering.index(x))  

Count/Frequency Encoding

Count encoding replaces each category with the number of times it appears in the dataset. Similarly, frequency encoding uses the percentage of observations in the dataset that have each category value. These can be useful for high-cardinality categorical variables with many possible values.

Target/Mean Encoding

Target encoding replaces each category with the mean of the target variable for observations having that category. For example, if the target is binary (0 or 1), target encoding will replace each category with the proportion of observations of that category where the target is 1.

Target encoding is a form of supervised learning and must be done carefully to avoid overfitting/target leakage. It‘s best to use it together with cross-validation.

Variable Transformation

Some models, like linear and logistic regression, assume that the input features are normally distributed. If a feature is not Gaussian, it‘s sometimes possible to apply a mathematical transformation to make it more normally distributed. Having approximately normal features can improve the performance of these models.

Some common variable transformations:

• Log transform: $\log(x)$
• Square root: $\sqrt{x}$
• Reciprocal: $\frac{1}{x}$
• Exponential: $e^x$

To illustrate, let‘s try to normalize the skewed Age variable in the Titanic data using a few different transformations:

import numpy as np
import matplotlib.pyplot as plt

plt.figure(figsize=(12,4))

plt.subplot(141)
df[‘Age‘].hist()
plt.title(‘Original‘)

plt.subplot(142)  
np.log(df[‘Age‘]).hist()
plt.title(‘Log‘)

plt.subplot(143)
np.sqrt(df[‘Age‘]).hist()  
plt.title(‘Square Root‘)

plt.subplot(144)
np.exp(df[‘Age‘]).hist()
plt.title(‘Exponential‘)

plt.tight_layout()
plt.show()

After visualizing the results, we can see that the exponential transformation does the best job of making Age more normally distributed. Choosing a good transformation often requires experimentation.

Handling Outliers

An outlier is a data point that is significantly different from the other observations. Outliers can have big impacts on model training and performance, so it‘s important to identify and handle them appropriately.

There are a few common approaches for dealing with outliers:

Outlier Removal

The simplest thing to do is remove outliers from the dataset – simply filter out or delete observations that are outliers. However, be careful not to remove too much data if there are many outliers, as this could leave you without enough data to effectively train your models.

Outlier Capping

Instead of completely removing outliers, we can cap their values at some maximum or minimum threshold. This preserves the observation while limiting the impact of the extreme values. Capping can be done based on standard deviations from the mean for normal distributions, or using percentiles for skewed distributions.

For example, here‘s how we could cap outliers for a feature at the 1st and 99th percentiles:

lower = df[‘feature‘].quantile(0.01) 
upper = df[‘feature‘].quantile(0.99)

df[‘feature‘] = np.where(
    df[‘feature‘] < lower, lower,
    np.where(
        df[‘feature‘] > upper, upper, 
        df[‘feature‘]
    )
)

Treat as Missing

Another option is to mark outliers as missing values, then use one of the missing data imputation techniques covered earlier to fill them in. This preserves the observation in the dataset while limiting the impact of the extreme outlier value.

Extracting Features from Dates

Date and time variables can be rich sources of useful features. There is often a lot of hidden information in dates that can be extracted to help machine learning models make better predictions.

Some informative features we can pull out of datetime variables include:

• Year
• Month
• Day of month
• Day of week
• Is weekend?
• Hour of day
• Minute

Let‘s see how to do some datetime feature extraction on the Lending Club loan dataset:

# Load data  
df = pd.read_csv(‘lendingclub.csv‘, parse_dates=[‘issue_d‘])

# Extract features
df[‘issue_year‘] = df[‘issue_d‘].dt.year
df[‘issue_month‘] = df[‘issue_d‘].dt.month
df[‘issue_day‘] = df[‘issue_d‘].dt.day
df[‘issue_dayofweek‘] = df[‘issue_d‘].dt.dayofweek
df[‘issue_is_weekend‘] = np.where(df[‘issue_dayofweek‘].isin([5,6]), 1, 0)

After extracting these features, we can use them in model training to potentially boost performance. The best datetime features to create will depend on the specific prediction problem.

Conclusions

We‘ve covered a lot of important concepts and techniques for feature engineering:

• Imputing missing data
• Encoding categorical variables
• Transforming variables to be more normal
• Handling outliers
• Extracting features from dates

Proper feature engineering is one of the most powerful ways to improve ML model accuracy. The features you create are like the ingredients in a recipe – better ingredients make a better final dish. Taking the time to carefully engineer informative features will pay dividends in better performing models.

The techniques we walked through are some of the most important, but there are many other methods as well. Some other useful feature engineering approaches include scaling, binning, polynomial feature expansion, domain-specific feature creation, and more. Effective feature engineering is both an art and science – it requires a combination of domain knowledge, intuition, and technical skills.

As a next step, I would encourage you to practice applying these feature engineering techniques to other real-world datasets. Kaggle is a great resource for finding datasets to work with. Remember, the features you create will depend on the specific data and the type of model you‘re trying to build.

Feature engineering is a skill that takes practice to master, but it is well worth the investment. Taking your feature engineering skills from zero to hero will make you a much more effective and valuable data scientist. Now go create some legendary features!