Feature Engineering Using Pandas for Beginners: The Essential Guide

Feature engineering is arguably the most critical part of the machine learning pipeline. As renowned data scientist Andrew Ng puts it, "Coming up with features is difficult, time-consuming, requires expert knowledge. ‘Applied machine learning‘ is basically feature engineering."

Indeed, the quality and quantity of features you develop can make or break the predictive performance of your models. One study by researchers at Google found that leveraging feature engineering improved the accuracy of a deep learning model for image classification by over 70% (Cubuk et al., 2018).

While feature engineering requires creativity and domain expertise, certain techniques are bread and butter across machine learning projects. The pandas library in Python provides an excellent toolkit to implement many of these techniques efficiently with minimal code.

In this comprehensive guide, we‘ll walk through 10 essential feature engineering techniques using pandas, including code examples, best practices, and real-world case studies. By the end, you‘ll have a solid foundation for improving your ML models through masterful feature development. Let‘s get started!

Exploratory Data Analysis and Profiling

Before we start creating new features, it‘s crucial to understand our raw data. Exploratory data analysis (EDA) is the process of investigating datasets to uncover patterns, check assumptions, and inform model design.

Pandas has several built-in functions that make EDA a breeze. For instance, the describe() method provides a statistical summary of numerical features:

df.describe()

	Age	Income	Loan_Amount
count	614	614	592
mean	38.87	5403.46	146.41
std	11.47	6109.08	85.59
min	21	150	9
25%	30	2877.50	100
50%	37	3812.50	128
75%	46	5795	168
max	80	81000	700

For categorical features, the value_counts() method shows the frequency of each unique value:

df[‘Education‘].value_counts()

Graduate        480
Not Graduate    134
Name: Education, dtype: int64

While these summaries are useful, it can be tedious to manually inspect every feature. This is where automated EDA tools like pandas profiling come in handy. With just a few lines of code, you can generate an interactive HTML report visualizing the distributions, correlations, and interactions of all features:

from pandas_profiling import ProfileReport

prof = ProfileReport(df)
prof.to_file(output_file=‘output.html‘)

Here‘s a snippet of the output:

Take time to thoroughly explore your data before engineering features. Visualize distributions, check for missing values and outliers, and examine relationships between variables. The insights you glean will help guide your feature choices down the line.

Scaling and Normalization

Many ML algorithms are sensitive to the scale and distribution of input features. If some features have much larger magnitudes than others, they may dominate the objective function and prevent the model from learning from other predictors.

Two common techniques for standardizing features are min-max scaling and z-score normalization. Min-max scaling transforms a feature to a fixed range, usually between 0 and 1:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
df[‘normalized_age‘] = scaler.fit_transform(df[[‘Age‘]])

Z-score normalization, on the other hand, rescales a feature to have zero mean and unit variance:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()  
df[‘standardized_income‘] = scaler.fit_transform(df[[‘Income‘]]) 

While pandas doesn‘t have built-in functions for these transformations, the scikit-learn library integrates seamlessly with pandas DataFrames. Experiment with different scalers and see how they impact your model‘s performance.

Encoding Categorical Variables

Most ML algorithms require numerical input features. Thus, encoding categorical variables is a crucial preprocessing step. We covered two common encoding schemes earlier – label encoding and one-hot encoding.

Label encoding replaces each category with an integer value:

df[‘Gender‘].replace({‘Male‘: 0, ‘Female‘: 1}, inplace=True)

This is suitable for ordinal variables with an intrinsic ranking or hierarchy.

For nominal variables, one-hot encoding creates new binary features for each category:

one_hot = pd.get_dummies(df[‘Education‘])
df = pd.concat([df, one_hot], axis=1)

However, one-hot encoding can greatly expand the feature space for high-cardinality variables. An alternative is feature hashing, which uses a hash function to map categories to a fixed number of dimensions:

from sklearn.feature_extraction import FeatureHasher

hasher = FeatureHasher(n_features=6, input_type=‘string‘)
hashed_features = hasher.fit_transform(df[‘Category‘])

The choice of encoding scheme depends on the nature of your categorical variables and the model you‘re using. When in doubt, try multiple approaches and compare results.

Handling Missing Data

Real-world datasets are messy and often contain missing values. While some ML algorithms can handle missing data directly, it‘s usually necessary to impute or remove missing values beforehand.

Pandas provides several functions for detecting and filling missing data. The isnull() method creates a boolean mask indicating missing values:

df.isnull().sum()

Age               0
Gender           13
Income            0
Loan_Amount      22
dtype: int64

Simple imputation fills missing values with a static number, like zero or the feature mean:

df[‘Age‘].fillna(df[‘Age‘].mean(), inplace=True)

A more sophisticated approach is multivariate imputation, which predicts missing values based on other features. Scikit-learn‘s IterativeImputer class does this efficiently:

from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

imputer = IterativeImputer()
df_imputed = imputer.fit_transform(df)

As a last resort, you can simply drop observations with missing data:

df.dropna(inplace=True)

However, this may significantly reduce your sample size and lead to bias if the missing values have a systematic pattern. Always check the percentage and randomness of missing data before settling on a strategy.

Working with Text Data

Unstructured text is a rich but challenging data type for feature engineering. The bag-of-words model is a simple yet effective approach for converting text into numerical features.

First, use pandas‘ str methods to clean and standardize the text:

df[‘review‘] = df[‘review‘].str.lower()  # Convert to lowercase
df[‘review‘] = df[‘review‘].str.replace(r‘\W‘, ‘ ‘)  # Remove punctuation
df[‘review‘] = df[‘review‘].str.replace(r‘\s+‘, ‘ ‘)  # Remove extra whitespace

Next, scikit-learn‘s CountVectorizer tokenizes the text and counts the occurrences of each word:

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(stop_words=‘english‘)
X = vectorizer.fit_transform(df[‘review‘])

The resulting matrix X has a column for each unique word and a row for each text document. The values are the counts of each word in each document.

For a more compact representation, you can use TF-IDF (term frequency-inverse document frequency) encoding instead:

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words=‘english‘)
X = tfidf.fit_transform(df[‘review‘])  

TF-IDF weights word counts by their rarity across documents, giving more prominence to unique and informative terms.

These text encoding techniques are just the beginning. More advanced approaches include word embeddings, topic modeling, and sentiment analysis. The key is to find a numerical representation that captures the underlying semantics and structure of the text.

Feature Selection

As you engineer more and more features, it‘s important to keep the feature space manageable. Having too many irrelevant or redundant features can lead to overfitting, increased computation time, and reduced interpretability.

Feature selection aims to identify a subset of predictors that are most relevant to the target variable. Univariate selection is a simple approach that evaluates each feature independently using a statistical test:

from sklearn.feature_selection import chi2

X_new = SelectKBest(chi2, k=10).fit_transform(X, y)

This applies the chi-squared test to each feature and selects the top k features with the highest scores.

For a more holistic assessment, recursive feature elimination (RFE) evaluates subsets of features by repeatedly training a model and pruning the least important features:

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

selector = RFE(estimator=LogisticRegression(), n_features_to_select=10)
X_new = selector.fit_transform(X, y)

This trains a logistic regression model on successively smaller subsets of features until the desired number of features is reached.

Finally, regularization techniques like Lasso and Ridge regression can perform feature selection and model fitting simultaneously by penalizing the coefficients of irrelevant features:

from sklearn.linear_model import Lasso

lasso = Lasso(alpha=0.1)  
lasso.fit(X, y)

The alpha parameter controls the strength of regularization. Features with non-zero coefficients after fitting are considered important.

Feature selection is both an art and a science. Domain knowledge should guide your initial feature choices, but data-driven techniques can help refine and validate your intuition. Always evaluate the impact of feature selection on your model‘s performance using cross-validation.

Interaction Features

Sometimes the most informative signals come from combinations of raw features. Interaction features, also known as feature crosses, capture relationships between predictors that might be overlooked when considering them individually.

Pandas provides several functions for computing interaction features. The most basic is simple multiplication:

df[‘Age_Income‘] = df[‘Age‘] * df[‘Income‘]

This creates a new feature representing the interaction between Age and Income.

For categorical variables, you can use the groupby() function to compute aggregate statistics:

df[‘mean_income_by_education‘] = df.groupby(‘Education‘)[‘Income‘].transform(‘mean‘)

This calculates the mean income for each education level and assigns it to each observation.

Scikit-learn‘s PolynomialFeatures class can automatically generate polynomial and interaction features up to a specified degree:

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False) 
X_new = poly.fit_transform(X)

With degree=2 and interaction_only=True, this creates pairwise interaction features without the quadratic terms.

While interaction features can capture complex relationships, they also expand the feature space considerably. Use them judiciously and consider regularization techniques to avoid overfitting.

Time Series Features

Time series data requires special handling in feature engineering. In addition to the standard techniques, you can leverage the temporal structure to create informative predictors.

Pandas has excellent support for time series via the DatetimeIndex and resampling functions. To convert a column to datetime format:

df[‘date‘] = pd.to_datetime(df[‘date‘])
df.set_index(‘date‘, inplace=True)

You can then extract various attributes:

df[‘dayofweek‘] = df.index.dayofweek
df[‘hour‘] = df.index.hour
df[‘month‘] = df.index.month

Resampling allows you to aggregate data at different time scales:

df_weekly = df.resample(‘W‘).mean()

This computes weekly averages of each feature.

Rolling window functions are useful for capturing trends and patterns:

df[‘rolling_mean_7d‘] = df[‘price‘].rolling(window=7).mean()

This calculates the 7-day rolling average price.

For multiple time series, you can use pivot() to reshape the data and apply functions across series:

df_pivoted = df.pivot(index=‘date‘, columns=‘symbol‘, values=‘price‘)
df_pivoted.pct_change()  # Compute percent change across all series

Time series feature engineering requires careful consideration of data leakage. Make sure your features only use data that would be available at the time of prediction. Rolling window functions and train-test splits are essential for backtesting.

Real-World Case Studies

To solidify our understanding of feature engineering, let‘s walk through a few real-world examples.

Loan Default Prediction

Suppose we‘re building a model to predict loan defaults. We have data on loan applicants, including demographics, employment history, credit scores, and loan details.

After conducting EDA, we might engineer the following features:

• Debt-to-income ratio: total monthly debt payments divided by monthly income
• Length of credit history: time since first credit account opened
• Number of recent credit inquiries: hard inquiries in the last 6 months
• Loan-to-value ratio: loan amount divided by appraised value of collateral

We could also create interaction features, such as credit score by loan purpose, to capture more nuanced risk patterns.

Customer Churn Prediction

Next, let‘s predict customer churn for a subscription-based business. We have data on customer demographics, subscription plans, usage behavior, and service interactions.

Potential engineered features include:

• Tenure: number of months since the customer signed up
• Usage intensity: average monthly usage over the last 3 months
• Service quality: number of service complaints filed in the last year
• Subscription value: monthly subscription fee multiplied by tenure

We might also derive features from unstructured data, such as the sentiment of customer support emails or the topics of product reviews.

Sales Forecasting

Finally, consider a retail company trying to forecast sales across multiple store locations. The data includes store attributes, product details, promotions, and economic indicators.

Some useful engineered features could be:

• Store age: number of years since the store opened
• Competitor presence: number of competing stores within a 5-mile radius
• Promotional intensity: percentage of products on promotion each week
• Macroeconomic factors: local unemployment rate, consumer confidence index

We would also want to create time series features, such as rolling average sales and year-over-year growth, to capture seasonal patterns and trends.

Conclusion

We‘ve covered a lot of ground in this guide to feature engineering with pandas. From EDA and preprocessing to interaction features and time series, pandas provides a powerful toolkit for transforming raw data into model-ready features.

The key to effective feature engineering is a deep understanding of your data and domain. Start with a thorough EDA, then use your insights to guide your feature choices. Don‘t be afraid to get creative – some of the most informative features come from unconventional combinations and transformations.

As you iterate, keep your feature space manageable through selection techniques and regularization. Validate your features on held-out data to avoid overfitting and leakage.

Remember, feature engineering is not a one-time task but an ongoing process. As you collect more data and refine your models, continue to explore new features and retire obsolete ones. The most successful data scientists are relentless feature tinkerers.

By mastering the art and science of feature engineering with pandas, you‘ll be well-equipped to tackle any machine learning challenge. So roll up your sleeves, dive into your data, and start engineering!