exclude object and category columns
As an artificial intelligence and machine learning expert, I can confidently say that data preprocessing is one of the most critical steps in any ML pipeline. In fact, it‘s often said that data scientists spend up to 80% of their time on data cleaning and feature engineering, and only 20% on actual modeling. This is where pandas, the beloved Python data manipulation library, shines.
With over 150,000 GitHub stars and 20,000 commits from over 2,000 contributors, pandas is by far the most popular and widely-used tool for data wrangling in Python. According to the Stack Overflow Developer Survey 2022, pandas is the 5th most commonly used library among professional developers, used by 41% of respondents. Itts columnar data structure and intuitive API make it a joy to work with tabular data, especially for machine learning tasks.
As an ML practitioner, being proficient with pandas can greatly speed up your workflow and allow you to extract the most value from your data. While pandas has a vast array of functionality, there are certain functions that prove especially useful for common ML data preprocessing steps. In this post, we‘ll dive into 10 pandas functions that I consider essential for any data scientist or ML engineer.
1. pipe()
Data cleaning often involves applying a series of transformations to your raw data. For example, you might want to drop duplicate rows, remove outliers, impute missing values, encode categorical variables, and scale features – all before feeding the data into your ML model. Doing these operations step-by-step can lead to verbose and hard-to-read code.
That‘s where the pipe() function comes in. pipe() allows you to chain together functions in a readable, functional-programming style manner. Here‘s an example:
def drop_duplicates(df):
return df.drop_duplicates()
def remove_outliers(df, cols):
for col in cols:
q1 = df[col].quantile(0.25)
q3 = df[col].quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 iqr
upper = q3 + 1.5 iqr
df = df[(df[col] >= lower) & (df[col] <= upper)]
return df
def encode_categoricals(df, cols):
for col in cols:
df[col] = pd.factorize(df[col])[0]
return df
df_cleaned = (diamonds.pipe(drop_duplicates)
.pipe(remove_outliers, [‘price‘, ‘carat‘, ‘depth‘])
.pipe(encode_categoricals, [‘cut‘, ‘color‘, ‘clarity‘]))
By defining each task as a standalone function, you can compose them together using pipe() to create a data cleaning "pipeline". This makes the code more modular, reusable and maintainable compared to doing the operations in one huge function. It‘s a best practice I highly recommend.
2. factorize()
Speaking of encoding categorical variables, it‘s a preprocessing step required by most ML algorithms, which operate on numeric feature vectors. While scikit-learn provides transformer classes like OrdinalEncoder and OneHotEncoder for this task, pandas has a convenient factorize() function that can achieve a similar effect in one line:
diamonds[‘cut_encoded‘] = pd.factorize(diamonds[‘cut‘])[0]
This will convert the "cut" feature from string categories to integer codes (0, 1, 2…). factorize() returns a tuple where the first element is the encoded values, and the second element is the distinct categories. You can pass the original and encoded features to a dictionary to map the integers back to the string categories after modeling:
cut_map = dict(zip(pd.factorize(diamonds[‘cut‘])[0],
pd.factorize(diamonds[‘cut‘])[1]))
I find factorize() to be a quick and dirty way to encode categorical variables, especially in the early stages of model building. Just be aware that it assigns integers arbitrarily, which may not be suitable for ordinal variables. It also doesn‘t handle unseen categories, so you‘ll need a more robust method like a scikit-learn transformer for production use.
3. explode()
A common data format I encounter is where multiple values are stored in a single cell as a list or tuple. For example, a survey dataset might have a column called "Interests" with values like [‘Sports‘, ‘Music‘, ‘Travel‘]. To properly use this data for ML, you‘ll need to "explode" these list-like values into individual rows.
pandas provides an explode() function for exactly this purpose:
df = pd.DataFrame({‘ID‘: [1, 2, 3],
‘Interests‘: [[‘Sports‘, ‘Music‘],
[‘Travel‘],
[‘Music‘, ‘Dance‘, ‘Art‘]]})
df_exploded = df.explode(‘Interests‘)
The resulting df_exploded will have one row for each individual interest, repeating the ID values as necessary. This "long form" data is much easier to work with for analysis and modeling. You can then use groupby() and aggregation functions to summarize the exploded data back to "wide form" if needed.
4. select_dtypes()
When building ML models, it‘s often necessary to apply different preprocessing steps to different types of features. For example, you might want to scale your numeric features and one-hot encode your categorical features. pandas‘ select_dtypes() function allows you to easily filter columns by their data type:
# select only numeric columns diamonds_numeric = diamonds.select_dtypes(include=np.number)diamonds_no_cat = diamonds.select_dtypes(exclude=[‘object‘, ‘category‘])
This is much cleaner than manually specifying column names, especially if your dataset has many features. You can pass the resulting subsets to scikit-learn transformers or apply pandas functions like fillna() or clip() to them separately. select_dtypes() supports the following data type options: ‘number‘, ‘object‘, ‘category‘, ‘datetime‘, ‘timedelta‘, ‘bool‘, and ‘string‘.
5. clip()
Handling outliers is a common data cleaning task in ML, as extreme values can have undue influence on models. While there are many statistical techniques for outlier detection and removal, a simple approach is to "clip" or "winsorize" the values to some specified range based on domain knowledge. pandas‘ clip() function allows you to do this easily:
# clip price to between $500 and $20,000 diamonds[‘price_clipped‘] = diamonds[‘price‘].clip(500, 20000)
Any price values below $500 will be set to $500, and any values above $20,000 will be capped at $20,000. You can also pass percentiles instead of absolute values:
diamonds[‘depth_clipped‘] = diamonds[‘depth‘].clip(lower=diamonds[‘depth‘].quantile(0.05),
upper=diamonds[‘depth‘].quantile(0.95))
This will clip the "depth" values to between the 5th and 95th percentiles. Clipping is a quick way to constrain values to a reasonable range without removing rows, but be careful not to clip too aggressively and lose important information. Always check the before-and-after distributions and consider alternative outlier treatments.
6. convert_dtypes()
One issue I often see when loading data into pandas is that string columns get automatically converted to object dtype, which is inefficient to store and operate on. It‘s best practice to explicitly convert object columns to string dtype where appropriate. That‘s where convert_dtypes() comes in:
diamonds = pd.read_csv(‘diamonds.csv‘).convert_dtypes()
With a single function call, convert_dtypes() will scan the DataFrame and convert any object columns to the more appropriate string or boolean dtypes where possible. It will also intelligently parse columns that look like numbers, dates or timedeltas. This is a simple way to enforce type consistency and optimize memory usage, which is especially important for large datasets. I recommend adding convert_dtypes() to your standard data loading process.
7. to_csv()
When doing ML experiments, it‘s important to save your cleaned and preprocessed datasets for reproducibility and future use. pandas‘ to_csv() function allows you to easily save a DataFrame to a CSV file:
diamonds_cleaned.to_csv(‘diamonds_cleaned.csv‘, index=False)
Some best practices when saving DataFrames to CSV:
- Use a descriptive filename that includes the dataset name, version and any key preprocessing steps
- Exclude the DataFrame index with index=False, unless it contains meaningful information
- Specify a na_rep value to represent missing data (default is empty string ‘‘)
- Use compression like gzip or bz2 for large files to save disk space
- Include a header row and use a standard delimiter like comma
Being disciplined about saving your transformed datasets will make your ML workflow much smoother and more efficient. You can easily load the CSVs into new notebooks or scripts without having to redo the preprocessing steps each time.
8. get_dummies()
Scikit-learn ML models require numeric feature matrices as input, so you‘ll need to convert any categorical variables to numeric form. A common encoding method is one-hot or dummy encoding, where each unique category value is converted to a binary vector. pandas has a built-in get_dummies() function for this:
diamonds_encoded = pd.get_dummies(diamonds,
columns=[‘cut‘, ‘color‘, ‘clarity‘],
drop_first=True)
This will create new binary columns like ‘cut_Good‘, ‘cut_Ideal‘, ‘cut_Premium‘… for each unique value in the specified categorical columns. The drop_first option is useful to avoid multicollinearity in linear models, as it drops the first category which is implicitly encoded by the other columns.
get_dummies() also has options to handle missing categories, specify prefix names, and encode multiple columns at once. Just be aware that one-hot encoding can greatly increase the dimensionality of your feature space, so it‘s not always the best choice. Consider alternative encodings like ordinal or target encoding for high-cardinality variables.
9. melt()
For ML, data is typically expected to be in a tabular "long form", with each row representing an individual observation and each column representing a feature. However, raw datasets aren‘t always stored this way. You might encounter "wide form" data, where categorical values are stored as columns instead of rows. pandas‘ melt() function allows you to easily reshape data from wide to long form:
df_wide = pd.DataFrame({‘ID‘: [1, 2, 3],
‘2020_Sales‘: [100, 200, 300],
‘2021_Sales‘: [150, 250, 350],
‘2022_Sales‘: [200, 300, 400]})
df_long = df_wide.melt(id_vars=[‘ID‘],
var_name=‘Year‘,
value_name=‘Sales‘)
The resulting df_long will have columns ‘ID‘, ‘Year‘ and ‘Sales‘, with one row for each ID-Year combination. This is a much more convenient format for analysis and modeling. You can also use melt() in combination with pivot() to reshape data in the opposite direction, from long to wide form.
10. groupby() + agg()
Grouping and aggregating is a powerful way to summarize and extract insights from your data. It‘s also a common technique for feature engineering, where you create new features by aggregating existing ones based on some grouping variable. pandas makes this easy with the groupby() and agg() functions.
For example, let‘s say we want to create new features in the diamonds dataset based on the average price and carat for each cut and color combination:
diamonds_agg = (diamonds.groupby([‘cut‘, ‘color‘])
.agg(price_avg=(‘price‘, ‘mean‘),
price_min=(‘price‘, ‘min‘),
price_max=(‘price‘, ‘max‘),
carat_avg=(‘carat‘, ‘mean‘))
.reset_index())
The resulting DataFrame diamonds_agg will have one row per unique cut-color pair, with columns for the aggregated price and carat statistics. You can then join this back to the original DataFrame to create new features:
diamonds = diamonds.merge(diamonds_agg, on=[‘cut‘, ‘color‘])
This is a simple but powerful technique to encode group-level information in your features, which can greatly improve the predictive power of your ML models. You can use any of pandas‘ built-in aggregation functions like sum(), median(), std() etc., or even specify custom aggregation functions. Just be careful not to introduce target leakage by including the target variable in the groupby.
Conclusion
As an AI/ML expert, I can‘t overstate the importance of pandas in the data science workflow. While there are newer and more scalable tools like dask, vaex and pyspark for handling big data, pandas remains the most widely used and actively developed library for data manipulation in Python. Its intuitive API, rich functionality and seamless integration with the rest of the scientific Python stack make it an essential tool for any data professional.
In this post, we‘ve covered 10 powerful pandas functions for common machine learning data preprocessing tasks, from handling missing data and outliers to feature encoding and aggregation. However, this is just the tip of the iceberg – pandas has many more functions and features that can make your life easier as a data scientist. The key is to develop a curiosity mindset and constantly explore what pandas can do.
As Wes McKinney, the creator of pandas, said in his book "Python for Data Analysis":
"The pandas library is constantly evolving, with new features and improvements being added all the time. The best way to stay on top of these changes is to read the documentation, subscribe to the mailing list, and engage with the community. Don‘t be afraid to experiment and try new things – that‘s how you‘ll really unleash the power of pandas."
So here‘s my challenge to you: pick one of the functions we‘ve covered in this post, and try to apply it to your own dataset. See what insights and improvements you can uncover. Then pick another function and repeat. Make a habit of regularly exploring and experimenting with pandas, and I promise you‘ll see a tremendous improvement in your data science skills and productivity.
Remember, pandas is just a tool – it‘s up to you to wield it with creativity and domain expertise. By combining the power of pandas with your own unique perspective and problem-solving abilities, you‘ll be well on your way to becoming a master data scientist and ML practitioner. So what are you waiting for? Fire up a Jupyter notebook and start pandas-ing!
