Predicting Future Sales with XGBRegressor: A Data Science Walkthrough

Accurately forecasting product demand is critical for retailers to optimize inventory and maximize revenue. In this post, we‘ll walk through the process of building a machine learning model to predict monthly sales for various items at a set of shops. We‘ll be using a powerful gradient boosting model called XGBRegressor.

Understanding XGBRegressor

XGBoost (eXtreme Gradient Boosting) is an optimized implementation of gradient boosting, an ensemble learning method that trains a sequence of predictive models, with each model attempting to correct the errors of the previous ones. In XGBoost, the models are decision trees.

Some key advantages of XGBoost are:

• Highly effective on structured data
• Scales well to large datasets
• Has built-in regularization to prevent overfitting
• Provides a robust, production-ready implementation

The Dataset

We‘ll be working with a dataset from a Kaggle competition which includes daily sales data for a set of shops and products, organized into monthly chunks. The dataset spans January 2013 to October 2015.

The files include:

• sales_train.csv – daily sales data per store and item
• items.csv – supplemental information about the items
• item_categories.csv – supplemental information about the item categories
• shops.csv – supplemental information about the shops
• test.csv – the test set for which we need to make forecasts (November 2015)

Our objective is to predict the total monthly sales for each shop and item combination in the test set. We‘ll be building an individual model for each combination.

Data Preprocessing

Let‘s take a look at the steps involved in preparing this data for modeling:

Data Cleaning

First, we need to clean the data to handle outliers, missing values, and inconsistencies:

• Remove items with a price > 300000 or items sold per day > 1000 as these are likely data errors
• Remove items with negative price as they may represent refunds/returns which we‘ll exclude for simplicity
• Consolidate duplicate shops (slightly differing names but same shop)
• Fill in missing values (very few in this dataset)

Feature Engineering

Next, we‘ll engineer some new features to provide richer, more informative signals for our model:

• Add shop city and category by extracting from the shop name
• Add item category and subcategory by splitting the item category name
• Create a monthly revenue feature (item price * items sold per month)
• Add lag features showing the previous month‘s sales
• Calculate monthly averages and trends:
  • Average monthly sales per item
  • Average monthly sales per shop
  • Average monthly sales per category
  • Average monthly sales per item/shop pair
• Determine first sale date per item and shop to capture product age
• Add date-related features like month number and number of days per month

Here‘s a code snippet showing how we can create lag features:

def lag_feature(df, lags, cols):
    for col in cols:
        for i in lags:
            shifted = df[[‘date_block_num‘, ‘shop_id‘, ‘item_id‘, col]]
            shifted.columns = [‘date_block_num‘, ‘shop_id‘, ‘item_id‘, col + ‘_lag_‘ + str(i)]
            shifted.date_block_num = shifted.date_block_num + i
            df = pd.merge(df, shifted, on=[‘date_block_num‘, ‘shop_id‘, ‘item_id‘], how=‘left‘)
    return df

lags = [1, 2, 3]  
matrix = lag_feature(matrix, lags, [‘item_cnt_month‘])

Train / Validation / Test Split

With our features built, we split the data into train, validation and test sets by time:

• Training set: Jan 2013 to Oct 2014 (22 months)
• Validation set: Nov 2014 to Oct 2015 (12 months)
• Test set: Nov 2015 (1 month, for Kaggle submission)

Training the Model

Now we‘re ready to train our XGBRegressor model:

model = XGBRegressor(
    max_depth=8,
    n_estimators=1000, 
    min_child_weight=300, 
    colsample_bytree=0.8, 
    subsample=0.8,
    eta=0.3,    
    seed=42)

model.fit(
    X_train, 
    Y_train, 
    eval_metric="rmse", 
    eval_set=[(X_train, Y_train), (X_valid, Y_valid)],
    verbose=True, 
    early_stopping_rounds=20)

Some key parameters:

• max_depth: maximum depth of each tree (regularization)
• n_estimators: number of trees to build
• min_child_weight: minimum sum of weights of all observations in a child (regularization)
• colsample_bytree: percentage of columns to be randomly sampled for each tree
• subsample: percentage of rows to be randomly sampled for each tree
• eta: learning rate shrinks the feature weights to make the boosting process more conservative

We use RMSE as the evaluation metric and set aside a validation set for early stopping to prevent overfitting. The model trains until the RMSE on the validation set fails to improve for 20 consecutive rounds.

Evaluation

Our trained model achieved an RMSE of 0.886 on the validation set. Taking a look at feature importance, we see that the most influential features are:

1. Previous month sales (lag 1)
2. Item monthly average sales
3. Shop monthly average sales
4. Item first sale date
5. Item category monthly average sales

This suggests that the most relevant signals for predicting an item‘s sales are its own previous sales, overall sales for that item/shop, and the item category.

Conclusion

XGBoost provides a powerful, scalable approach to sales forecasting that can incorporate various trends, seasonality, and product-related features. Proper data cleaning, feature engineering, and hyperparameter tuning can yield a highly precise model.

Potential next steps could include:

• Adding features based on shop location and regional economic data
• Systematically testing different time-series validation schemes
• Exploring deep learning sequence models as an alternative approach
• Developing an app to serve model predictions to shop owners

I hope this has been a helpful overview of the product demand forecasting process using XGBRegressor. Please check out the full code on my GitHub and feel free to reach out with any questions!