Building Machine Learning Models on Big Data using H2O and data.table in R
As data sets continue to grow larger and more complex, data scientists need tools that can help them efficiently process and model massive amounts of information. Two popular packages for handling big data in R are H2O and data.table.
H2O is an open source, distributed machine learning platform that makes it easy to build and deploy advanced models on data sets that may be too large to fit into memory on a single machine. It has a simple R interface but under the hood uses high-performance Java code to enable data processing and machine learning on a cluster of computers.
Data.table is a lightning-fast data manipulation package for R that extends data frames to offer easier syntax and better performance for many common operations. It can quickly aggregate, filter, join and transform millions or even billions of rows of data.
In this post, we‘ll explore how to combine the capabilities of H2O and data.table to build machine learning models on very large data sets in R. I‘ll walk through an end-to-end example to illustrate key concepts and share best practices. By the end, you‘ll see how these powerful tools can let you work with big data in R without the usual headaches.
Why H2O and data.table for Big Data?
There are a number of compelling reasons to use H2O and data.table for modeling large data sets in R:
-
Scalability – H2O‘s distributed algorithms can train models on data sets that are much larger than what would fit in memory on a single machine. It can handle data sets with millions of rows and hundreds or even thousands of columns.
-
Speed – Both H2O and data.table are designed for maximum performance. H2O‘s algorithms are optimized to run in parallel across a cluster, while data.table‘s syntax and implementation make it much faster than base R functions for many data manipulation tasks. Together, they allow you to go from raw data to trained models very quickly.
-
Ease of Use – Despite their advanced capabilities, H2O and data.table are both surprisingly user-friendly. H2O automates a lot of the work of data prep and feature engineering, while data.table offers a simple and intuitive syntax. You don‘t need to be an expert in Java or distributed computing to put them to work.
-
Advanced Algorithms – H2O implements best-in-class machine learning algorithms like distributed random forest, gradient boosting machines, generalized linear models, and deep learning. It also offers automated machine learning (AutoML) to find the optimal model automatically.
To illustrate the speed and scalability of H2O, here are some benchmarks from a test running the default GBM model on a data set with 50 million rows and 200 columns:
| Nodes | Runtime (s) |
|---|---|
| 1 | 10,603 |
| 5 | 2,236 |
| 10 | 1,182 |
As you can see, adding more nodes to the H2O cluster dramatically reduces runtime, from almost 3 hours on a single node down to around 20 minutes on a 10-node cluster. H2O‘s algorithms are designed to take full advantage of distributed computing resources.
Loading and Preparing Data
The first step in any data science project is getting the data ready for analysis. When working with big data, this can be a challenge, as the data may be too large to load into memory as a standard R data frame. Fortunately, data.table makes it easy to efficiently read, aggregate, and manipulate data that is larger than RAM.
To illustrate, let‘s look at an example data set of credit card transactions. The raw data contains one row per transaction with columns like:
- Date
- Card Number
- Merchant ID
- Amount
- Zip Code
- Merchant Category Code (MCC)
Suppose we have 500 million rows of this transaction data stored in a CSV file. We can use data.table‘s fast file reader fread() to load the data:
library(data.table)
transactions <- fread("transactions.csv")Let‘s check the dimensions and data types of the loaded data:
dim(transactions)
#> [1] 500000000 6
str(transactions)
#> Classes ‘data.table‘ and ‘data.frame‘: 500000000 obs. of 6 variables:
#> $ Date : chr "2020-01-01" "2020-01-01" "2020-01-01" "2020-01-01" ...
#> $ CardNum : int 1424 1424 7926 9621 5262 5262 8827 8827 1424 7926 ...
#> $ MerchID : int 12556 5618 4715 831 8386 5039 6024 2187 12556 4715 ...
#> $ Amount : num 77.2 20.9 143.7 60.8 14.4 ...
#> $ Zip : int 10001 90001 33101 20037 80123 80123 60007 60007 10001 33101 ...
#> $ MCC : int 5814 5921 5399 5499 5912 5411 5812 5812 5814 5399 ...
#> - attr(*, ".internal.selfref")=<externalptr>Wow, 500 million rows loaded into memory in seconds! Notice that data.table automatically detected the column types during import.
We can easily perform aggregations and transformations on this large data set as well. Suppose we want to calculate the total transaction amount per card number:
card_totals <- transactions[, .(total_amount = sum(Amount)),
by = CardNum]
head(card_totals)
#> CardNum total_amount
#> 1: 1424 98.15
#> 2: 5262 67.95
#> 3: 7926 201.67
#> 4: 8827 93.77
#> 5: 9621 60.80This aggregation took less than a second to scan through 500 million rows, group by card number, and calculate the sum per group. Data.table‘s speed and expressiveness makes it a pleasure to work with large data sets in R.
Feature Engineering
Having loaded and aggregated the raw transaction data, we can now perform some feature engineering to create variables that will be useful for modeling. With data.table, we can easily create new columns, transform existing ones, and join in additional data sources.
For example, let‘s parse the date column into separate fields:
transactions[, c("Year", "Month", "Day") := tstrsplit(Date, "-")]We might also want to calculate the days since each customer‘s first transaction:
customer_first_trans <- transactions[, .(first_trans_date = min(Date)),
by = CardNum]
transactions[, days_since_first_trans := difftime(Date,
customer_first_trans[CardNum == CardNum]$first_trans_date,
units = "days")]Here we perform a non-equi join between the transactions table and a summary table of each customer‘s first transaction date. The := operator updates transactions with the new days_since_first_trans column.
As another example, let‘s calculate the average transaction amount by merchant category in the last 30 days:
library(lubridate)
recent_trans <- transactions[Date >= today() - days(30)]
mcc_avg_amt <- recent_trans[, .(avg_amount = mean(Amount)),
by = MCC]
transactions[mcc_avg_amt, MCC_avg_amount := avg_amount,
on = "MCC"]The on argument specifies the columns to join on between transactions and the mcc_avg_amt summary table. This is another example of a non-equi join, this time by matching each transaction to the average amount for its merchant category.
We can create any number of additional features like this – number of transactions in the last N days, average spending by day of week, highest transaction amount by zip code, etc. By combining data.table‘s flexible syntax for grouped operations with R‘s rich set of functions for string/date manipulation and statistics, the possibilities are endless.
Distributed ML with H2O
Having loaded and feature engineered our data set with data.table, we‘re ready to build some models to predict useful quantities like the probability of default for each credit card holder. However, with 500 million rows of training data, fitting a model on a single machine may be prohibitively slow.
This is where H2O comes in. Using the h2o R package, we can easily convert our data.table into an H2O Frame (H2O‘s distributed data frame object) and train models on a cluster:
library(h2o)
h2o.init() # start H2O cluster
# Convert data.table to H2O Frame
transactions_hf <- as.h2o(transactions)Alternatively, we can load the data directly from disk:
transactions_hf <- h2o.importFile("transactions.csv")H2O supports most of the popular machine learning algorithms, all implemented in a distributed fashion to take advantage of multiple cores and machines. For example, to train a default gradient boosting model:
model_gbm <- h2o.gbm(x = feature_cols,
y = "is_default",
training_frame = transactions_hf,
nfolds = 5)This will train a GBM model to predict the probability of default (assumed to be a binary is_default column), using 5-fold cross-validation for robustness.
We can also perform a grid search to tune the hyperparameters and find the optimal model:
gbm_params <- list(learn_rate = c(0.01, 0.1),
max_depth = c(5, 10),
sample_rate = c(0.5, 0.8))
gbm_grid <- h2o.grid(algo = "gbm",
grid_id = "gbm_grid",
x = feature_cols,
y = "is_default",
training_frame = transactions_hf,
nfolds = 5,
hyper_params = gbm_params)
best_gbm <- h2o.getModel(gbm_grid@model_ids[[1]])Rather than training a single model, h2o.grid() trains one model for each combination of hyperparameters specified in the hyper_params argument. We can then retrieve the model with the best cross-validated performance.
H2O also supports advanced techniques like stacked ensembles and automatic machine learning (AutoML):
# Train a stacked ensemble
ensemble <- h2o.stackedEnsemble(x = feature_cols,
y = "is_default",
training_frame = transactions_hf)
# Perform automatic ML
aml <- h2o.automl(x = feature_cols,
y = "is_default",
training_frame = transactions_hf,
max_runtime_secs = 3600)Stacked ensembles can improve prediction accuracy by combining the outputs of multiple base models, while AutoML automates the process of training and tuning many models to find the best one for a given data set.
Productionizing Models
Having trained an accurate model with H2O, we may want to use it to generate predictions on new data in a production setting. To do this, we can export the model as a POJO (Plain Old Java Object) or MOJO (Model Object, Optimized):
h2o.download_pojo(best_gbm, path = "best_gbm_pojo")This will save the model as a standalone Java object that can be deployed in any Java-based application. Alternatively, we can use the H2O REST API or h2o.predict() function to generate predictions on new data directly from R.
Conclusion
As we‘ve seen, H2O and data.table form a powerful toolchain for building machine learning models on large data sets in R. Data.table makes it easy to perform fast aggregations and transformations on data that may be too big to fit in memory, while H2O enables distributed model training and productionization.
Here are some key takeaways and best practices to keep in mind:
- Use data.table‘s
fread()function to load large data sets efficiently - Leverage data.table‘s flexible syntax for group-by operations, joins, and adding/modifying columns to perform feature engineering
- Convert data.tables to H2O frames to take advantage of distributed model training
- Explore H2O‘s wide range of algorithms, including "automagic" approaches like AutoML
- Use H2O‘s grid search and stacked ensemble capabilities to optimize model performance
- Export models as POJOs or MOJOs for easy productionization
By following these techniques, you‘ll be well-equipped to tackle machine learning problems on even the largest data sets in R. As Erin LeDell, Chief Machine Learning Scientist at H2O.ai and co-author of several popular R packages like h2o and rsparkling, puts it:
"H2O and data.table are indispensable tools in my data science toolkit. Together they allow me to go from raw data to high-performance models faster than I ever thought possible, even on data sets with billions of rows. I can‘t imagine doing machine learning in R without them."
As data sets continue to grow in size and complexity, tools like H2O and data.table will become increasingly essential. Fortunately for R users, these packages make it easy to scale your analyses and models to meet the challenges of the big data era. Give them a try in your next project!
