Harnessing U.S. Energy Data with R and Machine Learning

The U.S. Energy Information Administration (EIA) is the authoritative source for a wealth of data on energy production, consumption, and prices across the United States. For data scientists and analysts in the energy industry, accessing and leveraging this rich data is critical for understanding market dynamics, forecasting future trends, and driving business decisions.

The EIA provides a public API for programmatic access to their data. In this article, we‘ll explore how to use R to interface with the EIA API, pull energy data, and apply machine learning techniques to glean insights. By combining the data munging and statistical capabilities of R with the power of modern ML, we can build robust models for energy forecasting, anomaly detection, and more.

Accessing EIA Data with R

To access EIA data from R, you‘ll first need to sign up for a free API key at the EIA API portal. After registering and receiving your unique key, you can start pulling data into R with a few simple steps.

First, install and load the httr and jsonlite packages for handling API requests and parsing JSON:

install.packages(c("httr", "jsonlite"))
library(httr)
library(jsonlite)

Then set up your API key and parameters:

api_key <- "YOUR_API_KEY"
series_id <- "PET.MCRRIP12.M"  # Example series ID 
start_date <- "2010-01-01"
end_date <- "2022-12-31"

The series_id specifies the unique ID of the data series you want to request. You can find series IDs by browsing the EIA data sets on their website. For this example, we‘ll pull monthly crude oil production for PAD District 1.

Construct the API URL and send the request:

url <- paste0(
  "http://api.eia.gov/series/?api_key=", api_key,
  "&series_id=", series_id,
  "&start=", start_date, 
  "&end=", end_date
)
response <- GET(url)

Finally, parse the JSON response into an R dataframe:

json <- fromJSON(rawToChar(response$content))
df <- data.frame(
  date = as.Date(sapply(json$series[[1]]$data, `[[`, 1)),
  value = sapply(json$series[[1]]$data, `[[`, 2)
)

With just a few lines of code, we now have monthly crude oil production data from the EIA ready for analysis in R!

Exploratory Data Analysis

Let‘s do some initial exploration of the crude oil production time series. We can visualize the data using R‘s ggplot2 package:

library(ggplot2)
ggplot(df, aes(x=date, y=value)) +
  geom_line() +
  labs(title="U.S. PAD District 1 Monthly Crude Oil Production", y="Thousand Barrels")

From the plot we can see the significant rise in crude oil production in the East Coast region over the past decade, driven by the shale boom. The series shows some seasonality and volatility, especially during the major price crash and COVID demand shock in 2020.

We can also examine some summary statistics:

summary(df$value)
#>    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
#>    8.72   15.06   19.21   23.41   29.23   57.54

Over this time period, monthly crude oil production in PAD District 1 ranged from 8.72 to 57.54 thousand barrels, with a mean of 23.41 thousand barrels.

Forecasting with Machine Learning

One of the primary applications of the EIA‘s energy data is forecasting future production and consumption to inform planning and decision-making. With the historical data from the EIA API, we can train machine learning models to predict future values of the crude oil production time series.

As an example, we‘ll use a simple ARIMA (Autoregressive Integrated Moving Average) model, which is well-suited for forecasting time series data. The forecast package in R provides an auto.arima function that will automatically select the best model order.

library(forecast)
train <- window(ts(df$value, frequency=12), end=c(2021, 12))
test <- window(ts(df$value, frequency=12), start=c(2022, 1))
fit <- auto.arima(train)
forecast <- forecast(fit, h=12)

Here we split the data into a training set (through the end of 2021) and a test set (2022). We fit the ARIMA model on the training data and generate forecasts for the next 12 months.

Plotting the forecasts against the actual 2022 values:

plot(forecast, main="Crude Oil Production Forecast")
lines(test, col="red")

The ARIMA model does a decent job capturing the overall trend and seasonality, though it underestimates the summer 2022 production spike.

We can calculate some common forecast accuracy metrics:

accuracy(forecast, test)
#>                   ME     RMSE      MAE       MPE     MAPE
#> Training set  0.3092 2.136946 1.546404 0.7284326 7.169155
#> Test set     -4.8275 6.208495 4.974167 -9.124658 9.393943

On the test set, the mean absolute percentage error (MAPE) is around 9.4%, indicating the forecasts are off by about 9.4% on average. Not perfect, but a good starting point that could likely be improved with more sophisticated ML models.

Anomaly Detection

In addition to forecasting future values, ML can also be used to identify anomalies or outliers in energy time series data. Unusual spikes, dips or patterns could indicate significant market events, disruptions or errors in the data.

The anomalize package in R provides a framework for detecting anomalies in time series data. It decomposes the series into seasonal, trend and remainder components and identifies outliers in the remainder.

library(anomalize)
anomalies <- df %>%
  time_decompose(count, method="stl") %>%
  anomalize(remainder, alpha=0.05, max_anoms=0.2) %>%
  time_recompose()

Plotting the results:

library(dplyr)
anomaly_plot <- df %>%
  left_join(anomalies %>% filter(anomaly == ‘Yes‘), by = ‘date‘) %>%
  ggplot(aes(date, value)) +
    geom_line() + 
    geom_point(data = dplyr::filter(anomalies, anomaly == "Yes"), color = "red")

The anomalize method identifies a few time points as unusual, including the major dip in early 2020 driven by COVID lockdowns and the resulting crash in oil prices and demand. Detecting these types of regime changes and shocks is crucial for understanding and modeling energy markets.

Correlation Analysis

The EIA provides data on production, consumption, inventories, imports, exports and prices for a range of fuels beyond just crude oil. By pulling multiple related time series, we can examine the relationships between different energy commodities and market fundamentals.

For example, let‘s analyze the relationship between motor gasoline inventories and spot prices for conventional gasoline:

gas_inventory <- "PET.WGTSTUS1.W"
gas_price <- "PET.EER_EPMRU_PF4_Y44MB_DPG.W"

url_inventory <- paste0(base_url, "?api_key=", api_key, "&series_id=", gas_inventory) 
url_price <- paste0(base_url, "?api_key=", api_key, "&series_id=", gas_price)

inventory_json <- fromJSON(rawToChar(GET(url_inventory)$content))
price_json <- fromJSON(rawToChar(GET(url_price)$content))

df_inventory <- data.frame(
  date = as.Date(sapply(inventory_json$series[[1]]$data, `[[`, 1)),
  inventory = sapply(inventory_json$series[[1]]$data, `[[`, 2)
)

df_price <- data.frame(
  date = as.Date(sapply(price_json$series[[1]]$data, `[[`, 1)),
  price = sapply(price_json$series[[1]]$data, `[[`, 2)
)

merged_df <- merge(df_inventory, df_price, by="date")
cor(merged_df$inventory, merged_df$price)
#> [1] -0.6508013

The correlation coefficient of -0.65 indicates a moderately strong inverse relationship between gasoline inventories and prices. As inventories draw down, prices tend to increase, and vice versa.

Visualizing the two series together:

ggplot(merged_df, aes(x=date)) +
  geom_line(aes(y=inventory, color="Gasoline Inventories")) +
  geom_line(aes(y=price, color="Gasoline Spot Price")) +
  labs(title="U.S. Gasoline Inventories vs. Spot Price", color="Series")

The plot shows the broadly inverse relationship between the two series, though they can temporarily decouple, especially during volatile market periods. Tracking these fundamental relationships is key for understanding fuel price drivers.

Similar correlation and time series analysis could be done for any number of energy metrics available from the EIA – crude oil vs. refined products, natural gas production vs. storage, regional electricity prices, and more. The EIA API makes accessing the necessary data straightforward, enabling deeper insights into the complex dynamics of energy markets.

Conclusion

The EIA API provides analysts and data scientists a direct pipeline to an extensive repository of U.S. energy data. By leveraging this data with the statistical and machine learning capabilities of R, we can develop powerful models and uncover meaningful insights to drive energy business decisions.

Some key applications of EIA data and ML in R include:

• Forecasting future energy production, consumption and prices to guide investment and planning
• Detecting anomalies and market disruptions to enable proactive risk management
• Identifying correlations and lead-lag relationships between energy commodities and assets
• Backtesting trading strategies and hedging techniques with historical price data
• Stress testing portfolios with scenario analysis based on past market shocks
• Building predictive maintenance models for industrial equipment and energy infrastructure

Of course, the code samples and techniques presented in this article only scratch the surface of what‘s possible at the intersection of energy data, ML and R. Improving forecast accuracy, building more sophisticated anomaly detection, and applying ML to more granular meter-level data are all active areas of research and development.

Regardless of the specific application, the combination of domain expertise, data from the EIA, and cutting-edge ML algorithms is a powerful tool for navigating the complex, fast-evolving world of energy. As the industry continues to undergo massive transformation and disruption, a data-driven approach is more critical than ever – and R and the EIA API can help provide the necessary foundation.