Time Series Forecasting with Extreme Learning Machines: A Comprehensive Guide
Time series forecasting is an essential tool with a wide range of applications, from product demand planning to weather prediction to healthcare. The goal is to make predictions about future values based on historical time series data.
Traditionally, statistical approaches like ARIMA (Auto Regressive Integrated Moving Average) models have been used for time series forecasting. However, these methods have limitations – they can only be applied to stationary data and fail to capture non-linear patterns and external factors.
In recent years, neural network models have emerged as a promising alternative for time series forecasting due to their ability to model complex non-linear relationships. One particularly powerful type of neural network is the Extreme Learning Machine (ELM).
What are Extreme Learning Machines?
ELMs are a special type of feedforward neural network with a single hidden layer. They were proposed by Guang-Bin Huang et al. in 2006 and have gained popularity due to their fast training speed and strong generalization ability.
The key difference between ELMs and traditional neural networks is in how the hidden layer parameters (weights and biases) are handled:
-
In traditional networks, these parameters are learned through an iterative process like backpropagation, which can be slow.
-
In contrast, the hidden layer parameters in ELMs are randomly initialized and fixed. Only the output weights are learned, which can be done very efficiently with a simple matrix operation.
This allows ELMs to be trained much faster than backpropagation networks, often orders of magnitude faster. ELMs have also been shown to achieve comparable or better accuracy on many tasks.
Another advantage is that ELMs are less prone to overfitting due to the random projection in the hidden layer, which acts as a regularizer. They also require less hyperparameter tuning since the hidden layer parameters don‘t need to be learned.
ELMs for Time Series Forecasting
So how can we apply ELMs to time series forecasting? The basic idea is to use a sliding window approach:
- Use the previous L timesteps (the "lag") as input
- Predict the value at the next timestep as output
- Slide the window forward by one step and repeat
This allows us to turn the time series forecasting problem into a standard supervised learning problem that can be tackled with ELMs or other machine learning models.
To demonstrate this, let‘s walk through a complete example of using ELMs for time series forecasting in Python. We‘ll use a product demand dataset containing daily sales data for a gaming console.
Loading and Visualizing the Data
First, we load the CSV data file and convert it to a NumPy array for easier manipulation:
import numpy as np
import matplotlib.pyplot as plt
data = np.genfromtxt(‘sales_data.csv‘, delimiter=‘,‘)
data = data.astype(‘float32‘)
plt.figure(figsize=(12,4))
plt.plot(data)
plt.xlabel(‘Time‘)
plt.ylabel(‘Sales‘)
plt.title(‘Product Demand‘)
plt.show()
We can see that the data has some clear seasonal patterns, which should help with forecasting.
Data Preparation
Next, we need to prepare the data for training the ELM. This involves:
- Splitting into train and test sets
- Normalizing the data
- Creating input/output pairs with a sliding window
The following code does all of that:
lag = 14
test_size = 90
train_data = data[:-test_size] test_data = data[-test_size:]
train_mean = train_data.mean()
train_std = train_data.std()
train_data = (train_data – train_mean) / train_std
test_data = (test_data – train_mean) / train_std
X_train, y_train = [], []
for i in range(lag, len(train_data)):
X_train.append(train_data[i-lag:i])
y_train.append(train_data[i])
X_train = np.array(X_train)
y_train = np.array(y_train)
X_test, y_test = [], []
for i in range(lag, len(test_data)):
X_test.append(test_data[i-lag:i])
y_test.append(test_data[i])
X_test = np.array(X_test)
y_test = np.array(y_test)
print(X_train.shape, y_train.shape)
print(X_test.shape, y_test.shape)
We use the last 90 days as the test set and everything before that as the training set.
The data is normalized by subtracting the training mean and dividing by the training standard deviation.
We use a lag of 14, so the input at each step contains sales values for the past 2 weeks and the target is the next day‘s value.
Building the ELM Model
We‘re now ready to build the ELM model:
input_size = X_train.shape[1]
hidden_size = 100
output_size = 1
input_weights = np.random.normal(size=[input_size, hidden_size])
biases = np.random.normal(size=[hidden_size])
def relu(x):
return np.maximum(0, x)
def hidden_nodes(X):
H = np.dot(X, input_weights)
H = H + biases
H = relu(H)
return H
output_weights = np.dot(pinv(hidden_nodes(X_train)), y_train)
def predict(X):
out = hidden_nodes(X)
out = np.dot(out, output_weights)
return out
The hidden layer size is set to 100. The input-to-hidden weights and biases are initialized randomly from a normal distribution.
We use ReLU as the activation function for the hidden layer. The output weights are computed analytically via the Moore-Penrose pseudoinverse of the hidden matrix.
The predict function passes the input through the ELM and returns the final prediction.
Evaluating Performance
Let‘s see how well the trained ELM does on the test set:
predictions = predict(X_test)
predictions = predictions train_std + train_mean
y_test = y_test train_std + train_mean
rmse = np.sqrt(np.mean((predictions – y_test)**2))
print(‘Test RMSE: %.3f‘ % rmse)
mape = np.mean(np.abs((y_test – predictions) / y_test)) * 100
print(‘Test MAPE: %.3f%%‘ % mape)
plt.figure(figsize=(12, 4))
plt.plot(y_test, label=‘Actual‘)
plt.plot(predictions, label=‘Predicted‘)
plt.xlabel(‘Time‘)
plt.ylabel(‘Sales‘)
plt.legend()
plt.title(‘ELM Predictions vs Actual‘)
plt.show()
Note that we need to denormalize the predictions and test targets before evaluating.
We look at two standard metrics – RMSE (root mean squared error) and MAPE (mean absolute percentage error).
With 100 hidden nodes, the ELM achieves a respectable RMSE of around 50 and MAPE of 8% on this dataset. Not bad for such a simple model!
We can also visualize the predictions vs actual values, which shows that the ELM is able to capture the overall trend and seasonality quite well.
Overcoming Limitations of ELMs
While ELMs have many advantages, one downside is that the random initialization can sometimes lead to suboptimal solutions. A few variants of the original ELM have been proposed to overcome this:
- Optimally Pruned ELM (OP-ELM): This version includes an additional pruning step to select the most relevant hidden nodes to improve model compactness and generalization.
- Online Sequential ELM (OS-ELM): This ELM variant allows incremental learning on new data without having to retrain the entire model from scratch.
- Others include the Error Minimized ELM (EM-ELM), Regularized ELM (R-ELM), and Kernel ELM.
The hyperparameters of the ELM can also be fine-tuned to optimize performance for a particular dataset. This includes the number of hidden nodes, type of activation function, and input weights distribution. Methods like random search or Bayesian optimization are often used.
Conclusion
Extreme Learning Machines offer an efficient and powerful approach for time series forecasting. Their extremely fast training time, good generalization ability, and ease of implementation make them an attractive choice compared to traditional neural networks.
With a bit of hyperparameter tuning, ELMs can achieve highly competitive results with much less computational cost. Variants like OP-ELM, OS-ELM, and others can help further boost accuracy while retaining the core speed advantage.
As time series data becomes increasingly ubiquitous, the ability to generate fast, reliable forecasts will only become more critical. ELMs are a promising tool to help tackle this challenge. Their successful application across a variety of domains showcases their potential.
This article provides a practical guide to getting started with ELMs for time series forecasting using a real-world sales demand dataset. You can adapt this example to your own time series problems and data. Try experimenting with different architectures, activation functions, and ELM variants to see what works best.
The field of time series forecasting is rapidly evolving, and ELMs are just one of many neural network approaches being explored, alongside RNNs, LSTMs, GRUs, and Temporal Convolutional Networks. Staying up to date with the latest developments will help you select the right tool for your forecasting task.
I encourage you to add ELMs to your forecasting toolbox and see how they can help you extract valuable insights from your time series data. Feel free to connect with me if you have any questions or want to share your results. Happy forecasting!
