Feature Selection in R with the Boruta Package

As a data scientist, you know that more data doesn‘t always mean better results. In fact, including irrelevant or redundant features in your machine learning models can actually lead to worse performance. That‘s where feature selection comes in – by systematically removing less useful variables from your dataset, you can train models that are simpler, faster and more accurate.

While there are many feature selection techniques to choose from, one of my personal favorites is the Boruta algorithm. Boruta is an all-relevant feature selection method, which aims to identify all features that are in some way relevant to the outcome variable. This contrasts with minimal-optimal methods like recursive feature elimination, which try to find a small subset of features that can build a model with the highest accuracy.

In this post, we‘ll dive into how the Boruta algorithm works, implement it on an example dataset using R, and compare it to some other common feature selection approaches. By the end, you‘ll have a solid understanding of when and how to leverage this powerful technique in your own work. Let‘s get started!

What is Feature Selection?

Before we jump into the specifics of Boruta, let‘s take a step back and clarify what feature selection is and why it‘s so important. In essence, feature selection is the process of reducing the number of input variables for a predictive model. The goal is to pare down a large feature space to only include the most relevant signals, thereby improving model performance.

There are several reasons you might want to perform feature selection:

• Simplicity: Models with fewer features are easier to understand and explain
• Speed: Less input data means faster training times and lower computational overhead
• Generalization: Removing noisy features can help models adapt better to new, unseen data
• Accuracy: In some cases, irrelevant features can actually decrease predictive power

In practice, feature selection is often an iterative process of building a model, evaluating its performance, removing features, and repeating. There are three main categories of feature selection algorithms:

1. Filter methods: Select features based on their statistical properties (correlation with target, variance, etc.)

2. Wrapper methods: Evaluate subsets of features by training a model and measuring performance

3. Embedded methods: Perform feature selection as part of the model training process itself

Boruta is a wrapper method, meaning it relies on an underlying machine learning algorithm (in this case Random Forest) to assess the importance of each feature. It then compares the importance of the real features to that of random "shadow" features, in order to determine which ones are truly relevant.

How Boruta Works

Now that we understand the motivation behind feature selection and the different types of algorithms, let‘s take a closer look at how Boruta works under the hood. At a high level, the procedure is:

1. Extend the dataset by adding copies of all features (these are called shadow features)

2. Shuffle the values in each shadow feature to remove their correlations with the target

3. Train a Random Forest model on the extended dataset

4. Calculate the feature importances (the default is Mean Decrease Accuracy)

5. Check whether each real feature has a higher importance than the best of its shadow features. If so, mark the feature as important.

6. Remove the features that have importances significantly lower than their best shadow features

7. Repeat steps 3-6 until all features are either marked as important or removed (or a max number of iterations is reached)

The idea is that by comparing the real features to randomized versions of themselves, we can determine whether their predictive power is truly better than random noise. Features that are consistently more important than the best shadow features are likely to be truly relevant to the outcome.

One key advantage of Boruta is that it captures all relevant features, even if they are weakly relevant or somewhat redundant with other features. This can be useful in applications like genetics, where the goal is often to identify all factors that are related to an outcome, not just the minimal set for optimal prediction.

Comparing to Other Methods

So how does Boruta stack up to other feature selection techniques? Let‘s consider a few common alternatives:

• Recursive Feature Elimination (RFE): This is another wrapper method that builds a model with all features, ranks them by importance, removes the least important, and repeats until a desired number of features is reached. RFE is great for finding a small subset of highly predictive features, but may eliminate relevant features that are redundant with others.

• Lasso Regression: Lasso is an embedded method that adds a regularization term to linear regression, shrinking the coefficients of less important features to zero. This can produce sparse, interpretable models. However, it struggles with highly correlated features and can be sensitive to the regularization strength.

• Univariate Selection: This is a filter method that evaluates each feature independently using a statistical test like chi-squared or ANOVA F-test. Features with the strongest relationships with the target are kept. While fast and simple, univariate selection can miss important interactions between features.

In general, Boruta tends to select more features than minimal-optimal methods like RFE or Lasso. It‘s a good choice when you want to identify all relevant signals and don‘t mind some redundancy. On the other hand, if you need the absolute smallest, most interpretable feature set for making predictions, other methods may be preferable.

Example in R

Theory is great, but to really understand how Boruta works, there‘s no substitute for trying it out yourself! Let‘s walk through an example using the famous Titanic dataset. We‘ll use R, but the same principles apply in Python or other languages.

First, make sure you have the Boruta package installed:

install.packages("Boruta")
library(Boruta)

Load the Titanic data into a data frame:

titanic <- read.csv("titanic.csv")
str(titanic)

Before running Boruta, we need to do some data cleaning – remove missing values, convert categoricals to factors, etc:

titanic <- titanic[complete.cases(titanic),]
titanic$Survived <- as.factor(titanic$Survived)
titanic$Pclass <- as.factor(titanic$Pclass)
titanic$Sex <- as.factor(titanic$Sex)
titanic$Embarked <- as.factor(titanic$Embarked)

Now we‘re ready to run Boruta! We‘ll use the Titanic Survival as the target variable and the rest as features:

boruta.train <- Boruta(Survived ~ ., data = titanic, doTrace = 2)
print(boruta.train)

This will print out a summary of the Boruta object, including how many features were confirmed important, tentative, or rejected. We can also plot the feature importances:

plot(boruta.train, xlab = "", xaxt = "n")
lz<-lapply(1:ncol(boruta.train$ImpHistory),function(i)
  boruta.train$ImpHistory[is.finite(boruta.train$ImpHistory[,i]),i])
names(lz) <- colnames(boruta.train$ImpHistory)
Labels <- sort(sapply(lz,median))
axis(side = 1,las=2,labels = names(Labels),
     at = 1:ncol(boruta.train$ImpHistory), cex.axis = 0.7)

Here we can see the distribution of importance scores for each feature, compared to its shadow copies. Features that are consistently more important than their shadows are marked green (confirmed), those that are consistently less important are red (rejected), and those that are sometimes more/less important are marked yellow (tentative).

To get the final confirmed feature subset:

final.boruta <- TentativeRoughFix(boruta.train)
getSelectedAttributes(final.boruta, withTentative = F)

And that‘s it! We‘ve successfully used Boruta to identify the relevant features in the Titanic dataset. Feel free to experiment with different datasets and parameters to get a feel for how the algorithm behaves.

Interpreting Results

Once you‘ve run Boruta on your data, how do you interpret the results? The key things to look at are:

• Number of confirmed features: This tells you how many features were consistently more important than random noise. The more confirmed features, the more complex your model will likely need to be.

• Number of rejected features: Conversely, the rejected features are those that were consistently less important than noise. These can safely be eliminated from your model.

• Number of tentative features: Tentative features are those whose importance was sometimes higher and sometimes lower than shadow features. You may want to investigate these further before deciding whether to include them.

• Importance distributions: The plot of importance scores can give you a sense of how much more relevant the confirmed features are compared to the rejected ones. Wider gaps imply stronger feature relevance.

It‘s important to remember that Boruta is a heuristic method, not a statistical test. The final decision of which features to use should be based on domain knowledge as well as empirical results. Just because a feature is marked as confirmed does not necessarily mean it will improve your model‘s performance.

Best Practices

To get the most out of Boruta, there are a few best practices to keep in mind:

1. Scale your data: Since Boruta uses Random Forest under the hood, it‘s best to standardize your numeric features to a consistent range. This ensures that the importances are comparable across features.

2. Tune the number of iterations: If you have a lot of tentative features, you may need to increase the maxRuns parameter to allow Boruta more chances to confirm them as important or unimportant. The default is 100.

3. Handle imbalanced classes: If your target variable is highly skewed, consider using stratified sampling or class weights to ensure the Random Forest model is not biased towards the majority class.

4. Use shadow features judiciously: While shadow features are essential to the Boruta algorithm, having too many can slow down the runtime considerably. If you have a very large feature space, consider using a subset of shadow features instead.

5. Combine with other methods: Boruta is not a silver bullet – it can still select irrelevant or redundant features, especially if they are highly correlated with truly important ones. Consider using Boruta as a first pass, then further refining your feature set with another method like RFE or Lasso.

Conclusion

Congratulations, you now have a solid understanding of the Boruta algorithm and how to use it for feature selection in R! We‘ve covered the key concepts behind the method, compared it to some popular alternatives, walked through an example implementation, and discussed best practices for interpreting and using the results.

To recap, Boruta is an all-relevant feature selection method that compares the importance of real features to that of randomized shadow features. It aims to identify all features that are somehow related to the target variable, even if they are weakly relevant or redundant. This makes it well-suited for applications where the goal is to understand the full set of factors influencing an outcome, rather than just building the most parsimonious predictive model.

On the other hand, if you need a minimal feature set for optimal performance, other methods like recursive feature elimination or lasso regression may be preferable. The key is to understand the strengths and limitations of each approach, and choose the one that best aligns with your specific goals and constraints.

Of course, Boruta is just one of many feature selection techniques out there. Other methods worth exploring include genetic algorithms, simulated annealing, and information gain. Ultimately, the best approach will depend on your particular dataset, domain, and objectives.

I hope this post has given you a useful framework for thinking about feature selection, and some practical tools for implementing it in your own work. Happy modeling!