Tuned vs. Untuned Classification Models: How Much Does Hyperparameter Optimization Matter?

Classification is one of the most common types of machine learning problems, with widespread applications from spam filtering to medical diagnosis to fraud detection. At their core, classification models learn patterns that distinguish between different categorical class labels, allowing them to predict the correct class for new unlabeled examples.

While the concepts behind classification are fairly straightforward, actually building high-performing real-world classifiers can be challenging. One key aspect of model development that is often overlooked by beginners is hyperparameter tuning – the process of optimizing the settings of a model to maximize its performance on a given dataset.

In this post, we‘ll take a deep dive into hyperparameter tuning for classification models. We‘ll start by explaining what hyperparameters are and why they matter. Then we‘ll look at some empirical evidence comparing the performance of tuned and untuned models across a variety of algorithms and datasets. Finally, we‘ll discuss some best practices for hyperparameter tuning and give recommendations for when tuning is worthwhile versus using untuned models.

What are Hyperparameters and Why Tune Them?

Hyperparameters refer to the configuration settings of a machine learning model that are set prior to training. In contrast to the model parameters that are learned from the training data (e.g. regression coefficients, neural network weights), hyperparameters must be specified by the practitioner developing the model.

Examples of common hyperparameters include:

• The regularization strength in logistic regression or SVMs
• The maximum depth or minimum samples per leaf in decision trees
• The number of trees in a random forest
• The architecture (number and size of hidden layers) of a neural network
• The learning rate of the optimizer in deep learning models

These settings can have a major impact on the bias-variance tradeoff of the model – how well it fits the training data versus how well it generalizes to new data. An underfit model will have poor performance because it fails to learn the underlying patterns, while an overfit model essentially memorizes the training examples and performs poorly on unseen data.

The goal of hyperparameter optimization is to find the sweet spot that achieves maximum generalization performance. By systematically evaluating different combinations of hyperparameters, we can often substantially improve upon the default out-of-the-box settings provided by most machine learning libraries. Especially for complex models with many hyperparameters, the difference between an untuned and well-tuned model can be drastic.

Approaches to Hyperparameter Tuning

There are a few different ways to approach the hyperparameter optimization problem:

Manual tuning involves changing the hyperparameters by hand and evaluating the model each time to gauge the effect. This can be a good way to build intuition, but it quickly becomes infeasible as the number of hyperparameters and range of values grows.

Grid search is a brute force approach that exhaustively evaluates a specified set of hyperparameter values. For example, you might try 3 different values for 4 different hyperparameters, training a model for each of the 81 possible combinations. Grid search is simple to implement but can be very computationally expensive.

Random search is similar to grid search but samples hyperparameter settings randomly instead of exhaustively. This allows covering a wider range of values in a more efficient manner. Research has shown random search tends to outperform grid search in most cases.

Bayesian optimization takes a more sophisticated approach, building a probabilistic model of the objective function (the performance metric to be optimized) and using it to intelligently choose the next set of hyperparameters to evaluate. This enables focusing the search on the most promising settings and requires fewer total model evaluations.

Performance of Tuned vs. Untuned Models

So how much difference does hyperparameter tuning actually make in practice? Let‘s take a look at some empirical evidence from the research literature.

A 2018 study by Probst et al. compared the performance of tuned and untuned models on 38 datasets across 6 common classification algorithms:

• Logistic regression
• Decision tree
• Random forest
• Gradient boosted trees
• Support vector machine (SVM)
• k-nearest neighbors (kNN)

They used random search with 100 iterations to tune 2-5 key hyperparameters for each model. The results showed that tuning improved the average accuracy by 1-5% across the algorithms, with the biggest gains for the more complex models like random forest and gradient boosting. However, the untuned models were still quite competitive overall, often coming within a percentage point or two of the optimized models.

Another study by Thornton et al. in 2013 used a sophisticated automated tuning approach called Auto-WEKA on 21 datasets with 10 different classification algorithms. They found tuning improved the average accuracy by 5.9% compared to the best individual untuned algorithm on each dataset. However, the untuned models did better than Auto-WEKA in 1/3 of the datasets, showing that tuning is not always beneficial.

Looking at SVM models specifically, a 2010 paper by Verbeke et al. found tuning improved accuracy by 3-4% on average across 10 datasets compared to the default settings. Interestingly, they found the polynomial kernel tended to outperform the more popular RBF kernel after tuning.

For deep learning models, hyperparameter tuning can potentially lead to even greater improvements due to the large number of architectural and optimization settings that interact in complex ways. However, full systematic tuning is often prohibitively expensive computationally. In a 2012 paper, Bergstra and Bengio found that random search worked nearly as well as grid search for tuning neural networks while requiring much less compute time.

To Tune or Not to Tune?

Based on the research, it‘s clear that hyperparameter optimization can provide meaningful accuracy gains in many cases, but it‘s not always necessary or cost-effective. So when should you bother tuning your models?

I‘d recommend starting with untuned models in the early stages of a project when you are more concerned with getting a baseline and iterating quickly. The reduced development time is often worth a small hit in performance. You can always circle back and tune the most promising models later.

Tuning tends to be more important when:

• Squeezing out every last bit of performance is critical (e.g. in a Kaggle competition)
• You have a complex model with many hyperparameters that interact in non-obvious ways
• The untuned model is clearly over- or under-fitting the data
• You have enough computing resources to run many training iterations
• Interpretability is not a major concern (tuned models are often harder to interpret)

On the other hand, untuned models can be preferable when:

• You need to quickly deploy a "good enough" model
• Computational resources are limited
• Simplicity and ease of understanding are a priority
• The model is already performing well with default settings

If you do decide to tune your models, I‘d recommend starting with random search as a good default approach. It‘s simple to implement, works well in most cases, and is more efficient than grid search. You can always move to a more sophisticated automated tuning method later if needed.

Conclusion

Hyperparameter tuning is an important tool in the machine learning practitioner‘s toolbox for achieving top-notch model performance. While it can provide significant gains in accuracy, it‘s not always necessary, and the benefits must be weighed against the computational cost.

For most classification problems, I‘d recommend starting with untuned models to establish a baseline and then moving to random search tuning on the most promising algorithms if the performance is not satisfactory. Well-designed default settings from mature ML libraries are often quite competitive with carefully tuned models while requiring much less development time.

The key is striking the right balance and knowing when tuning is likely to be worthwhile for your particular use case. As with most things in machine learning, there‘s no one-size-fits-all answer. But hopefully this post has given you a better understanding of the tradeoffs involved and how to approach the tuning process!