4 Ways to Evaluate Your Machine Learning Model: Cross Validation Techniques (with Python Code)

Introduction

When building machine learning models, one of the most critical steps is evaluating how well the model performs on unseen data. Just because a model achieves high accuracy on the training data doesn‘t necessarily mean it will generalize well to new examples. This is where cross validation comes into play.

Cross validation is a resampling technique that allows you to assess the performance and stability of your machine learning model. The fundamental idea is to split your data into multiple subsets, use some for training the model and the others for testing, and repeat this process multiple times. This provides a more robust estimate of the model‘s true performance compared to evaluating on a single train/test split.

In this article, we‘ll dive into four commonly used cross validation techniques:

1. Holdout method
2. Leave-one-out cross validation (LOOCV)
3. K-fold cross validation
4. Stratified k-fold cross validation

We‘ll explain how each method works, discuss their strengths and weaknesses, and show how to implement them in Python with code examples. By the end, you‘ll have a solid understanding of cross validation and be able to choose the appropriate technique for your machine learning projects. Let‘s get started!

1. Holdout Method

The holdout method is the simplest form of cross validation. It involves splitting your dataset into two subsets:

• Training set – used to train the model
• Test set (or holdout set) – used to evaluate the model‘s performance on unseen data

A common split is to use 70-80% of the data for training and the remaining 20-30% for testing. The split is usually done randomly to ensure the subsets are representative of the original data distribution.

Here‘s how to implement the holdout method in Python using scikit-learn:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score

# Load example dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train model
model = LogisticRegression()
model.fit(X_train, y_train)

# Evaluate on test set  
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Holdout Accuracy: {accuracy:.3f}")

The holdout method is fast and easy to implement. However, its evaluation can have high variance, especially with small datasets, since performance depends heavily on which examples end up in the train and test sets. Repeated holdout with different random splits can help reduce this variance.

2. Leave-One-Out Cross Validation (LOOCV)

Leave-one-out cross validation takes the holdout method to the extreme. With LOOCV, you use a single example from the dataset as the test set and train the model on all other examples. You repeat this for every example in the dataset.

So if you have N examples total, you will train N separate models, each one using N-1 examples for training and the remaining example for testing. The final performance estimate is then the average of these N test accuracies.

Implementing LOOCV in Python is straightforward with scikit-learn:

from sklearn.model_selection import LeaveOneOut
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score

iris = load_iris() 
X, y = iris.data, iris.target

model = LogisticRegression()
loo = LeaveOneOut()
accuracies = []

for train_index, test_index in loo.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    accuracies.append(accuracy_score(y_test, y_pred))

print(f"LOOCV Accuracy: {np.mean(accuracies):.3f}")  

LOOCV is a nearly unbiased estimate of the model‘s performance since each example gets a chance to be in the test set. However, it can have high variance due to the models being highly correlated (each one is trained on nearly identical data). It‘s also very computationally expensive, as you need to train N separate models. As such, it‘s not commonly used in practice except with very small datasets.

3. K-Fold Cross Validation

K-fold cross validation strikes a balance between the holdout method and LOOCV. The procedure is as follows:

1. Randomly split the dataset into K equal-sized subsets ("folds").
2. For each fold k = 1, 2, …, K:
   • Use fold k as the test set
   • Use the remaining K-1 folds as the training set
   • Fit the model on the training set and evaluate on the test set
   • Record the test accuracy
3. Estimate overall performance as the average of the K test accuracies

Common choices for K are 5 or 10. Larger K means lower bias but higher variance in the performance estimate.

Here‘s the code to run K-fold cross validation in Python:

from sklearn.model_selection import KFold
from sklearn.linear_model import LogisticRegression  
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target

model = LogisticRegression()
kf = KFold(n_splits=5)
accuracies = []

for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index] 
    y_train, y_test = y[train_index], y[test_index]

    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    accuracies.append(accuracy_score(y_test, y_pred))

print(f"K-Fold Accuracy: {np.mean(accuracies):.3f}")

K-fold cross validation generally provides a less biased estimate than the holdout method but with higher variance than LOOCV. It‘s a good choice in most situations as it balances bias and variance while being computationally feasible.

4. Stratified K-Fold Cross Validation

Stratified k-fold cross validation is a variation of k-fold that ensures each fold has approximately the same class distribution as the original dataset. This is important when working with imbalanced datasets where some classes have far fewer examples than others.

Without stratification, some folds may end up with few or no examples of the minority class, leading to poor results. Stratified k-fold ensures all classes are well-represented in each fold.

The implementation in Python requires just one small change to use StratifiedKFold instead of KFold:

from sklearn.model_selection import StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris 
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target

model = LogisticRegression()  
skf = StratifiedKFold(n_splits=5)
accuracies = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

    model.fit(X_train, y_train) 
    y_pred = model.predict(X_test)
    accuracies.append(accuracy_score(y_test, y_pred))

print(f"Stratified K-Fold Accuracy: {np.mean(accuracies):.3f}")  

Stratified k-fold is recommended over standard k-fold for classification problems, especially with imbalanced classes. For regression, since there are no distinct classes, regular k-fold is used.

Comparing Cross Validation Techniques

So which cross validation method should you use? The answer depends on your specific dataset and goals. Here are some general guidelines:

• Holdout method: Fast and simple, but can have high variance. Use for quick evaluation or very large datasets.

• LOOCV: Nearly unbiased, but can have high variance and is computationally expensive. Use for small datasets (<1000 examples).

• K-fold: Balances bias and variance, computationally feasible. Use as the default for medium-sized datasets.

• Stratified k-fold: Same as k-fold but ensures class balance. Use for classification with imbalanced classes.

In practice, k-fold (stratified for classification) with K=5 or 10 is a good choice in most cases. It provides a reliable estimate of model performance without too much computational overhead.

Bias-Variance Tradeoff

When choosing a cross validation method and interpreting the results, it‘s important to keep the bias-variance tradeoff in mind.

• Bias refers to the difference between the model‘s expected predictions and the true values. High bias means the model is underfitting and not capturing the underlying patterns in the data.

• Variance refers to the variability of the model‘s predictions for different training sets. High variance means the model is overfitting and is too sensitive to the noise in the training data.

There is often a tradeoff between bias and variance. For example, the holdout method tends to have higher bias than k-fold since it trains on less data. But it also tends to have lower variance since performance is evaluated on a single test set.

On the other hand, LOOCV has low bias since it trains on nearly all the data but high variance since the models are highly correlated. K-fold strikes a balance, with bias and variance depending on the choice of K.

The goal is to find the sweet spot that minimizes both bias and variance to get a model that generalizes well. Cross validation helps estimate what this performance will be on unseen data.

Conclusion

Cross validation is an essential tool for evaluating machine learning models. By using techniques like the holdout method, k-fold, and LOOCV, you can get a more robust and reliable estimate of a model‘s performance compared to evaluating on a single dataset split.

The choice of cross validation technique depends on the size and characteristics of your dataset, as well as computational constraints. In general, k-fold (stratified for classification) with K=5 or 10 is a good default that balances bias and variance.

When interpreting cross validation results, keep the bias-variance tradeoff in mind. The goal is to find a model that achieves low bias and low variance, indicating it has learned the underlying patterns in the data without overfitting.

By understanding and applying these cross validation techniques, you can more effectively evaluate and select machine learning models, leading to better performance on real-world tasks. Experiment with different techniques and see how they impact your model evaluation process!