A Comprehensive Guide to Gaussian Naive Bayes Classification with Hyperparameter Tuning

Naive Bayes is a family of probabilistic algorithms that apply Bayes‘ theorem with strong independence assumptions between the features. Despite their seemingly oversimplified assumptions, Naive Bayes classifiers have been found to perform well in many real-world applications, particularly in high-dimensional datasets. They require a small amount of training data to estimate the necessary parameters and are extremely fast compared to more sophisticated methods.

One popular variant is the Gaussian Naive Bayes algorithm, which assumes that the continuous values associated with each class are distributed according to a Gaussian distribution. In this post, we‘ll dive deep into the inner workings of Gaussian Naive Bayes, implement it from scratch in Python, and share tips to tune its hyperparameters for optimal performance.

The Mathematical Foundations of Gaussian Naive Bayes

At its core, Naive Bayes is a probabilistic model that uses Bayes‘ theorem to calculate the posterior probability of a class C given the observed features X:

$P(C|X) = \frac{P(X|C) P(C)}{P(X)}$

where:

• P(C|X) is the posterior probability of class C given the predictor X
• P(X|C) is the likelihood of predictor X given class C
• P(C) is the prior probability of class C
• P(X) is the prior probability of predictor X

Since P(X) is constant for all classes, we only need to maximize the numerator P(X|C)P(C). The "naive" assumption of class-conditional independence allows us to express the likelihood as a product of simpler probabilities:

$P(X|C) = \prod_{i=1}^{n} P(X_i|C)$

where $X_i$ is the ith feature and n is the number of features.

In Gaussian Naive Bayes, we assume that each feature follows a Gaussian distribution for each class:

$P(X_i|C) = \frac{1}{\sqrt{2\pi\sigma^2_C}}\exp(-\frac{(X_i – \mu_C)^2}{2\sigma^2_C})$

where $\mu_C$ and $\sigma^2_C$ are the mean and variance of feature $X_i$ in class C. These parameters are estimated from the training data using maximum likelihood:

$\mu_C = \frac{1}{NC}\sum{i=1}^{N_C} X_i$

$\sigma^2_C = \frac{1}{NC}\sum{i=1}^{N_C}(X_i – \mu_C)^2$

where $N_C$ is the number of samples in class C.

The class priors P(C) are typically estimated by the relative frequencies of the classes in the training set:

$P(C) = \frac{N_C}{N}$

where N is the total number of samples.

Comparing Naive Bayes Variants

While GaussianNB assumes normally distributed features, other Naive Bayes variants exist for different types of data:

• MultinomialNB: Assumes the features are generated from a simple multinomial distribution. Suitable for discrete counts, such as word frequencies in text classification.
• BernoulliNB: Assumes the features are binary (0s and 1s). Useful when your feature vectors are binary, such as a bag of words encoding.
• CategoricalNB: Assumes categorical features with possibly different number of categories per feature. A more computationally efficient implementation of multinomial Naive Bayes.

Here is a summary of when to use each variant depending on the characteristics of your data:

Algorithm	Data Type	Example Use Cases
GaussianNB	Continuous	Iris species, medical diagnosis
MultinomialNB	Discrete counts	Text classification, spam filtering
BernoulliNB	Binary	Click prediction, sentiment analysis
CategoricalNB	Categorical	Census data, survey responses

It‘s important to preprocess your data appropriately for each variant. For example, binarizing your features before applying BernoulliNB, or encoding categories as integers for CategoricalNB.

Applications of Gaussian Naive Bayes

Gaussian Naive Bayes has been successfully used across a variety of domains thanks to its simplicity, efficiency, and surprisingly good performance. Some example applications include:

• Medical diagnosis: Classifying patients as having a disease or not based on their observed symptoms. GaussianNB works well here since many medical measurements like blood pressure, BMI, and cholesterol are continuous-valued.

• Spam filtering: Determining if an email is spam or not based on the frequencies of certain words or characters. While MultinomialNB and BernoulliNB are more commonly used for this discrete data, GaussianNB can still perform well if the input is transformed to continuous values.

• Financial risk modeling: Predicting the risk level (high, medium, low) of a loan applicant defaulting based on their income, credit score, and other numeric factors. The normality assumption of GaussianNB fits well with these kinds of financial metrics.

• Face recognition: Identifying individuals based on numeric features extracted from their facial images, such as distances and angles between key points. GaussianNB‘s efficiency makes it suitable for real-time face recognition systems.

• Weather forecasting: Classifying weather conditions (sunny, cloudy, rainy) based on continuous measurements like temperature, humidity, and pressure. The simplicity of GaussianNB allows it to quickly learn and adapt to changing weather patterns.

Implementing Gaussian Naive Bayes from Scratch

To gain a deeper understanding of how Gaussian Naive Bayes works, let‘s implement it from scratch in Python. We‘ll break it down into the following steps:

1. Separate the training data by class
2. Compute the mean and variance of each feature for each class
3. Compute the prior probability of each class
4. Define a function to calculate the Gaussian probability density
5. Define a function to predict the class with the highest posterior probability

import numpy as np

class GaussianNB:
    def __init__(self):
        self.classes = None
        self.class_priors = None
        self.feature_means = {}
        self.feature_vars = {}

    def fit(self, X, y):
        self.classes = np.unique(y)
        self.class_priors = {c: np.sum(y == c) / len(y) for c in self.classes}

        for c in self.classes:
            X_c = X[y == c]
            self.feature_means[c] = np.mean(X_c, axis=0)
            self.feature_vars[c] = np.var(X_c, axis=0)

    def predict(self, X):
        return np.array([self._predict_one(x) for x in X])

    def _predict_one(self, x):
        posteriors = {}
        for c in self.classes:
            mu = self.feature_means[c]
            var = self.feature_vars[c]
            likelihood = np.prod(self._gaussian_density(x, mu, var), axis=-1)
            posteriors[c] = self.class_priors[c] * likelihood
        return max(posteriors, key=posteriors.get)

    def _gaussian_density(self, x, mu, var):
        return 1/np.sqrt(2*np.pi*var) * np.exp(-(x-mu)**2 / (2*var))

Let‘s test it on the Iris dataset:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

gnb = GaussianNB()
gnb.fit(X_train, y_train)
y_pred = gnb.predict(X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))

Output:

Accuracy: 0.9666666666666667

Our from-scratch implementation achieves the same performance as scikit-learn‘s GaussianNB on this simple dataset. However, scikit-learn‘s optimized version is much faster and more numerically stable.

Hyperparameter Tuning with GridSearchCV

The key hyperparameter to tune for GaussianNB is var_smoothing, which controls the amount of smoothing applied to the feature variances to avoid numerical instability and overfitting. We can use scikit-learn‘s GridSearchCV to find the optimal var_smoothing value:

from sklearn.model_selection import GridSearchCV
from sklearn.naive_bayes import GaussianNB

param_grid = {‘var_smoothing‘: np.logspace(0,-9, num=100)}
grid = GridSearchCV(GaussianNB(), param_grid, cv=5)
grid.fit(X_train, y_train)

print("Best score:", grid.best_score_)
print("Best params:", grid.best_params_)

Output:

Best score: 0.9666666666666668
Best params: {‘var_smoothing‘: 1.0}

For the Iris dataset, the default var_smoothing=1e-9 works well. But in general, it‘s good practice to tune this parameter, especially for high-dimensional data where numerical instabilities are more likely.

Preprocessing and Feature Selection

As with any machine learning algorithm, preprocessing your data correctly is crucial to getting good results with Gaussian Naive Bayes. Here are some important steps:

• Normalization: Since GaussianNB assumes normally distributed features, it‘s a good idea to normalize your data to have zero mean and unit variance. You can use scikit-learn‘s StandardScaler for this.

• Handling missing values: GaussianNB cannot handle missing values in the input data. You can either remove samples with missing values (if they are few) or impute them with a technique like mean imputation using SimpleImputer.

• Encoding categorical variables: If your data has categorical features, you need to encode them as numbers before feeding them to GaussianNB. One common approach is one-hot encoding using scikit-learn‘s OneHotEncoder.

• Feature selection: In high-dimensional datasets with many irrelevant or redundant features, feature selection can improve both the accuracy and efficiency of GaussianNB. Univariate feature selection methods like SelectKBest and SelectPercentile work well with Naive Bayes.

Here‘s an example pipeline that puts these steps together:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder 
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.feature_selection import SelectKBest, f_classif

num_pipeline = Pipeline([
    (‘imputer‘, SimpleImputer(strategy=‘median‘)),
    (‘scaler‘, StandardScaler())
])

cat_pipeline = Pipeline([
    (‘imputer‘, SimpleImputer(strategy=‘most_frequent‘)),
    (‘onehot‘, OneHotEncoder(handle_unknown=‘ignore‘))    
])

preprocess_pipeline = ColumnTransformer([
    (‘num‘, num_pipeline, num_features),
    (‘cat‘, cat_pipeline, cat_features)  
])

model_pipeline = Pipeline([
    (‘preprocessing‘, preprocess_pipeline),
    (‘feature_selection‘, SelectKBest(f_classif, k=10)),
    (‘classifier‘, GaussianNB(var_smoothing=1e-9))
])

Putting your feature preprocessing and model training steps into a single pipeline ensures that the same transformations are applied consistently to training and test data, reducing the chances of data leakage.

Scaling Gaussian Naive Bayes

Naive Bayes models are often used in real-time systems that require very fast predictions on new data points, such as online learning and anomaly detection. Even though training a Naive Bayes model is already quite fast, there are a few ways to speed up predictions even further:

• Probability calibration: The predict_proba() method of GaussianNB returns the raw posterior probabilities for each class, which involves expensive exp() calls. If you only need the predicted class labels, use the predict() method instead.

• Vectorization: Make sure your input data is stored as a NumPy array or SciPy sparse matrix, which enables fast vectorized operations in the underlying C code of scikit-learn.

• Parallel processing: The partial_fit() method of MultinomialNB and BernoulliNB allows you to train the model incrementally on batches of data, which can be parallelized across multiple cores or machines. GaussianNB does not support partial_fit() since it requires the entire dataset to estimate the feature means and variances.

• Model compression: If your trained GaussianNB model is taking up too much memory, you can try reducing the precision of the floating-point numbers used to store the model parameters. For example, converting from float64 to float32 will halve the model size with minimal impact on accuracy.

Remember that premature optimization is the root of all evil. Always profile your code to identify the true performance bottlenecks before attempting any optimization.

Current Research and Future Directions

Despite its simplicity and strong assumptions, the Naive Bayes family of algorithms remains a popular and effective choice for many classification problems. However, there are still many active areas of research aimed at improving Naive Bayes:

• Bayesian network classifiers: Relaxing the independence assumption by learning a more flexible graphical structure over the features, such as Tree-Augmented Naive Bayes (TAN) and Averaged One-Dependence Estimators (AODE).

• Semi-supervised learning: Exploiting unlabeled data to improve the performance of Naive Bayes classifiers when labeled data is scarce, using techniques like Expectation-Maximization (EM) and Co-Training.

• Transfer learning: Adapting Naive Bayes models trained on one domain to a different but related domain, by learning a transformation of the feature space or a weighting of the training examples.

• Bayesian deep learning: Combining the simplicity and interpretability of Naive Bayes with the representational power of deep neural networks, by stacking multiple Naive Bayes layers or using variational inference to learn a deep generative model.

As Naive Bayes continues to be applied to newer and more complex domains, we can expect to see further innovations that build upon its core strengths while addressing its limitations.

Conclusion

I hope this post has given you a comprehensive overview of the Gaussian Naive Bayes algorithm, from its mathematical foundations and computational efficiencies to its practical applications and future research directions. We‘ve seen how to implement it from scratch, tune its hyperparameters, preprocess and scale the input data, and integrate it into real-world systems.

While Gaussian Naive Bayes is not the most sophisticated or flexible machine learning algorithm out there, its simplicity and efficiency make it a valuable tool in any data scientist‘s toolkit. By deeply understanding its strengths and weaknesses, you can make informed decisions about when and how to apply it to your own classification problems.

As with any algorithm, the key to success with Gaussian Naive Bayes is to iterate and experiment with different preprocessing, hyperparameter, and evaluation schemes on your specific dataset. Don‘t be afraid to question its assumptions, try out new techniques, and learn from both successes and failures. Happy classifying!