Bayesian Decision Theory – Discriminant Functions and Normal Density (Part 3)

Introduction

Welcome back to our multi-part series on Bayesian decision theory! In the previous two articles, we laid the foundation by introducing the core concepts of Bayesian decision making, including prior probabilities, class-conditional densities, posterior probabilities, and Bayes optimal classifiers. We also examined the decision-making process for different scenarios involving various states of nature and actions.

In this article, we will dive deeper into the inner workings of Bayesian classifiers by exploring discriminant functions and their relationship to decision surfaces. We will pay special attention to the two-category case and investigate how the normal (Gaussian) density comes into play. Finally, we will implement an interactive web-based discriminant function calculator to gain hands-on experience with these concepts. Let‘s get started!

Classifiers, Discriminant Functions, and Decision Surfaces

At its core, a Bayesian classifier aims to assign input feature vectors to different classes based on a decision rule that minimizes the overall risk or maximizes the posterior probability. This classification process can be represented in various ways, but one of the most common approaches is to use a set of discriminant functions g_i(x), where i = 1, …, c for a c-class problem.

The classifier computes the value of each discriminant function g_i(x) for an input vector x and assigns x to class ω_i if g_i(x) > g_j(x) for all j ≠ i. In other words, the classifier selects the class corresponding to the discriminant function with the highest value.

The choice of discriminant functions is not unique, as they can be scaled or shifted without affecting the decision boundary. Common choices include:

• g_i(x) = -R(α_i | x), the negative conditional risk
• g_i(x) = P(ω_i | x), the posterior probability
• g_i(x) = ln p(x | ω_i) + ln P(ω_i), the log-likelihood plus the log prior

The decision boundary between classes in feature space is determined by the points where the discriminant functions are equal, i.e., g_i(x) = g_j(x). These boundaries partition the feature space into c decision regions R₁, R₂, …, R_c, where the classifier assigns input vectors in region R_i to class ω_i.

The Two-Category Case

For a two-class problem, also known as a dichotomizer, we can simplify the decision-making process by defining a single discriminant function:

g(x) ≡ g₁(x) – g₂(x)

The classifier then assigns x to class ω₁ if g(x) > 0 and to class ω₂ otherwise. This simplification allows us to focus on a single function and its sign rather than comparing multiple functions.

Some common discriminant functions for the two-category case include:

• g(x) = P(ω₁ | x) – P(ω₂ | x), the difference of posteriors
• g(x) = ln (p(x | ω₁) / p(x | ω₂)) + ln (P(ω₁) / P(ω₂)), the log-likelihood ratio plus the log prior ratio

The Normal (Gaussian) Density

In many real-world applications, the class-conditional densities p(x | ω_i) are assumed to follow a normal or Gaussian distribution. The normal density is an attractive choice due to its mathematical properties and the central limit theorem, which states that the sum of a large number of independent random variables tends towards a Gaussian distribution.

For a univariate normal density, the probability density function (PDF) is given by:

p(x) = (1 / (σ√(2π))) exp(-(x – μ)² / (2σ²))

where μ is the mean, and σ² is the variance. The normal density is fully characterized by these two parameters.

In the multivariate case, the normal density for a d-dimensional vector x is given by:

p(x) = (1 / ((2π)^(d/2) |Σ|^(1/2))) exp(-1/2 (x – μ)^T Σ^(-1) (x – μ))

where μ is the d-dimensional mean vector, Σ is the d × d covariance matrix, |Σ| is the determinant of Σ, and Σ^(-1) is the inverse of Σ.

The covariance matrix Σ plays a crucial role in the multivariate normal density. It is symmetric and positive semidefinite, and its diagonal elements σ_ii represent the variances of the individual components, while the off-diagonal elements σ_ij represent the covariances between components. If σ_ij = 0, then the corresponding components x_i and x_j are statistically independent.

Discriminant Functions for Normal Densities

When the class-conditional densities are normally distributed, we can derive explicit expressions for the discriminant functions. Let‘s consider a two-category case with classes ω₁ and ω₂, where the class-conditional densities are:

p(x | ω₁) = N(μ₁, Σ₁)
p(x | ω₂) = N(μ₂, Σ₂)

Here, N(μ, Σ) denotes a normal density with mean μ and covariance Σ.

The log-likelihood ratio discriminant function for this case is:

g(x) = ln (p(x | ω₁) / p(x | ω₂))
= -1/2 (x – μ₁)^T Σ₁^(-1) (x – μ₁) + 1/2 (x – μ₂)^T Σ₂^(-1) (x – μ₂) + ln (|Σ₂| / |Σ₁|)

If the covariance matrices are equal (Σ₁ = Σ₂ = Σ), the discriminant function simplifies to:

g(x) = (μ₁ – μ₂)^T Σ^(-1) x – 1/2 (μ₁^T Σ^(-1) μ₁ – μ₂^T Σ^(-1) μ₂)

This is a linear function of x, and the decision boundary is a hyperplane.

If the covariance matrices are diagonal (Σ_i = diag(σ_i1², …, σ_id²)), the discriminant function becomes:

g(x) = Σ_j=1^d ((x_j – μ_1j)² / σ_1j² – (x_j – μ_2j)² / σ_2j²) + Σ_j=1^d ln (σ_2j / σ_1j)

This corresponds to a quadratic decision boundary.

Discriminant Function Calculator

To help visualize and experiment with discriminant functions, we have developed an interactive web-based discriminant function calculator. The calculator allows you to input the parameters of two normally distributed classes (means, variances, and prior probabilities) and displays the resulting discriminant function and decision boundary.

You can access the discriminant function calculator at the following link: [Insert link to your web-based calculator]

The calculator provides a user-friendly interface to explore different scenarios and gain intuition about the behavior of discriminant functions for normally distributed classes.

Real-World Applications

Discriminant functions and Bayesian decision theory find applications in various domains, including:

1. Spam email filtering: Classify incoming emails as spam or non-spam based on features like word frequencies, sender information, and email structure.

2. Medical diagnosis: Determine the likelihood of a patient having a specific disease based on symptoms, test results, and risk factors.

3. Credit scoring: Assess the creditworthiness of loan applicants using features such as income, credit history, and employment status.

4. Fraud detection: Identify fraudulent transactions or insurance claims by analyzing patterns and anomalies in data.

5. Speech recognition: Classify spoken words or phonemes based on acoustic features extracted from audio signals.

In each of these applications, the goal is to make optimal decisions based on available information and minimize the overall risk or cost associated with misclassifications.

Conclusion

In this article, we explored the concepts of discriminant functions, decision surfaces, and the normal density in the context of Bayesian decision theory. We focused on the two-category case and derived discriminant functions for normally distributed classes with different covariance structures.

By understanding these concepts and utilizing tools like the discriminant function calculator, you can gain a deeper understanding of how Bayesian classifiers work and how to apply them to real-world problems. In the next article, we will delve into more advanced topics, such as parameter estimation, model selection, and the connections between Bayesian decision theory and other machine learning approaches.

Stay tuned for more insights and practical examples in the exciting world of Bayesian decision theory!

References

1. Duda, R. O., Hart, P. E., & Stork, D. G. (2012). Pattern classification. John Wiley & Sons.
2. Bishop, C. M. (2006). Pattern recognition and machine learning. Springer.
3. Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT Press.