Principal Component Analysis Demystified |

Principal Component Analysis (PCA) is one of the most popular dimensionality reduction techniques used in data science and machine learning. It allows you to reduce the number of variables in a dataset while retaining most of the important information. This is extremely valuable when dealing with high-dimensional datasets, making PCA a must-have tool in the data scientist‘s toolkit.

In this blog post, we‘ll dive into the workings of PCA from a conceptual and practical standpoint. We‘ll strive to demystify how PCA works "under the hood" and demonstrate how to effectively leverage PCA in a typical machine learning workflow using Python and pandas. Whether you‘re new to PCA or looking to deepen your understanding, this post will equip you with the knowledge and code snippets needed to apply PCA with confidence. Let‘s get started!

The Intuition Behind PCA

At its core, PCA aims to find a new set of variables that are linear combinations or mixtures of the initial variables. These new variables, called principal components, have some special properties:

1. They are uncorrelated with each other
2. The first principal component accounts for the maximum amount of variance in the data. The second principal component accounts for the next highest variance and is orthogonal to the first. This continues until you exhaust all the variance in the data.

So PCA takes our original, possibly correlated variables and transforms them into a new set of uncorrelated variables ordered by how much variance they explain from the data.

An intuitive way to think about PCA is to imagine a 3D dataset plotted on x, y, and z axes. PCA will find a new coordinate system defined by principal component axes PC1, PC2, PC3 such that the data, when projected onto these axes, retains most of its original structure. The new axes pass through the areas of maximum variance in the data cloud.

The Mathematics of PCA

Now let‘s dig a little deeper into the mathematics behind PCA. Don‘t worry, we won‘t go too deep into the linear algebra rabbit hole. The key mathematical concepts to understand are:

• Variance and Covariance
• Eigenvectors and Eigenvalues

PCA works by calculating the eigenvectors and eigenvalues of the covariance matrix. The eigenvectors define the directions of the new axes (principal components) and the eigenvalues define the magnitude or amount of variance carried by each axis.

The principal components are new variables that are constructed as linear combinations or weighted averages of the initial variables. The weights in this linear combination are the elements of the eigenvectors. The eigenvectors form a new orthogonal basis for the data.

Mathematically, if X is our original data matrix, then the covariance matrix is given by:

Cov(X) = (X-mean(X))‘(X-mean(X)) / (n-1)

The eigenvectors and eigenvalues are calculated from this covariance matrix. If e_1, e_2, …, e_p are the eigenvectors and λ_1, λ_2, …, λ_p are the corresponding eigenvalues, then the ith principal component is given by:

PC_i = e_i1 X_1 + e_i2 X_2 + … + e_ip * X_p

where e_ij is the jth element of the ith eigenvector.

Step-by-Step: Applying PCA

Now that we understand the intuition and mathematics behind PCA, let‘s walk through the steps to apply PCA to a dataset.

1. Standardize the data: PCA is sensitive to the scaling of the original variables. It‘s crucial to standardize the data to have zero mean and unit variance. This ensures that all variables contribute equally to the analysis. In Python, this can be easily done using StandardScaler from scikit-learn.

2. Calculate the covariance matrix: The covariance matrix captures the correlations between all pairs of variables. It‘s a square symmetric matrix where the (i,j)th element is the covariance between the ith and jth variables. In numpy, the cov function can be used to compute the covariance matrix.

3. Find the eigenvectors and eigenvalues of the covariance matrix: The eigenvectors and eigenvalues are key to determining the principal components. In numpy, the eigh function can be used to compute these.

4. Sort eigenvalues in descending order and choose the top k eigenvectors: The eigenvalues indicate the amount of variance carried by each principal component. Sort the eigenvalues from highest to lowest and choose the top k eigenvectors corresponding to the top k eigenvalues. The number k is the number of dimensions you want to reduce the data to.

5. Construct a projection matrix from the top k eigenvectors: The selected eigenvectors (principal components) form the columns of a projection matrix W. This matrix will be used to transform the original data into the new subspace.

6. Transform the original data using the projection matrix: The original data matrix X is transformed into the new k-dimensional feature subspace by the matrix multiplication: Y = X * W. Each data point is now represented by k features instead of p.

Here‘s a code snippet demonstrating these steps using Python and scikit-learn:

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import pandas as pd

# Load data into a pandas dataframe
data = pd.read_csv(‘data.csv‘)

# Separate features and target
X = data.drop(‘target‘, axis=1)
y = data[‘target‘]

# Standardize the features
scaler = StandardScaler()
X_std = scaler.fit_transform(X)

# Create a PCA instance and fit_transform the data
pca = PCA(n_components=k)
X_pca = pca.fit_transform(X_std)

# Convert to dataframe
component_names = [f"PC{i+1}" for i in range(k)]
X_pca_df = pd.DataFrame(X_pca, columns=component_names)

The choice of k, the number of principal components to keep, is an important parameter. It can be chosen based on the cumulative explained variance ratio. The explained variance ratio of each component indicates the percentage of variance it accounts for. A common approach is to choose k such that the cumulative explained variance is above a certain threshold, like 80% or 90%.

pca = PCA().fit(X_std)
cumulative_variance_ratio = np.cumsum(pca.explained_variance_ratio_)
k = np.argmax(cumulative_variance_ratio >= 0.9) + 1

Interpreting Principal Components

Interpreting the principal components can provide valuable insights into the data. Each principal component is a linear combination of the original features, and the coefficients (or loadings) of this linear combination can be examined to understand what the component represents.

In the iris dataset, for example, if the first principal component has high positive coefficients for petal length and petal width, it indicates that this component largely captures the information in these two variables.

Visualizing the data in the reduced PCA space can also be enlightening. Plotting the data points with the principal components as axes can reveal interesting structures, clusters, or separations in the data that were not apparent in the original high-dimensional space.

import matplotlib.pyplot as plt
import seaborn as sns

sns.scatterplot(x=‘PC1‘, y=‘PC2‘, hue=y, data=X_pca_df)
plt.xlabel(‘Principal Component 1‘)
plt.ylabel(‘Principal Component 2‘)
plt.title(‘Iris Dataset PCA Visualization‘)
plt.show()

PCA in Scikit-Learn

Scikit-learn makes applying PCA extremely simple with the PCA class. The key parameters to be aware of are:

• n_components: This can be an integer specifying the number of components to keep, or a float between 0 and 1 indicating the desired explained variance ratio.
• svd_solver: This specifies the solver to use for Singular Value Decomposition (SVD). The default is ‘auto‘.
• whiten: If True, the components are multiplied by the square root of n_samples and then divided by the singular values to ensure uncorrelated outputs with unit component-wise variances.

Here‘s an example usage:

from sklearn.decomposition import PCA

pca = PCA(n_components=2, svd_solver=‘auto‘, whiten=False)
X_pca = pca.fit_transform(X_std)

Some other useful methods and attributes of the PCA object are:

• explainedvariance: The amount of variance explained by each of the selected components
• explained_varianceratio: Percentage of variance explained by each of the selected components
• components_: Principal axes in feature space, representing the directions of maximum variance in the data

Conclusion

We‘ve covered a lot of ground in this post, starting from the intuition behind PCA, moving on to the underlying mathematics, and finally demonstrating how to implement PCA in Python using scikit-learn.

To summarize, PCA is a technique that reduces the dimensionality of a dataset by finding new uncorrelated variables (principal components) that successively maximize variance. The principal components are calculated from the eigenvectors of the covariance matrix of the original data.

When applying PCA, it‘s important to standardize the data first, choose an appropriate number of components (based on explained variance), and interpret the components by examining their coefficients and visualizing the transformed data.

PCA has wide-ranging applications from data visualization to feature engineering and can be a valuable tool in any data scientist‘s toolkit. I hope this post has provided you with a solid understanding of PCA and the confidence to apply it in your own projects.