Simplifying the Math Behind Principal Component Analysis
Principal Component Analysis (PCA) is one of the most popular techniques for dimensionality reduction of data. It allows us to identify patterns in data by reducing the number of dimensions, while retaining as much important information as possible. The goal is to find the best summary of the data using a limited number of principal components.
At its core, PCA relies on the eigendecomposition of the covariance matrix of the data to find new uncorrelated variables, called principal components, that successively maximize variance. The principal components are linear combinations of the original variables and are defined by the eigenvectors of the covariance matrix.
The Goal and Intuition Behind PCA
The goal of PCA is to project the data onto a lower dimensional subspace that captures most of the variability or information in the data. Imagine a 2D data distribution that actually spans a 1D subspace, i.e. a line, with some small noise or variability in the orthogonal direction. PCA seeks to find the orientation of that line in order to project the data onto it and reduce the dimensions while losing minimal information.
PCA can be thought of as a rotation of the data to align it with the directions of maximal variance. The new rotated features or principal components are uncorrelated and ordered by the amount of variance they capture from the data. By projecting onto a subset of the top principal components, PCA provides a lower dimensional representation that focuses on the most expressive or informative dimensions of the data.
The Mathematical Details
Suppose we have a data matrix X with n samples and p features. The first step is to center the data by subtracting the mean of each feature, so that the mean of the centered data is zero.
Next, we compute the covariance matrix of the centered data:
$\Sigma = \frac{1}{n} X^T X$
This covariance matrix is a square symmetric matrix that captures the pairwise covariances between all features. The diagonal elements are the variances of each individual feature.
We then perform eigendecomposition on the covariance matrix:
$\Sigma = U \Lambda U^T$
Where U is a matrix whose columns are the unit eigenvectors of Σ, and Λ is a diagonal matrix with the corresponding eigenvalues. The eigenvectors are called the principal components and are orthogonal (uncorrelated). The eigenvalues represent the amount of variance captured by each principal component.
We typically sort the columns of U and the diagonal elements of Λ in decreasing order of eigenvalues. The eigenvector with the largest eigenvalue is the first principal component and accounts for the most variance in the data. The top k eigenvectors capture the most variance.
To get the lower dimensional representation, we project the data onto the subspace spanned by the top k eigenvectors:
$Z = XU_k$
Where U_k contains the top k eigenvectors as columns. The resulting Z is an n x k matrix with k feature dimensions, which captures the maximal variance of the data.
The number of principal components k is a hyperparameter that determines the amount of dimensionality reduction. It can be chosen based on the cumulative percentage of variance explained by the top components, e.g. select k components that cumulatively explain 90% of the variance.
Interpreting the Results
After finding the principal components, the eigenvectors can be inspected to understand how the original features combine to create the new dimensions. The entries of each eigenvector represent the relative contribution or weight of each original feature to that principal component. Features with the same sign contribute positively to it, while opposite signs mean the features contrast each other.
The eigenvalues indicate the relative importance and amount of variance captured by each principal component. A large gap between eigenvalues suggests there are key structural dimensions in the data, while small gaps indicate the remaining dimensions may just be modeling noise.
The data can be visualized along the top two or three principal components to see if they reveal interesting structure, such as clusters or manifolds in the data. Outliers can also be identified more easily after PCA.
Strengths and Weaknesses
PCA has several advantages:
- Reduces dimensionality while preserving much of the original information and structure in the data
- Removes correlated features by projecting onto uncorrelated principal components
- Makes data more interpretable by aligning it with directions of maximal variance
- Can improve performance of downstream machine learning models and data visualization
However, PCA also has limitations:
- It is a linear transformation, so cannot capture nonlinear structure in the data
- It is unsupervised and does not take into account class labels if performing classification
- Interpretation can be difficult for high dimensional data
- Results depend on scaling of the features, typically need to standardize data first
Conclusions
In summary, PCA is an invaluable tool for dimensionality reduction and data compression. By projecting data onto principal components formed by the eigenvectors of the covariance matrix, PCA identifies the most informative and variable dimensions in the data. This makes PCA useful for noise reduction, data visualization, feature extraction and simplifying data to improve machine learning models.
The key elements to understand are the construction of the principal components from eigenvectors of the covariance matrix, the role of eigenvalues in ordering components by variance explained, and the projection of data onto the top components to reduce dimensions while preserving maximal information. By deeply understanding these mathematical steps, data scientists can effectively apply PCA to uncover important structure and patterns in high dimensional data.
