A Deep Dive into Dimensionality Reduction: Techniques, Math, Code, and Insights
Dimensionality reduction is a crucial tool in the data scientist‘s arsenal, especially in the era of big data where datasets often have thousands or even millions of features. In a 2020 survey of data scientists by Kaggle, 46% of respondents reported using dimensionality reduction techniques in their work [1].
The goal of dimensionality reduction is to transform high-dimensional data into a lower-dimensional representation while preserving the most important information. This has several benefits:
-
It mitigates the curse of dimensionality, where the amount of data needed to generalize accurately grows exponentially with the number of features [2].
-
It reduces computational and memory requirements of machine learning algorithms.
-
It helps visualize high-dimensional data in 2D or 3D.
-
It can improve the interpretability of the data and the models learned from it.
In this post, we‘ll take a deep dive into several popular dimensionality reduction techniques, exploring their mathematical underpinnings, their pros and cons, and their applications. We‘ll also see these techniques in action through Python code examples.
Principal Component Analysis (PCA)
PCA is the granddaddy of dimensionality reduction techniques, dating back to Karl Pearson in 1901 [3]. The goal of PCA is to find a lower-dimensional linear subspace that maximizes the variance of the projected data.
The Math of PCA
Given a dataset $X$ with $n$ samples and $p$ features, PCA seeks to find a linear transformation $W$ that maps $X$ to a new dataset $Y$ with $q$ features, where $q < p$. The transformation $W$ is chosen to maximize the variance of $Y$.
Mathematically, this can be formulated as an optimization problem:
$$
\max{W} \; \text{Var}(Y) = \max{W} \; \text{Var}(XW)
$$
subject to $W^TW = I$.
The solution to this problem is given by the eigenvectors of the covariance matrix of $X$ corresponding to the $q$ largest eigenvalues [4]. These eigenvectors form the columns of $W$, and are called the principal components.
PCA in Python
Here‘s how to perform PCA in Python using scikit-learn:
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)This code finds the 2 principal components of the data X and transforms X into the reduced space.
When to Use PCA
PCA is a good choice when:
- Your data is linear
- You want an unsupervised technique
- You want to maximize the variance of the projected data
However, PCA has some limitations:
- It only finds linear subspaces
- It is not optimized for class separability
- It can be sensitive to scaling of the features
Linear Discriminant Analysis (LDA)
LDA is a supervised dimensionality reduction technique that is optimized for class separability. It was introduced by Ronald A. Fisher in 1936 for taxonomic classification [5].
The Math of LDA
Given a dataset $X$ with $n$ samples, $p$ features, and $K$ classes, LDA seeks to find a linear transformation $W$ that maximizes the ratio of the between-class variance to the within-class variance.
Mathematically, this is formulated as:
$$
\max_{W} \; J(W) = \frac{|W^T S_B W|}{|W^T S_W W|}
$$
where $S_B$ is the between-class scatter matrix and $S_W$ is the within-class scatter matrix.
The solution to this problem is given by the eigenvectors of $S_W^{-1}S_B$ corresponding to the largest eigenvalues [6].
LDA in Python
Here‘s how to perform LDA in Python using scikit-learn:
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
lda = LinearDiscriminantAnalysis(n_components=2)
X_lda = lda.fit_transform(X, y)This code finds the 2 most discriminative directions in the data X based on the class labels y.
When to Use LDA
LDA is a good choice when:
- You have a supervised classification problem
- Your classes are linearly separable
- You want to reduce dimensionality while preserving class separability
However, LDA also has some limitations:
- It assumes that each class is Gaussian with a common covariance matrix
- It can at most reduce the dimensionality to K-1, where K is the number of classes
- It doesn‘t perform well if the classes are not linearly separable
t-Distributed Stochastic Neighbor Embedding (t-SNE)
t-SNE is a nonlinear dimensionality reduction technique that is particularly well-suited for visualizing high-dimensional data. It was introduced by Laurens van der Maaten and Geoffrey Hinton in 2008 [7].
The Math of t-SNE
t-SNE converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data.
In the high-dimensional space, the similarity of datapoint $x_j$ to datapoint $xi$ is the conditional probability, $p{j|i}$, that $x_i$ would pick $x_j$ as its neighbor if neighbors were picked in proportion to their probability density under a Gaussian centered at $x_i$ [7].
In the low-dimensional space, the similarity of datapoint $y_j$ to datapoint $y_i$ is modeled using a Student‘s t-distribution with one degree of freedom (hence the "t" in t-SNE).
The cost function minimized by t-SNE is the sum of Kullback-Leibler divergences over all datapoints:
$$
C = \sum_{i} KL(P_i || Q_i) = \sum_i \sumj p{j|i} \log \frac{p{j|i}}{q{j|i}}
$$
where $P_i$ is the conditional probability distribution over all other datapoints given datapoint $i$ in the high-dimensional space, and $Q_i$ is the conditional probability distribution over all other map points given map point $i$ in the low-dimensional space.
t-SNE in Python
Here‘s how to perform t-SNE in Python using scikit-learn:
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, perplexity=30)
X_tsne = tsne.fit_transform(X)This code reduces the data X to 2 dimensions using t-SNE with a perplexity of 30.
When to Use t-SNE
t-SNE is a good choice when:
- You want to visualize high-dimensional data in 2D or 3D
- You want to preserve the local structure of the data
- You suspect that your data has a nonlinear structure
However, t-SNE also has some drawbacks:
- It‘s computationally expensive, especially for large datasets
- It doesn‘t preserve the global structure of the data well
- It doesn‘t provide an explicit mapping function, so you can‘t embed new data points
Autoencoders
Autoencoders are a type of neural network used for unsupervised learning of efficient data encodings. They were first introduced by Geoffrey Hinton and James McClelland in 1986 [8].
The Math of Autoencoders
An autoencoder consists of two parts: an encoder function $h=f(x)$ and a decoder that produces a reconstruction $r=g(h)$. The encoder maps the input data $x$ to a hidden representation $h$, typically of lower dimensionality. The decoder then maps this hidden representation back to the original input space, producing the reconstruction $r$.
The autoencoder is trained to minimize the reconstruction error, which is typically the mean squared error between the input and the reconstruction:
$$
L(x, g(f(x))) = ||x – g(f(x))||^2
$$
By minimizing this reconstruction error, the autoencoder learns to capture the most important features of the data in the hidden representation.
Autoencoders in Python
Here‘s how to create a simple autoencoder in Python using Keras:
from keras.layers import Input, Dense
from keras.models import Model
input_layer = Input(shape=(input_dim,))
encoded = Dense(encoding_dim, activation=‘relu‘)(input_layer)
decoded = Dense(input_dim, activation=‘sigmoid‘)(encoded)
autoencoder = Model(input_layer, decoded)
encoder = Model(input_layer, encoded)
autoencoder.compile(optimizer=‘adam‘, loss=‘binary_crossentropy‘)
autoencoder.fit(X, X, epochs=50, batch_size=256, shuffle=True)
encoded_data = encoder.predict(X)This code defines an autoencoder with a single hidden layer of dimension encoding_dim, trains it on the data X, and uses the encoder part to create a reduced-dimension representation of X.
When to Use Autoencoders
Autoencoders are a good choice when:
- You want a non-linear dimensionality reduction technique
- You‘re working with image, audio, or text data
- You want to learn feature representations in an unsupervised manner
However, autoencoders also have some limitations:
- They can be difficult to train, especially deep autoencoders
- They don‘t have a closed-form solution like PCA or LDA
- The learned features can be hard to interpret
Choosing the Right Technique
With so many dimensionality reduction techniques available, it can be hard to know which one to choose. Here are some guidelines:
| Technique | Linearity | Supervision | Computational Cost | Interpretability |
|---|---|---|---|---|
| PCA | Linear | Unsupervised | Low | High |
| LDA | Linear | Supervised | Low | High |
| t-SNE | Non-linear | Unsupervised | High | Low |
| Autoencoder | Non-linear | Unsupervised | High | Low |
In general, PCA is a good starting point for most problems. If you have labeled data and want to preserve class separability, LDA is a good choice. If you want to visualize high-dimensional data, t-SNE is often used. If you‘re working with complex data like images or text, autoencoders can be powerful.
Of course, these are just guidelines and the best approach will depend on your specific data and problem. It‘s often a good idea to try multiple techniques and compare the results.
Evaluating Dimensionality Reduction
After applying dimensionality reduction, how do you know if it worked well? Here are some ways to evaluate the quality of the dimensionality reduction:
-
Reconstruction error: For techniques like PCA and autoencoders that provide a way to reconstruct the original data from the reduced representation, you can measure the reconstruction error. The lower the error, the better the dimensionality reduction.
-
Classification accuracy: If the goal is to improve classification performance, you can evaluate the accuracy of a classifier trained on the reduced data.
-
Visualization: For techniques like t-SNE that are used for visualization, you can qualitatively assess how well the visualization captures the structure of the data.
-
Explained variance: For PCA, you can look at the percentage of variance explained by each principal component. A good dimensionality reduction should explain a large percentage of the variance with a small number of components.
Conclusion
Dimensionality reduction is a vital tool for working with high-dimensional data in machine learning and data science. By understanding the mathematics behind techniques like PCA, LDA, t-SNE, and autoencoders, you can make informed decisions about when and how to use them.
Remember, dimensionality reduction is not a silver bullet. It‘s a tool that should be used judiciously and with an understanding of its limitations. But when used appropriately, it can lead to faster training times, improved model performance, and better insights into your data.
