Visualizing and Manipulating High Dimensional Data with HyperTools: An AI/ML Perspective

As artificial intelligence and machine learning models become increasingly sophisticated, the datasets they rely on are also growing in size and complexity. It‘s not uncommon for modern AI/ML pipelines to involve data with hundreds or even thousands of dimensions. While high dimensional data can encode rich patterns and structures, it also poses significant challenges for visualization and interpretation.

Visualizing high dimensional datasets is critical for several reasons:

1. Understanding the structure of the data: Plotting the data in 2 or 3 dimensions can reveal important clusters, outliers, and correlations that may not be apparent from raw feature values alone.

2. Debugging and quality control: Visualizations can help identify issues with the data itself, such as missing values, scaling inconsistencies, or mislabeled samples. They provide a sanity check before feeding data into complex ML models.

3. Model selection and evaluation: Comparing visualizations of the raw data, model inputs, and model outputs can provide insights into how well a model is capturing the underlying structure of the data and where it may be falling short.

4. Communicating results: Clear, intuitive visualizations are essential for conveying the insights and decisions derived from high dimensional data to both technical and non-technical stakeholders.

However, the curse of dimensionality makes it challenging to directly visualize data in more than 3 dimensions. This is where dimensionality reduction techniques like PCA, t-SNE, and UMAP come in – they provide a way to compress high dimensional data into a lower dimensional representation while preserving key patterns and structures.

HyperTools is a powerful Python library that makes it easy to apply these dimensionality reduction techniques and create meaningful visualizations of high dimensional data with just a few lines of code. Developed by a team of neuroscience and data science researchers at Dartmouth College, HyperTools has quickly gained adoption in the AI/ML community due to its simplicity, flexibility, and attractive visual aesthetics.

Key Features and Advantages

So what sets HyperTools apart from other dimensionality reduction and visualization libraries? Here are some of its key features and advantages from an AI/ML perspective:

Support for a wide range of dimensionality reduction techniques:
HyperTools provides a unified interface for applying PCA, t-SNE, UMAP, MDS, Isomap, and more. This makes it easy to experiment with different techniques and select the one that provides the clearest separation and structure for a given dataset. The consistent API also makes it simple to switch techniques as datasets and requirements evolve.

Seamless integration with core data science libraries:
HyperTools is built on top of NumPy, SciPy, Pandas, Scikit-Learn, and Matplotlib, and can handle a variety of data formats and types. This means it slots easily into most existing data science and ML workflows with minimal friction. Here‘s an example of using HyperTools to visualize a Pandas DataFrame:

import pandas as pd
from hypertools import plot

# Load data into a DataFrame 
df = pd.read_csv(‘my_data.csv‘)

# Plot the data with HyperTools
plot(df, ‘.‘, group=df.target, labels=list(df.columns))

Powerful alignment and normalized modeling capabilities:
One of HyperTools‘ unique features is its support for hyperalignment (see: http://haxbylab.dartmouth.edu/publications/HyperAlign2020Final.pdf), a technique for projecting multiple datasets into a common space while preserving their unique structure. This is extremely valuable for comparing and combining datasets collected under different conditions. HyperTools also supports fitting normalized models to visualize the common structure across multiple datasets.

Built-in clustering and text analysis:
HyperTools can automatically perform k-means clustering on input data and use the cluster assignments to color-code points in the low dimensional plots. It can also apply topic modeling techniques like NMF and LDA to raw text data to extract and visualize the semantic content. This saves significant time and effort when working with unstructured or partially labeled data.

Attractive visual aesthetics:
While eye-catching visuals are not always the top priority in AI/ML workflows, they certainly don‘t hurt when it comes time to present results to colleagues and decision-makers. HyperTools produces attractive, publication-quality plots with sensible defaults while still providing full customization options. The animated 3D plots are particularly engaging and effective for interactive data exploration.

Active development and support:
HyperTools has a growing community of users and contributors, with active support from the core development team. The library is continuously being updated with new features, optimizations, and bug fixes. Recent improvements include support for custom color maps, legends, and plot annotations, as well as performance enhancements for very large datasets.

Real-world Applications and Use Cases

To further illustrate the value of HyperTools for AI/ML, let‘s walk through a few real-world applications and use cases.

Visualizing high dimensional biological data:
One of the most active areas of AI/ML research and application is in the biomedical domain, where datasets often involve millions of high dimensional measurements from sources like genomics, proteomics, and neuroimaging. HyperTools is widely used in this field to visualize patterns in gene expression, brain activity, and other biological signals.

For example, a recent study used HyperTools to visualize and compare fMRI brain activity patterns across different tasks and subjects (https://www.nature.com/articles/s41593-019-0436-x/). By projecting the high dimensional fMRI data into a common low dimensional space with hyperalignment, the researchers were able to identify stable task-general and task-specific neural patterns that could be used to decode cognitive states.

Exploring high dimensional textual data:
HyperTools is also useful for visualizing patterns in unstructured text data such as customer reviews, social media posts, or news articles. The library‘s data transformation utilities can be used to convert raw text into a high dimensional vector representation using techniques like word embedding or TF-IDF, which can then be compressed, clustered, and plotted in a low dimensional space. Here‘s an example of applying a PCA topic model to visualize different Reddit communities and subreddits:

import hypertools as hyp

# Load reddit comment data
data = hyp.load(‘reddit_comments_10000.csv‘)

# Apply PCA topic modeling to extract 10 latent topics  
topic_model = hyp.analyze(data, pre_normalize=‘across‘, model=‘PCA‘, ndims=10)  

# Plot the topic-reduced data and color by subreddit
hyp.plot(topic_model.get_data(), ‘.‘, labels=topic_model.get_labels(),
         title=‘Reddit Topic Model‘, normalize=‘across‘)

Comparing performance of different ML models:
When developing AI/ML models, it is often useful to visualize and compare the predictions of different models on the same dataset. HyperTools can be used to project the high dimensional model outputs into a common space, making it easy to see which models are capturing similar patterns and which are behaving differently.

For instance, let‘s compare the performance of three different sklearn classifiers on the classic Iris dataset:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from hypertools import plot

iris = datasets.load_iris()
X, y = iris.data, iris.target

rf = RandomForestClassifier().fit(X, y)  
svm = SVC().fit(X, y)
knn = KNeighborsClassifier().fit(X, y)

preds = [rf.predict(X), svm.predict(X), knn.predict(X)]

plot(preds, ‘.‘, labels=[‘RandomForest‘, ‘SVM‘, ‘kNN‘])

The resulting plot shows that the RandomForest and kNN predictions are more similar to each other than to the SVM. This kind of visualization can be a useful aid in model selection and ensembling.

Limitations and Future Directions

While HyperTools is a powerful and flexible library, it‘s important to be aware of its limitations. One key limitation is that the quality and interpretability of the visualizations depends heavily on the choice of dimensionality reduction technique and parameters. There is no one-size-fits-all approach, and different techniques may reveal (or obscure) different patterns in the data. It‘s important to experiment with multiple approaches and to use domain knowledge to guide the selection.

Another limitation is that HyperTools currently lacks some of the features and customization options of more mature plotting libraries like Seaborn or Plotly. While the HyperTools team is working to expand the library‘s capabilities, for now users may need to rely on other tools for certain advanced plotting needs.

Looking to the future, the challenge of high dimensional data visualization is only going to grow as AI/ML models become more sophisticated and datasets become larger and more complex. Some promising future directions include:

• Integration with deep learning frameworks: As deep learning becomes the dominant paradigm for AI/ML, there is a growing need for tools that can visualize the high dimensional feature spaces learned by neural networks. Future versions of HyperTools could include utilities for visualizing activations, attention weights, and other internal states of models implemented in PyTorch, TensorFlow, etc.

• Interactive visualization dashboards: While HyperTools currently supports interactive plots via the mpld3 library, there is room for more fully featured interactive dashboard capabilities that would allow users to explore high dimensional data in real-time by adjusting dimensionality reduction parameters, filtering and highlighting subsets of data, and so on.

• Scalability to massive datasets: As datasets grow to the terabyte and petabyte scale, new approaches will be needed to efficiently compress and visualize data that exceeds the memory limits of a single machine. Integrating HyperTools with distributed computing frameworks like Spark or Dask could help address this challenge.

Ultimately, the goal of high dimensional data visualization is to help humans make sense of the increasingly complex information being processed by AI/ML systems. Libraries like HyperTools play a vital role in bridging the gap between the abstract world of high dimensional data and the intuitive visual comprehension abilities of the human brain. As an AI/ML practitioner, I believe that fluency with tools like HyperTools will only become more essential in the years ahead.