The Key Concepts To Investigating Your Dataset: An AI/ML Perspective

As artificial intelligence and machine learning become increasingly integrated into organizations, the importance of proper data investigation cannot be overstated. While much attention is given to model selection and optimization, the reality is that the success of any AI/ML project hinges on the quality and understanding of the data that feeds it.

Renowned computer scientist Andrew Ng sums it up well: "It‘s not who has the best algorithm that wins. It‘s who has the most data."[^1] But I would add that it‘s not just about having the most data – it‘s about having the most well-understood and insightful data. That‘s where exploratory data analysis (EDA) comes in.

The Role of EDA in Machine Learning Projects

Exploratory data analysis is the critical first step in the data science process. Before you can build predictive models or optimize decision-making, you need to develop a deep understanding of the dataset you‘re working with. This allows you to:

• Formulate sharp, relevant questions to guide your analysis
• Validate that the data is suitable for the task at hand
• Expose data quality issues early on
• Identify the most relevant features and relationships
• Inform your choice of models and algorithms

Despite its importance, EDA is often rushed or skipped over entirely in the race to build models. But as data scientist Tamara Louie points out, "Skipping EDA is like going on a road trip without a map. You might eventually arrive at your destination, but you‘ll likely take a lot of wrong turns along the way."[^2]

So what exactly should you investigate during EDA? Let‘s dive into some of the key concepts and techniques.

Descriptive Statistics

One of the first steps in understanding a dataset is to examine the distributions of individual variables. Descriptive statistics provide a quantitative summary of a feature‘s central tendency, dispersion, and shape.

Common measures of central tendency include:

• Mean: the average value
• Median: the middle value
• Mode: the most frequent value

These metrics give you a sense of where most of the data is concentrated. But they don‘t tell you anything about how spread out the data is. That‘s where measures of dispersion come in:

• Range: the difference between the maximum and minimum values
• Variance: the average squared deviation from the mean
• Standard deviation: the square root of variance

A low variance or standard deviation indicates that most data points are clustered tightly around the mean, while a high value suggests the data is more spread out.

Skewness and kurtosis describe the shape of a distribution. Skewness measures the asymmetry of the data, with positive values indicating a long right tail and negative values a long left tail. Kurtosis represents the "peakedness" of a distribution. High kurtosis means more of the variance is a result of infrequent extreme deviations.[^3]

Here‘s how you can calculate these statistics in Python using the scipy library:[^4]

from scipy.stats import kurtosis, skew

mean = np.mean(data)
median = np.median(data) 
std_dev = np.std(data)
skewness = skew(data)
kurt = kurtosis(data)

While these metrics are useful, they are sensitive to extreme values in the data. Be sure to investigate outliers that may be skewing the results before drawing conclusions.

Data Visualization

In tandem with calculating descriptive statistics, visualizing your data is a powerful way to identify patterns, relationships, and anomalies. Different plot types serve different EDA purposes.

Univariate plots show the distribution of a single variable. Common choices include:

• Histograms: show the frequency of values in specified ranges (bins)
• Box plots: display the median, quartiles, and outliers
• Density plots: smooth version of a histogram, shows probability density

Here‘s an example of creating a histogram in Python using Matplotlib:[^5]

import matplotlib.pyplot as plt

plt.hist(data)
plt.xlabel(‘Value‘) 
plt.ylabel(‘Frequency‘)
plt.title(‘Histogram of Data‘)
plt.show()

To understand relationships between multiple variables, use bivariate and multivariate visualizations:

• Scatter plots: show relationship between two continuous variables
• Line plots: track changes in a variable over time
• Heat maps: display magnitude of a phenomenon between two variables

When working with high-dimensional data, techniques like principal component analysis (PCA) can help reduce the dimensionality to make the data more visualizable.

As a best practice, make your visualizations as intuitive as possible. Use clear labels, legends, and appropriate scales. Interactive plots that allow zooming or hovering can help users explore granular insights. Tools like Plotly and Bokeh make it easy to create interactive dashboards in Python.[^6]

Investigating Correlations

Many machine learning algorithms assume that the input features are independent of each other. In reality, features are often correlated. Examining these correlations helps inform feature selection and model design.

The most common measure of correlation is the Pearson correlation coefficient. It ranges from -1 to +1, with -1 indicating a perfect negative linear relationship, +1 a perfect positive linear relationship, and 0 no linear relationship. You can calculate the Pearson correlation between two variables x and y in Python as follows:[^7]

from scipy.stats import pearsonr

corr, _ = pearsonr(x, y)
print(‘Pearson correlation: %.3f‘ % corr)

For non-linear relationships, rank correlation methods like Spearman‘s rank or Kendall‘s tau are more appropriate. These measures assess how well the relationship between two variables can be described by a monotonic function.

With multiple features, a correlation matrix is a handy way to see all pairwise correlations at once. Plotting a heatmap of the correlation matrix can reveal interesting structure in the data:

import seaborn as sns

corr_matrix = df.corr()
sns.heatmap(corr_matrix, annot=True)
plt.show()

Keep in mind that correlation does not imply causation. Just because two variables are related does not mean that one causes the other. Subject matter expertise is needed to posit causal mechanisms that can then be validated through experimentation.

Assessing Data Quality

Real-world data is messy. It often contains missing values, outliers, inconsistencies, and errors. Identifying these issues upfront is critical to ensuring your models are learning from high-quality information.

Missing data is one of the most common quality issues. Strategies for dealing with missing data depend on the type of data and the extent of missingness:[^8]

• Dropping observations with missing values (if only a few)
• Imputing missing values with measures of central tendency
• Using algorithms that can handle missing data (e.g. random forests)
• Treating missing values as a separate category (for categorical data)

Outliers, or extreme values far from the rest of the distribution, can also be problematic. They can skew summary statistics and have outsized influence on models. While there is no universal definition of an outlier, common detection methods include:

• Z-score: assumes a normal distribution, values >3 std deviations from mean are outliers
• Interquartile range (IQR): outliers are values below Q1 – 1.5IQR or above Q3 + 1.5IQR
• DBSCAN: density-based clustering algorithm that identifies outliers as points in low-density regions[^9]

Not all outliers are invalid though. Use your domain knowledge to determine if an extreme value represents a data entry error or a legitimate rare event before deciding how to handle it.

Other data quality issues to check for:

• Duplicate records
• Impossible values (e.g. negative age)
• Values with wrong data type
• Inconsistent formatting or naming conventions

Fixing data quality problems is often an iterative process. Document the types of issues you find and the steps taken to resolve them. This will help refine your data collection and preprocessing procedures for future projects.

Validating Assumptions

Many machine learning models make assumptions about the structure of the input data. Violating these assumptions can result in unreliable or misleading results. As part of EDA, it‘s important to check whether your data meets the assumptions of the algorithms you intend to use.

Some common assumptions made by ML models include:[^10]

• Linearity: there is a linear relationship between the features and target variable
• Normality: the data follows a Gaussian distribution
• Homoscedasticity: the variance of the residuals is constant
• Independence: the observations are independent of each other

Different models make different assumptions. Linear regression, for example, assumes linearity, normality, and homoscedasticity. Decision trees, on the other hand, do not assume any particular distribution.

Plotting the data can help you visually check for violations of assumptions. For example, creating a Q-Q plot lets you see if your data deviates from normality:

from scipy import stats

stats.probplot(data, dist="norm", plot=plt)
plt.show()

If the points deviate significantly from the diagonal line, the normality assumption may be violated. In that case, you could consider transforming the data or using a non-parametric model instead.

Residual plots are useful for checking the assumptions of regression models. Plotting the residuals against the predicted values should show a random scatter if the assumptions hold. Patterns in the residuals (e.g. increasing variance) are a red flag.

Consulting with subject matter experts is also key for validating assumptions. They can provide context on whether the assumptions make sense given what‘s known about the real-world process being modeled.

Leveraging Domain Expertise

While EDA is a highly quantitative process, it‘s a mistake to neglect qualitative insights. Domain experts can offer invaluable context to guide your analysis and help interpret results.

Some specific ways to leverage subject matter expertise:

• Identify the most relevant features based on knowledge of the business or system
• Validate that the relationships exposed in the data match real-world understanding
• Provide possible explanations for surprising or counterintuitive findings
• Inform data collection to fill in gaps or blind spots

Domain expertise is especially important when the goal is to use ML models to drive decision-making. Blindly trusting a model‘s output without sanity checking against human judgment can lead to costly errors. As Harvard Business School professor Tarun Khanna puts it, "AI systems are powerful and useful, but you should not put blind faith in them unless you have figured out a way to validate what they are doing."[^11]

Effective EDA requires close collaboration between data scientists and subject matter experts. Foster an open dialogue and encourage stakeholders to question results that don‘t match their experience or intuition. At the same time, be willing to challenge long-held assumptions if the data tells a different story. The most impactful insights often come at the intersection of quantitative analysis and human domain knowledge.

Conclusion

Investigative EDA is the foundation of successful machine learning projects. By taking the time to thoroughly understand the data – its distributions, relationships, quality issues, and real-world context – data scientists can avoid costly pitfalls and build models that generate reliable, actionable insights.

The concepts and techniques covered in this post are just a starting point. The specific tools used for EDA will vary depending on the data type, domain, and analytical goals. But the general principles apply across applications: let your data guide your analysis, validate assumptions rigorously, and never underestimate the power of domain expertise.

When done well, EDA is like constructing a solid foundation before building a house. It may not be the most glamorous part of the project, but it‘s what everything else rests on. Invest in understanding your data upfront and you‘ll reap the benefits in every subsequent stage of the ML workflow.

[^1]: Andrew Ng, Machine Learning Yearning
[^2]: Tamara Louie, https://www.linkedin.com/pulse/why-you-cant-skip-exploratory-data-analysis-tamara-louie
[^3]: NIST Engineering Statistics Handbook, https://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm
[^4]: Descriptive Statistics in Python, https://machinelearningmastery.com/descriptive-statistics-in-python/
[^5]: Histograms in Matplotlib, https://www.tutorialspoint.com/matplotlib/matplotlib_histogram_plot.htm
[^6]: Generating Interactive Plots in Python, https://mode.com/blog/python-interactive-plot-libraries/
[^7]: Pearson Correlation in Python, https://realpython.com/numpy-scipy-pandas-correlation-python/
[^8]: Handling Missing Data, https://towardsdatascience.com/7-ways-to-handle-missing-values-in-machine-learning-1a6326a4a71b
[^9]: Outlier Detection Techniques in Machine Learning, https://www.kdnuggets.com/2021/05/outlier-detection-techniques-machine-learning.html
[^10]: Assumptions of Machine Learning Models, https://www.kdnuggets.com/2021/07/assumptions-machine-learning.html
[^11]: Tarun Khanna, https://hbr.org/2022/01/ai-should-augment-human-intelligence-not-replace-it