A Beginner‘s Guide to Descriptive Statistics: Understanding Your Data

As a data scientist, one of the first and most important steps in any data analysis project is exploring and understanding the data you‘re working with. Before you start building fancy machine learning models, you need to get a solid grasp of what your data actually looks like.

This is where descriptive statistics comes in. Descriptive statistics is all about summarizing and describing the main features of a dataset, giving you a high-level overview of what you‘re dealing with. It‘s a crucial first step that lays the foundation for the rest of your analysis and modeling.

In this article, we‘ll dive deep into the essential concepts of descriptive statistics. We‘ll cover what the main types of descriptive statistics are, how to calculate and interpret them, and how to use them to gain insights into your data. Whether you‘re a complete beginner or just need a refresher, by the end of this guide you‘ll have a solid understanding of this fundamental data science skill. Let‘s get started!

The Three Pillars of Descriptive Statistics

At a high level, there are three main aspects of a dataset that descriptive statistics aims to quantify:

1. The central tendency or "average" of the data
2. The dispersion or spread of the data
3. The shape or distribution of the data

Each of these corresponds to specific descriptive statistics that capture relevant summaries of those aspects. Let‘s look at each one in turn.

Measures of Central Tendency

Measures of central tendency tell you about the "center" or typical value in your dataset. The three main measures are:

Mean: The arithmetic average, calculated by summing all the values and dividing by the number of values. The mean is intuitive but sensitive to outliers.

Median: The "middle" value that separates the upper and lower halves of the data. The median is robust to outliers.

Mode: The most frequently occurring value. There can be no mode, one mode, or multiple modes in a dataset.

For example, let‘s say we have a dataset of exam scores:
[70, 85, 92, 85, 76, 98, 60]

The mean score is (70+85+92+85+76+98+60) / 7 = 80.86
The median score is 85
The mode is 85 (appears twice)

In this case, the mean and median are fairly close. But if there was an outlier, say a score of 30, the mean would drop to 73.88 while the median would stay at 85. This highlights how the mean is pulled toward outliers while the median is more robust.

Measures of Dispersion

While measures of central tendency tell you the typical value, measures of dispersion quantify how far away or spread out the data points are from that center. Key measures include:

Range: The difference between the maximum and minimum value. Easy to calculate but sensitive to outliers.

Variance: The average squared deviation from the mean. Quantifies spread but not intuitive to interpret.

Standard Deviation: The square root of variance. More interpretable as it‘s in the original units of the data. A low standard deviation means data is closely clustered around the mean.

Interquartile Range (IQR): The range between the 25th and 75th percentile. Robust to outliers.

Continuing our exam score example, the statistics are:

Range: 98 – 60 = 38
Variance: 166.48
Standard deviation: 12.90
IQR: 89.5 – 73 = 16.5

The standard deviation of 12.90 points indicates a moderate spread around the mean of 80.86. The IQR of 16.5 points tells us that the middle 50% of scores fall in a 16.5 point range.

Measures of dispersion give you a sense of the variability in your data. High dispersion means data points are far from the center and each other. Low dispersion means data is tightly clustered. This has implications for analysis, e.g. highly dispersed data may need transformations to meet model assumptions.

Measures of Shape

The two key measures that describe the shape and symmetry of a distribution are:

Skewness: Quantifies how asymmetric a distribution is. Positive skew means the tail is on the right, negative means the tail is on the left.

Kurtosis: Quantifies the thickness of the tails and peak of the distribution relative to a normal distribution. High kurtosis means thick tails and a sharp peak.

With our exam score data, we‘d need a larger sample size to meaningfully quantify the shape. But in general, a distribution of exam scores is often negatively skewed (tail on the left). This is because there‘s a hard upper limit (100%) that most won‘t reach, but no hard lower limit. Kurtosis of exam scores tends to be low, with thin tails as most scores cluster around the mean.

Measures of shape are important to check because many statistical methods assume normally-distributed data. Skewed data can bias mean-based statistics. Knowing your data‘s shape helps you assess if it meets model assumptions and if transformations are needed.

Identifying Outliers and Errors

An essential use case of descriptive statistics is identifying potential outliers and data errors. Outliers are data points that are far away from other observations. They may be due to genuine rare events, or they could be errors in data entry or measurement.

Some signs of potential outliers:

• Values that are far beyond the range of the rest of the data
• Values that are > 3 standard deviations from the mean
• Values that are < Q1 – 1.5IQR or > Q3 + 1.5IQR (Q1=25th percentile, Q3=75th)

For example, if we had an exam score of 120 in our dataset, that would be an obvious outlier (or more likely, a data error since it exceeds the max possible score).

Outliers can have a big impact on your statistical analysis. They can dramatically shift the mean and increase measures of dispersion. In a regression, outliers can alter the slope of the line and affect significance tests. So it‘s crucial to identify potential outliers early on using descriptive statistics. Then you can investigate if they are genuine novelties or errors that need to be fixed or removed.

Visualizing Descriptive Statistics

Visualizations are a powerful way to convey the key descriptive statistics of your data. Some common plots:

Histogram: Shows the distribution of a continuous variable. The x-axis is the value range, the y-axis is the frequency or count. Histograms let you see the shape, center, and spread.

Box Plot: Summarizes a distribution with 5 numbers: minimum, Q1, median, Q3, and maximum. The "box" spans Q1 to Q3 (IQR), with a line at the median. Dots beyond the "whiskers" are potential outliers. Box plots are great for comparing distributions across groups.

Scatter Plot: Plots data points with two continuous variables, one on each axis. Scatter plots help visualize relationships between variables and spot outliers.

For our exam score data, a histogram could show the distribution is roughly symmetric with a peak around 80-90. A box plot would show the median line around 85, the IQR box spanning from mid-70s to around 90, and a potential low outlier around 60.

Visualizations bring descriptive statistics to life and make them easily understandable. They should be a key part of your data exploration workflow alongside calculating the raw numbers.

Descriptive Statistics in Action

To solidify these concepts, let‘s look at descriptive statistics in a real-world dataset. We‘ll use the "tips" dataset from the seaborn library, which records tips given at a restaurant.

Numeric Data

For the numeric "total_bill" variable, we get these statistics:

Mean: $19.79
Median: $17.80
Standard Deviation: $8.90
IQR: $12.35

The mean is higher than the median, suggesting the data is right-skewed. The standard deviation and IQR indicate a substantial spread, with bills ranging from $3 to $50. There may be some high outliers pulling up the mean.

Categorical Data

For the categorical "day" variable, we can look at counts and proportions:

Thursday: 62 (19.4%)
Friday: 19 (5.9%)
Saturday: 87 (27.2%)
Sunday: 76 (23.8%)

This shows that Saturday and Sunday are the busiest days, making up over 50% of the total. Fridays are the slowest. This suggests weekends are the peak times for this restaurant.

Visualization

Plotting a histogram of "total_bill" confirms the distribution is right-skewed, with a long tail of high values. A scatter plot of "total_bill" vs. "tip" shows a positive linear relationship, with a few potential outliers.

This example illustrates how descriptive statistics and visualizations work hand-in-hand to surface insights about your data. The specific numbers quantify the key features, while the plots make those features tangible.

Pitfalls to Watch Out For

While descriptive statistics are incredibly useful, there are some common pitfalls to be aware of:

1. Relying too heavily on the mean. The mean can be misleading if there are outliers or skew. Always check the median and visualize the distribution.

2. Ignoring outliers. Outliers can drastically change your statistics. While they may be genuine, it‘s important to investigate if they are errors or unduly affecting your analysis.

3. Not considering subgroups. The overall statistics may obscure differences between subgroups. Always check for subgroup differences, e.g. customer segments, time periods, etc.

4. Misinterpreting skew. People often assume positive skew means values are large and negative skew means small. But it‘s about where the tail is. Always look at plots to clarify.

5. Not considering units. Descriptive statistics are only meaningful if you understand the units. A mean of "4.5" is very different if it‘s in dollars vs. millions of dollars.

By being aware of these issues, you can use descriptive statistics responsibly to draw accurate insights from your data.

Tools for Descriptive Statistics

Many software tools have built-in functions for descriptive statistics. In Python, key libraries are:

• NumPy: Offers mean, median, percentile, std, var functions
• SciPy: Has skew, kurtosis, iqr, describe functions
• Pandas: Can calculate stats on DataFrame columns with .describe(), .mean(), .median() etc.

In R, most of the key stats are built-in functions like mean(), median(), sd(), summary(). The psych package offers additional tools.

For visualization, Matplotlib in Python and ggplot2 in R are popular choices. Seaborn is a Python library specifically for statistical visualization built on Matplotlib.

While it‘s good to know the formulas, these tools make computing descriptive statistics quick and easy in practice. The challenge is more about knowing how to interpret and use the output.

Conclusion

We‘ve covered a lot of ground in this guide to descriptive statistics. To recap, the key points are:

• Descriptive statistics quantitatively summarize the main features of a dataset
• The three main types are measures of central tendency, dispersion, and shape
• These statistics help identify outliers, assess assumptions, and surface insights
• Visualizations like histograms and box plots powerfully convey descriptive statistics
• There are some common pitfalls to watch out for when using descriptive statistics
• Many software tools have built-in functions for quickly computing descriptive statistics

Descriptive statistics are a fundamental tool in any data scientist‘s toolkit. By taking the time to thoroughly explore your data with these techniques up front, you set yourself up for a more insightful and robust analysis down the line.

But this is just the beginning. Inferential statistics takes these concepts a step further to generalize from a sample to a population. Predictive modeling uses these summaries as building blocks for machine learning models.

No matter where your data science journey takes you, a strong foundation in descriptive statistics will serve you well. Armed with this knowledge, you‘re well on your way to becoming a data analysis pro. Happy analyzing!