Traversing the Trinity of Statistical Inference Part 1: Estimation

Statistical inference is the process of using data from a sample to draw conclusions about the larger population from which the sample was taken. It is a foundational concept in statistics that is used across many domains, from scientific research to business to public policy.

At a high level, statistical inference can be broken down into three main activities:

1. Estimation – Using sample data to estimate the unknown parameters (characteristics) of a population
2. Confidence Intervals – Quantifying the uncertainty in our estimates
3. Hypothesis Testing – Assessing evidence in favor or against claims made about a population

In this three-part series, we will explore each of these fundamental statistical concepts in turn, building intuition along the way through examples and visualizations.

The Goal of Estimation

In this first post, we will dive into estimation, arguably the most fundamental of the inferential trifecta. The basic idea of estimation is straightforward – we want to learn about characteristics of a population, but measuring the entire population is usually impractical or impossible.

Instead, we take a representative sample from the population and use the sample data to estimate the unknown population parameters. In other words, we want to use characteristics of the sample (which we measure) to estimate characteristics of the wider population (which are unknown).

Some examples of parameters we may want to estimate include:

• The average height of students at a university
• The proportion of defective products from a manufacturing process
• The click-through rate of an online advertisement
• The average effect of a new drug compared to the current standard treatment

Suppose we want to estimate the average height of students at a large university. Measuring the height of all 30,000 students would be a daunting undertaking. Instead, we could randomly select say 500 students and measure their heights. We then use the average height in our sample of 500 to estimate the average height of all 30,000 students.

Already, some key questions arise – how good is our estimate likely to be? How far off might it be from the true average height? How can we quantify our uncertainty? We‘ll come back to these questions shortly.

Parameters vs Statistics

Before diving into the mechanics of estimation, let‘s clarify some terminology. A parameter is a numerical characteristic of a population – for instance the average height of all 30,000 students in our example. A parameter‘s value is typically unknown and is what we‘re trying to estimate.

A statistic, on the other hand, is a numerical characteristic of a sample. The average height of the 500 students in our sample is an example of a statistic. We calculate statistics from our sample data and use them to estimate parameters.

In our example, the average height of all students is a parameter, while the average height of sampled students is a statistic. We use the sample statistic to estimate the population parameter.

Point Estimates vs Interval Estimates

There are two main flavors of estimates – point estimates and interval estimates. A point estimate is a single "best guess" of a parameter. The sample average height of 500 students is a point estimate for the true average height of all 30,000 students.

An interval estimate specifies a range of values that the parameter is likely to fall in. A 95% confidence interval for the average height being between say 5‘6" and 5‘8" is an example. Interval estimates incorporate the uncertainty of estimation – we‘ll explore them in detail in the next post on confidence intervals.

Desirable Properties of Estimators

An estimator is a rule or formula that tells us how to calculate an estimate from sample data. For instance, using the sample average as an estimate for a population mean is an estimator. Given a sample, an estimator gives us an estimate.

There are many possible estimators we could use, but some have better properties than others. A few key desirable properties for estimators include:

1. Unbiasedness – An unbiased estimator is one whose expected value is equal to the true parameter being estimated. In other words, if we were to take many samples and calculate the estimator‘s value for each, the average of those values would be spot on to the true parameter. Unbiased estimators are centered at the right spot.

2. Efficiency – An efficient estimator has a small standard error relative to other unbiased estimators. The standard error is the standard deviation of an estimator‘s sampling distribution – it quantifies how much the estimator‘s value would vary from sample to sample. A smaller standard error means the estimator‘s values are more tightly clustered near the true parameter.

3. Consistency – A consistent estimator gets closer to the true parameter as the sample size gets larger. With a large enough sample, a consistent estimator will be very close to the true value.

4. Sufficiency – A sufficient estimator makes full use of the relevant information in the sample. Adding more variables to a sufficient estimator doesn‘t give you any additional information about the parameter.

Maximum likelihood estimators are a class that have many of these desirable properties. The basic idea is to choose the parameter values that make the observed data most probable.

Methods of Estimation

A few common methods for finding point estimates include:

1. Maximum Likelihood Estimation (MLE) – Choose the parameter values that maximize the likelihood of observing the sample data. MLEs have many desirable properties.

2. Method of Moments – Equate sample moments (mean, variance, etc.) to population moments and solve for the parameters. This often gives estimates that are easy to calculate.

3. Least Squares – Choose the parameter values that minimize the squared differences between observed data and the model predictions. Commonly used in regression.

Each method has its own strengths and weaknesses. MLEs have desirable properties but can be computationally intensive. Moment estimates are often easy to calculate but have weaker statistical properties. The choice of method depends on the specifics of the problem.

Estimates vs Estimators

It‘s important to be clear about the difference between an estimator and an estimate. An estimator is a rule that tells us how to calculate an estimate from data. An estimate is the realized value of the estimator for a specific sample.

For example, say we‘re estimating a population mean and we decide to use the sample average as our estimator. That‘s our rule – our estimator is the sample average.

Now, say we collect a sample of 100 data points and take their average, obtaining 27.5. That value, 27.5, is our estimate. If we took another sample of 100, we would likely get a different estimate because our sample would be different.

The estimator (sample average) is the rule or formula, while the estimate (27.5 in this case) is the output of that rule for a given sample. This distinction becomes especially important when considering the behavior of estimators over many samples.

Sampling Distributions and the Standard Error

An estimator like the sample average will give different estimates for different samples. This variability can be characterized by the sampling distribution of the estimator – the distribution of the estimator‘s values over all possible samples of a given size.

Understanding the sampling distribution allows us to quantify the uncertainty in our estimates. The center of the sampling distribution tells us about the bias of the estimator. The spread tells us about its efficiency.

A key measure of an estimator‘s precision is the standard error – the standard deviation of its sampling distribution. A smaller standard error means the estimates are more tightly clustered around the true parameter, indicating a more precise estimator.

The standard error is often estimated using the standard deviation of the sample, divided by the square root of the sample size. This makes sense intuitively – more data gives us a more precise estimate.

Example: Estimating a Population Mean and Proportion

Let‘s cement these concepts with an example. Say we want to estimate the average age of participants in a large study, along with the proportion of participants over 65.

We take a random sample of 500 participants and calculate the sample average age to be 52.7 years, with a standard deviation of 8.2 years.

For the same sample, we calculate the proportion of participants over 65 to be 0.18, or 18%.

Our point estimate for the true average age in the whole study population is 52.7 years. Our point estimate for the proportion over 65 is 0.18.

To quantify the uncertainty in our estimate of the average age, we can calculate the standard error:

SE = 8.2 / sqrt(500) = 0.37

This tells us that our estimate is likely to be within about 0.37 years of the true population mean.

For the proportion, the standard error is:

SE = sqrt(0.18 * 0.82 / 500) = 0.017

Our estimate of the proportion is likely to be within about 0.017 of the true value.

We could also give interval estimates. A 95% confidence interval for the average age would be roughly 52.7 ± 2*0.37, or (51.96, 53.44). We‘ll dive deeper into confidence intervals in the next post.

Conclusion

Estimation is a core part of statistics, allowing us to learn about populations from samples. The key ideas are:

1. We use sample statistics to estimate population parameters.
2. There are point estimates and interval estimates.
3. Good estimators are unbiased, efficient, consistent, and sufficient.
4. Maximum likelihood estimation is a general method for finding estimators with good properties.
5. An estimator gives different estimates for different samples – this variation is captured by the sampling distribution and standard error.

In the next post, we‘ll build on these concepts to understand confidence intervals, a powerful tool for quantifying the uncertainty in our estimates. After that we‘ll round out our inferential journey by seeing how estimation allows us to test hypotheses about populations.

Estimation is just the start of a grand adventure in statistical inference. Stay tuned for the next installments!