Exploratory Data Analysis Using SAS and Python: A Comprehensive Guide

Exploratory data analysis, or EDA for short, is a crucial step in the data science process that focuses on summarizing and visualizing data to uncover insights, spot anomalies, and test assumptions. Performing EDA allows data scientists and analysts to gain a deeper understanding of the data they are working with before diving into machine learning or statistical modeling.

While EDA can be performed using a variety of tools, SAS and Python are two of the most popular technologies for data exploration. In this article, we‘ll dive deep into conducting EDA using SAS programming, while also comparing and contrasting with performing similar analyses in Python. Whether you‘re a seasoned data scientist or just getting started with exploratory analysis, this guide will equip you with the skills you need to become an EDA pro.

Getting Started with EDA in SAS

Before we can start exploring our data, we first need to read it into SAS. We‘ll be using a dataset of bank loan data containing information on 100,514 loans and 19 variables. The data is stored in a CSV file that we can input using the PROC IMPORT procedure:

PROC IMPORT DATAFILE="bank_loans.csv" 
    DBMS=CSV
    OUT=loans;
    GETNAMES=YES;
RUN;

With our data imported, we can start by examining the contents of the dataset using PROC CONTENTS, which provides metadata on the variables including types, formats, and labels:

PROC CONTENTS DATA=loans;
RUN; 

From the output, we can see there are 19 variables, a mix of numeric and character types. Some key variables include loan amount, interest rate, credit score, and loan status.

Next, we can generate descriptive statistics on the numeric variables using PROC MEANS:

PROC MEANS DATA=loans N NMISS MIN Q1 MEDIAN Q3 MAX MEAN STDDEV;
RUN;

This gives us a good overview of the central tendency, variability, and ranges of the numeric fields. We can spot some potential outliers, like maximum debt to income ratio of 9999.

Dealing with Missing Data

Missing data is a common issue when working with real-world datasets. In our bank loans data, we can check the number and percent of missing values for each variable using PROC FREQ:

PROC FREQ DATA=loans;
    TABLES _ALL_ / MISSING NOCUM NOPERCENT;
RUN;

Looks like around 8% of credit score values are missing. We have a few options for handling this:

1. Delete observations with missing values
2. Impute missing values
3. Use missing as a separate category

Deciding on an approach requires carefully considering the mechanisms behind why data is missing.

Let‘s try imputing the missing credit scores using the median value. We can do this using PROC STDIZE:

PROC STDIZE DATA=loans 
    OUT=loans_imputed
    METHOD=MEDIAN;
    VAR credit_score;
RUN;

Another option for multiple imputation would be using PROC MI. The best approach depends on the particular dataset and analytic goals.

Visualizing Distributions with Univariate Plots

Univariate visualizations let us understand the distribution of individual variables. The two most common univariate plots are histograms and box plots.

We can easily create histograms in SAS using PROC UNIVARIATE:

PROC UNIVARIATE DATA=loans_imputed;
    VAR loan_amount income;
    HISTOGRAM loan_amount income;
RUN; 

The histograms reveal that both loan amount and income have right skewed distributions. We may want to consider transforming these variables to make them more normally distributed.

Box plots are another helpful tool for visualizing distributions and detecting potential outliers. We can make box plots in SAS using PROC SGPLOT:

PROC SGPLOT DATA=loans_imputed;
    VBOX credit_score / GROUP=loan_status;
RUN;

This box plot compares the distribution of credit scores between loans that were paid off on time versus loans that defaulted. Borrowers who defaulted tended to have lower credit scores.

Exploring Relationships with Bivariate Plots

In addition to understanding variables on their own, we also want to explore the relationships between pairs of variables. Two useful bivariate visualizations are scatter plots and correlation matrices.

To create a scatter plot in SAS, we use PROC SGPLOT with the SCATTER statement:

PROC SGPLOT DATA=loans_imputed;
    SCATTER X=debt_to_income Y=interest_rate;
RUN;

The scatter plot shows a moderate positive relationship between debt to income ratio and interest rate, which makes sense – lenders charge higher rates to riskier borrowers.

To get a broader picture of relationships between multiple variables at once, we can create a correlation matrix using PROC CORR:

PROC CORR DATA=loans_imputed PLOTS=MATRIX;
    VAR loan_amount income credit_score debt_to_income;
RUN;

The correlation matrix shows that loan amount and income have the strongest positive correlation, while debt to income ratio is negatively correlated with the other variables.

Performing EDA in Python

Now that we‘ve seen how to conduct exploratory analysis in SAS, let‘s compare the process to performing EDA in Python. Python has a rich ecosystem of open source libraries for data science, including pandas, NumPy, Matplotlib, and Seaborn.

We can perform very similar analyses in Python as we did in SAS. Here‘s how we might generate descriptive statistics on our loans data using Python:

import pandas as pd
loans_df = pd.read_csv(‘bank_loans.csv‘)
loans_df.describe()

And here‘s how we can make a box plot of credit score by loan status:

import seaborn as sns
import matplotlib.pyplot as plt
sns.boxplot(x=‘loan_status‘, y=‘credit_score‘, data=loans_df)
plt.show()

As you can see, the actual code to perform EDA between SAS and Python is quite different, since SAS uses procedures while Python is object-oriented. However the general process and techniques are very similar across the two languages.

Some advantages of using Python for EDA include:

• Free and open source
• Integrates easily with other Python tools for machine learning
• Large community and support

On the other hand, benefits of SAS include:

• Specialized analytics procedures
• Handles large datasets well
• Common in regulated industries and academia

Tips for Effective EDA

Exploratory data analysis is as much an art as a science. While we‘ve covered some of the key techniques, here are a few tips to keep in mind to get the most out of your EDA:

1. Start with a clear plan, but be ready to adapt as you go. It helps to have questions in mind to guide your analysis, but stay open to new paths of exploration.

2. Use visualizations whenever possible. Graphical representations are much easier for the human brain to process than tables of numbers.

3. Look for surprises in your data. What seems counterintuitive or contradicts your assumptions? Investigating outliers can lead to important insights.

4. When you find interesting relationships, ask yourself if they could be explained by confounding variables. Slice and dice your data in different ways to test your hypothesis.

5. Document your work and share it with others. EDA is often a collaborative process and recording your steps will make your findings reproducible.

Conclusion

In this guide, we‘ve covered the fundamentals of exploratory data analysis in SAS and Python. EDA is a key component of the data science workflow that enables analysts to uncover insights and inform subsequent modeling decisions.

Both SAS and Python offer powerful capabilities for exploring and visualizing data. Which language you choose will depend on your specific needs and organizational context. Regardless of your toolset, mastering the art of EDA will make you a more effective data scientist.

As you embark on EDA in your own projects, remember to always anchor your work in the scientific method and let your curiosity be your guide. Now get out there and happy exploring!