Unleashing the Power of Data Science at the Command Line with Scikit-Learn

Introduction

When you think of doing data science, what comes to mind? If you‘re like most aspiring data scientists, you probably envision firing up a Jupyter notebook, importing your favorite libraries like pandas, numpy, and scikit-learn, and diving into modeling. But what if I told you there was another way – a way that leveraged the speed, simplicity, and power of the command line?

That‘s right, I‘m talking about doing data science at the command line. Now before you scoff at the idea of shunning your beloved Python IDEs and notebooks, hear me out. The command line has been around for decades and is an incredibly efficient way to process data and automate repetitive tasks. By combining it with some nifty data science tools, you can supercharge your workflow and solve problems faster than ever.

In this post, I‘ll walk you through the process of doing data science at the command line, with a focus on using scikit-learn for machine learning. We‘ll cover the essential steps of the OSEMN data science process, which include:

• Obtaining data from various sources
• Scrubbing and cleaning data
• Exploring and visualizing data
• Modeling data with machine learning algorithms
• Interpreting results

Along the way, I‘ll introduce you to some powerful command line tools that will make your life easier, like csvkit, datamash, and skll. I‘ll also show you how to use R and ggplot to create visualizations directly from the terminal.

As an avid user of the command line for data science myself, I‘ll share my personal tips, insights, and opinions on how to make the most of this approach. My goal is to open your mind to the possibilities of the command line and give you a new set of tools to add to your data science toolbelt.

So without further ado, let‘s dive in!

Setting Up Your Data Science Environment

Before we get started, let‘s make sure you have everything you need to follow along with the examples in this post. At a minimum, you‘ll need:

• A Unix-based operating system (Linux or macOS)
• Basic familiarity with the command line
• Python 3.x installed
• R installed

If you‘re on Windows, you have a few options:

• Use the Windows Subsystem for Linux (WSL) to run a Linux environment directly on Windows
• Use a tool like Cygwin to provide a Unix-like environment on Windows
• Use a virtual machine or cloud instance running Linux

Personally, I recommend using either WSL or a cloud VM if you‘re on Windows. It‘ll make your life much easier in the long run.

Once your OS is set up, you‘ll need to install a few key command line tools. I suggest starting with:

• csvkit – for working with CSV data
• datamash – for fast, robust data processing
• jq – for working with JSON data
• skll – for machine learning experiments with scikit-learn
• ggplot2 for R – for creating visualizations

You can install most of these with pip or your system‘s package manager.

Here are the commands to install them:

pip install csvkit
sudo apt-get install datamash
pip install skll
brew install jq        [on macOS]
sudo apt-get install jq   [on Linux]

For R and ggplot2, first install R, then run:

R
> install.packages("ggplot2")

Once you have those basics installed, you‘re ready to start doing data science at the command line! Exciting, isn‘t it?

Obtaining and Exploring Data

Now that your environment is set up, let‘s obtain a dataset and start exploring.

For this demo, we‘ll use the classic Adult Census Income dataset. We can download it directly using the command line:

curl -O https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data

This downloads the file adult.data to your current directory.

Let‘s take a peek at the data using csvlook from the csvkit library:

csvlook adult.data

This displays a nice table preview of the data. We can see there are 14 columns with data on age, education, occupation, income, and more.

To get some quick summary statistics, we can use datamash:

cat adult.data | datamash --header-in mean 1 median 1 iqr 1 

This calculates the mean, median, and interquartile range of the first column (age). We see the average age is about 38, the median is 37, and the middle 50% are between 28 and 48.

We can use csvcut to select just the columns we want:

csvcut -c age,education,occupation,income adult.data | csvlook

To filter rows meeting certain criteria, use csvgrep:

csvgrep -c income -m ‘>50K‘ adult.data 

This shows only the rows where income is >50K.

As you can see, command line tools like csvkit and datamash make it easy to quickly slice and dice your data to start understanding it better.

But what about handling other file formats? No problem, there are handy command line tools for those too:

• in2csv converts Excel files to CSV
• jq let‘s you process JSON
• xml2json converts XML to JSON for further processing

The beauty of the command line is you can combine these tools to wrangle your data however you need. The sky‘s the limit!

Visualizing Data with R and ggplot

They say a picture is worth a thousand words, so let‘s create some visualizations of our Adult dataset. But instead of jumping over to a Jupyter notebook and firing up matplotlib, we‘ll do it directly from the command line using R and ggplot.

First, let‘s see the distribution of ages with a histogram:

csvcut -c age adult.data | Rio -ge ‘g+geom_histogram(binwidth=1)‘ > agehist.png

Here we use csvcut to extract only the age column, then pipe it to Rio, which lets us execute R code. The magic happens in the -ge flag, which tells Rio to run the code and save the output to agehist.png.

We can make a quick bar chart of education levels:

csvcut -c education adult.data | body sort | uniq -c | Rio -ge ‘g+geom_col()‘ > edubars.png

After selecting the education column, we sort the values, count the unique ones with uniq -c, then plot the counts as a bar chart.

Finally, let‘s compare age distributions by income level:

csvcut -c age,income adult.data | Rio -ge ‘g+geom_density(aes(age,color=income))‘ > ageincome.png  

With a few keystrokes, we made a nice density plot of the age distributions colored by income level. We can clearly see that higher incomes are associated with older individuals.

The ability to harness the power of R and ggplot directly from the command line is incredibly handy. You can make a plot with just a few commands, then iterate and refine it, all without ever leaving the terminal.

Machine Learning with Scikit-Learn

For the grand finale, let‘s build a machine learning model to predict income level based on the other features in our data. Of course we‘ll be using the venerable scikit-learn library, but through the clever skll command line interface.

To start, we need to prepare our data by:

1. Adding column names to the data file
2. Adding an id column
3. Splitting into train and test sets
4. Creating a config file to specify the model

Here are the commands to do all that:

head -n 1 adult.names | sed ‘s/|/,/g‘ > adult.csv
cat adult.data >> adult.csv
cp adult.csv adultdata.csv
sed -i ‘1s/^/id,/‘ adultdata.csv
gshuf adultdata.csv | csvcut -c -11,15-50 | csvcut -c 1-14,income > adult_clean.csv
cat adult_clean.csv | csvcut -c income | csvstat
cat adult_clean.csv | body shuffle | head -n 30000 > train.csv
cat adult_clean.csv | body shuffle | tail -n 2222 > test.csv

First we extract the column names from the adult.names file and put them in adult.csv along with the data. Then we clean things up by removing some unnecessary columns and doing some light preprocessing. Finally we split the data into train and test sets.

Now for the skll config file:

[General]
experiment_name = AdultIncome
task = evaluate

[Input]
train_directory = train
test_directory = test
featuresets = [["train.csv",
                "test.csv"]]
learners = ["LinearSVC"]
label_col = income

[Tuning]
grid_search = true
objective = f1_score_micro

[Output]
logs = output
results = output
predictions = output

We specify the train and test data locations, the model to use (LinearSVC), and the target column. We‘ll do a grid search to tune the model hyperparameters and output the results to the output directory.

Time to train the model!

run_experiment income.cfg

After a minute, we have our results:

cat output/AdultIncome_summary.tsv

The LinearSVC model achieved 84% accuracy on the test set. Not too shabby for a few lines of configuration and a single command to train!

We can explore other models by simply adding them to the learners list in the config file, for example:

learners = ["LinearSVC", "LogisticRegression", "RandomForestClassifier"]

Rerunning the experiment will train all those models and compare the results.

There you have it – a complete machine learning pipeline, from data processing to model training to evaluation, done entirely through the command line!

Conclusion

Wow, we covered a lot of ground in this post! We saw how to:

• Obtain data from the web
• Wrangle and explore data with csvkit and datamash
• Create visualizations with R and ggplot
• Train and evaluate machine learning models with scikit-learn

And we did it all without ever leaving the command line. Hopefully this has given you a taste of just how powerful the command line can be for data science.

With a solid understanding of shell scripting, some key command line tools in your belt, and trusty scikit-learn by your side, you can quickly gather data, extract insights, and build models, all with just a few keystrokes. No fiddling with complex IDEs or waiting for notebooks to load – just efficient, reproducible data science.

I find this streamlined workflow incredibly freeing. Once you learn a few key tools and commands, you can mix and match them to process data however you need. With pipes and filters, the command line lets you quickly explore data and prototype ideas. And with version control, it‘s easy to track your experiments and collaborate with others.

That said, the command line isn‘t the right fit for every task. Sometimes you need the interactivity of a notebook, or the visual interface of a tool like Tableau. And if you‘re working with massive datasets, Spark is probably a better choice.

But for small to medium-sized data, the command line is often the fastest and simplest way to get the job done. And with the skll package, it‘s easy to do machine learning with the familiar scikit-learn API.

So next time you‘re wrangling a CSV file or training a model, I encourage you to give the command line a shot. You might be surprised at how much you can accomplish without ever opening a notebook. And you just might find that the command line becomes your go-to data science environment.

I hope this post has gotten you excited to try doing data science at the command line. It‘s a fun and effective way to up your data science game. Give it a try and let me know how it goes!