Essential Bash Commands Every Data Scientist Should Know

As a data scientist, you likely spend a significant amount of time working with data – collecting it, cleaning it, analyzing it, visualizing it. While tools like Python, R, and SQL are the go-to for in-depth data manipulation and modeling, sometimes you just need to do quick exploration or transformation of data files. That‘s where knowing some bash commands comes in very handy.

Bash (which stands for Bourne Again Shell) is a Unix shell and command language that provides a powerful interface for interacting with your operating system. Most data science work is done on Unix-based systems like Linux and macOS, so familiarity with bash is essential. In fact, a recent survey by Kaggle found that 54% of data scientists use bash scripting regularly.

With just a few keystrokes, you can navigate folders, peek inside files, search for patterns, join datasets, and more – no coding required! This can be a huge time-saver, especially when dealing with large datasets that are cumbersome to load into memory with Python or R.

Let‘s dive into some of the most useful bash commands for wrangling data and enhancing productivity as a data scientist. These examples use a 1.5 GB dataset of NYC taxi trips from 2013, downloaded and unzipped like so:

wget https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2013-12.csv.gz
gunzip yellow_tripdata_2013-12.csv.gz

Navigating Directories

First, you need to be comfortable moving around your filesystem. These basic commands will orient you:

• pwd (print working directory) – shows the current directory path
• ls (list) – shows the files and subdirectories in the current directory
  • ls -lh provides more details including file sizes in human-readable format
• cd (change directory) – navigates into a different directory
  • cd .. moves up one level
  • cd ~ goes to your home directory
  • cd - returns to the previous directory you were in
• mkdir (make directory) – creates a new directory
• rmdir (remove directory) – deletes an empty directory

Viewing File Contents

Next up is peeking inside files without having to open them in a text editor or load into Python/R:

• cat (concatenate) – outputs the entire contents of one or more files
  • cat file1.txt file2.txt > combined.txt joins files together
• less – allows scrolling through the contents of a file
• head – shows just the top lines of a file (default is 10)
  • head -n5 file.txt shows the first 5 lines
• tail – shows the last lines of a file (default is 10)
  • tail -f logfile shows new lines as they are written to the file
• wc (word count) – counts the lines, words, and characters in a file
  • wc -l displays just the number of lines

On our taxi dataset:

$ wc -l yellow_tripdata_2013-12.csv
14776615 yellow_tripdata_2013-12.csv

$ head -n3 yellow_tripdata_2013-12.csv  
vendor_id,pickup_datetime,dropoff_datetime,passenger_count,trip_distance,pickup_longitude,pickup_latitude,rate_code,store_and_fwd_flag,dropoff_longitude,dropoff_latitude,payment_type,fare_amount,surcharge,mta_tax,tip_amount,tolls_amount,total_amount
CMT,2013-12-01 00:00:00,2013-12-01 00:05:00,3,1.10,-73.989320,40.732875,1,N,-73.978760,40.747150,CSH,6.5,0.5,0.5,0,0,7.5
VTS,2013-12-01 00:00:00,2013-12-01 00:07:00,1,1.26,-73.973305,40.793921,1,,,,CRD,8,0.5,0.5,1.66,0,10.66

So we can see this file has over 14 million rows, and from the header and first couple lines, get a quick sense of the column names and values.

Searching Within Files

Finding needles in haystacks becomes trivial with these search tools:

• grep (global regular expression print) – searches for a pattern in files
  • grep "error" *.log finds all lines with "error" in log files
  • grep -i makes the search case-insensitive
  • grep -v inverts the search, showing lines that don‘t match
  • grep -r searches recursively in all subdirectories
  • grep -c counts matching lines instead of showing them
• awk – for more advanced text processing, extraction by column, etc
  • awk -F‘,‘ ‘{print $2}‘ prints the 2nd column of comma-separated lines
• sed (stream editor) – for find & replace and other transformations
  • sed ‘s/foo/bar/g‘ replaces foo with bar

Let‘s search our taxi data for rows with more than 6 passengers:

$ grep -c ",[7-9]," yellow_tripdata_2013-12.csv
30521

$ time grep ",[7-9]," yellow_tripdata_2013-12.csv > output.csv
real    0m25.027s

So there are a little over 30k trips with 7 or more passengers in this dataset (0.2% of total trips). Grep is quite fast – it took only 25 seconds to search all 14 million lines and write out the matching rows to a new file! Just for comparison, loading the whole 1.5 GB CSV file into a Pandas dataframe in Python takes nearly 3 minutes on my machine.

Filtering and Transforming Data

Narrowing down data files and reshaping them is a breeze using these commands:

• cut – extracts specific columns from delimited files
  • cut -d‘,‘ -f1-3 gets the first three comma-separated columns
• sort – orders rows by a given column
  • sort -t‘,‘ -k4 -n sorts numerically by the 4th column of CSV data
  • sort -r reverses the order
  • sort -u sorts and filters out duplicate rows
• uniq – finds or removes duplicates
  • uniq -c prefixes each row with a count of occurrences
• paste – joins files horizontally by rows
  • paste -d‘,‘ file1.txt file2.txt combines two files row-by-row with comma separator
• join – merges two files on a common column
• tr – translates/deletes characters
  • tr ‘[:lower:]‘ ‘[:upper:]‘ converts to uppercase

For a quick example, let‘s find the most frequent pickup locations:

$ cut -d‘,‘ -f6-7 yellow_tripdata_2013-12.csv | sort | uniq -c | sort -nr | head
  12363 -73.982524,40.768867
  11875 -73.981160,40.775898
  10470 -73.980016,40.751661
   9709 -73.991638,40.756483
   9494 -73.974634,40.767822
   9189 -73.993896,40.750585
   9151 -73.982734,40.76466
   8621 -73.99182,40.7501
   7799 -73.96974,40.799527
   7673 -73.986122,40.760304

First, we use cut to extract just the latitude and longitude columns. Then we sort lexicographically and pipe into uniq -c to get counts of unique coordinates. Finally, we do a numeric reverse sort to get the most frequent locations. All without ever leaving the command line!

Analyzing and Visualizing

We can even do some quick analysis and plotting right in the terminal:

• datamash – simple statistics on tabular data
  • datamash max 1 finds the maximum value in the first column
• xsv – csvkit‘s Swiss Army knife for analyzing and manipulating CSV data
  • xsv stats yellow_tripdata_2013-12.csv calculates summary statistics for each column
• gnuplot and feedgnuplot – for interactive plotting and piping data from other commands
  • cut -d‘,‘ -f4,5 file.csv | feedgnuplot --points makes a scatterplot of columns 4 and 5

Let‘s get some basic stats on our taxi fares:

$ cut -d‘,‘ -f13 yellow_tripdata_2013-12.csv | datamash min 1 max 1 mean 1 median 1 q1 1 q3 1
2.5,419.3,12.450598752203938,9.5,6,14.5

$ cut -d‘,‘ -f13 yellow_tripdata_2013-12.csv | feedgnuplot --histogram 0 --binwidth 5 --xlen .05  > fares.png

The datamash command instantly gives us some quick summary statistics – we can see that fares range from $2.50 to $419.30, with a mean of $12.45, median of $9.50, and interquartile range of $6 to $14.50.

Then, piping the fares to feedgnuplot, we can generate a histogram plot and redirect it to an image file:

Not too shabby for a little command line fu!

Leveraging Bash for Big Data

Where bash really shines is in processing huge datasets that may be too large to fit in memory for Python or R. In general, bash loops will outperform Python for loops on big data thanks to its speedy built-in Unix commands.

For example, let‘s say we want to calculate the average speed in mph of each taxi ride. In Python, we might do something like:

import pandas as pd

df = pd.read_csv(‘yellow_tripdata_2013-12.csv‘)

df[‘duration‘] = (pd.to_datetime(df[‘dropoff_datetime‘]) - pd.to_datetime(df[‘pickup_datetime‘])).dt.total_seconds() / 3600
df[‘speed_mph‘] = df[‘trip_distance‘] / df[‘duration‘]

print(df[‘speed_mph‘].mean())

This takes about 5 minutes to run on my machine, and consumes over 6 GB of RAM since it reads the whole 1.5 GB file into a dataframe.

Compare that to this bash command:

paste <(cut -d‘,‘ -f2,3 yellow_tripdata_2013-12.csv) <(cut -d‘,‘ -f5 yellow_tripdata_2013-12.csv) |
  awk -F‘[,:]‘ ‘{ 
    h = $2; m = $3; s = $4
    h2 = $6; m2 = $7; s2 = $8
    secs = (h2*3600 + m2*60 + s2) - (h*3600 + m*60 + s)
    mph = $11 / (secs / 3600)
    total += mph; n++ 
  } END {
    print total / n 
  }‘

This runs in under a minute and uses less than 300 MB of memory! Here‘s what it does step-by-step:

1. paste is used to extract and combine the pickup time, dropoff time, and distance columns into a single stream
2. awk then processes each row:

• Parsing out the hours, minutes, seconds from the timestamps
• Converting to total seconds and finding the trip duration
• Dividing distance by duration in hours to calculate speed
• Keeping a running total of speeds and row count

3. After processing all rows, it prints out the average speed

Essentially, awk lets us vectorize the calculations across rows without needing to load the whole dataset into memory like Pandas does. Lots of big data processing tasks follow a similar pattern of streaming data through a command pipeline.

Integrating with Distributed Frameworks

Bash can also be a valuable ally when working with distributed computing frameworks like Hadoop or Spark. Many common operations can be done without ever needing to write a lick of Java or Scala.

For instance, to process multiple data files with a Hadoop streaming job:

hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar \
    -input /user/me/input/* \
    -output /user/me/output \
    -mapper /path/to/mapper.sh \
    -reducer /path/to/reducer.sh

Here the mapper and reducer are just bash scripts that read from stdin and write to stdout, similar to the commands we‘ve already covered. You can use this to do parsing, filtering, aggregations, etc. over huge datasets.

Similarly, you can use bash with Spark via the --archives flag to distribute a zip file of bash scripts to each executor node:

spark-submit --archives my-bash-scripts.zip# my-spark-job.py

Then inside your PySpark code, you can call the bash scripts using the subprocess module:

import os, subprocess

def bash_transform(x):
    script = os.path.join(SparkFiles.getRootDirectory(), ‘my-bash-scripts.zip#transform.sh‘)
    proc = subprocess.Popen(script, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    out, err = proc.communicate(str(x).encode())
    return out.decode().strip()

rdd.map(bash_transform)  

This can be an easy way to parallelize existing bash commands without having to re-implement them in Python.

Conclusion

The command line is a data scientist‘s secret weapon. Mastering bash will streamline your workflow, allowing you to quickly explore and transform data, automate repetitive tasks, and level up your big data game.

Some key points to remember:

• Pipes (|) are the key to stringing together commands into powerful data processing pipelines
• Favor text formats like CSV over binary formats for easy command line processing
• Combine bash with Python or R for a hybrid approach, using the best tool for each task
• Brush up on your regex skills to unlock even more text filtering power

Adopt these practices and you‘ll be well on your way to becoming a command line ninja! The only limit is your imagination.

Here are some resources to continue your bash journey:

• Data Science at the Command Line (book)
• Pure Bash Bible (GitHub repo)
• Command Line Data Manipulation (DataCamp course)
• Bash Redirections Cheat Sheet (PDF)
• Explain Shell (web tool to explain any command)

Now fire up your terminal and get bashing!