commands to execute for each item

Introduction

Bash scripting is an essential skill for any Linux user or system administrator. One of the most powerful tools in your bash scripting toolkit is the for loop, which allows you to automate repetitive tasks and work efficiently with large datasets. In this comprehensive guide, we‘ll dive deep into the world of bash for loops, exploring their syntax, usage, and advanced techniques. Whether you‘re a beginner looking to learn the basics or an experienced scripter seeking to refine your skills, this article will provide you with the knowledge and examples you need to master for loops and take your bash scripting to the next level.

Understanding For Loops in Bash

At its core, a for loop is a way to iterate over a list of items and perform a set of commands for each item. The basic syntax of a bash for loop looks like this:


for item in list
do

done


Here, item is a variable that takes on the value of each element in list one at a time. The commands inside the do...done block are executed for each iteration of the loop.

The list can be specified in several ways:

• As a simple list of strings separated by spaces

for color in red green blue; do echo $color; done

• As an array variable

fruits=(apple banana orange); for fruit in "${fruits[@]}"; do echo $fruit; done

• As the output of a command using command substitution

for file in $(ls); do echo $file; done

Range Loops

One of the most common use cases for bash for loops is iterating over a sequence or range of numbers. There are a few different ways to create numeric ranges in bash:

Using the seq command

The seq command generates a sequence of numbers from a starting value to an ending value, with an optional step value:

for num in $(seq 1 5); do echo $num; done

This will output the numbers 1 through 5. You can also specify a step value as a third argument:

for num in $(seq 0 2 10); do echo $num; done

This will print the even numbers from 0 to 10.

Brace expansion

Bash supports a shorthand syntax for generating ranges using brace expansion:

for num in {1..5}; do echo $num; done

This is equivalent to the seq example above. You can also include a step value using double dots:

for num in {0..10..2}; do echo $num; done

Again, this will print the even numbers between 0 and 10.

C-style syntax

Bash also supports a C-like syntax for for loops using double parentheses:

for ((i=0; i<5; i++)); do echo $i; done

This loop will run 5 times with the variable i taking on values from 0 to 4. The syntax within the double parentheses is similar to C and supports the usual arithmetic and logical operators.

Here‘s a more complex example demonstrating the power of C-style for loops:


for ((i=1, j=5; i<=5; i++, j--))
do
echo "$i/$j = $(echo "scale=2; $i/$j" | bc)"
done


This loop calculates the division of two sequences (1/5, 2/4, 3/3, 4/2, 5/1) with two decimal places of precision using bc for floating-point math.

Advanced For Loop Techniques

Now that you‘re comfortable with the basics of bash for loops, let‘s explore some more advanced usage and techniques.

Looping over command output

We saw a simple example of this earlier using ls, but you can loop over the output of any command by enclosing it in $(...):

for user in $(cut -f1 -d: /etc/passwd); do echo $user; done

This example loops through the first field of each line in the /etc/passwd file, outputting the usernames.

Reading lines from a file

You can loop through the lines of a file using the read command with a while loop or a process substitution:


while IFS= read -r line
do
echo "$line"
done < input.txt


Or using process substitution:

for line in $(< input.txt); do echo $line; done

The first method is generally preferred as it handles file input more cleanly.

Parallel loops

Have a lot of data to process? Bash for loops can be run in parallel for improved performance on multi-core systems using tools like GNU Parallel:


seq 1 100 | parallel -j4 ‘echo processed item {}‘


This will run four parallel jobs processing the numbers 1 to 100. Just replace echo with your actual processing task.

Nested loops

For loops can be nested to perform complex multi-dimensional processing such as iterating over all files in a series of directories:


for dir in /; do
for file in "$dir"; do
[[ -f $file ]] && echo "$file"
done
done


This example uses a nested loop to list all regular files in each subdirectory, skipping other types of filesystem entries.

Use Case Examples

Now that you‘ve seen the flexibility and power of bash for loops, let‘s walk through some practical scripting examples.

Bulk renaming files

Suppose you have a directory full of files named image001.jpg, image002.jpg, ... and you want to prepend today‘s date in YYYYMMDD format. Here‘s a one-liner to do it:


for img in image*.jpg; do
mv "$img" "$(date +%Y%m%d)-$img"
done


Monitoring disk usage

Here‘s a script to monitor disk usage on all mounted filesystems and send an alert if any exceed 90% usage:


for fs in $(df -h | grep -vE ‘(tmpfs|udev)‘ | awk ‘{print $6}‘); do
usage=$(df -h "$fs" | tail -n 1 | awk ‘{print $5}‘ | cut -d‘%‘ -f1)
if [[ $usage -ge 90 ]]; then
echo "Alert: $fs is $usage% full!"
fi
done


You could run this as a cron job for proactive monitoring.

Analyzing log files

Suppose you have a web server access log and want to find the top 10 IPs by request count:


for ip in $(cut -f1 -d‘ ‘ access.log | sort -u); do
requests=$(grep -c "$ip" access.log)
echo "$ip,$requests"
done | sort -t‘,‘ -k2nr | head


This script loops through the unique IPs in the log file, counts the occurrences of each, sorts by the request count descending, and takes the top 10.

Tips and Best Practices

Here are some tips to keep in mind when writing bash for loops:

• Double quote variables to avoid misparsing due to spaces or special characters: "$var"
• Use $() instead of backticks ` ` for command substitution; it‘s more readable and nestable
• Prefer ${array[@]} over $array for looping through array elements to handle elements containing whitespace correctly
• Be careful with filenames containing spaces or special characters; consider using find -print0 with read -d $‘\0‘ to handle these
• Avoid unnecessary cat or ls by using process substitution or find -exec instead

And some general bash scripting best practices:

• Always start your scripts with #!/bin/bash and use set -euo pipefail to avoid subtle bugs
• Use functions to organize your code into logical and reusable components
• Prefer $(...) over (...) for arithmetic; the latter is a subshell and slower
• Use comments and meaningful variable names to improve readability and maintainability

Conclusion

In this guide, we‘ve explored the ins and outs of bash for loops, from basic syntax to advanced usage and best practices. You‘ve seen how for loops can be used to iterate over simple lists, ranges of numbers, command output, and file contents, and be combined with other bash features for powerful scripting.

The examples we walked through demonstrated just a few of the countless ways for loops can automate everyday tasks and parse/process data, but the possibilities are limitless. I encourage you to take what you‘ve learned and experiment with for loops in your own bash scripting- I think you‘ll be amazed by how much you can accomplish!

Bash for loops are an essential tool in any scripter‘s toolbox, and I hope this guide has given you the knowledge and confidence to fully leverage them. Now get out there and happy looping!