The for Command - Shell Scripting

Iterating through a series of commands is a common programming practice. Often you need to repeat a set of commands until a specific condition has been met, such as processing all of the files in a directory, all of the users on a system, or all of the lines in a text file.

The bash shell provides the for command to allow you to create a loop that iterates through a series of values. Each iteration performs a defined set of commands using one of the values in the series. The basic format of the bash shell for command is:

for var in list
do
commands
done

You supply the series of values used in the iterations in the list parameter. There are several different ways that you can specify the values in the list.

In each iteration, the variable var contains the current value in the list. The first iteration uses the first item in the list, the second iteration the second item, and so on until all of the items in the list have been used.The commands entered between the do and done statements can be one or more standard bash shell commands. Within the commands the $var variable contains the current list item value for the iteration.

I mentioned that there are several different ways to specify the values in the list. The following sections show the various ways to do that.

Reading values in a list

The most basic use of the for command is to iterate through a list of values defined within the for command itself:

$ cat test1
#!/bin/bash
# basic for command
for test in Alabama Alaska Arizona Arkansas California Colorado
do
echo The next state is $test
done
$ ./test1
The next state is Alabama
The next state is Alaska
The next state is Arizona
The next state is Arkansas
The next state is California
The next state is Colorado
$

Each time the for command iterates through the list of values provided, it assigns the test variable the next value in the list. The $test variable can be used just like any other script variable within the for command statements. After the last iteration, the $test variable remains valid throughout the remainder of the shell script. It retains the last iteration value (unless you change its value):

$ cat test1b
#!/bin/bash
# testing the for variable after the looping
for test in Alabama Alaska Arizona Arkansas California Colorado
do
echo "The next state is $test"
done
echo "The last state we visited was $test"
test=Connecticut
echo "Wait, now we’re visiting $test"
$ ./test1b
The next state is Alabama
The next state is Alaska
The next state is Arizona
The next state is Arkansas
The next state is California
The next state is Colorado
The last state we visited was Colorado
Wait, now we’re visiting Connecticut
$

The $test variable retained its value, and also allowed us to change the value and use it outside of the for command loop, as any other variable would.

Reading complex values in a list

Things aren’t always as easy as they seem with the for loop. There are times when you run into data that causes problems. Here’s a classic example of what can cause shell script programmers problems:

$ cat badtest1
#!/bin/bash
# another example of how not to use the for command
for test in I don’t know if this’ll work
do
echo "word:$test"
done
$ ./badtest1
word:I
word:dont know if thisll
word:work
$

Ouch, that hurts. The shell saw the single quotation marks within the list values and attempted to use them to define a single data value, and it really messed things up in the process.

There are two ways to solve this problem:

  • Use the escape character (the backslash) to escape the single quotation mark.
  • Use double quotation marks to define the values that use single quotation marks.

Neither solution is all that fantastic, but each one does help solve the problem:

$ cat test2
#!/bin/bash
# another example of how not to use the for command
for test in I don’t know if "this’ll" work
do
echo "word:$test"
done
$ ./test2
word:I
word:don’t
word:know
word:if
word:this’ll
word:work
$

In the first problem value, I added the backslas character to escape the single quotation mark in the don’t value. In the second problem value, I enclosed the this’ll value in double quotation marks. Both methods worked fine to distinguish the value.Yet another problem you may run into is multi-word values. Remember, the for loop assumes that each value is separated with a space. If you have data values that contain spaces, you’ll run into yet another problem:

$ cat badtest2
#!/bin/bash
# another example of how not to use the for command
for test in Nevada New Hampshire New Mexico New York North Carolina
do
echo "Now going to $test"
done
$ ./badtest1
Now going to Nevada
Now going to New
Now going to Hampshire
Now going to New
Now going to Mexico
Now going to New
Now going to York
Now going to North
Now going to Carolina
$

Oops, that’s not exactly what we wanted. The for command separates each value in the list with a space. If there are spaces in the individual data values, you must accommodate them using double quotation marks:

$ cat test3
#!/bin/bash
# an example of how to properly define values
for test in Nevada "New Hampshire" "New Mexico" "New York"
do
echo "Now going to $test"
done
$ ./test3
Now going to Nevada
Now going to New Hampshire
Now going to New Mexico
Now going to New York
$

Now the for command can properly distinguish between the different values. Also, notice that when you use double quotation marks around a value, the shell doesn’t include the quotation marks as part of the value.

Reading a list from a variable

Often what happens in a shell script is that you accumulate a list of values stored in a variable and then need to iterate through the list. You can do this using the for command as well:

$ cat test4
#!/bin/bash
# using a variable to hold the list
list="Alabama Alaska Arizona Arkansas Colorado"
list=$list" Connecticut"
for state in $list
do
echo "Have you ever visited $state?"
done
$ ./test4
Have you ever visited Alabama?
Have you ever visited Alaska?
Have you ever visited Arizona?
Have you ever visited Arkansas?
Have you ever visited Colorado?
Have you ever visited Connecticut?
$

The $list variable contains the standard text list of values to use for the iterations. Notice that the code also uses another assignment statement to add (or concatenate) an item to the existing list contained in the $list variable. This is a common method for adding text to the end of an existing text string stored in a variable.

Reading values from a command

Yet another way to generate values for use in the list is to use the output of a command. You use the backtick characters to execute any command that produces output, then use the output of the command in the for command:

$ cat test5
#!/bin/bash
# reading values from a file
file="states"
for state in `cat $file`
do
echo "Visit beautiful $state"
done
$ cat states
Alabama
Alaska
Arizona
Arkansas
Colorado
Connecticut
Delaware
Florida
Georgia
$ ./test5
Visit beautiful Alabama
Visit beautiful Alaska
Visit beautiful Arizona
Visit beautiful Arkansas
Visit beautiful Colorado
Visit beautiful Connecticut
Visit beautiful Delaware
Visit beautiful Florida
Visit beautiful Georgia
$

this example uses the cat command to display the contents of the file states. You’ll notice that the states file includes each state on a separate line, not separated by spaces. The for command still iterates through the output of the cat command one line at a time, assuming that each state is on a separate line. However, this doesn’t solve the problem of having spaces in data. If you list a state with a space in it, the for command will still take each word as a separate value. There’s a reason for this, which we’ll look at in the next section.

Changing the field separator

The cause of this problem is the special environment variable IFS, called the internal field separator. The IFS environment variable defines a list of characters the bash shell uses as fieldseparators. By default, the bash shell considers the following characters as field separators:

  • A space
  • A tab
  • A newline

If the bash shell sees any of these characters in the data, it’ll assume that you’re starting a new data field in the list. When working with data that can contain spaces (such as filenames), this can be annoying, as you saw in the previous script example.

To solve this problem, you can temporarily change the IFS environment variable values in your shell script to restrict the characters the bash shell recognizes as field separators. However, there is somewhat of an odd way of doing this. For example, if you want to change the IFS value to only recognize the newline character, you need to do this:

IFS=$’ ’

Adding this statement to your script tells the bash shell to ignore spaces and tabs in data values.Applying this to the previous script yields the following:

$ cat test5b
#!/bin/bash
# reading values from a file
file="states"
IFS=$’ ’
for state in `cat $file`
do
echo "Visit beautiful $state"
done
$ ./test5b
Visit beautiful Alabama
Visit beautiful Alaska
Visit beautiful Arizona
Visit beautiful Arkansas
Visit beautiful Colorado
Visit beautiful Connecticut
Visit beautiful Delaware
Visit beautiful Florida
Visit beautiful Georgia
Visit beautiful New York
Visit beautiful New Hampshire
Visit beautiful North Carolina
$

Now the shell script is able to use values in the list that contain spaces.There are other excellent applications of the IFS environment variable. Say that you want to iterate through values in a file that are separated by a colon (such as in the /etc/passwd file). All you need to do is set the IFS value to a colon:

IFS=:

If you want to specify more than one IFS character, just string them together on the assignment line:

IFS=$’ ’:

This assignment uses the newline, colon, semicolon, and double quotation mark characters as field separators. There’s no limit to how you can parse your data using the IFS characters.

Reading a directory using wildcards

Finally, you can use the for command to automatically iterate through a directory of files. To do this, you must use a wildcard character in the file or pathname. This forces the shell to use file globbing. File globbing is the process of producing file or path names that match a specified wildcard character.

This feature is great for processing files in a directory, when you don’t know all of the filenames:

$ cat test6
#!/bin/bash
# iterate through all the files in a directory
for file in /home/rich/test/*
do
if [ -d file ]
then
echo "$file is a directory"
elif [ -f "$file" ]
then10
echo "$file is a file"
fi
done
$ ./test6
/home/rich/test/dir1 is a directory
/home/rich/test/myprog.c is a file
/home/rich/test/myprog is a file
/home/rich/test/myscript is a file
/home/rich/test/newdir is a directory
/home/rich/test/newfile is a file
/home/rich/test/newfile2 is a file
/home/rich/test/testdir is a directory
/home/rich/test/testing is a file
/home/rich/test/testprog is a file
/home/rich/test/testprog.c is a file
$

The for command iterates through the results of the /home/rich/test/* listing. The code tests each entry using the test command (using the square bracket method) to see if it’s a directory (using the -d parameter) or a file (using the -f parameter). (See Chapter 9, ‘‘Using Structured Commands’’.)Notice in this example I did something different in the if statement tests:

if [ -d $file ]

In Linux it’s perfectly legal to have directory and filenames that contain spaces. To accommodate that, you should enclose the $file variable in double quotation marks. If you don’t, you’ll get an error if you run into a directory or filename that contains spaces:

./test6: line 6: [: too many arguments
./test6: line 9: [: too many arguments

the bash shell interprets the additional words as arguments within the test command, causing an error.You can also combine bot directory search method and the list method in the same for statement, by listinaseries of directory wildcards in the for command:

$ cat test7
#!/bin/bash
# iterating through multiple directories
for file in /home/rich/.b* /home/rich/badtest
do
if [ -d "$file" ]
then
echo "$file is a directory"
elif [ -f "$file" ]
then
echo "$file is a file"
else
echo "$file doesn’t exist"
fi
done
$ ./test7
/home/rich/.backup.timestamp is a file
/home/rich/.bash_history is a file
/home/rich/.bash_logout is a file
/home/rich/.bash_profile is a file
/home/rich/.bashrc is a file
/home/rich/badtest doesn’t exist
$

the for statement first uses file globbing to iterate through the list of files that result from the wildcard character, then it iterates through the next file in the list. You can combine any number of wildcard entries in the list to iterate through.


All rights reserved © 2018 Wisdom IT Services India Pvt. Ltd DMCA.com Protection Status

Shell Scripting Topics