Shell Scripting Tricks

Terminology

Here is some basic shell terminology you need to know.

• Parameter

  A parameter is essentially just a variable, except BASH distinguishes between Positional Parameters (the parameters supplied to the script on the command line, like: ./my_script.sh these are positional parameters & Variable Parameters (the parameters defined inside the script, like: my_variable_parameter='hello';

• Word

  A word is just a string.

• Pattern

  A regex matching pattern.

What This Tutorial Covers

What You Need For This Tutorial

Tests

By tests, I'm just referring to how 'if' evaluates expressions. The way shell scripts evaluate expressions is particularly unique.

• Use the 'test' command to evaluate an expression.

if test "hi" = "hi"; then echo "there"; fi
        

• Use brackets to evaluate an expression.

if [ "hi" = "hi" ]; then echo "there"; fi
        

• Use double brackets to evaluate an expression with regex. There are more differences between this and the single bracket, however I find that the ability to use regex and the inclusional of additional operators over flags are the biggest differences.

if [[ "hi" =~ "^h" ]]; then echo "there"; fi
        

Output Redirection & Piping

File descriptors are just numbers that are shortcuts to file streams. You always start with three, stdin (0), stdout (1), stderr (2). You can use redirection to move these around.

• Redirect stderr (2) to stdout (1). On the left of the '>' chevron, you just need the number of the file descriptor. On the right of the '>' chevron you need to prepend the file descriptor number with an ampersand.

2>&1
        

• Open the file descriptor (3) for reading to 'readFile.txt' and open file descriptor (4) for writing to 'writeFile.txt'. Then grep for the word 'hello' in 'readFile.txt' and write the results to 'writeFile.txt'.

3< readFile.txt 
4> writeFile.txt 
grep "hello" <&3 >&4
        

• Close the file descriptor (3).

3<&-
        

• Open file descriptor (3) for both reading and writing.

3<> myFile.txt
        

Redirection is just moving output from one file stream to another file stream, while Pipe is the redirection of the output from one command to another command.

• Run a command and redirect stdout to a file. Without a number, the '>' chevron defaults to redirecting stdout (1). Commands are written to pull input from stdin and push output to either stdout or stderr. So what's really happening here is that the 'echo' command is printing 'hello there' to stdout and then we are redirecting whatever stdout receives to the file.

echo 'hello there' > 'my file.txt'
        

• In this example, we redirect stderr to the same file

rmdir some_dir_that_does_not_exist 2> 'my file.txt'
        

• Redirect command output and append it to a file

echo 'hello there' >> 'my file.txt'
        

• Redirect a multline string to a command. This is known as a HereDoc or Here Document. The EOF is a macro for the 'End Of File' character. As it's commonly not seen in strings until the end (obviously), it's usually used to mark the beginning and end of a HereDoc string.

cat <<EOF
hi
there
how's it going
EOF
        

• Redirect the string into the command. This is known as a Here String. It's usefulness is that it avoids using subshells, which occur when using pipes (both pipes and subshells are discussed below in different sections).

cut -d' ' -f2 <<< 'why hello there'
        

• Pipe stdout from one command into stdin of another command. Behind the scenes, pipes actually run all commands concurrently in their own subshells, but, each subshell waits to receive input, so in a sense, the commands run sequentially from left to right.

echo 'hello there' | cut -d' ' -f1
        

You can also pipe stderr using a special pipe syntax. Usually only available in later shell versions.

rmdir some_dir_that_does_not_exist |& cut -d':' -f3
        

Shell Expansions & Subshells

Expansions are when some code is converted to something else when evaluated. Here I'll go over several types of expansions (except parameter expansion, which is described in the next section below).

Filename Expansion, otherwise known as Globbing, are some characters you can use to expand a filename into multiple files.

• Remove all files that start with 'hello'

rm hello*
        

• Remove all files that start with 'hell', have any fifth character, and end with 'txt'

rm hell?.txt
        

• Remove all files that start with 'hell', have either an 'a' or an 'o' as the fifth character, and end with 'txt'

rm hell[ao].txt
        

Tilde & Dash Expansion

• Change directory to current user's home

cd ~
        

• Change directory to specified user's home

cd ~root
        

• Change directory to previous directory

cd -
        

Brace Expansion

• Run the mkdir command twice to create both directories: /var/log & /var/tmp

mkdir /var/{log,tmp}
        

Subshells, Command Substitution, & Parallel Processing.

• Run the echo command in a child process. This is known as a Subshell.

(echo 'hello there')
        

• Store the result of the echo command in a variable. This is known as Command Substitution.

result=$(echo 'hello there')
        

• Run the echo command in a child process in the background (continue executing the script without waiting for the child process to complete). You'll see job IDs and process IDs printed when running background processes.

(echo 'hello there') &
        

• Run multiple echo commands in parallel child processes. Then use the wait command to wait for each child process to finish before continuing.

(sleep 1 && echo 'hello there') &
(echo 'how are you today?') &

wait

echo 'i am fine, thank you'
        

Arithmetic Expressions

• Arithmetic expressions must be run in double parentheses.

((2 + 2))
        

• And just like Command Substitution, you can expand the result for storing using the dollar sign.

result=$((2 + 2))
        

# Set new positional parameters (they replace the old ones, they don't append)
set -- these are new positional parameters

# If a command in your script fails, exit the script
set -e

# You can always change the '-' to a '+' to turn the feature off
set +e

# If you try to expand a variable that does exist, exit the script
set -u

# If a command in a series of piped commands fail, 
# return its exit status instead of the exit status of the last command
set -o pipefail

# Print the commands that are run prepended by a '+' sign,
# followed by their output. Useful for debugging.
set -x

# Do not allow redirection to overwrite existing files.
set -C
      

unset myVariable
      

# The name of the script or command
$0

# The arguments passed to the script
$1 $2 $3

# All of the arguments passed to the script
$*

# The entire command line, or in other words, 
# the script name and all of its arguments
$@

# The number of arguments passed to the script
$#

# The options (flags) the shell has on to set how it behaves
$-

# Exit value of the last command
$?

# Process ID of the last command
$!