Shell Scripting Fundamentals

July 25, 2022 · 23 min read

If you have any questions, feel free to comment below. Click the block can copy the code.
And if you think it's helpful to you, just click on the ads which can support this site. Thanks!

Shell is the language we use to communicate with the operating system through the command line.

Shell scripts can run directly on the command line, or a set of logic can be organized into a file for reuse. The command line can be thought of as a “shell script executing line by line.”

There are many common kinds of shell scripts in Linux, including:

• Bourne Shell (/usr/bin/sh or /bin/sh)
• Bourne Again Shell (/bin/bash)
• C Shell (/usr/bin/csh)
• K Shell (/usr/bin/ksh)
• zsh
• …

Linux systems generally use bash by default.

The file should begin with #! /bin/bash to specify bash as the script interpreter.

Script example #

Create a test. sh file with the following contents:

#! /bin/bash
Echo "Hello World!"

Ways to run it #

As an executable file

ls -l test.sh # Check permissions
chmod +x test.sh  # Give the script executable permission

./test.sh  # Run from the current path
# Hello World!  # Script output

/home/acs/test.sh  # Run using an absolute path
# Hello World!  # Script output

~/test.sh  # Run from the home-directory path
# Hello World!  # Script output

Run with an interpreter

bash test. Sh
Hello World!  # Script output

Comments #

Single-line comments #

Everything after # on each line is a comment.

# This is a single-line comment
echo 'Hello World'  #  This is also a comment

Multiline comments #

Format:

:<<EOF
First line of comments
Second line of comments
Third line of comments
EOF

EOF can be replaced with any other string. For example:

:<<abc
First line of comments
Second line of comments
Third line of comments
Abc

:<<!
First line of comments
Second line of comments
Third line of comments
!

Defining variables #

When defining a variable, no $ sign is needed; there must be no spaces on either side of the equals sign. For example:

Name 1='timerring'  # Define a string with single quotes
Name 2="timerring"  # Define a string with double quotes
Name 3=timerring    # Quotes can also be omitted; this still represents a string

Variables are strings when defined, but when a variable needs to be an integer, it is automatically converted to an integer.

Using variables #

To use a variable, add the $ symbol or the $ {} symbol. Braces are optional and mainly help the interpreter identify variable boundaries.

name=timerring
echo $name  # Output timerring
echo ${name}  # Output timerring
echo ${name}acwing  # Output timerringacwing

Read-only variables #

Use readonly or declare to make a variable read-only. A variable declared read-only cannot be deleted with unset.

name=timerring
readonly name
declare -r name  # Either form works

name=abc  # Produces an error because name is now read-only

Deleting variables #

unset can delete a variable. If a variable does not exist, its value is an empty string.

name=timerring
unset name
echo $name  # Outputs an empty string

Variable types #

1. User-defined variables (local variables): variables that child processes cannot access
2. Environment variables (global variables): variables that child processes can access

A process can launch multiple child processes. Enter bash on the command line to launch a new process, and enter exit or Ctrl+d to leave the new bash.

Convert a user-defined variable into an environment variable:

name=timerring  # Define a variable
export name  # First method
declare -x name  # Second method

Convert an environment variable into a user-defined variable:

export name=timerring  # Define an environment variable
declare +x name  # Convert to a user-defined variable

Ways to pass parameters between two scripts:

1. Call one script from inside another script.
2. Pass parameters using environment variables that multiple scripts can share.

#s1:
#!/bin/bash
# Set an environment variable
export MY_PARAM1="arg1"
./s2.sh

#s2:
#!/bin/bash
# Read the environment variable in script2.sh
echo "First argument: $MY_PARAM1"

Strings #

Strings can use single quotes, double quotes, or no quotes.

Differences between single and double quotes:

• Contents within single quotes are output as-is, without execution or variable expansion;
• Contents within double quotes can be executed and variables can be expanded; for strings, no quotes and double quotes have the same effect.

name=timerring  # No quotes
echo 'hello, $name \"hh\"'  # Single-quoted string, outputs hello, $name \"hh\"
echo "hello, $name \"hh\""  # Double-quoted string, outputs hello, timerring "hh"

Get string length

name="timerring"
echo ${ #name }  # Outputs 3

Extract a substring

name="hello, timerring"
echo ${name:0:5}  # Extract 5 characters starting at 0

Type #

Type+command can explain where a command comes from (built-in commands, third-party commands, etc.), for example:

type readonly # readonly is a shell builtin (shell built-in command)
Type ls # ls is aliased to ‘ls –color+auto’

Default variables #

File parameter variables #

When executing a shell script, you can pass parameters to it. $1 is the first parameter, $2 is the second, and so on. In particular, $0 is the filename (including the path: an absolute path is shown if entered, and a relative path is shown if entered).

When passing parameters to a script, parameter numbers with more than one digit need to be enclosed in braces: echo {10}

For example, create the file test.sh:

#! /bin/bash

Echo "Filename:"$0
Echo "First parameter:"$1
Echo "Second parameter:"$2
Echo "Third parameter:"$3
Echo "Fourth parameter:"$4

Then run the script:

chmod +x test. Sh 
./test. Sh 1 2 3 4

# output
# Filename:./test. Sh
# First parameter:1
# Second parameter:2
# Third parameter:3
# Fourth parameter:4

Other parameter-related variables #

Arrays #

An array can hold multiple values of different types. Only one-dimensional arrays are supported, and the array size need not be specified when initializing it. Array indices start at 0.

Definition #

An array is written with parentheses, with spaces between elements. For example:

array=(1 abc "def" timerring)

You can also directly define the value of an array element:

array[0]=1
array[1]=abc
array[2]="def"
array[3]=timerring

Reading an element’s value #

Format:

${array[index]}

For example:

array=(1 abc "def" timerring)
echo ${array[0]} # 1
echo ${array[1]} # abc
echo ${array[2]} # def
echo ${array[3]} # timerring

Reading the entire array #

${array[@]}  # First form
${array[*]}  # Second form

Array length #

Similar to strings

${#array [@]}  # First form
${#array [*]}  # Second form

Expr command #

The expr command evaluates an expression, in this format:

expr EXPRESSION

Notes on expressions #

• Separate each item with spaces
• Put a backslash before shell-specific characters (if an expression fails, try escaping them)
• Quote strings containing spaces or other special characters
• expr prints the result to stdout. For a logical relational expression, stdout contains 1 if true and 0 otherwise.
• expr’s exit code: for a logical relational expression, it is 0 if true and 1 otherwise.

String expressions #

• length STRING : Returns the length of STRING
• index STRING CHARSET : Position of the first occurrence in STRING of any individual character in CHARSET, starting at 1. Returns 0 if STRING contains none of those characters.
• substr STRING POSITION LENGTH : Returns a substring of STRING starting at POSITION with a maximum length of LENGTH. Returns an empty string if POSITION or LENGTH is negative, 0, or nonnumeric.

Example:

str="Hello World!"
# It is best to quote $str here; otherwise this is equivalent to echo `expr length Hello World!`, where World! is treated as another argument and an error occurs
echo `expr length "$str"`  # `` is not single quotes; it is the short slanted mark on the ESC key, meaning execute this command; outputs 12
# echo expr length '$str' # Outputs 4: the literal string $str, because single quotes are used here
echo `expr index "$str" aWd`  # Outputs 7; indexing starts at 1
echo `expr substr "$str" 2 3`  # Outputs ell

Integer expressions #

expr supports ordinary arithmetic operations. Arithmetic expressions have lower precedence than string expressions and higher precedence than logical relational expressions. Note that bash can only perform integer arithmetic; it treats floating-point numbers as strings.

• + - : Addition and subtraction. Both operands are converted to integers; a failed conversion produces an error.
• * / % : Multiplication, division, and modulo. Both operands are converted to integers; a failed conversion produces an error. Note: because * is a special character, it must also be escaped when used.
• () can change precedence, but must be escaped with backslashes

a=3
b=4

echo `expr $a + $b`  # Outputs 7
echo `expr $a - $b`  # Outputs -1
echo `expr $a \* $b`  # Outputs 12; * must be escaped
echo `expr $a / $b`  # Outputs 0; integer division
echo `expr $a % $b` # Outputs 3
echo `expr \( $a + 1 \) \* \( $b + 1 \)`  # Outputs 20; value is (a + 1) * (b + 1)

Additionally, $[] or $(()) can be used; both perform mathematical operations. They support + - * / % (“addition, subtraction, multiplication, division, modulo”).

$ a=5; b=7; c=2  
$ echo $(( a+b*c ))  
19  
$ echo $(( (a+b)/c ))  
6  
$ echo $(( (a*b)%c))  
1

Logical relational expressions #

• | : If the first operand is nonempty and nonzero, returns its value; otherwise returns the second operand’s value, provided it is also nonempty or nonzero, and returns 0 otherwise. If the first operand is nonempty or nonzero, the second is not evaluated.
• &: If both operands are nonempty and nonzero, returns the first; otherwise returns 0. If the first operand is 0 or empty, the second is not evaluated.
• < <= = == != >= > : Compares the operands, returning 1 if true and 0 otherwise. == is synonymous with =. “expr” first tries to convert both operands to integers for an arithmetic comparison; if conversion fails, it compares characters using the character set’s collation rules.
• () can change precedence, but must be escaped with backslashes

Example:

a=3
b=4

echo `expr $a \> $b`  # Outputs 0; > must be escaped
echo `expr $a '<' $b`  # Outputs 1; special characters can also be quoted
echo `expr $a '>=' $b`  # Outputs 0
echo `expr $a \<\= $b`  # Outputs 1

c=0
d=5

echo `expr $c \& $d`  # Outputs 0
echo `expr $a \& $b`  # Outputs 3
echo `expr $c \| $d`  # Outputs 5
echo `expr $a \| $b`  # Outputs 3

read command #

The read command reads one line of data from standard input. Its exit code is 1 on end-of-file and 0 otherwise.

Parameter descriptions

• -p: Can be followed by a prompt
• -t: Followed by a number of seconds specifying how long to wait for input; the command is automatically ignored after that time The order of -t and -p does not affect the result. Example:

read name  # Read the value of name
timerring  # Standard input
echo $name  # Output the value of name
timerring  # Standard output

read -p "Please input your name: " -t 30 name  # Read name's value; wait for 30 seconds
Please input your name: timerring  # Standard input
echo $name  # Output the value of name
timerring  # Standard output

# If the message after -p needs escaping, write it this way
read -p $'Please input your name: \n' name

Summary of meanings of $ #

echo command #

Enter help echo to view the relevant echo documentation. echo outputs strings. Command format:

echo STRING

Display a plain string #

echo "Hello AC Terminal"
echo Hello AC Terminal  # Quotes can be omitted

Display escaped characters #

echo "\"Hello AC Terminal\""  # Note: only double quotes can be used; with single quotes, escaping does not occur
echo \"Hello AC Terminal\"  # Double quotes can also be omitted

Display a variable #

name=timerring
echo "My name is $name"  # Outputs My name is timerring

Display a newline #

echo -e "Hi\n"  # -e enables escaping
echo "timerring"

# output：
# Hi
# 
# timerring

A newline can also be produced as follows:

This is a bash C-style string; $'...' represents escape characters in strings. Bash’s C-style strings are a special string notation that allows escape characters inside strings, much like string notation in the C programming language.

echo $'good\nnight'

# good
# night

Display without a newline #

echo -e "Hi \c" # -e enables escaping; \c suppresses the newline
echo "timerring"

# Hi timerring

Direct output to a file #

echo "Hello World" > output.txt  # Overwrite output.txt with the content

Output a string as-is, without escaping or expanding variables (use single quotes) #

name=timerring
echo '$name\"'

# $name\"

Display the result of a command #

echo `date` # In fact, this displays the result via ``; similar to ` expr 2 \* 3 ` in the earlier example
# Equivalent to the following:
echo $(date)

# Thu Aug 3 17:35:49 CST 2023

printf command #

The printf command formats output, much like the printf function in C/C++. By default, it does not add a newline at the end of a string.

Command format:

printf format-string [arguments...]

Usage examples #

printf "%10d.\n" 123  # Use 10 positions, right-aligned
printf "%-10.2f.\n" 123.123321  # Use 10 positions, 2 decimal places, left-aligned
printf "My name is %s\n" "timerring"  # Format a string for output
printf "%d * %d = %d\n"  2 3 `expr 2 \* 3` # Pass the expression's value as an argument

# output
#        123.
# 123.12    .
# My name is timerring
# 2 * 3 = 6

Test command and conditional notation [] #

Logical operators && and || #

&& means AND; || means OR Both use short-circuit evaluation:

• expr1 && expr2: When expr 1 is false, expr2 is skipped
• expr1 || expr2: When expr1 is true, expr2 is skipped An expression’s exit code of 0 means true; nonzero means false. Note that this is the exit code!

Test command #

Enter help test on the command line to see how to use the test command. The test command checks file types and compares variables. The test command returns its result via its exit code, not stdout. 0 means true; nonzero means false.

For example:

test 2 -lt 3  # True, return value 0
echo $?  # Output the previous command's return value, 0

ls  # List all files in the current directory
# homework  output.txt  test.sh  tmp
test -e test.sh && echo "exist" || echo "Not exist"
exist  # test.sh exists
test -e test2.sh && echo "exist" || echo "Not exist"
Not exist  # testh2.sh does not exist

Checking file types #

test -e filename  # Check whether the file exists
echo $?  # Output the previous command's return value

Checking file permissions #

test -r filename  # Check whether the file is readable

Comparing integers #

test $a -eq $b  # Whether a equals b

Comparing strings #

Checking multiple conditions #

test -r filename -a -x filename

Conditional notation [] #

[] is used almost exactly like test and is more common in if statements. Also, [[]] is an enhanced version of [] with more features.

[ 2 -lt 3 ]  # True, return value 0
echo $?  # Output the previous command's return value, 0

ls  # List all files in the current directory
# homework  output.txt  test.sh  tmp
[ -e test.sh ] && echo "exist" || echo "Not exist"
exist  # test.sh exists
[ -e test2.sh ] && echo "exist" || echo "Not exist"
Not exist  # testh2.sh does not exist

Note:

• Separate every item inside [] with spaces
• Variables inside the brackets are best enclosed in double quotes
• Constants inside the brackets are best enclosed in single or double quotes

name="timer ring"
[ $name == "timer ring" ]  # Wrong; equivalent to [ timer ring == "timer ring" ], with too many arguments
[ "$name" == "timer ring" ]  # Correct

Conditional statements #

if…then form #

Similar to the if-else statement in C/C++. if condition checks the exit code. An exit code of 0 is true. Therefore, an expression must follow if. Add spaces wherever possible between parts of a conditional statement.

Single-level if #

if condition
then
    statement1
    statement2
    ...
fi

Example:

a=3
b=4

if [ "$a" -lt "$b" ] && [ "$a" -gt 2 ]
then
    echo ${a}is in range
fi

Result: 3 is in range

Single-level if-else #

if condition
then
    statement1
    statement2
    ...
else
    statement1
    statement2
    ...
fi

Example:

a=3
b=4

if ! [ "$a" -lt "$b" ]
then
    echo ${a}is not less than ${b}
else
    echo ${a}is less than ${b}
fi

Output result: 3 is less than 4

Multilevel if-elif-elif-else #

if condition
then
    statement1
    statement2
    ...
elif condition
then
    statement1
    statement2
    ...
elif condition
then
    statement1
    statement2
else
    statement1
    statement2
    ...
fi

Example:

a=4

if [ $a -eq 1 ]
then
    echo ${a}equals 1
elif [ $a -eq 2 ]
then
    echo ${a}equals 2
elif [ $a -eq 3 ]
then
    echo ${a}equals 3
else
    echo other
fi

Output: other

case…esac form #

Similar to the switch statement in C/C++.

case $variable_name in
    value1)
        statement1
        statement2
        ...
        ;;  # Similar to break in C/C++
    value2)
        statement1
        statement2
        ...
        ;;
    *)  # Similar to default in C/C++
        statement1
        statement2
        ...
        ;;
esac

Example:

a=4

case $a in
    1)
        echo ${a}equals 1
        ;;  
    2)
        echo ${a}equals 2
        ;;  
    3)                                                
        echo ${a}equals 3
        ;;  
    *)
        echo other
        ;;  
esac

Output result: other

Loop statements #

for…in…do…done #

for var in val1 val2 val3
do
    statement1
    statement2
    ...
done

Example 1: output a 2 cc, one element per line:

for i in a 2 cc
do
    echo $i
done

Example 2: output all filenames in the current path, one filename per line:

for file in `ls`
do
    echo $file
done

Example 3: output 1-10

for i in $(seq 1 10)
do
    echo $i
done

Example 4: use {1..10} or {a..z} in either forward or reverse order; they cannot be used directly in the terminal, but can be used in a loop.

for i in {a..z}
do
    echo $i
done

for ((…;…;…)) do…done #

for ((expression; condition; expression))
do
    statement1
    statement2
done

Example: output 1-10, one number per line:

for ((i=1; i<=10; i++))
do
    echo $i
done

while…do…done loop #

while condition
do
    statement1
    statement2
    ...
done

Example: the end-of-file character is Ctrl+d; after entering it, read returns false.

while read name
do
    echo $name
done

until…do…done loop #

Ends when the condition is true.

until condition
do
    statement1
    statement2
    ...
done

Example: ends when the user enters yes or YES; otherwise keeps waiting for input.

until [ "${word}" == "yes" ] || [ "${word}" == "YES" ]
do
    read -p "Please input yes/YES to stop this program: " word
done

break command #

It differs slightly from break in CPP: It exits the current loop level. Unlike C/C++, break cannot exit a case statement (which is exited with ;;); therefore, the break below exits the nearest for loop directly.

while read name
do
    for ((i=1;i<=10;i++))
    do
        case $i in
            8)
                break
                ;;
            *)
                echo $i
                ;;
        esac
    done
done

Each time this example reads a non-EOF string, it outputs 1-7.
You can end the program with the Ctrl+d end-of-file character or terminate the process directly with Ctrl+c.

continue command #

Skip the current loop iteration.

For example: this program outputs all odd numbers from 1-10.

for ((i=1;i<=10;i++))
do
    if [ `expr $i % 2` -eq 0 ]
    then
        continue
    fi
    echo $i
done

Handling an infinite loop #

If AC Terminal can open the program, simply enter Ctrl+c.

Otherwise, terminate the process directly:

1. Use top or ps aux to find the process PID
2. Enter kill -9 PID to terminate the process

Functions #

Functions in bash resemble functions in C/C++, but the value returned by return is different: it is an exit code from 0-255, where 0 means a normal exit.

• To obtain a function’s output, use echo to print it to stdout, then retrieve the stdout result using $(function_name).
• A function’s return value can be retrieved using $?.

[function] func_name() {  # The function keyword can be omitted
    statement1
    statement2
    ...
}

Without capturing the return or stdout value #

func() {
    name=timerring
    echo "Hello $name"
}

# Calling a function in shell does not require parentheses
func
# Hello timerring

Capturing the return and stdout values #

When return is omitted, it defaults to return 0.

func() {
    name=timerring
    echo "Hello $name"

    return 123
}

output=$(func)
ret=$?

echo "output = $output"
echo "return = $ret"

# output = Hello timerring
# return = 123

Function input parameters #

Inside a function, $1 is the first input parameter, $2 the second, and so on.

Note: inside a function, $0 is still the filename, not the function name. $() reads a function’s stdout without printing it directly

You can trace the whole process: eventually, at sum=$(func $(expr $1 - 1)), $1 is 1. It enters func (0), which outputs echo 0. Because $() reads stdout, sum becomes 0. Then echo $(expr $sum + $1) echoes 0 + 1 at this level. Returning to the previous level, its sum=$(func $(expr $1 - 1)) captures that result, so $sum is 1. That level then echoes 0 + 1 + 2 because its $1 is 2. The result is captured at the next level, and so on, returning level by level until the final echo prints the result.

func() {  # Recursively calculate $1 + ($1 - 1) + ($1 - 2) + ... + 0
    word=""
    while [ "${word}" != 'y' ] && [ "${word}" != 'n' ]
    do
        read -p "Enter func($1)? Please enter y/n:" word
    done

    if [ "$word" == 'n' ]
    then
        echo 0
        return 0
    fi  

    if [ $1 -le 0 ] 
    then
        echo 0
        return 0
    fi  

    sum=$(func $(expr $1 - 1))
    echo $(expr $sum + $1)
}

echo $(func 10)

# 55, computed recursively

Local variables within functions #

You can define a local variable inside a function; its scope is limited to that function. You can define local variables in recursive functions.

local variable_name=variable_value

#! /bin/bash

func() {
    local name=timerring
    echo $name
}
func
# timerring

echo $name
# -bash: name: command not found

The first line is the name variable inside the function; the second refers to name outside the function, where the variable does not exist.

exit command #

• The exit command exits the current shell process and returns an exit status; $? can retrieve that status.
• The exit command accepts an integer argument representing the exit status. If omitted, the default status is 0.
• The exit status can only be an integer from 0~255; only 0 indicates success, while all other values indicate failure.

Create a script test.sh with the following contents:

#! /bin/bash

if [ $# -ne 1 ]  # If exactly one argument is passed, exit normally; otherwise exit abnormally.
then
    echo "arguments not valid"
    exit 1
else
    echo "arguments valid"
    exit 0
fi

Run the script:

chmod +x test.sh 
./test.sh acwing
# arguments valid
echo $?  # One parameter was passed, so exit normally; exit code is 0
# 0
./test.sh 
# arguments not valid
echo $?  # Number of parameters passed is not 1, so exit abnormally; exit code is 1
# 1

Both return and exit return an exit code; the difference is that return ends the current function, whereas exit ends the entire shell script.

File redirection #

Every process opens three file descriptors by default:

• stdin standard input, reads data from the command line, file descriptor 0
• stdout standard output, writes data to the command line, file descriptor 1
• stderr standard error output, writes data to the command line, file descriptor 2 File redirection can redirect these three streams to other files.

List of redirection commands #

Input and output redirection #

echo -e "Hello \c" > output.txt  # Redirect stdout to output.txt
echo "World" >> output.txt  # Append the string to output.txt

read str < output.txt  # Read the string from output.txt

echo $str  # Output result: Hello World

Redirect stdin and stdout simultaneously #

Create a bash script:

#! /bin/bash

read a
read b

echo $(expr "$a" + "$b")

Create input.txt with the following contents:

3
4

Run the command:

chmod +x test.sh  # Add executable permission
./test.sh < input.txt > output.txt  # Read from input.txt and write output to output.txt
# Order does not affect the final result; file descriptors can also be included
./test.sh 1> output.txt 0<input.txt
cat output.txt  # View the contents of output.txt
# 7

Including external scripts #

Like the include operation in C/C++, bash can include code from other files.

. filename  # Note the space between the dot and the filename
or
source filename

Example Create test1.sh with these contents:

#! /bin/bash

name=timerring  # Define the variable name

Then create test2.sh with these contents:

#! /bin/bash
# The included file may have a path, such as an absolute path
# source /home/test1.sh

source test1.sh # or . test1.sh
echo My name is: $name  # You can use variables from test1.sh

Run the command:

chmod +x test2.sh 
./test2.sh 
# My name is: timerring

The terminal is also one big bash script, so all contents of .bashrc are executed by default when you log in. After modifying .bashrc, you often run source .bashrc to execute the file again.

Note: a similar file, .vimrc, is not a bash script; it uses vim’s special configuration syntax, so running source .vimrc will certainly produce an error.

Related readings

• SSH Remote Access and File Transfer