Linux - Pipes, Redirection, and Advanced Command Chaining in Linux

Linux provides a powerful mechanism for connecting commands together and controlling where their input and output go. Pipes, redirection, and command chaining allow users to combine simple commands into more useful operations without writing a complete program. These features are especially important for system administration, log analysis, automation, and troubleshooting.

1. Standard Input, Output, and Error

Before understanding pipes and redirection, it is important to understand the three standard streams used by Linux programs.

Standard input (stdin) is normally connected to the keyboard. It provides data that a command reads.

Standard output (stdout) is normally displayed on the terminal. It contains the normal results produced by a command.

Standard error (stderr) is also normally displayed on the terminal, but it is used for error and diagnostic messages.

Linux represents these streams using file descriptors:

0 = Standard input
1 = Standard output
2 = Standard error

For example:

ls /home

The directory listing is sent to standard output.

If the specified directory does not exist:

ls /unknown-directory

the error message is sent to standard error.

The distinction between stdout and stderr becomes particularly useful when redirecting command output.

2. Output Redirection Using >

The > operator redirects standard output from the terminal into a file.

For example:

ls > files.txt

Instead of displaying the directory contents on the screen, Linux writes them to files.txt.

If files.txt already exists, its previous contents are normally overwritten.

For example:

echo "Linux Administration" > notes.txt

creates notes.txt and stores the specified text in it.

The important point is that > does not append to an existing file. It replaces its contents.

3. Appending Output Using >>

The >> operator is used when you want to add output to the end of an existing file.

echo "First entry" > log.txt
echo "Second entry" >> log.txt

The first command creates or overwrites the file, while the second command adds another line to the end.

This is particularly useful for maintaining log files.

For example:

date >> activity.log

Each execution adds the current date and time to the existing log.

4. Redirecting Standard Input Using <

The < operator allows a command to receive input from a file rather than directly from the keyboard.

For example:

sort < names.txt

Here, sort receives its input from names.txt.

Conceptually, the operation is:

names.txt
    |
    v
standard input
    |
    v
sort

This is useful when a command expects input but the required information is already stored in a file.

5. Redirecting Errors Using 2>

Because standard error has file descriptor 2, errors can be redirected separately from normal output.

For example:

ls /valid /invalid 2> errors.txt

Normal output is displayed on the terminal, while the error generated for /invalid is stored in errors.txt.

This separation is useful when running commands against many files or directories and you want to keep a record of failures.

6. Redirecting Both Output and Errors

Sometimes both standard output and standard error need to be stored in the same file.

A commonly used syntax is:

command > output.txt 2>&1

For example:

ls /home /invalid > result.txt 2>&1

Here:

> result.txt

redirects standard output to result.txt.

2>&1

redirects standard error to the same destination as standard output.

Modern Bash also supports:

command &> result.txt

This provides a shorter way to redirect both output streams.

7. The Pipe Operator |

A pipe is one of the most powerful features of the Linux command line.

The | operator takes the standard output of one command and sends it directly as the standard input of another command.

For example:

ls | sort

The first command produces a directory listing. Instead of displaying that output directly, the pipe sends it to sort.

The process can be visualized as:

ls
 |
 v
sort
 |
 v
Terminal

This allows multiple simple commands to work together.

8. Using Multiple Pipes

More than two commands can be connected using multiple pipes.

For example:

cat access.log | grep "404" | sort | uniq

The processing takes place from left to right.

cat access.log
       |
       v
grep "404"
       |
       v
sort
       |
       v
uniq

Each command receives the output of the previous command as its input.

This approach is commonly used when processing large amounts of text, particularly server logs and system information.

However, cat is not always necessary. The same operation can often be written more efficiently as:

grep "404" access.log | sort | uniq

9. The tee Command

The tee command is useful when you want to see command output on the terminal while simultaneously saving it to a file.

For example:

ls | tee directory.txt

The output is displayed on the terminal and also written to directory.txt.

It can be especially useful when troubleshooting or recording the output of an operation.

You can also append rather than overwrite by using:

ls | tee -a directory.txt

The -a option tells tee to append the output.

10. Command Substitution

Command substitution allows the output of one command to be used as part of another command.

The modern syntax is:

$(command)

For example:

echo "Today is $(date)"

The shell first executes:

date

and then places its output inside the echo command.

Another example is:

mkdir "backup-$(date +%Y%m%d)"

If the date is September 9, 2026, this could create:

backup-20260909

Command substitution is particularly useful in shell scripts and automation because it allows dynamically generated information to become part of another command.

11. Command Chaining with ;

The semicolon allows multiple commands to be placed on the same command line.

For example:

pwd; ls; date

The shell executes each command sequentially.

Importantly, the next command is executed regardless of whether the previous command succeeds or fails.

For example:

mkdir test; cd test

The shell attempts to execute cd test even if creating the directory failed.

Therefore, ; is appropriate when the commands are independent of one another.

12. Conditional Execution Using &&

The && operator executes the second command only if the first command succeeds.

For example:

mkdir project && cd project

If mkdir project succeeds, the shell executes cd project.

If directory creation fails, the second command is not executed.

This is particularly useful when commands depend on the successful completion of earlier operations.

For example:

mkdir backup && cp important.txt backup/

The file is copied only if the backup directory was successfully created.

13. Conditional Execution Using ||

The || operator executes the second command only if the first command fails.

For example:

cd project || echo "Directory not found"

If cd project succeeds, the echo command is skipped.

If the directory does not exist or the directory cannot be accessed, the error message is displayed.

This makes || useful for basic failure handling.

14. Combining && and ||

These operators can be combined to create simple success/failure logic.

For example:

mkdir project && echo "Directory created" || echo "Creation failed"

The intention is:

Try to create directory
        |
        +-- Success --> display "Directory created"
        |
        +-- Failure --> display "Creation failed"

However, this pattern should be used carefully because the overall logic depends on the exit status of each command.

For more complicated operations, explicit shell constructs such as if statements are generally clearer.

15. Grouping Commands

Linux shells allow commands to be grouped together.

Parentheses execute commands in a subshell:

(cd /tmp && ls)

The commands inside the parentheses execute in a separate shell environment.

Curly braces can also group commands:

{ pwd; date; }

Grouping becomes particularly useful when applying redirection to several commands.

For example:

{ echo "System information"; date; uname -a; } > system-info.txt

The output from all three commands is redirected into the same file.

16. Here Documents

A here document allows multiple lines of input to be supplied directly to a command.

The general structure is:

command <<EOF
line 1
line 2
line 3
EOF

For example:

cat <<EOF
Linux
Ubuntu
Fedora
Debian
EOF

The shell supplies the text between the two EOF markers as standard input to cat.

Here documents are especially useful in shell scripts when a command needs a block of multiline input.

The terminating word does not have to be EOF. It can be another identifier, provided the opening and closing markers match.

17. Here Strings

A here string provides a single string as standard input.

The syntax is:

command <<< "text"

For example:

wc -w <<< "Linux command line"

The supplied text becomes the command's standard input.

Here strings are useful when you have a small amount of data and do not want to create a temporary file or use an ordinary pipe.

18. Practical Example: Combining Several Techniques

Suppose you want to find HTTP 404 errors in a web server log, sort them, count repeated entries, and save the result.

You could use:

grep "404" access.log | sort | uniq -c > 404-summary.txt

The process works as follows:

access.log
    |
    v
grep "404"
    |
    v
sort
    |
    v
uniq -c
    |
    v
404-summary.txt

Here, grep selects matching lines, sort organizes them, uniq -c counts identical consecutive lines, and > saves the final output to a file.

This demonstrates the central philosophy of the Linux command line: instead of requiring one large program to perform every operation, several small utilities can be connected together.

19. Why Pipes and Redirection Are Important

Pipes and redirection are fundamental to Linux administration because they allow commands to be combined efficiently.

They are commonly used for:

  • Processing system and application logs

  • Searching configuration files

  • Filtering command output

  • Generating reports

  • Automating administrative tasks

  • Saving command results

  • Separating normal output from errors

  • Passing data between programs

  • Building shell scripts

  • Troubleshooting systems

A good understanding of these mechanisms makes the Linux command line significantly more powerful.

Summary

Linux treats many input and output operations as streams. Redirection changes where those streams come from or where they go, while a pipe connects the output of one command directly to the input of another. Operators such as ;, &&, and || control how commands are executed in sequence, while command substitution, grouping, here documents, and here strings provide more advanced ways to construct command-line operations.

The most important concepts to remember are:

>       Redirect output and overwrite a file
>>      Redirect output and append to a file
<       Read input from a file
2>      Redirect standard error
2>&1    Send standard error to standard output's destination
|       Pass output to another command
tee     Display output and save it simultaneously
$(...)  Use command output inside another command
;       Execute commands sequentially
&&      Execute next command after success
||      Execute next command after failure
<<      Provide multiline input
<<<     Provide a string as input

Together, these features form the foundation for constructing powerful Linux command-line workflows and shell scripts.