Unix - UNIX Pipes, Filters, and Command Chaining

UNIX is built around a powerful philosophy: small programs should perform one task well and be combined to accomplish more complex tasks. Pipes, filters, and command chaining are essential features that allow users to connect commands and process data efficiently without creating temporary files.

Introduction to Pipes

A pipe is a mechanism that sends the output of one command directly as the input to another command. Instead of displaying the output on the screen, the output is passed to the next command for further processing.

The pipe operator is represented by the vertical bar symbol (|).

Basic Syntax

command1 | command2

In this syntax:

  • command1 generates output.

  • The pipe transfers the output.

  • command2 receives the output as input.

Example

ls | more

Here:

  • ls lists files and directories.

  • more displays the output one screen at a time.

The output of ls becomes the input of more.

Understanding Filters

Filters are commands that read input from standard input, process it, and write the result to standard output. They are often used with pipes.

Characteristics of Filters

  • Accept input from another command.

  • Perform processing on the data.

  • Produce modified output.

  • Can be combined with other filters.

Common UNIX Filters

grep

Searches for specific patterns in text.

Example:

cat employees.txt | grep "Manager"

This displays only lines containing the word "Manager".

sort

Sorts lines alphabetically or numerically.

Example:

cat names.txt | sort

Output appears in sorted order.

uniq

Removes duplicate adjacent lines.

Example:

sort names.txt | uniq

This first sorts the file and then removes duplicate entries.

wc

Counts lines, words, and characters.

Example:

cat report.txt | wc

Output may display:

50 300 2000

Meaning:

  • 50 lines

  • 300 words

  • 2000 characters

head

Displays the beginning of a file.

Example:

cat logfile.txt | head

Shows the first 10 lines.

tail

Displays the end of a file.

Example:

cat logfile.txt | tail

Shows the last 10 lines.

Multiple Pipes

Several commands can be connected using multiple pipes.

Example

cat employees.txt | grep "Sales" | sort

Process flow:

  1. cat reads the file.

  2. grep selects records containing "Sales".

  3. sort arranges the results alphabetically.

Each command processes the output of the previous command.

Practical Examples of Pipes

Counting Files

ls | wc -l

Explanation:

  • ls lists files.

  • wc -l counts lines.

The result indicates the number of files and directories.

Finding Specific Processes

ps -ef | grep apache

Explanation:

  • ps -ef displays running processes.

  • grep apache finds Apache-related processes.

Displaying Largest Files

ls -l | sort

This combines file listing and sorting.

Counting Matching Entries

grep "error" logfile.txt | wc -l

This counts the number of lines containing the word "error".

Command Chaining

Command chaining allows multiple commands to execute in sequence.

UNIX provides several operators for command chaining.

Semicolon (;)

Executes commands one after another regardless of success or failure.

Syntax:

command1 ; command2

Example:

date ; who

The second command runs even if the first command encounters an issue.

Logical AND (&&)

Runs the second command only if the first command succeeds.

Syntax:

command1 && command2

Example:

mkdir project && cd project

Explanation:

  • Directory is created.

  • If creation succeeds, the system enters the directory.

If the first command fails, the second command is skipped.

Logical OR (||)

Runs the second command only if the first command fails.

Syntax:

command1 || command2

Example:

cd backup || echo "Directory not found"

If the directory does not exist, the message is displayed.

Combining AND and OR

Example:

mkdir test && cd test || echo "Operation failed"

Process:

  • Create directory.

  • Enter directory if creation succeeds.

  • Display failure message if any step fails.

Redirecting Input and Output

Pipes often work together with redirection.

Output Redirection (>)

Saves output to a file.

ls > files.txt

Append Output (>>)

Adds output to an existing file.

date >> logfile.txt

Input Redirection (<)

Uses a file as input.

sort < names.txt

Combining Redirection and Pipes

cat data.txt | grep "Linux" > result.txt

Explanation:

  • Read data from file.

  • Search for "Linux".

  • Save matching lines into another file.

Standard Streams in UNIX

Pipes and filters work with three standard streams.

Standard Input (stdin)

Default input source, usually the keyboard.

File descriptor:

0

Standard Output (stdout)

Default output destination, usually the terminal.

File descriptor:

1

Standard Error (stderr)

Used for error messages.

File descriptor:

2

Example:

ls nonexistentfile 2> errors.txt

The error message is stored in errors.txt.

Advanced Pipeline Example

Consider a file containing employee records.

cat employees.txt | grep "IT" | sort | uniq | wc -l

Execution sequence:

  1. Read employee data.

  2. Select IT department records.

  3. Sort records.

  4. Remove duplicates.

  5. Count remaining entries.

This demonstrates how multiple filters can work together to perform complex processing with a single command.

Advantages of Pipes and Filters

Efficient Resource Usage

Data is transferred directly between commands without creating temporary files.

Modular Design

Each command performs a specific task, making solutions easier to understand and maintain.

Faster Processing

Large datasets can be processed continuously as they flow through the pipeline.

Reusability

The same commands can be combined in different ways to solve various problems.

Automation Support

Pipes and filters are extensively used in shell scripts for automated data processing.

Best Practices

  • Use pipes instead of temporary files whenever possible.

  • Combine simple commands to build powerful workflows.

  • Use filters such as grep, sort, awk, sed, and uniq effectively.

  • Test individual commands before creating complex pipelines.

  • Redirect errors separately when troubleshooting.

Conclusion

Pipes, filters, and command chaining form the backbone of UNIX command-line efficiency. Pipes enable commands to communicate directly, filters transform and refine data, and command chaining controls execution flow. Together, these features allow users and administrators to perform sophisticated tasks using simple, reusable commands, making UNIX one of the most powerful operating environments for automation, text processing, and system administration.