The pipe operator (|) is one of the most powerful features in Linux, enabling you to chain multiple commands together. This article explores how pipes work, how they differ from redirection, and demonstrates practical filter command usage.

How Pipe Operators Work

A pipe connects the standard output of one command to the standard input of another. This allows you to build complex data processing pipelines from simple, single-purpose commands.

Basic syntax:

command1 | command2
command1 | command2 | command3 | commandN

When chaining commands, the output from the left side becomes input for the right side. The key requirement is that each command must handle the data format produced by the previous command in the chain.

Pipe Commands vs Redirection

Both pipes and redirection alter data flow, but they serve different purposes:

Mechanism	Function	Example
Redirection >	Connects command output to a file	ls > file.txt
Pipe `	`	Connects output of one command to input of another

Redirection uses files as intermediaries, while pipes create direct command-to-command data streams.

Practical Pipe Examples

Filtering ls output:

ls | grep documents

Sorting logged-in users:

who | sort

Redirecting pipe results to a file:

ls -la | grep backup > output.txt

Combining Pipes with Input/Output Redirection

Input redirection with pipes:

tr a-z A-Z < input.txt | sort

Full pipeline with both redirections:

tr a-z A-Z < input.txt | sort | uniq > output.txt

Understanding Filter Commands

A filter is any command that reads from standard input and writes to standard output. Filters are designed to work seamlesssly with pipes.

Common Filter Commands

Command	Purpose
awk	Pattern scanning and text processing language
cut	Extract specific columns from each line
grep	Search for patterns in text
head	Display first lines of a file
tail	Display last lines of a file
sort	Sort lines of text
uniq	Remove duplicate lines
wc	Count lines, words, and characters
sed	Stream editor for text transformation
tr	Character translation and deletion
tee	Read from stdin and write to stdout/files
tar	Archive file menipulation

Real-World Filter Examples

Extract usernames from system accounts:

awk -F: '{print $1}' /etc/passwd | sort | head -5

Output:

_apt
backup
bin
daemon

The -F: option sets the field separator to colon.

Find most frequently used commands:

history | awk '{print $2}' | sort | uniq -c | sort -rn | head -5

Output:

17 cat
14 vim
11 sudo
7 ll
7 cd

Check system memory:

free | grep Mem | awk '{print $2}'

Extract specific fields from passwd file:

grep "bin/bash" /etc/passwd | cut -d: -f1,6

Count logged-in users:

who | wc -l

Search for processes:

ps aux | grep nginx

Find files containing IP addresses:

grep -r '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' /etc/

Archive and restore directories:

# Copy directory structure
tar cf - /home/userdata | ( cd /backup/; tar xf - )

# Remote copy with compression
tar czf - /home/userdata | ssh server ">( cd /destination; tar xzf - )"

Compare archive sizes:

# Uncompressed size
tar cf - /etc | wc -c

# Gzip compressed size
tar czf - etc.tar | wc -c

# Bzip2 compressed size
tar cjf - etc.tar | wc -c

Database Backup with Pipes

Traditional multi-step approach:

mysqldump -u admin -p'password' mydb > /tmp/mydb.backup
gzip /tmp/mydb.backup
scp /tmp/mydb.backup.gz admin@backup-server:/mysql/

Single pipeline approach:

mysqldump -u admin -p'password' mydb | gzip -9 | ssh admin@backup-server "cat > /mysql/mydb.gz"

This compresses and transfers the database in one streamlined command.

Key Takeaways

• Pipes connect commands directly without intermediate files
• Filters read from stdin and write to stdout
• Combining simple filters creates powerful data processing pipelines
• Always consider data format compatibility between chained commands