Understanding the Stream Editor (sed)

The Stream Editor (sed) is a powerful utility in Unix/Linux systems for processing text files. It allows you to perform complex text manipulation operations using simple commands. This guide explores the fundamental operations of sed including deletion, insertion, and modification.

Basic Setup

Before working with sed, let's create a sample file for our demonstrations:

cp /etc/passwd ./
cat -n passwd

Deletion Operations

Sed provides several ways to delete lines from files:

Removing Specific Lines

To delete the first line of a file:

sed -i '1d' passwd

To delete a range of lines (lines 1-3):

sed -i '1,3d' passwd

Pattern-Based Deletion

To remove lines containing specific patterns, such as non-login users:

sed -i '/\/sbin\/nologin/d' passwd

To delete a range of lines matching specific patterns (from lines starting with "mail" to lines starting with "ftp"):

sed -i '/^mail/,/^ftp/d' passwd

Insertion Operations

Sed allows you to insert text before or after specific lines:

Appending Text After Matching Lines

To add text after lines containing "/bin/bash":

sed -i '/\/bin\/bash/a This is user which can login to system' passwd

Inserting Text Before Matching Lines

To insert text before lines between "root" and "nginx":

sed -i '/^root/,/^nginx/i AAAAAAAAAAAAAAAAAAAA' passwd

Inserting File Contents

To insert the contents of another file after lines matching a pattern:

cat list
sed -i '/root/r list' passwd

Writting Matching Lines to a New File

To save lines matching a pattern to a new file:

sed '/\/bin\/bash/w /tmp/user_login.txt' passwd

Modification Operations

Sed's substitution capabilities are among its most powerful features:

Basic Substitution

To replace all occurrences of "/bin/bash" with "/BIN/BASH":

sed -i 's/\/bin\/bash/\/BIN\/BASH/g' passwd

Limited Substitution

To replace only the first occurrence of "root" with "ROOT":

sed -i 's/root/ROOT/' passwd

To replace only the second occurence in each line:

cat str.txt
sed -i 's/HADOOP/hadoop/2' str.txt

Case-Insensitive Substitution

To perform case-insensitive replacements:

sed -i 's/pattern/string/ig' filename

Backreferences in sed

Backreferences allow you to reference matched patterns in your replacement strings:

Ampersand Reference

The ampersand (&) represents the entire matched pattern:

cat file.txt
sed -i 's/Had..p/&s/g' file.txt

Numbered Backreferences

Using parentheses in the pattern and \1, \2, etc. in the replacement:

sed -i 's/\(had..ps\)/\1R/g' str.txt

While & references the entire match, numbered backreferences allow more flexible manipulation of matched groups.

Variable Usage in sed

When using variables in sed commands:

• If your pattern contains variables, use double quotes around the command
• When including custom variables with sed, if the command is in single quotes, the variable should also be in single quotes

sed Command Summary

Query Operations

• Print (p): Display lines matchnig a pattern

Deletion Operations

• Delete (d): Remove lines matching a pattern or line numbers

Insertion Operations

• Append (a): Add text after matching lines
• Insert (i): Add text before matching lines
• Read (r): Insert contents of a file after matching lines
• Write (w): Save matching lines to a new file

Modification Operations

• Substitute (s): Replace patterns with text

• Flags for substitution:

  • g: Replace all occurrences in a line
  • 2g: Replace only the first two occurrences in a line
  • i: Case-insensitive matching

Additional Commands

• Line number (=): Display line numbers of matching lines

Advanced Backreference Examples

To find strings starting with "1", followed by any two characters, and ending with "e", then append "r":

sed "s/1..e/&r/g" file

The same operation using numbered backreferences:

sed "s/\(1..e\)/\1r/g" file