AWK Field and Record Manipulation

AWK processes text line by line, treating each line as a record and each whitespace-separated value as a field. Key built-in variables control how data is parsed and displayed.

• $0 — Represents the entire current line. awk '{print $0}' /etc/passwd outputs every line unchanged.
• NR — Total number of input records processed across all files. awk '{print NR, $0}' /etc/passwd prefixes each line with its global line number.
• FNR — Current file’s record number. Useful when processing multiple files: awk '{print FNR, $0}' /etc/passwd /etc/hosts resets the counter per file.
• NF — Number of fields in the current record. To print the last field: awk -F: '{print $NF}' /etc/passwd.
• FS — Input field separator. Default is whitespace. Override with -F: or BEGIN{FS=":"}. Complex delimiters: -F'[ :\t]' splits on space, colon, or tab.
• OFS — Output field separator. Replaces comas in print statements. Example: awk 'BEGIN{FS=":"; OFS="+++"} {print $1,$2}' /etc/passwd joins fields with "+++".
• RS — Record separator. Default is newline. Change to space: awk 'BEGIN{RS=" "} {print $0}' /etc/hosts.
• ORS — Output record separator. Concatenates lines: awk 'BEGIN{ORS=" "} {print $0}' /etc/hosts outputs all content on one line.

Formatted Output with printf

Unlike print, printf gives fine control over formatting:

awk -F: '{printf "%-15s %-8s %6d\n", $1, $2, $3}' /etc/passwd

• %-15s: Left-align string in 15-character field
• %6d: Right-align integer in 6-character field
• \n: Explicit newline required — printf does not auto-terminate lines

Pattern Matching and Conditional Logic

AWK supports multiple matching styles:

• Regex matching: /^root/ matches lines starting with "root". $1 ~ /^root/ matches first field.
• Comparision operators: $3 == 0, $3 > 1000, $7 != "/bin/bash".
• Logical operators: && (AND), || (OR), ! (NOT). Example: awk -F: '$1 ~ /root/ && $3 <= 15' /etc/passwd.
• Conditional blocks: ``` awk -F: '{ if($3 > 1000) print $1 " is a regular user" }' /etc/passwd

  
   Multi-branch: ```
  awk -F: '{
        if($3 == 0) admin++
        else if($3 >= 1000) regular++
        else system++
      } END {
        print "Admin:", admin, "System:", system, "Regular:", regular
      }' /etc/passwd
  

Looping in AWK

AWK supports while and for loops for field iteration:

# Print all fields of each line
awk -F: '{ i=1; while(i <= NF) { print $i; i++ } }' /etc/passwd

# C-style for loop: print each line 5 times
awk -F: '{ for(i=1; i<=5; i++) print $0 }' /etc/passwd

# Print all fields using for loop
awk -F: '{ for(i=1; i<=NF; i++) print $i }' /etc/passwd

SED: Stream Editing Fundamentals

SED operates in a cycle: read line → apply edits → output → repeat.

Core flags:

• -n: Suppress automatic output
• -i: Modify file in-place
• -r: Enable extended regex
• -e: Chain multiple commands

SED Commands and Examples

• d — Delete lines: sed '3d' file removes line 3
• p — Print: Use with -n: sed -n '5,10p' file
• a — Append after line: sed '3a\New line' file
• i — Insert before line: sed '3i\Insert here' file
• c — Change line: sed '3c\Replaced line' file
• s — Susbtitute: sed 's/old/new/g' replaces globally
• n — Read next line: sed -n '/pattern/{n;p}' prints next line after match
• = — Print line number: sed -n '/root/=' file
• q — Quit processing: sed '/^bin/q' stops at first line starting with "bin"

SED Substitution Flags

• g: Global replacement (all occurrences)
• i: Case-insensitive
• w file: Write matching lines to file
• y/abc/xyz/: Character translation (no regex)

Examples:

sed 's/ERROR/CRITICAL/gw errors.log' logfile
sed -r 's/(user)([0-9]+)/\2_\1/g' users.txt
sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' text.txt

SED Hold and Pattern Spaces

SED uses two buffers:

• Pattern space: Current line being processed
• Hold space: Persistent storage (initially empty)

Key commands:

• h: Copy pattern → hold (overwrite)
• H: Append pattern → hold
• g: Copy hold → pattern (overwrite)
• G: Append hold → pattern
• x: Swap hold and pattern

Example: Reverse first and last line of a file:

sed '1h; $G; $x' file

Example: Duplicate every line:

sed 'G' file

Example: Move first line to end:

sed '1h; 1d; $G' file

Advanced SED: Multi-Command and Variable Substitution

Chain commands with semicolons or braces:

sed '1,3d; s/root/admin/g' file
sed '2{s/old/new/g; s/alpha/beta/g}' file

Use shell variables with double quotes:

newname="admin"
sed "s/root/$newname/g" file
sed "\$a$newname" file  # Escape $ to avoid shell expansion

Use ! to invert matching:

sed '3!d' file          # Only keep line 3
sed '/pattern/!d' file  # Delete lines NOT matching pattern