Introduction to AWK

AWK is a powerful text processing language designed for pattern scanning and data extraction. Unlike grep for searching and sed for editing, AWK excels at data analysis and report generation. It processes input line by line, splitting each line into fields using whitespace as the default separator, then performs various operations on these fields.

There are three main AWK implementations: awk, nawk, and gawk. The GNU version (gawk) is most common used. The language was named after its creators: Alfred Aho, Peter Weinberger, and Brian Kernighan.

Basic Syntax and Usage

The fundamental AWK syntax follows this pattern:

awk 'pattern { action }' filename

Patterns specify what to match in the input data, while actions define the operations to perform when matches occur. Curly braces group commands, and patterns are regular expressions enclosed in slashes.

Invocation Methods

AWK can be called in three ways:

# Command line
awk [-F separator] 'commands' input-file

# Shell script
#!/bin/awk -f
commands

# External script file
awk -f script.awk input-file

Practical Examples

Display Recent Login Accounts

last -n 5 | awk '{print $1}'

AWK processes each line, splits it into fields, and $1 represents the first field (username).

Process /etc/passwd File

# Show usernames
cat /etc/passwd | awk -F: '{print $1}'

# Show usernames and shells with tab separator
cat /etc/passwd | awk -F: '{print $1 "\t" $7}'

# Add headers and footers
cat /etc/passwd | awk -F: 'BEGIN {print "name,shell"} {print $1 "," $7} END {print "blue,/bin/nosh"}'

Pattern Matching

# Find lines containing "root"
awk -F: '/root/' /etc/passwd

# Find lines starting with "root"
awk -F: '/^root/' /etc/passwd

# Find root lines and show shells
awk -F: '/root/{print $7}' /etc/passwd

Built-in Variables

AWK proivdes several built-in variables:

• FS - Input field separator
• OFS - Output field separator
• RS - Record separator
• ORS - Output record separator
• NF - Number of fields in current record
• NR - Current record number
• FILENAME - Current filename

Variable Usage Example

awk -F: '{print "File:" FILENAME ", Line:" NR ", Fields:" NF ", Content:" $0}' /etc/passwd

Programming Features

Variables and Counting

# Count users in /etc/passwd
awk 'BEGIN {user_count = 0} {user_count++} END {print "Total users:", user_count}' /etc/passwd

# Calculate file sizes
ls -l | awk 'BEGIN {total_size = 0} {total_size += $5} END {print "Size:", total_size/1024/1024 "MB"}'

Conditional Statements

# Sum file sizes excluding directories
ls -l | awk 'BEGIN {size = 0} 
{if($5 != 4096) {size += $5}} 
END {print "Size:", size/1024/1024 "MB"}'

Arrays

# Store and display usernames
awk -F: '{users[NR] = $1} END {for(i=1; i<=NR; i++) print i, users[i]}' /etc/passwd

Data Analysis Examples

Sum and Average Calculation

# Sample data file
cat data.log | awk '{total += $NF} END {print "Sum:", total; print "Average:", total/NR}'

Row to Column Transformation

# Convert every 3 rows to 1 column
awk '{if(NR%3 != 0) ORS=" "; else ORS="\n"} 1' input.txt

Memory Usage Analysis

# Calculate memory usage percentage
cat memory.log | grep -A3 "^zzz" | grep -v "^-|^z" | 
awk '{if(NR%3 != 0) ORS=" "; else ORS="\n"} 1' | 
awk -F'[: ]+' '{print (1 - $8/$2) * 100}' | 
awk '{sum += $1} END {print "Avg usage:", sum/NR "%"}'