Introduction to SystemTap

SystemTap is a powerful dynamic tracing tool that allows developers and system administrators to monitor and analyze the behavior of both the Linux kernel and user-space applications without modifying the code or restarting the process. By defining custom probes, users can track various aspects such as function calls, CPU usage, disk I/O, and more. When combined with visualization tools like FlameGraphs, SystemTap becomes an invaluable asset for performance analysis and debugging.

How SystemTap Works

At its core, SystemTap operates by translating scripts into C code, wich is then compiled into a kernel module. Once loaded, this module hooks into the kernel and activates the specified probes. When an event occurs, the corresponding handler runs, collects data, and stores or outputs it. After the session ends, the module is removed from the kernel. This process is managed by the stap command-line utility.

Installation

Before installing SystemTap, ensure your kernel supports it by installing the appropriate debug and development packages:

• kernel-debuginfo-common
• kernel-debuginfo
• kernel-devel

These packages must match your current kernel version. Once installed, install SystemTap via:

yum install systemtap

Verify the installation with a simple test:

stap -ve 'probe begin { printf("Hello, World\n"); exit(); }'

If you encounter version mismatch errors, ensure the kernel version in /usr/src/kernels/<version>/include/generated/compile.h matches the output of uname -a. You may also need to clear the SystemTap cache located at /root/.systemtap/cache/.

Basic Probes and Events

SystemTap provides several built-in events that can be used to monitor system and application behavior:

• begin: Fired when the SystemTap session starts.
• end: Fired when the SystemTap session ends.
• kernel.function("sys_xxx"): Triggers when entering a specific system call.
• kernel.function("sys_xxx").return: Triggers when exiting a system call.
• timer.ms(300): Fires every 300 milliseconds.
• timer.profile: Periodically fires on each CPU.
• process("a.out").function("foo*"): Triggers on functions in a.out starting with "foo".
• process("a.out").statement("*@main.c:200"): Triggers on line 200 of main.c in a.out.

Common Data Output

SystemTap scripts can output various system and process metrics:

• tid(): Current thread ID.
• pid(): Curent process ID.
• uid(): Current user ID.
• execname(): Name of the executing process.
• cpu(): Current CPU index.
• gettimeofday_s(): Current timestamp in seconds.
• get_cycles(): Snapshot of the hardware cycle counter.
• pp(): Name of the probe point.
• ppfunc(): Name of the triggered function.
• $$var: Access a local variable named var in the context.
• print_backtrace(): Prints the kernel stack trace.
• print_ubacktrace(): Prints the user-space stack trace.

Writing SystemTap Scripts

SystemTap scripts resemble C in syntax and support basic control structures, string operations, and aggregate statistics.

Control Structures

function if_expr() {
    i = 0
    if (i == 1)
        printf("[if] i = %d\n", i);
    else
        printf("[else] i = %d\n", i);
}

function while_expr() {
    i = 0;
    while (i != 2)
        printf("[while] i = %d\n", i++);
}

function for_expr() {
    for (i = 0; i < 2; i++)
        printf("[for] i = %d\n", i);
}

String Operations

function str() {
    uid = uid();
    s_uid = sprint(uid);
    f_uid = "user" . s_uid
    printf("uid: %d-%s-%s\n", uid, s_uid, f_uid);
}

Global Variables and Aggregates

global t;
global tpl[400];

t["key"]++;
t["key", pid()] <<< 1;

Aggregation Statistics

global stats;

probe kernel.function("sys_write") {
    stats[execname(), pid()] <<< 1;
}

probe end {
    foreach([name, pid] in stats-) {
        printf("Process %s (PID %d) wrote %d bytes\n", name, pid, @count(stats[name, pid]));
    }
}

Common Commands

SystemTap provides several command-line options for script execution and probe discovery:

stap -e 'script here' # Run inline script
stap -l 'probe point' # List matching probes
stap -L 'probe point' # List probes and local variables

Example: Tracing a Program's Functions

probe process("/path/to/myapp").function("*") {
    printf("Function: %s\n", ppfunc());
}

Example: Tracing Functions in a Specific File

probe process("/path/to/myapp").statement("*@source.cpp") {
    printf("Function: %s\n", ppfunc());
}

Example: Printing Context Variables

probe process("/path/to/myapp").function("consume") {
    printf("Function: %s, Args: n=%d, to=%p\n", ppfunc(), $n, $to);
}

Advanced Usage

Function Call Frequency

global freq;

probe process("/path/to/myapp").function("*") {
    freq[ppfunc()]++;
}

probe end {
    foreach(func in freq- limit 10)
        printf("%s: %d\n", func, freq[func]);
}

System Call Frequency

global syscalls;

probe kernel.function("sys_open") {
    syscalls[execname()]++;
}

probe end {
    foreach(name in syscalls- limit 10)
        printf("%s: %d\n", name, syscalls[name]);
}

Call Stack Visualization

global functions;

function init() {
    functions["main"] = 1;
    functions["init_app"] = 1;
    functions["start_server"] = 1;
}

probe begin {
    init();
}

probe process("/path/to/myapp").function("*").call {
    if (functions[ppfunc()] == 1)
        printf("%s -> %s\n", thread_indent(4), ppfunc());
}

probe process("/path/to/myapp").function("*").return {
    if (functions[ppfunc()] == 1)
        printf("%s <- %s\n", thread_indent(-4), ppfunc());
}

Generating FlameGraphs

To visualize CPU usage as a FlameGraph, use the following script:

#!/bin/bash

FLAMEGRAPH="/path/to/FlameGraph"
STAP_TOOL="/path/to/openresty-systemtap-toolkit"

if [ $# -ne 1 ]; then
    echo "Usage: $0 PID"
    exit 1
fi

$STAP_TOOL/sample-bt -p $1 -t 60 -u > /tmp/bt-sample
$FLAMEGRAPH/stackcollapse-stap.pl /tmp/bt-sample > /tmp/bt_stap.out
$FLAMEGRAPH/flamegraph.pl /tmp/bt_stap.out > /tmp/flamegraph.svg

rm /tmp/bt-sample /tmp/bt_stap.out

This script collects stack traces for 60 seconds, processes them, and generates an SVG FlameGraph showing CPU time distribution by function call depth and duration.