When debugging kernel-level issues, traditional approaches like eBPF might not be available on older kernel versions. In such cases, perf probe provides a practical alternative for tracing kernel functions without writing custom kernel modules. Before adding probes, identify available symbols through /proc/kallsyms. For instance, to trace splice_to_pipe: ```

Add basic function probe

perf probe --add 'splice_to_pipe'

With kernel-debuginfo, capture variable values directly

perf probe --add 'splice_to_pipe pipe->nrbufs pipe->buffers spd->nr_pages'

After setting probes, record events to a specific process: ```
# Record single event for 30 seconds
perf record -e 'probe:splice_to_pipe' -p $(pidof myapp) -gR sleep 30

# Record multiple events simultaneously
perf record -e 'probe:tcp_splice_data_recv,probe:kill_fasync,probe:pipe_wait' -p $(pidof myapp) -gR sleep 30

View results with: ``` perf report --stdio

Useful probe management commands: ```
# List existing probes
perf probe --list

# Remove a specific probe
perf probe --del probe:splice_to_pipe

# View probe points with syntax highlighting
perf probe -L splice_to_pipe

Note: Capturing variable values requires kernel-debuginfo to be installed.Problem Context

When transferring approximately 24KB of data, the splice() system call blocks on kernel 3.10.0 but returns immediately on kernel 6.5.6. This behavior occurs specifically when performing fd -> pipe transfers without consuming from the pipe. The blocking threshold differs significantly: - Kernel 3.10.0: blocks around 24KB - Kernel 6.5.6: blocks around 200KB (far exceeding the 65536 byte pipe capacity) This investigation focuses on: - Verifying whether the number of pages in skbuff exceeds pipe capacity on 3.10.0 - Checking wait_for_space conditions and page counts on 6.5.6 Analysis on Kernel 3.10.0

The test environment consists of a CentOS 7.9 virtual machine (kernel 3.10.0-1160.62.1.el7.x86_64) running on a Fedora 39 host with QEMU 7.2.4. ### Understanding splice_to_pipe Logic

The core loop in splice_to_pipe (fs/splice.c) reveals the blocking mechanism: ``` ssize_t splice_to_pipe(struct pipe_inode_info *pipe, struct splice_pipe_desc *spd) { for (;;) { if (pipe->nrbufs < pipe->buffers) { // Write data to pipe pipe->nrbufs++; page_idx++; transferred += buf->len;

        if (!--spd->nr_pages)
            break;
        if (pipe->nrbufs < pipe->buffers)
            continue;
        break;  // Line 230: exit when pipe full
    }

    if (spd->flags & SPLICE_F_NONBLOCK) {
        if (!transferred)
            transferred = -EAGAIN;
        break;
    }

    // Wake readers and wait
    if (should_wakeup) {
        wake_up_interruptible_sync(&pipe->wait);
        kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
    }

    pipe->waiting_writers++;
    pipe_wait(pipe);  // Blocking point
    pipe->waiting_writers--;
}

// Release remaining pages
while (page_idx < total_pages)
    spd->spd_release(spd, page_idx++);

return transferred;

}

The suspected blocking condition occurs when: - `pipe->nrbufs >= pipe->buffers` (pipe is full) - `SPLICE_F_NONBLOCK` flag is not set - The process enters `pipe_wait()`### Tracing Single Splice Call with 24KB Data

Generate test data and send: ```
$ dd if=/dev/zero of=/tmp/testdata bs=1k count=24
$ ncat -nv 192.168.32.245 10022 < /tmp/testdata

Set up comprehensive probes: ``` perf probe --add 'splice_to_pipe pipe->nrbufs pipe->buffers spd->nr_pages' perf record -e 'probe:splice_to_pipe' -p $(pidof myapp) -gR sleep 30

Results show two invocations: ```
# Sample 1
nrbufs=0x0 buffers=0x10 nr_pages=17

# Sample 2
nrbufs=0x10 buffers=0x10 nr_pages=2

The first call fills the pipe completely (nrbufs reaches 0x10 = 16 buffers). The second call attempts to write remaining data but finds the pipe full, triggering the blocking path. ### Extended Tracing

Add more probe points to trace the complete call chain: ``` perf probe --add 'tcp_read_sock desc->count' perf probe --add 'tcp_splice_data_recv rd_desc->count len offset' perf probe --add 'pipe_wait pipe->nrbufs pipe->buffers' perf probe --add 'sock_spd_release spd->nr_pages i'

The trace reveals: - `tcp_splice_data_recv` reads 0x6000 (24KB) in one call - Only 0x5890 bytes written to pipe before it fills - Remaining 0x770 bytes occupy 2 pages - Total page count: 18 pages for 24KB of data ### Root Cause: Memory Fragmentation

The 24KB data requiring 18 pages (each page is 4KB) endicates severe memory fragmentation. The socket buffer (skbuff) stores data non-contiguously, with each fragment requiring a separate pipe buffer slot. Since the default pipe capacity is 16 buffers, 18 pages cannot fit, causing the second `splice_to_pipe` invocation to block. Analysis on Kernel 6.5.6
------------------------

On modern kernels, pipe implementation uses `head/tail/max_usage` instead of `nrbufs/buffers`. Testing with 64KB data: ```
perf probe --add 'splice_to_pipe spd->nr_pages pipe->head pipe->tail pipe->max_usage'
perf probe --add '__splice_segment poff plen'

Results show 64KB data occupies only 11-12 pages, not the theoretical 16 pages. The __splice_segment trace reveals the kernel now tracks data by physical page offsets and lengths, allowing more efficient packing. Conclusion

The blocking behavior on kernel 3.10.0 stems from memory fragmentation causing skbuff data to span more pages than the pipe can accommodate. Modern kernels (6.x) handle this better through improved memory organization, where contiguous memory regions are more common and data spans fewer physical pages. This is a memory allocation characteristic rather than a splice implementation issue.