Performance Anomaly Investigation

During performance profiling on an Arm64 embedded platform, we encountered a puzzling behavior with dma-buf memory regions. Benchmarking memcpy() operations revealed a tenfold performance disparity:

This asymmetry contradicted initial hypotheses about cache misses. Profiling with perf showed unexpected results: the slow dma-buf read operation exhibited <0.5% cache miss rate, while the fast heap-to-dma-buf transfer showed 16% misses. Furthermore, the bottleneck in the slow path traced to a simple register subtraction instruction within a 128-byte copy loop—an operation that should be negligible.

To isolate the issue, we created microbenchmarks for pure read and write operations:

void benchmark_read_bandwidth(void* region, int length) {
    char* limit = (char*)region + length;
    char* ptr = (char*)region;
    uint64_t v0, v1;
    
    while (ptr < limit) {
        asm volatile(
            "ldp %0, %1, [%2], #16\n"
            : "=r"(v0), "=r"(v1), "+r"(ptr)
            :
            : "memory"
        );
    }
}

void benchmark_write_bandwidth(void* region, int length) {
    char* limit = (char*)region + length;
    char* ptr = (char*)region;
    const uint64_t val0 = 0xDEADBEEF;
    const uint64_t val1 = 0xCAFEBABE;
    
    while (ptr < limit) {
        asm volatile(
            "stp %0, %1, [%2], #16\n"
            :
            : "r"(val0), "r"(val1), "+r"(ptr)
            : "memory"
        );
    }
}

The results confirmed read operations on dma-buf were drastically slower:

This pointed to a fundamental architectural difference in how Arm64 handles memory types for DMA-accessible regions.

DMA and Buffer Sharing Fundamentals

Direct Memory Access offloads I/O data transfers from the CPU, allowing peripherals to read/write memory independently. However, this creates cache coherence challenges: CPU writes may reside in cache layers without reaching physical memory, causing DMA controllers to read stale data.

The Linux dma-buf subsystem addresses this by providing a framework for sharing buffers across drivers, kernel/user spaces, and processes. Graphics memory allocated through DRM/GEM, for instance, uses dma-buf for zero-copy buffer exchange between applications and compositors in UMA architectures.

Arm64 Memory Ordering and Write Buffering

Weakly-Ordered Memory Model

Arm64 employs a weakly-ordered memory model where memory operations may complete out of program order while preserving sequential consistency. Consider this instruction sequence:

STR R12, [R1]    // Write to memory
LDR R0, [SP], #4 // Read from stack
LDR R2, [R3, #8] // Read with offset

The CPU may execute these as:

1. Store R12 into the write buffer (CPU continues immediately)
2. Initiate cache line fill for the stack read (miss)
3. Complete the offset read (hit) while the previous operations are pending
4. Receive stack data from memory
5. Flush write buffer to memory

This reordering eliminates pipeline stalls, significantly improving throughput compared to strict ordering.

Write Buffering and Combining

The write buffer sits between L1 cache and main memory, accepting stores that miss cache. It enables write combining: multiple stores to contiguous addresses merge into a single transaction. For example, four 8-byte writes to adjacent locations can coalesce into one 32-byte write, reducing memory traffic.

Memory Type Classification

Arm64 defines two primary memory types, each with distinct attributes affecting performance and coherency:

Normal Memory

Covers most system memory including code segments, data segments, RAM, and ROM. It supports:

• Full caching with write-back or write-through policies
• Weak ordering for optimal performance
• Compiler and CPU reordering optimizations

Linux kernel defines several variants:

// arch/arm64/include/asm/memory.h
#define MT_NORMAL        0  // Standard cached memory
#define MT_NORMAL_TAGGED 1  // Memory Tagging Extension
#define MT_NORMAL_NC     2  // Non-cacheable

Device Memory

Designed for memory-mapped peripherals and DMA regions. Key characteristics:

• Never cacheable – all accesses go directly to physical memory
• Restricts reordering and combining based on attributes
• Provides deterministic access timing

Six attribute combinations exist based on three behavioral controls:

• G/nG (Gathering): Enables/disables write combining
• R/nR (Re-ordering): Permits/prohibits access reordering
• E/nE (Early Write Acknowledgement): Write completion signaled at write buffer vs. endpoint

Linux typically uses:

// arch/arm64/include/asm/memory.h
#define MT_DEVICE_nGnRnE 3  // Most restrictive
#define MT_DEVICE_nGnRE  4  // Moderate restrictions

Most dma-buf implementations default to MT_DEVICE_nGnRE for compatibility.

Memory Attribute Configuration

System software programs the Memory Attribute Indirection Register (MAIR_ELn) with eight attribute encodings. During page table walks, the AttrIndx[2:0] field in level 3 descriptors indexes into MAIR_ELn to determine page-level behavior. This granular control allows mixing memory types within virtual adress spaces.

Cache Coherence Architecture

Shareability Domains

Arm64 defines four coherency domains:

1. Non-shareable: Single CPU core (private L1)
2. Inner Shareable: CPU cluster sharing L2 cache
3. Outer Shareable: Multiple clusters sharing L3 cache
4. System Shareable: All memory agents (CPUs, GPU, DMA, NPU)

Coherence Protocols

Inner domain coherence uses the MESI protocol, transparant to software. Outer domain coherence requires protocols like AMBA AXI Coherency Extensions (ACE). System-level coherence demands ACE-Lite or equivalent protocols connecting peripherals to a coherence fabric.

Early SoCs lacking system-level coherence hardware relied on software cache maintenance (flush/invalidate) or non-cacheable Device memory for DMA buffers. Modern platforms with full hardware coherence can safely allocate Normal memory for dma-buf, enabling CPU caching and eliminating performance cliffs.

Vendor-specific optimizations leverage this: some Qualcomm platforms use Normal memory for graphics allocations, and Vulkan applications can request VK_MEMORY_PROPERTY_HOST_CACHED_BIT on device-local heaps when hardware coherence is present.

The performance anomaly we observed stems directly from defaulting to Device memory: writes appear fast due to write buffer combining, while reads suffer uncached latency. Profiling misled us because cache miss metrics don't capture uncached access penalties, and the subtract instruction appeared hot due to pipeline stalls waiting for read data.