Overview

This guide covers setting up a differential testing environment where:

• QEMU serves as the reference implementation (REF)
• NEMU serves as the device under test (DUT)

Test environment: Ubuntu 18.04.5 LTS

Building NEMU

First, install the required dependencies:

apt install build-essential man gcc gdb git libreadline-dev libsdl2-dev zstd libzstd-dev
git clone https://github.com/OpenXiangShan/NEMU.git
cd NEMU/
export NEMU_ROOT=/home/test/NEMU
export NEMU_HOME=/home/test/NEMU
make riscv64-benos_defconfig
make menuconfig

Within the menuconfig interface, apply these debug-oriented build settings:

Build Options -> Optimization Level：Select O0 (disable compiler optimizations)
Build Options -> Enable link-time optimization：Uncheck (disable LTO)
Build Options -> Enable debug information：Check (enable debug symbols)

Enable the dynamic library-based differential testing mode:

Testing and Debugging -> [*] Enable differential testing -> Reference design (QEMU, communicate with dynamic linking)

In NEMU/scripts/build.mk, remove all -Werror flags to prevent compilation failures:

CFLAGS  := -O2 -MMD -Wall $(INCLUDES) $(CFLAGS)
CXXFLAGS  := -O2 -MMD -Wall --std=c++17 $(XINCLUDES) $(CFLAGS)

Build the project:

make -j$(nproc)

Upon successful compilation, the executable NEMU/build/riscv64-nemu-interpreter is generated.

Building qemu-dl-diff

4.1 Udpate ISA Configuration

Modify NEMU/scripts/isa.mk to set the target architecture:

ISA ?= riscv64
ISAS = $(shell ls $(NEMU_HOME)/src/isa/)
ifeq ($(filter $(ISAS), $(ISA)), )
$(error Invalid ISA=$(ISA). Supported: $(ISAS))
endif
NAME := $(ISA)-$(NAME)
CFLAGS += -D__ISA_$(ISA)__=1

4.2 Patch Interrupt Handler

In NEMU/tools/qemu-dl-diff/src/diff-test.c, disable the interrupt raising call:

void difftest_raise_intr(uint64_t interrupt_num) {
    /* isa_raise_intr(interrupt_num); */
}

This function is originally defined in NEMU/tools/qemu-dl-diff/src/isa/x86/intr.c (the x86 dependency is not investigated further).

4.3 Link Dynamic Library Support

In NEMU/scripts/build.mk, append -ldl to the LDFLAGS section:

.DEFAULT_GOAL = app

ifdef SHARE
SO = -so
CFLAGS  += -fPIC -D_SHARE=1
LDFLAGS += -rdynamic -shared -fPIC -Wl,--no-undefined -lz -ldl
endif

Ensure -ldl is added precisely at this location, not elsewhere in the file.

4.4 Enable Debug Symbols

In NEMU/scripts/build.mk, modify CFLAGS, CXXFLAGS, and LDFLAGS to use -O0 and -g:

CFLAGS  := -O0 -g -MMD -Wall $(INCLUDES) $(CFLAGS)
CXXFLAGS  := -O0 -g -MMD -Wall --std=c++17 $(XINCLUDES) $(CFLAGS)
LDFLAGS := -O0 -g $(LDFLAGS)

4.5 Compile the Dynamic Library

Navigate to the qemu-dl-diff tool directory and build:

cd tools/qemu-dl-diff/
make

The dynamic library riscv64-qemu-so appears in NEMU/tools/qemu-dl-diff/build/.

Running NEMU with Differential Testing

The dynamic library launch configuration is defined in NEMU/tools/qemu-dl-diff/src/isa/riscv64/init.c:

char *isa_qemu_argv[] = {
  "/usr/bin/qemu-system-riscv64",
  "-nographic", "-S", "-serial", "none", "-monitor", "none",
  NULL
};

This requires QEMU to be installed at /usr/bin/qemu-system-riscv64. Place both benos_payload.bin and riscv64-qemu-so in the NEMU/build/ directory.

Execute with differential testing enabled:

./riscv64-nemu-interpreter -b benos_payload.bin -d ./riscv64-qemu-so

5.1 Issue: dlopen Failure

Attempting to load the executable with dlopen results in:

cannot dynamically load executable

Root cause: dlopen behavior varies across glibc versions. On systems with glibc 2.27, this approach works. Systems with glibc 2.31 or newer reject opening executables directly, enforcing stricter usage rules for dlopen.

Potential workarounds: Implementing dlopen interception or custom dlopen replacement add significant complexity.

5.2 Issue: Missing .debug Files

On compatible systems where dlopen succeeds, an assertion failure occurs. Debugging with gdb --args ./riscv64-nemu-interpreter -b benos_payload.bin -d ./riscv64-qemu-so reveals the issue: a debug symbol file is expected at /usr/lib/debug/.build-id/97/612b9c612a179a100ded49c146a82904b8c5d8.debug.

This file contains debug symbols parsed during QEMU initialization. Package-installed QEMU typically creates this symlink automatically. However, older distributions like Ubuntu 18.04 max out at QEMU 2.11.1, which lacks qemu-system-riscv64 support:

QEMU emulator version 2.11.1 (Debian 1:2.11+dfsg-1ubuntu7.42)

This leaves two encompatible requirements: a system new enough to run qemu-system-riscv64, yet old enough to have glibc before version 2.27 for dlopen compatibility. Testing across multiple distribution versions becomes necessary.