System Installation and Storage Configuration

Download the ImmortalWrt image for the bcm2711 target (Raspberry Pi 4B) from the official releases. Use a tool like BalenaEtcher to flash the rpi-4-ext4-factory.img.gz image onto the TF card.

After flashing, the root partition defaults to a minimal size. To utilize the full storage capacity for models, expand the partition using fdisk on the host machine.

Unmount the second partition and launch fdisk on the disk device (e.g., /dev/sdb):

sudo umount /dev/sdb2
sudo fdisk /dev/sdb

Inside fdisk, print partitions to note the start sector of partition 2. Delete partition 2, then create a new primary partition starting at the exact same sector to preserve data. Accept the default end sector to fill the disk. Write the changes and exit.

Apply the changes to the partition table and resize the filesystem:

sudo partprobe /dev/sdb
sudo e2fsck -f /dev/sdb2
sudo resize2fs /dev/sdb2

Insert the card into the Raspberry Pi and power it on. Connect via Ethernet to 192.168.1.1. SSH access is available as root without a password.

Cross-Compiling Dependencies

The ImmortalWrt system lacks development headers for OpenSSL. To build software requiring SSL (like CMake), cross-compile OpenSSL and CMake using the ImmortalWrt SDK and Toolchain.

OpenSSL Compilation

Check the installed OpenSSL version on the device:

opkg list-installed | grep libopenssl

Download the corresponding source version. Configure it for cross-compilation pointing to your toolchain:

./config no-asm shared no-async \
    --prefix=/opt/openssl-dev \
    --cross-compile-prefix=aarch64-openwrt-linux-musl- \
    linux-aarch64

Set the STAGING_DIR environment variable required by the toolchain scripts, then build:

export STAGING_DIR=/path/to/sdk/staging_dir
make -j$(nproc) && make install

CMake Compilation

Create a toolchain file toolchain.cmake to define cross-compilation settings:

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(TOOLCHAIN_ROOT /path/to/toolchain-aarch64_cortex-a72_gcc-13.3.0_musl)
set(OPENSSL_DEV_ROOT /path/to/openssl-dev)

set(CMAKE_C_COMPILER ${TOOLCHAIN_ROOT}/bin/aarch64-openwrt-linux-musl-gcc)
set(CMAKE_CXX_COMPILER ${TOOLCHAIN_ROOT}/bin/aarch64-openwrt-linux-musl-g++)
set(CMAKE_FIND_ROOT_PATH ${OPENSSL_DEV_ROOT})

Build CMake using the toolchain file:

cmake -S . -B build \
  -DCMAKE_TOOLCHAIN_FILE=toolchain.cmake \
  -DOPENSSL_ROOT_DIR=/path/to/openssl-dev \
  -DCMAKE_INSTALL_PREFIX=/opt/cmake

cmake --build build -j$(nproc)

Transfer the compiled binaries to the Raspberry Pi using scp.

Device Environment Setup

Configure the wireless interface to connect to a router for internet access. Install the SFTP server to facilitate file transfers:

opkg update
opkg install openssh-sftp-server

Install the standard C++ library required by the cross-compiled binaries:

opkg install libstdcpp

If the compiled CMake binary fails to find shared libraries, update the dynamic linker path:

export LD_LIBRARY_PATH=/usr/local/openssl-dev/lib:$LD_LIBRARY_PATH

Building llama.cpp

Install compilation dependencies on the device:

opkg install git make gcc libstdcpp

Clone the llama.cpp repository. Since ImmortalWrt uses musl libc, linking against libdl is unnecessary and causes build errors. Edit ggml/CMakeLists.txt and comment out the linkage:

# if (CMAKE_SYSTEM_NAME MATCHES "Linux")
#     target_link_libraries(ggml PRIVATE dl)
# endif()

Configure the build with static linking to avoid runtime library conflicts. Disable tests and cURL support to minimize dependencies:

cmake -B build \
  -DBUILD_SHARED_LIBS=OFF \
  -DLLAMA_BUILD_TESTS=OFF \
  -DLLAMA_CURL=OFF \
  -DGGML_OPENMP=OFF \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -j4

Model Execution

Download GGUF format models (e.g., Qwen3-1.7B) and place them in the root directory. Use llama-cli to test text generation:

./build/bin/llama-cli \
  -m /root/models/Qwen3-1.7B-Q8_0.gguf \
  -p "Explain quantum computing briefly:" \
  -n 128 \
  -t 4 \
  --temp 0.7

To run the model as an API service, use llama-server. For Qwen3 models, specific chat templates can be applied:

./build/bin/llama-server \
  -m /root/models/Qwen3-1.7B-Q8_0.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --ctx-size 4096 \
  --threads 4

For multimodal models like Qwen3-VL, specify the mmproj file:

./build/bin/llama-server \
  -m /root/models/Qwen3-VL-2B-Instruct-Q6_K.gguf \
  --mmproj /root/models/mmproj-F16.gguf \
  --ctx-size 4096 \
  --temp 0.8

Benchmark the inference speed using llama-bench to analyze performance across different thread counts:

./build/bin/llama-bench \
  -m /root/models/Qwen3-1.7B-Q8_0.gguf \
  -p 512,1024 \
  -n 128,256 \
  -t 1,2,4 \
  -r 3