During a weekend, I conducted a small experiment on virtualizing a PCI device using the MDEV framework. Here's a brief record of the process:
The wdg-czl project is a virtual PCI device driver based on the Linux kernel's mediated device (mdev) framework. It demonstrates how to share a virtual PCI device between a host and a virtual machine. The project consists of two core modules: a host-side mdev driver and a guest-side PCI driver. The Mediated Device (mdev) is a mechanism provided by the Linux kernel that allows creating virtual devices in userspace, which can then be passed through to a virtual machine via the VFIO framework.
The architecture of the demo is illustrated below. (This diagram references the kernel documentation: Documentation/driver-api/vfio-mediated-device.rst)
The demo framework references the mediated passthrough scheme from the article below. The difference is that the watchdog in this experiment is purely an emulated device. Its BAR IO space actually corresponds to memory buffers, so PCI passthrough is not required. "Mediated passthrough allows performance-critical accesses to be passed directly to the GPU (e.g., video memory), while intercepting non-performence-critical parts (like CSR and some MMIO registers) and emulating them in the mdev module. This makes the system perceive multiple 'seemingly' complete GPU PCIe devices, i.e., vGPUs, which can also support native GPU drivers, such as Intel's integrated GPU GVT-g."
https://blog.itpub.net/28285180/viewspace-2962108/ (A quick 5-minute tech talk on GPU virtualization)
The program consists of three parts: a userspace test program in the Guest OS, a kernel-space vWDG driver in the Guest OS, and a kernel-space vWDG mdev device emulation in the Host OS. The Host VMM (QEMU) does not require modification. The function call chain is as follows:
User Operation
|
v
+------------------+
| create/delete | <-- mdev type management
| mdev devices |
+------------------+
|
v
+------------------+
| QEMU vfio-pci | <-- Device passthrough
+------------------+
|
v
+------------------+
| Guest PCI Driver | <-- Device driver
+------------------+
|
v
+------------------+
| /dev/czl-wdg-X | <-- User interface
+------------------+
|
v
+------------------+
| test.c | <-- Functional tests
+------------------+
Design Philosophy of the MDEV Framework
The 'm' in mdev stands for "Mediated," which describes the architectural design of inserting a controllable, manageable, and virtualizable software mediation layer between the device and the virtual machine. This device can be an emulated memory device, like the WDG in this article, or a real physical device, like a PCIe device. MDEV is arguably more flexible than VFIO passthrough. The virtual machine sees the MDEV device, not the real physical device. The MDEV device acts as a bridge between the real physical device and the VM. All calls from the VM to the MDEV device are bridged to the real physical device (similar to the Bridge pattern in software design). In contrast, VFIO passthrough exposes the VM directly to the physical device. The existence of the MDEV device allows software intervention, primarily in the following aspects:
- Mediated Access: All VM accesses to the device go through the host kernel's mediation layer, not directly to the hardware.
- Mediated Resources: Physical resources are partitioned and virtualized by the software layer, with each VM getting a portion.
- Mediated Control: Device state, configuration, and DMA operations are controlled and arbitrated by the mediation layer.
- Mediated Security: The mediation layer ensures isolation between VMs and prevents out-of-bounds access.
- Mediated Performance: QoS, scheduling, and monitoring are managed by the mediation layer.
- Mediated Migration: State save/restore is performed through the mediation layer, supporting device migration.
Code Implementation
Host Kernel Watchdog PCI Driver (mdev module):
#include <linux/init.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/kernel.h>
#include <linux/fs.h>
#include <linux/poll.h>
#include <linux/slab.h>
#include <linux/cdev.h>
#include <linux/sched.h>
#include <linux/wait.h>
#include <linux/uuid.h>
#include <linux/vfio.h>
#include <linux/iommu.h>
#include <linux/sysfs.h>
#include <linux/ctype.h>
#include <linux/file.h>
#include <linux/mdev.h>
#include <linux/pci.h>
#define IO_BAR0_SIZE 32
#define IO_CONF_SIZE 0x100
#define CZL_WDG_DEVICE_VENDOR_ID 0xbeef
#define CZL_WDG_DEVICE_DEVICE_ID 0x1001
#define API_DBG(fmt, ...) do { \
printk("%s line %d, "fmt, __func__, __LINE__, ##__VA_ARGS__); \
} while (0)
struct czl_wdg_dev {
dev_t wdg_devt;
struct class *wdg_class;
struct cdev wdg_cdev;
struct device dev;
};
struct mdev_region_info {
u64 start;
u64 phys_start;
u32 size;
u64 vfio_offset;
};
struct wdg_mdev_state {
u8 *config;
u8 *iobase;
struct mdev_device *mdev;
struct mdev_region_info region_info[VFIO_PCI_NUM_REGIONS];
u32 bar_mask[VFIO_PCI_NUM_REGIONS];
struct list_head next;
struct vfio_device_info dev_info;
int index;
struct mutex ops_lock;
};
static const struct file_operations czl_wdg_fops = {
.owner = THIS_MODULE,
};
static struct mutex wdg_mdev_list_lock;
static struct list_head wdg_mdev_devices_list;
#define WDG_VFIO_PCI_OFFSET_SHIFT (40)
#define WDG_VFIO_PCI_OFFSET_TO_INDEX(off) (off >> WDG_VFIO_PCI_OFFSET_SHIFT)
#define WDG_VFIO_PCI_INDEX_TO_OFFSET(index) \
((u64)(index) << WDG_VFIO_PCI_OFFSET_SHIFT)
#define WDG_VFIO_PCI_OFFSET_MASK \
(((u64)(1) << WDG_VFIO_PCI_OFFSET_SHIFT) - 1)
#define MAX_WDGS (16)
static struct czl_wdg_dev czl_wdg;
(The full host driver code continues with functions for sysfs attributes, mdev device creation/removal, PCI config space emulation, BAR read/write handling, and VFIO ioctl implementation. The complete code can be found in the original article.)
QEMU Guest Kernel Watchdog PCI Driver
Since the added watchdog device is a PCIe device, the first task after the driver entry point executes is to initialize the PCIe device-related operations. This typically involves calling pci_register_driver to register the PCIe devices the driver is interested in with the kernel, and providing a pci device probe callback function. When the kernel matches the hardware corresponding to the driver, it calls the probe function. Therefore, from a certain perspective, the probe callback function of a PCIe device is the main entry point for most PCIe drivers. If no hardware is plugged into the motherboard, the driver remains in a silent state, unbound from any device. The driver only becomes active upon detecting the PCIe hardware. (Of course, this process is specific to PCIe hardware device drivers. For exceptions, like the RXE driver in drivers/infiniband/sw, which simulates hardware behavier in software without involving real hardware, the initialization flow will necessarily differ. Such drivers typically register a device model structure in software, often calling the kernel's device_add API when adding the device. This is why RXE devices must be created manually using rdma link, as they are not hardware devices and do not support automatic hardware bus detection.)
The driver and device are bound based on the vendor ID/device ID matching rule in the PCIe configuration space.
#include <linux/init.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/kernel.h>
#include <linux/fs.h>
#include <linux/poll.h>
#include <linux/slab.h>
#include <linux/cdev.h>
#include <linux/sched.h>
#include <linux/wait.h>
#include <linux/uuid.h>
#include <linux/vfio.h>
#include <linux/iommu.h>
#include <linux/sysfs.h>
#include <linux/ctype.h>
#include <linux/file.h>
#include <linux/mdev.h>
#include <linux/pci.h>
#include <linux/idr.h>
static int devno;
static DEFINE_IDR(wdg_minors);
static DEFINE_MUTEX(wdg_minors_lock);
#define WDG_MINORS_COUNT 256
struct wdg_pci_state {
struct pci_dev *pdev;
struct device *dev;
int iobase;
int iolen;
int major;
int minor;
};
static struct class *wdg_class;
static const struct pci_device_id czl_pci_table[] = {
{ PCI_DEVICE(0xbeef, 0x1001), },
{ 0, }
};
// ... (Guest driver probe, remove, file operations, etc.) ...
(The full guest driver code continues with the probe function, which sets up the PCI device, allocates a minor number, creates a character device, and allows userspace interaction. The complete code is listed in the original article.)
QEMU Guest Userspace WDT Test Case
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <stddef.h>
#include <stdint.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdarg.h>
void dump_buf(unsigned char *buf, int len)
{
int i;
for (i = 0; i < len; i++) {
if (i % 16 == 0)
printf("\n0x%04x: ", i);
printf("0x%02x ", buf[i]);
}
printf("\n");
}
int main(void)
{
int wdgfd;
int status;
unsigned char buf[32];
wdgfd = open("/dev/czl-wdg-0", O_RDWR);
if (wdgfd < 0) {
printf("open failure.\n");
return -1;
}
while (1) {
memset(buf, 0x00, 32);
status = read(wdgfd, buf, 32);
if (status < 0) {
printf("read failure.\n");
return -1;
}
printf("Read %d bytes.\n", status);
dump_buf(buf, 32);
memset(buf, 0x5a, 32);
lseek(wdgfd, 0, SEEK_SET);
status = write(wdgfd, buf, 32);
if (status < 0) {
printf("write failure.\n");
return -1;
}
printf("Write %d bytes.\n", status);
sleep(1);
}
close(wdgfd);
return 0;
}
Test Procedure:
**1. Install the WDG MDEV Driver:**The host kernel's VFIO driver is typically built-in. If the kernel is recompiled with drivers as modules, the dependency of mdev on VFIO becomes apparent. The module dependency on the host side can be represented by the following diagram.
Update (for 5.15.x kernels and later): The kernel supports custom mdev_driver implementations parallel to the built-in vfio_mdev_driver. The vfio_mdev.ko module becomes optional. In very early versions (e.g., 5.4), mdev.ko did not depend on vfio.ko. This dependency was introduced later when the logic of vfio_mdev.ko was merged into mdev.ko. With the introduction of IOMMUFD in the 6.x kernel series, this coupling was resolved again, and MDEV no longer depends on VFIO at the API level.
2. Create an mdev Device:
# Create two mdev devices
echo "f422fd86-35c0-11ef-8e50-9342c1138a56" > /sys/devices/virtual/czl_wdg/czl_wdg/mdev_supported_types/czl_wdg-1/create
echo "c04de378-35d8-11ef-95c3-339660dfc874" > /sys/devices/virtual/czl_wdg/czl_wdg/mdev_supported_types/czl_wdg-2/create
3. Pass the mdev Device to QEMU:
qemu-system-x86_64 -m 4096 -smp 4 --enable-kvm \
-drive file=/path/to/disk.img \
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/f422fd86-35c0-11ef-8e50-9342c1138a56 \
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/c04de378-35d8-11ef-95c3-339660dfc874
**4. Install the WDG PCI Driver in the Guest VM:**After the VM boots, the two passthrough MDEV devices might be bound to a generic driver like "serial". They need to be unbound first, and then the custom WDG driver can be loaded:
# Unbind from the default driver
echo -n 0000:00:04.0 > /sys/bus/pci/drivers/serial/unbind
echo -n 0000:00:05.0 > /sys/bus/pci/drivers/serial/unbind
# Insert the custom driver
sudo insmod czl-mdev-drv.ko
**5. Run Tests:**After loading the driver, the test program can read from and write to the WDG PCI device's BAR0 address space. The calls in the guest are effectively mediated back to the host's mdev driver, allowing the implementation of custom business logic based on the BAR space callbacks.
**Interrupt Injection Emulation:**Interrupts can be injected from the host MDEV driver into the guest using the eventfd mechanism via KVM. The image below shows the guest OS driver successfully receiving an interrupt injected by the host OS MDEV framework callback.
**DMA Remapping Support:**The vWDG in this example is a purely emulated device without its own storage, so PCI passthrough is not needed. For pure memory-emulated devices, mdev_set_iommu_device is not called, so the device does not get an IOMMU domain. To enible DMA, the MDEV device must be associated with a parent device that has DMA capabilities (like a PCIe device). By changing the parent device to a real PCIe device (e.g., an NVIDIA GPU) and calling mdev_set_iommu_device, the MDEV device can borrow its parent's IOMMU for mapping GVA (Guest Virtual Address) or GPA (Guest Physical Address) to HPA (Host Physical Address).
Summary
The relationship between the objects in this test project is illustrated below:
+------------------+ +------------------+ +------------------+
| Host | | QEMU | | Guest |
+------------------+ +------------------+ +------------------+
| | | | | |
| +------------+ | | | | +------------+ |
| | mdev Core | | | vfio-pci | | | PCI Driver | |
| | (mdev) | | | Module | | |czl-mdev-drv| |
| +------------+ | | | | +------------+ |
| | | Passthrough: | | |
| +------------+ | | vfio-pci,sysfsdev| | +------------+ |
| | vfio_mdev | |<-------->| =/sys/bus/mdev/ |<--------| | /dev/czl- | |
| | Module | | | devices/UUID | | | wdg-X | |
| +------------+ | | | | +------------+ |
| | | PCI BAR Space | | |
| +------------+ | | Mapping | | +------------+ |
| |czl-mdev- | | | | | | test.c | |
| | wdg.ko | | | | | +------------+ |
| +------------+ | | | | |
| Create mdev: | | | | |
| /sys/.../create | | | | |
+------------------+ +------------------+ +------------------+