Fundamentals of Virtualized Storage

Storage I/O consistently represents a primary bottleneck in virtualized architectures. To mitigate latency and throughput constraints, administrators should distribute disk workloads across multiple physical drives or implement centralized storage solutions like SAN or NFS to enable high availability and live migration capabilities.

Storage Backends and Image Formats

Libvirt abstracts underlying storage into distinct backends and file formats. Understanding these distinctions is critical for performance tuning and migration workflows.

• Backend Types:
  • dir: Standard filesystem directory
  • fs: Pre-formatted block device (ext4, xfs, etc.)
  • netfs: Network-exported filesystem (NFS, CIFS)
  • disk: Raw physical disk device
  • iscsi: iSCSI target
  • logical: LVM volume group
• Allocation Strategies:
  • Fixed (Thick): Pre-allocates the full specified size on the host filesystem, regardless of actual guest data usage.
  • Dynamic (Thin): Expands on-demand up to a maximum capacity, conserving host storage until data is written.
  • Differencing: Stores only blocks that differ from a parent/base image, enabling rapid cloning.
• Image Formats:
  • raw: Unstructured binary image. Zero metadata overhead, excellent compatibility, but lacks native snapshot or compression support. Relies on host filesystem sparse file support.
  • qcow2: Copy-on-Write format. Natively supports thin provisioning, snapshots, zlib compression, and AES encryption. The recommended format for most KVM deployments.
  • vmdk, vhdx, vdi: Formats native to VMware, Hyper-V, and VirtualBox respectively, oftan used during cross-platform migrations.

You can verify supported formats by running: qemu-img --help | grep -i supported

Image Management with qemu-img

The qemu-img utility handles lifecycle operations for disk images. Core subcommands include create, info, check, resize, convert, and snapshot.

Creating and Inspecting Images

# Create a 5G thin-provisioned image
[root@kvm-host ~]# qemu-img create -f qcow2 golden-base.qcow2 5G
Formatting 'golden-base.qcow2', fmt=qcow2 size=5368709120 encryption=off cluster_size=65536 lazy_refcounts=off

# Verify allocation vs virtual size
[root@kvm-host ~]# qemu-img info golden-base.qcow2
image: golden-base.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 196K
cluster_size: 65536

By default, qemu-img creates sparse files. Tools like ll display the virtual size, while du reveals actual disk consumption. Filesystem-level sparse behavior can be manipulated during copy operations:

# Force sparse preservation during copy
[root@kvm-host ~]# cp --sparse=always golden-base.qcow2 sparse-copy.qcow2

# Force full allocation during copy (eliminates holes)
[root@kvm-host ~]# cp --sparse=never golden-base.qcow2 full-copy.qcow2

Preallocation Modes for QCOW2

The preallocation option controls how host space is reserved:

• off: Default. No host space reserved until guest writes occur.
• metadata: Allocates space for QCOW2 metadata tables. File remains sparse but reduces runtime fragmentation overhead.
• falloc: Reserves disk blocks without zeroing them. Faster than full while guaranteeing space availability.
• full: Allocates and zeroes the entire image. Eliminates allocation latency during guest operation but increases creation time.

Backing Files and Differencing Disks

Differancing disks rely on a read-only parent image. Child images only capture write operations, drastically reducing storage consumption for homogeneous VM fleets.

# Create child image linked to parent
[root@kvm-host ~]# qemu-img create -f qcow2 -o backing_file=golden-base.qcow2 web-tier-disk.qcow2 20G
Formatting 'web-tier-disk.qcow2', fmt=qcow2 size=21474836480 backing_file='golden-base.qcow2' ...

# Verify chain
[root@kvm-host ~]# qemu-img info web-tier-disk.qcow2
image: web-tier-disk.qcow2
file format: qcow2
virtual size: 20G
backing file: golden-base.qcow2

Multiple VMs can share the same backing_file. This approach standardizes OS deployments but introduces I/O overhead due to the read redirection through the chain.

Format Conversion for Migration

Migrating from VMware requires converting .vmdk images to KVM-native formats:

[root@kvm-host ~]# qemu-img convert -O qcow2 source-server.vmdk target-server.qcow2
[root@kvm-host ~]# qemu-img info target-server.qcow2
image: target-server.qcow2
file format: qcow2
virtual size: 40G (42949672960 bytes)
disk size: 12.4G

Note that converted images may temporarily appear larger due to metadata reconstruction and alignment adjustments.

Resizing Images

# Expand virtual capacity
[root@kvm-host ~]# qemu-img resize target-server.qcow2 +10G
Image resized.

# Verify new virtual size
[root@kvm-host ~]# qemu-img info target-server.qcow2
virtual size: 50G (53687091200 bytes)

Important: Resizing only modifies the virtual disk geometry. Inside the guest OS, you must still extend the partition table and resize the filesystem. QCOW2 does not support shrinking images natively.

Snapshot Architectures

KVM supports three snapshot categories:

• Disk Snapshots: Capture block-level state. Ideal for backup workflows.
• Memory Snapshots: Preserve RAM and device state. Implemented via virsh save. Requires a running VM.
• Checkpoints: Combine disk and memory states for exact point-in-time recovery, ensuring transactional consistency.

Disk snapshots are further classified by storage layout (internal vs. external) and guest state (powered-off vs. running). Internal snapshots embed metadata within the QCOW2 file, while external snapshots use a new child file.

Managing Internal Disk Snapshots

# List existing snapshots
[root@kvm-host ~]# qemu-img snapshot -l golden-base.qcow2

# Create a snapshot
[root@kvm-host ~]# qemu-img snapshot -c pre-update-golden golden-base.qcow2

# Revert to snapshot (VM must be shut down)
[root@kvm-host ~]# qemu-img snapshot -a pre-update-golden golden-base.qcow2

# Remove snapshot
[root@kvm-host ~]# qemu-img snapshot -d pre-update-golden golden-base.qcow2

Internal snapshots are restricted to native QCOW2 images. Images converted from other formats may exhibit corruption during snapshot operations.

Libvirt Storage Pools

Storage pools provide a centralized abstraction layer for managing disks, partitions, LVM groups, or network shares. While not mandatory, pools simplify volume creation and permission handling.

Pool Lifecycle Commands

• pool-list: Display active/inactive pools
• pool-info: Show capacity, allocation, and autostart status
• pool-define-as: Create persistent pool definition
• pool-build: Initialize underlying storage structure
• pool-start / pool-autostart: Activate and enable boot persistence
• pool-destroy / pool-undefine: Deactivate and remove configuraton

Directory-Based Pool Example

# Define and initialize a directory pool
[root@kvm-host ~]# mkdir -p /data/vm-storage
[root@kvm-host ~]# chown root:root /data/vm-storage
[root@kvm-host ~]# chmod 700 /data/vm-storage

[root@kvm-host ~]# virsh pool-define-as vm_data_dir dir --target "/data/vm-storage"
Pool vm_data_dir defined

[root@kvm-host ~]# virsh pool-build vm_data_dir
Pool vm_data_dir built
[root@kvm-host ~]# virsh pool-start vm_data_dir
Pool vm_data_dir started
[root@kvm-host ~]# virsh pool-autostart vm_data_dir
Pool vm_data_dir marked as autostarted

Libvirt automatically generates XML definitions in /etc/libvirt/storage/. Modifications should be performed via virsh pool-edit to prevent configuration drift.

Partition and LVM Pools

For block-level pools, libvirt handles mounting and VG initialization automatically:

# Filesystem-backed pool
[root@kvm-host ~]# virsh pool-define-as local_ext4 fs --source-dev "/dev/sdb1" --target "/mnt/vm-ext4"

# LVM-backed pool (builds VG if missing)
[root@kvm-host ~]# virsh pool-define-as vm_lvm_pool logical --source-dev "/dev/sdc" --target "/dev/vg_kvm"
[root@kvm-host ~]# virsh pool-build vm_lvm_pool
[root@kvm-host ~]# virsh pool-start vm_lvm_pool

NFS Network Pool

# Define NFS export as storage backend
[root@kvm-host ~]# virsh pool-define-as nfs_share netfs \
  --source-host 192.168.1.50 --source-path "/exports/vm-images" --target "/mnt/nfs-vms"
Pool nfs_share defined

[root@kvm-host ~]# virsh pool-start nfs_share
Pool nfs_share started

Storage Volume Operations

Volumes represent allocatable units within a pool. They manifest as regular files, block devices, or logical volumes depending on the pool type.

Creating and Cloning Volumes

# Create a QCOW2 volume in a directory pool
[root@kvm-host ~]# virsh vol-create-as vm_data_dir app-server-disk.qcow2 50G --format qcow2

# Inspect volume metrics
[root@kvm-host ~]# virsh vol-info /data/vm-storage/app-server-disk.qcow2
Name:           app-server-disk.qcow2
Type:           file
Capacity:       50.00 GiB
Allocation:     196.00 KiB

# Clone an existing volume
[root@kvm-host ~]# virsh vol-clone app-server-disk.qcow2 app-server-disk-backup.qcow2 --pool vm_data_dir

Deleting Volumes

[root@kvm-host ~]# virsh vol-delete app-server-disk-backup.qcow2 --pool vm_data_dir
Volume app-server-disk-backup.qcow2 deleted

Attaching Volumes to Running VMs

Volumes can be hot-plugged using XML definitions or direct CLI parameters.

# XML definition for temporary disk
[root@kvm-host ~]# cat /tmp/extra-disk.xml
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none'/>
  <source file='/data/vm-storage/app-server-disk.qcow2'/>
  <target dev='vdb' bus='virtio'/>
</disk>

# Attach persistently
[root@kvm-host ~]# virsh attach-device production-web /tmp/extra-disk.xml --persistent
Device attached successfully

Alternatively, use the simplified parameter interface:

[root@kvm-host ~]# virsh attach-disk production-web /data/vm-storage/app-server-disk.qcow2 vdb --persistent
Disk attached successfully

# Verify attachment
[root@kvm-host ~]# virsh domblklist production-web
Target     Source
-------------------------------------------
vda        /data/vm-storage/golden-base.qcow2
vdb        /data/vm-storage/app-server-disk.qcow2