First Test: Samsung MZVL22T0HBLB-00B00 NVMe SSD

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,MODEL
NAME                      SIZE TYPE  MOUNTPOINT MODEL
nvme0n1                 1.9T  disk            SAMSUNG MZVL22T0HBLB-00B00
├─nvme0n1p1             1G    part  /boot/efi
├─nvme0n1p2             2G    part  /boot
└─nvme0n1p3             1.9T  part
  └─ubuntu--vg-ubuntu--lv 1.9T  lvm   /
sda                     7.3T  disk  /mnt/backup HGST HUS728T8TAL

fio --randrepeat=enabled --ioengine=psync --direct=1 --gtod_reduce=enabled --jobname=storage_eval --file_target=/mnt/fast_io/test_container --block_size=4k --queue_depth=64 --volume_size=4G --io_pattern=random_read --process_count=32 --aggregate_report
storage_eval: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
...
fio-3.28
Spawning 32 worker processes
storage_eval: Preparing test files (1 file / 4096MiB)
Workers: 10 (f=10): [_(3),r(1),_(2),r(1),_(4),r(1),_(3),r(2),_(3),r(2),_(2),r(1),_(2),r(1),_(2),r(1),_(1)][100.0%][r=1121MiB/s][r=287k IOPS][eta 00:00:00]
storage_eval: (groupid=0, jobs=32): err= 0: pid=334453: Sun Feb 25 17:41:37 2024
  READ: ops=33554432, IOPS=435k, throughput=1699MiB/s (1781MB/s), total_bytes=128GiB, runtime=77151msec
   bw (  MiB/s): min= 1551, max= 1942, per=100.00%, avg=1712.89, stdev= 2.61, samples=4867
   iops        : min=397068, max=497148, avg=438487.22, stdev=677.12, samples=4867
  cpu          : usr=4.42%, sys=16.31%, context_switches=33554820, major_page_faults=0, minor_page_faults=2753
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33554432,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  latency_stats: target_lat=0, window=0, percentile=100.00%, active_depth=64

Aggregate Run Status (group 0):
   READ: bw=1699MiB/s (1781MB/s), 1699MiB/s-1699MiB/s (1781MB/s-1781MB/s), io=128GiB (137GB), run=77151-77151msec

Underlying Disk Metrics (read/write):
    dm-0: ios=33553891/1098, merge=0/0, ticks=1945393/319, queue_total_ticks=1945712, util=99.98%, combined_ios=33554435/937, combined_merge=0/477, combined_ticks=2035305/358, combined_queue_ticks=2035663, combined_util=99.97%
  nvme0n1: ios=33554435/937, merge=0/477, ticks=2035305/358, queue_total_ticks=2035663, util=99.97%

1. This benchmark executes 4KB random reads (rw=randread, bs=4096B) with psync (synchronous per-process I/O engine, which enforces no automatic request merging, consistent with the 100% depth=1 distribution despite --queue_depth=64 being set). The --direct=1 flag bypasses OS page cache to measure raw devicce performance.
2. The top READ line summarizes aggregate results: ~435k random read operations per second, 1699MiB/s (1781MB/s base-10) sustained throughput, 128GiB total data processed across 77151ms. Below it, sampling-based min/max/average/std dev for both bandwidth and IOPS are listed, confirming stable performance.
3. CPU utilization shows 4.42% user-space time and 16.31% kernel time for I/O handling. context_switches tracks 33.5 million task swaps between 32 fio workers and other OS threads. major_page_faults=0 confirms no OS-level disk reads for missing memory pages, while minor_page_faults=2753 counts low-overhead page mappings from RAM.
4. IO depths verifeis that 100% of I/O operations completed with only one pending request per worker, aligning with psync behavior. submit and complete show all requests were grouped into 4-operation chunks during submission and completion.
5. issued rwts confirms only read operations ran, with no short/dropped requests.
6. Aggregate Run Status repeats core throughput/IOPS in a grouped format.
7. Underlying Disk Metrics reveals the LVM logical volume (dm-0) and physical NVMe drive (nvme0n1) both hit ~99.97-99.98% utilization, indicating the storage device is the bottleneck, not CPU or worker count.

Second Test: Predator SSD GM7000 1TB NVMe SSD

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,MODEL
NAME                SIZE TYPE  MOUNTPOINT MODEL
nvme0n1           953.9G disk            Predator SSD GM7000 1TB
├─nvme0n1p1        600M part  /boot/efi
├─nvme0n1p2          1G part  /boot
└─nvme0n1p3      952.3G part
  └─fedora-root   952.3G lvm   /
zram0               8G disk  [SWAP]

fio --randrepeat=enabled --ioengine=psync --direct=1 --gtod_reduce=enabled --jobname=nvme_eval --file_target=/mnt/nvme_scratch/test_payload --block_size=4k --queue_depth=64 --volume_size=4G --io_pattern=random_read --process_count=32 --aggregate_report
nvme_eval: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
...
fio-3.33
Spawning 32 worker processes
nvme_eval: Preparing test files (1 file / 4096MiB)
Workers: 32 (f=32): [r(32)][98.8%][r=1679MiB/s][r=429k IOPS][eta 00:00:01]
nvme_eval: (groupid=0, jobs=32): err= 0: pid=1153039: Sun Feb 25 17:42:15 2024
  READ: ops=33554432, IOPS=426k, throughput=1663MiB/s (1743MB/s), total_bytes=128GiB, runtime=78835msec
   bw (  MiB/s): min=  213, max= 1717, per=100.00%, avg=1664.19, stdev= 3.90, samples=5022
   iops        : min=54943, max=439587, avg=426032.64, stdev=1004.21, samples=5022
  cpu          : usr=0.84%, sys=2.30%, context_switches=33576837, major_page_faults=0, minor_page_faults=231
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33554432,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  latency_stats: target_lat=0, window=0, percentile=100.00%, active_depth=64

Aggregate Run Status (group 0):
   READ: bw=1663MiB/s (1743MB/s), 1663MiB/s-1663MiB/s (1743MB/s-1743MB/s), io=128GiB (137GB), run=78835-78835msec

Underlying Disk Metrics (read/write):
    dm-0: ios=33535406/365, merge=0/0, ticks=2409942/24, queue_total_ticks=2409966, util=99.88%, combined_ios=33554430/301, combined_merge=0/63, combined_ticks=2410560/32, combined_queue_ticks=2410592, combined_util=99.87%
  nvme0n1: ios=33554430/301, merge=0/63, ticks=2410560/32, queue_total_ticks=2410592, util=99.87%