Note: The setup with CentOS 8 (kernel 4.18) is not viable. Switch the client and MDS to Ubuntu (tested successfully on Ubuntu 18.04 with kernel 4.15) while keeping the setup steps the same.

With the rapid development of networking and parallel computing, file I/O has become a bottleneck for overall system performence. Traditional NFS versions forced all clients to connect to a single server, which handled every request—unsuitable for parallel computing. Parallel NFS (pNFS), part of the NFSv4.1 standard, enhances scalability by enabling clients to transfer data directly to storage devices, boosting throughput. When the server also supports pNFS, clients can access data via multiple servers simultaneously, significantly improving read/write speeds. It supports three storage protocols/layouts: file, object, and block. pNFS client code is merged into Linus Torvalds’ mainline, but server-side code remains absent.

2 Environment Preparation

2.1 iSCSI Client Software Installation

All nodes except iSCSI servers require iSCSI cliant tools.

# CentOS
yum install -y iscsi-initiator-utils

# Ubuntu
apt install open-iscsi
# Commands are identical for both

2.2 iSCSI Server Setup (Example: 192.168.96.14)

# CentOS: Install targetcli
yum install targetcli -y

# Ubuntu: Install targetcli-fb (alternative for Ubuntu)
# apt install targetcli-fb

Using targetcli to create volumes and configure client access:

# Enter targetcli shell
targetcli

# Create a block backstore (using /dev/sdb as example)
/backstores/block create my_storage /dev/sdb

# Create an iSCSI target
/iscsi create iqn.2024-06.example.com:storage

# Associate backstore with target and LUN
/iscsi/iqn.2024-06.example.com:storage/tpg1/luns create /backstores/block/my_storage

# Retrieve iSCSI Initiator names from non-server nodes (e.g., 96.228, 96.208)
# Then associate with the target's ACLs
/iscsi/iqn.2024-06.example.com:storage/tpg1/acls create iqn.1994-05.com.redhat:ad9e7f1bf9f  # For 96.228
/iscsi/iqn.2024-06.example.com:storage/tpg1/acls create iqn.1994-05.com.redhat:dc651ec4d08b  # For 96.208

exit  # Configuration saves automatically

Repeat for the second iSCSI server (192.168.96.244) with a different device (e.g., /dev/sdc):

/backstores/block create my_storage2 /dev/sdc
/iscsi create iqn.2024-06.example.com:storage2
/iscsi/iqn.2024-06.example.com:storage2/tpg1/luns create /backstores/block/my_storage2
/iscsi/iqn.2024-06.example.com:storage2/tpg1/acls create iqn.1994-05.com.redhat:ad9e7f1bf9f
/iscsi/iqn.2024-06.example.com:storage2/tpg1/acls create iqn.1994-05.com.redhat:dc651ec4d08b

MDS Configuration

1. Connect to iSCSI Targets

   # Discover targets
   iscsiadm -m discovery -t st -p 192.168.96.14
   iscsiadm -m discovery -t st -p 192.168.96.244
   
   # Log in to targets
   iscsiadm -m node -p 192.168.96.14 --login
   iscsiadm -m node -p 192.168.96.244 --login
   
   # Verify connections
   iscsiadm -m session -P 3 | grep -E 'Portal|Attached'
   

2. Format and Mount iSCSI Devices

   # Create mount points
   mkdir -p /mnt/pnfs_export/dir{1,2}
   
   # Format as XFS (supports pNFS)
   mkfs.xfs /dev/sda -f
   mkfs.xfs /dev/sdb -f
   
   # Mount devices
   mount /dev/sda /mnt/pnfs_export/dir1
   mount /dev/sdb /mnt/pnfs_export/dir2
   
   # Verify mounts
   df -h
   

3. Configure NFS Exports
   Edit /etc/exports:

   /mnt/pnfs_export *(pnfs,rw,sync,fsid=0,no_root_squash,no_subtree_check)
   /mnt/pnfs_export/dir1 *(pnfs,rw,sync,fsid=1,no_root_squash,no_subtree_check)
   /mnt/pnfs_export/dir2 *(pnfs,rw,sync,fsid=2,no_root_squash,no_subtree_check)
   

Client Configuration

1. Connect to iSCSI Targets (same as MDS)

   iscsiadm -m discovery -t st -p 192.168.96.14
   iscsiadm -m discovery -t st -p 192.168.96.244
   iscsiadm -m node -p 192.168.96.14 --login
   iscsiadm -m node -p 192.168.96.244 --login
   iscsiadm -m session -P 3 | grep -E 'Portal|Attached'
   

2. Start NFS Block Layout Service

   # Load block layout driver
   modprobe blocklayoutdriver
   
   # Start and enable nfs-blkmap
   systemctl start nfs-blkmap.service
   systemctl enable nfs-blkmap.service
   systemctl status nfs-blkmap.service
   

3. Mount NFS Share

   # Create mount points
   mkdir -p /mnt/pnfs/dir{1,2}
   
   # Mount NFSv4 share
   mount -t nfs4 192.168.96.228:/ /mnt/pnfs/
   
   # Verify mount
   df -h
   

Verify pNFS Setup

CentOS Client (Issue Observed)

cat /proc/self/mountstats | grep -E 'READ|WRITE|LAYOUT|pnfs'

• dir1/dir2 show pnfs and LAYOUT_SCSI, but cp commands freeze.

Ubuntu Client (Works)

After switching to Ubuntu:

• cp succeeds, LAYOUT operations increase, and file access works.