Environment Setup

Package Installation

Download RPM packages:

http://bits.gluster.org/pub/gluster/glusterfs/3.4.2/x86_64/

Required packages:
glusterfs-3.4.2-1.el6.x86_64.rpm
glusterfs-api-3.4.2-1.el6.x86_64.rpm
glusterfs-cli-3.4.2-1.el6.x86_64.rpm
glusterfs-fuse-3.4.2-1.el6.x86_64.rpm
glusterfs-libs-3.4.2-1.el6.x86_64.rpm
glusterfs-server-3.4.2-1.el6.x86_64.rpm

Dependencies:
yum install rpcbind libaio lvm2-devel

Server Installation

Install server packages:
rpm -ivh glusterfs-*.rpm

Client packages:
rpm -ivh glusterfs-3.4.2-1.el6.x86_64.rpm \
         glusterfs-libs-3.4.2-1.el6.x86_64.rpm \
         glusterfs-fuse-3.4.2-1.el6.x86_64.rpm

Utility Installation

yum install sysstat atop iperf
gcc -o postmark postmark-1.52.c

System Configuration

Hostname setup:
/etc/hosts
192.168.1.101 server1
192.168.1.102 server2
192.168.1.103 server3

Disable firewall:
service iptables stop
chkconfig iptables off

SELinux configuration:
/etc/selinux/config
SELINUX=disabled

Disk partitioning:
parted /dev/sdb mklabel gpt
mkfs.ext4 -L /brick1 /dev/sdb1

Service Configuration

Auto-mount setup:
/etc/fstab
LABEL=/brick1 /brick1 ext4 defaults 1 1

Start Gluster service:
service glusterd start
chkconfig glusterd on

Volume Operations

Peer management:

gluster peer probe server2
gluster peer status

gluster peer detach server3

Volume creation:

gluster volume create testvol server1:/brick1/v1
gluster volume start testvol
gluster volume info

mount -t glusterfs server1:/testvol /mnt

Volume expansion:

gluster volume add-brick testvol server2:/brick1/v2
gluster volume rebalance testvol start

Volume removal:

umount /mnt
gluster volume stop testvol
gluster volume delete testvol
rm -rf /brick1/v*

Replicated Volume

gluster volume create repvol replica 2 server1:/brick1/r1 server2:/brick1/r2
gluster volume start repvol

Tuning Parameters

gluster volume set testvol performance.cache-size 1GB
gluster volume set testvol network.ping-timeout 30
gluster volume set testvol nfs.disable on

Troubleshooting

Check service status:

service glusterd status
ps -ef | grep gluster

Log locaitons:

/var/log/glusterfs/
/var/lib/glusterd/

Force data repair:

find /mnt -type f -print0 | xargs -0 head -c1 >/dev/null

Performance Tuning

Recommended settinsg:

echo deadline > /sys/block/sda/queue/scheduler
echo 4096 > /sys/block/sda/queue/read_ahead_kb

XFS options:
mkfs.xfs -i size=512 -n size=8192 -d su=256k,sw=10 /dev/sdb1