File System Selection for Scalable Storage

In modern production environments, while NFS offers simplicity for shared storage, its limitations become evident as data scales exponentially. Distributed file systems that support horizontal scalability, high capacity, and fault tolerance are increasingly necessary. Common options include MogileFS, FastDFS, MooseFS, GlusterFS, Ceph, Lustre, TFS, GFS, and HDFS.

• MogileFS: A key-value metadata system without FUSE support; requires API access. Optimized for serving massive numbers of small files (e.g., web images).
• FastDFS: A Chinese-developed improvement over MogileFS with better performance but still lacks FUSE.
• MooseFS: Lightweight, supports FUSE, enabling standard POSIX operations. Suffers from a single-point-of-failure master (mitigatable via DRBD).
• GlusterFS: Stackable architecture with no dedicated metadata server; excels at large-file workloads but struggles with small-file I/O due to local filesystem bottlenecks.
• Ceph: Fully distributed with kernel integration (since Linux 2.6.34), but historically unstable due to reliance on immature Btrfs components.
• Lustre: Enterprise-grade but complex and kernel-dependent, requiring custom kernel builds.

Key-value systems lack directory semantics, forcing API usage for basic operations—unappealing for general-purpose use. Among FUSE-compatible options, Ceph and Lustre are often ruled out for stability or complexity. GlusterFS suits virtualization and large files but underperforms with small files. MooseFS emerges as a balanced choice: simple, POSIX-compliant, and sufficiently robust for many use cases, especially where community support and documentation are valued.

MooseFS Architecture Overview

Components

1. Master Server: Manages metadata, coordinates chunk servers, handles file operations, and enforces replication policies.
2. Metalogger Server: Backs up metadata change logs (changelog_ml.*.mfs) for disaster recovery.
3. Chunk Servers: Store actual file data in chunks, replicate based on configured goals, and serve client I/O.
4. Clients: Mount the filesystem via FUSE, interacting as with a local filesystem.

Advantages

• Simple deployment and maintenance
• Online scalability without service interruption
• POSIX compatibility—no application modifications needed
• High disk utilization and configurable trash retention
• Load distribution across all chunk servers
• Configurable replication (goal) per file/directory

Limitations

• Single master node (mitigated via DRBD + Pacemaker)
• Metadata server demands significant RAM

High-Availability MooseFS Deployment

Environment

OS: CentOS 7.5 x86_64
Master: 192.168.5.71
Metalogger: 192.168.5.72
VIP: 192.168.5.77
Chunk Servers: 192.168.5.73, 192.168.5.74, 192.168.5.75

Hosts Configuration

cat >> /etc/hosts << EOF
192.168.5.71 mfs71
192.168.5.72 mfs72
192.168.5.73 mfs73
192.168.5.74 mfs74
192.168.5.75 mfs75
EOF

Master Server Setup

useradd mfs -s /sbin/nologin
yum install -y gcc gcc-c++ zlib-devel
cd /opt
wget https://github.com/moosefs/moosefs/archive/v3.0.101.tar.gz
tar -zxvf v3.0.101.tar.gz
cd moosefs-3.0.101
./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs \
            --disable-mfschunkserver --disable-mfsmount
make && make install

cd /usr/local/mfs/etc/mfs
cp mfsmaster.cfg.sample mfsmaster.cfg
cp mfsexports.cfg.sample mfsexports.cfg
cp mfstopology.cfg.sample mfstopology.cfg

Edit mfsexports.cfg:

192.168.5.0/24 / rw,alldirs,maproot=0
192.168.5.0/24 . rw

Start services:

/usr/local/mfs/sbin/mfsmaster start
/usr/local/mfs/sbin/mfscgiserv start  # Web UI at http://192.168.5.71:9425

Chunk Server Configuration

Prepare storage (e.g., /mnt/mfsdata), then:

cd /usr/local/mfs/etc/mfs
cp mfschunkserver.cfg.sample mfschunkserver.cfg
echo "/mnt/mfsdata" > mfshdd.cfg
chown -R mfs:mfs /mnt/mfsdata

# In mfschunkserver.cfg:
MASTER_HOST = 192.168.5.71
MASTER_PORT = 9420

/usr/local/mfs/sbin/mfschunkserver start

Metalogger Setup

cd /usr/local/mfs/etc/mfs
cp mfsmetalogger.cfg.sample mfsmetalogger.cfg

# In mfsmetalogger.cfg:
MASTER_HOST = 192.168.5.71
MASTER_PORT = 9419
META_DOWNLOAD_FREQ = 1  # Hourly backups

/usr/local/mfs/sbin/mfsmetalogger start

Client Mounting

yum install -y fuse fuse-devel
modprobe fuse

mkdir -p /mnt/{mfs,mfsmeta}
/usr/local/mfs/bin/mfsmount /mnt/mfs -H 192.168.5.71
/usr/local/mfs/bin/mfsmount -m /mnt/mfsmeta -H 192.168.5.71

# Persistent mount via /etc/fstab:
/usr/local/mfs/bin/mfsmount /mnt/mfs fuse mfsmaster=192.168.5.71,mfsport=9421,_netdev 0 0

Core Feature Testing

Trash Bin Functionality

echo "test" > /mnt/mfs/file.txt
rm /mnt/mfs/file.txt
find /mnt/mfsmeta/trash -name "*file.txt*"  # e.g., /mnt/mfsmeta/trash/004/00000004|file.txt
cat '/mnt/mfsmeta/trash/004/00000004|file.txt'  # Note quotes due to '|'

# Set trash retention (seconds):
mfssettrashtime 3600 /mnt/mfs  # 1 hour
mfssettrashtime -r 0 /mnt/mfs   # Disable trash (immediate deletion)

Replication Control

mkdir /mnt/mfs/{test1,test2}
mfssetgoal 1 /mnt/mfs/test1      # New files: 1 copy
mfssetgoal -r 2 /mnt/mfs/test2   # Recursive: all contents → 2 copies

cp /etc/hosts /mnt/mfs/test1/
mfsfileinfo /mnt/mfs/test1/hosts  # Shows 1 replica
mfsfileinfo /mnt/mfs/test2/hosts  # Shows 2 replicas

Snapshot Operations

mfsmakesnapshot /mnt/mfs/test2 /mnt/mfs/test2-snap
mfsmakesnapshot /mnt/mfs/file.txt /mnt/mfs/file.txt-snap

# Modify original:
echo "new" >> /mnt/mfs/file.txt
mfsfileinfo /mnt/mfs/file.txt        # New chunk ID
mfsfileinfo /mnt/mfs/file.txt-snap   # Original chunk ID preserved

Disaster Recovery

If the master fails catastrophically:

1. Stop the master and remove corrupted metadata:

/usr/local/mfs/sbin/mfsmaster stop
rm -rf /usr/local/mfs/var/mfs/*

3. Copy latest backup from metalogger:

scp mfs72:/usr/local/mfs/var/mfs/metadata_ml.mfs.back \
      mfs72:/usr/local/mfs/var/mfs/changelog_ml.0.mfs \
      /usr/local/mfs/var/mfs/
chown -R mfs:mfs /usr/local/mfs/var/mfs

5. Recover and restart:

/usr/local/mfs/sbin/mfsmaster -a  # Auto-rebuilds metadata.mfs

Fault Tolerance Validation

With replication goal = 2:

• Stoping one chunk server: clients continue reading/writing using the remaining replica.
• Writing during outage: new files store on available servers; missing replicas sync once the server recovers (within REPLICATIONS_DELAY_DISCONNECT seconds).

Storage Expansion

Add new chunk servers by repeating the chunk server setup. MooseFS automatically redistributes data across all available nodes to balance storage load.