Real-Time Directory Synchronization

In production environments, maintaining consistent file states across multiple servers is critical. A common requirement involves continuously mirroring changes from a primary directory—such as an NFS export—to a designated path on a backup server.

Core Synchronization Approach

Two widely adopted methods enable this behavior:

• inotifywait + rsync: A lightweight, script-driven pipeline where filesystem events trigger targeted rsync transfers.
• sersync: An enhanced, daemon-based tool built atop inotify, offering improved reliability and built-in failover handling.

Workflow Overview:

1. inotifywait monitors the source directory for relevant filesystem events.
2. Upon detection, it emits structured event metadata to a pipe or script.
3. That output drives an rsync command that synchronizes only the affected content to the remote destination.

Kernel-Level inotify Tuning

Linux kernels ≥ 2.6.13 support inotify natively. For sustained high-volume monitoring, adjust these /proc/sys/fs/inotify/ parameters:

• max_queued_events: Maximum number of unprocessed events (default: 16384). Increase to prevent Event Queue Overflow.
• max_user_watches: Total files/directories one user may monitor (default: 8192). Raise for deep or wide directory trees.
• max_user_instances: Max concurrent inotify handles per user (default: 128).

Apply persistent settings via /etc/sysctl.conf:

fs.inotify.max_queued_events = 66666
fs.inotify.max_user_watches = 100000

Verify with:

cat /proc/sys/fs/inotify/{max_queued_events,max_user_watches,max_user_instances}

inotify-tools Command Suite

Install inotify-tools (typically available via EPEL):

• inotifywait: Blocks until filesystem events occur; ideal for scripting synchronization triggers.
• inotifywatch: Aggregates event counts over time (less common in sync pipelines).

Key inotifywait Options

Time and Format Tokens

--timefmt supports:

• %Y: 4-digit year
• %m: Month (01–12)
• %d: Day of month (01–31)
• %H: Hour (00–23)
• %M: Minute (00–59)
• %S: Second (00–60)

--format tokens include:

• %T: Formatted timestamp (requires --timefmt)
• %w: Watched directory path
• %f: Filename triggering the event
• %e: Comma-separated list of event types
• %Xe: Events joined by custom delimiter X

Common Event Types

Practical Monitoring Examples

Monitor /data once (exits after first event):

inotifywait /data

Monitor recursively and continuously:

inotifywait -mr /data

Log formatted events with timestamps and semicolon-separated events:

inotifywait -mr --timefmt "%Y-%m-%d %H:%M:%S" \
             --format "%T %w%f event: %;e" \
             -e create,delete,moved_to,close_write,attrib \
             /data

rsync Fundamentals

rsync provides efficient, delta-based synchronization. Its syntax is:

rsync [OPTIONS] SOURCE... DESTINATION

Basic usage examplse:

# Copy entire /etc tree (including directory itself)
rsync -av /etc 192.168.232.20:/opt/

# Copy only contents of /etc (no /etc/ wrapper)
rsync -av /etc/ 192.168.232.20:/opt/

# Enforce exact mirror: delete extraneous files on destination
rsync -av --delete /data/ 192.168.232.20:/opt/

Essential rsync Flags

Transfer Modes

• Local Mode: rsync -a /src /dst — no network involved.
• SSH Mode: rsync -avz /src user@host:/dst — encrypted, no daemon required.
• Daemon Mode: rsync -avz /src rsync://user@host/module/path — requires rsyncd running; uses port 873 by default.

Daemon Configuration (Server Side)

Sample /etc/rsyncd.conf:

uid = root
gid = root
max connections = 0
ignore errors
exclude = lost+found/
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
reverse lookup = no
hosts allow = 10.0.0.0/24

[backup]
path = /data/
comment = Production data mirror
read only = no
auth users = rsyncuser
secrets file = /etc/rsync.pass

Set strict permissions on credential file:

chmod 600 /etc/rsync.pass

Content of /etc/rsync.pass (one line: username:password):

rsyncuser:securePass123

Test connectivity:

rsync --password-file=/etc/rsync.pass rsync://rsyncuser@192.168.232.20/backup

Automated Real-Time Sync Script

On the source host, deploy this robust sync loop:

#!/bin/bash
SOURCE_DIR="/data/"
RSYNC_TARGET="rsyncuser@192.168.232.20::backup"
LOG_FILE="/var/log/changelist.log"
PASS_FILE="/etc/rsync.pass"

inotifywait -mrq \
  --timefmt "%Y-%m-%d %H:%M:%S" \
  --format "%T %w %f" \
  -e create,delete,moved_to,close_write,attrib \
  "$SOURCE_DIR" | while IFS= read -r TIMESTAMP DIR FILE; do
    FULL_PATH="${DIR}${FILE}"
    if rsync -az --delete --password-file="$PASS_FILE" "$SOURCE_DIR" "$RSYNC_TARGET"; then
        echo "[$TIMESTAMP] Synced $FULL_PATH" >> "$LOG_FILE"
    else
        echo "[$TIMESTAMP] Sync failed for $FULL_PATH" >> "$LOG_FILE"
    fi
done

Ensure initial state consistency before launching the script. Run it as a background service using systemd, screen, or nohup.