System logs are essential for understanding operating system behavior, troubleshooting issues, and analyzing performance. However, log files accumulated over time can grow significantly and consume substantial disk space, potentially impacting system performance. This article presents an automated solution using Bash scripting to monitor and maintain log files, ensuring they remain manageible in size.

Log files generated by ELK stack or other logging systems can expand rapidly. Without regular maintenance, these files may eventually fill the disk, causing system instability and service disruptions.

The Bash script continuously monitors a specified directory, checking each log file's size against a predefined threshold. When a file exceeds this limit, the script removes older entries to keep the file within acceptable bounds.

Cron Job Scheduling:

Configure automatic execution using crontab:

crontab -e
0 0 * * * /usr/local/scripts/cleanup_logs.sh

The script should run at intervals greater than the time required to process the smallest log file. Daily execution is recommended for most environments.

Implementation Script:

#!/bin/bash

# Target directory containing logs
log_directory="/var/log/application"

# Size threshold in bytes (500MB)
max_bytes=$((500*1024*1024))

while :; do
    within_limit=true
    
    log_files=("$log_directory"/*)
    
    for current_file in "${log_files[@]}"; do
        if [[ -f "$current_file" ]]; then
            file_bytes=$(stat -c%s "$current_file")
            
            if (( file_bytes >= max_bytes )); then
                # Remove approximately 86400 lines (1 day worth at ~1000 lines/sec)
                # Adjust line count based on actual log generation rate
                sed -i '1,86400d' "$current_file"
                within_limit=false
            fi
        fi
    done
    
    if $within_limit; then
        break
    fi
    
    # Check interval in seconds (1800 = 30 minutes)
    sleep 1800
done

The script iterates through all files in the specified directory, calculates each file's size in bytes, and truncates the oldest entries when the threshold is exceeded. The deletion amount (number of lines removed) should be calibrated based on the log generation rate to ensure adequate space is freed with each execution.