Storage and Filesystem Management

Handling Disk Space Exhaustion

When the root partition runs out of space, basic shell features like tab-completion may fail. To resolve this, inspect large directories and remove unnecessary data.

# Check usage in temporary and user directories
cd /tmp && du -sh *
cd /home && sudo du -sh *

File and Directory Inspection

To list files with human-readable sizes, including hidden ones:

ls -alh

To determine directory sizes up to a specific depth:

du -h --max-depth=1
sudo du -sh /home/*

Partition Mounting and Repair

If a filesystem becomes read-only due to an improper shutdown, use fsck to repair it. Note the distinction between fsck.ext4 (checks/repairs existing filesystems) and mkfs.ext4 (creates a new filesystem, erasing data).

# Repair a corrupted ext4 partition
fsck.ext4 -y /dev/sdXN

# Remount a disk to a new directory
sudo umount /dev/sdXN
sudo mkdir /mnt/data
sudo mount /dev/sdXN /mnt/data
sudo chmod -R 777 /mnt/data

Disk Performance Benchmarking

For read speed, use hdparm. For write speed, use dd.

# Read test
sudo hdparm -Tt /dev/sda

# Write test
time dd if=/dev/zero of=/tmp/testfile bs=1G count=2 oflag=dsync

Resolving File Operation Errors

To delete files starting with a dash (-), prefix with ./ or use --:

rm ./-example_file.txt
rm -- -example_file.txt

To remove stubborn .nfs files reporting "Device or resource busy", terminate the processes holding them using fuser:

fuser -k .nfs000000000example

File Compression and Archiving

Format	Compress	Extract
.tar.gz	tar -czvf archive.tar.gz dir/	tar -xzvf archive.tar.gz -C target/
.tar	tar -cvf archive.tar dir/	tar -xvf archive.tar
.zip	zip -r archive.zip dir/	unzip archive.zip
.gz	gzip -c file > file.gz	gunzip -c file.gz > file_new
.tar.zst	tar -I zstd -cvf archive.tar.zst dir/	tar -I zstd -xvf archive.tar.zst

Process and Resource Monitoring

System Metrics

# CPU details
lscpu
cat /proc/cpuinfo

# Memory usage (MB or GB)
free -m

# Disk usage
df -h  # Overall
du -h  # Current directory

# GPU hardware
lspci | grep -i vga
lspci | grep -i nvidia

Process Management

Identify running processes using ps, pstree, or netstat, then terminate them gracefully or forcefully.

# Find a specific process
ps aux | grep target_process

# Terminate processes
kill -15 <PID>   # Graceful
kill -9 <PID>    # Forceful
pkill target_process
killall target_process

Within top or htop, press Shift+P to sort by CPU and Shift+M to sort by memory utilization.

Managing Zombie Processes

Zombie processes (Z state) cannot be killed directly because they are already dead; the parent process must read their exit status. Killing the parent process clears the zombie entry from the process table.

# Force kill parents of zombie processes (use with extreme caution)
kill -9 $(ps -A -ostat,ppid | awk '/[zZ]/ && !seen[$2]++ {print $2}')

Clearing Stuck Semaphores

Improperly terminated database processes can leave semaphores allocated. Clear them by user:

ipcs -s | grep username | awk '{print $2}' | xargs -n 1 ipcrm -s

Memory Analysis: VIRT, RES, and SHR

• VIRT (Virtual Size): The total memory space the process can access, including swapped out and allocated but unused memory.
• RES (Resident Size): The actual physical RAM consumed by the process.
• SHR (Shared): Memory shared with other processes (e.g., shared libraries). Private memory is roughly RES - SHR.

For deep memory analysis, parse /proc/<pid>/smaps. Below is a C++ implementation that aggregates memory metrics from the smaps file:

#include <iostream>
#include <fstream>
#include <string>
#include <unordered_map>
#include <sstream>

class ProcSmapsReader {
public:
    static std::unordered_map<std::string, unsigned long> aggregateStats(const std::string& pid) {
        std::string filePath = "/proc/" + pid + "/smaps";
        std::unordered_map<std::string, unsigned long> metrics = {
            {"Size", 0}, {"Rss", 0}, {"Pss", 0}, {"Shared_Clean", 0},
            {"Shared_Dirty", 0}, {"Private_Clean", 0}, {"Private_Dirty", 0}, {"Swap", 0}
        };

        std::ifstream file(filePath);
        std::string line;
        while (std::getline(file, line)) {
            for (auto& pair : metrics) {
                if (line.find(pair.first + ":") == 0) {
                    std::istringstream iss(line.substr(pair.first.length() + 1));
                    unsigned long val;
                    iss >> val;
                    pair.second += val;
                    break;
                }
            }
        }
        return metrics;
    }
};

int main() {
    std::string myPid = std::to_string(getpid());
    auto stats = ProcSmapsReader::aggregateStats(myPid);
    for (const auto& entry : stats) {
        std::cout << entry.first << ": " << entry.second << " kB\n";
    }
    return 0;
}

Shell Productivity and Scripting

Parallel Execution in Bash

To process loop iterations concurrently, append & to spawn background tasks, followed by wait to pause until all complete.

#!/bin/bash
START_EPOCH=$(date +%s)
ITER_COUNT=500
counter=1

while [ $counter -le $ITER_COUNT ]
do
    (
        sleep 0.5
        echo "Task $counter finalized"
    ) &
    counter=$((counter + 1))
done

wait
END_EPOCH=$(date +%s)
echo "Total Duration: $((END_EPOCH - START_EPOCH)) seconds"

Persistent Background Processes

To keep scripts running after closing the terminal, use nohup and &, or disown the process.

nohup ./execute_task.sh > output.log 2>&1 &
# Alternatively
./execute_task.sh &
disown

Terminal Keyboard Shortcuts

Shortcut	Action
Ctrl + C	Terminate foreground process
Ctrl + Z	Suspend foreground process (resume with fg or bg)
Ctrl + D	Logout/EOF
Ctrl + S	Pause terminal output
Ctrl + Q	Resume terminal output
Ctrl + Shift + T	Open new terminal tab
Alt + [1-9]	Switch terminal tabs

Terminal Cursor and Prompt Customization

Restore a missing terminal cursor:

echo -e "\033[?25h" # Show cursor
echo -e "\033[?25l" # Hide cursor

Modify the PS1 environment variable to change the prompt format and color. Add to ~/.bashrc for persistence.

# Green prompt showing user, host, and full working directory
export PS1='\[\e[32;40m\][\u@\h:$PWD]\$\[\e[0m\] '

Text Processing

Searching with Grep

# Find lines starting with "error" (case-insensitive)
grep -i "^error" application.log

Manipulating with Sed

# Delete first and last lines
sed -i '1d' app.log
sed -i '$d' app.log

# Delete lines 5 through 10
sed -i '5,10d' app.log

# Replace all instances of 'old_val' with 'new_val'
sed -i 's/old_val/new_val/g' app.log

# Append text after line 3
sed -i '3a\INSERTED_LINE' app.log

Writting with Echo

# Overwrite file
echo "Initialization complete" > status.txt

# Append to file
echo "New entry" >> status.txt

# Write text containing quotes
echo "User \"admin\" logged in" >> status.txt

Vim Essentials

Normal mode operations for text manipulation:

• Deletion: dd (line), ndd (n lines), dw (word), d$ (to end of line), dG (to end of file).
• Yank (Copy): yy (line), nyy (n lines), y$ (to end of line).
• Paste: p (after cursor), P (before cursor).
• Navigation: Ctrl+f (page down), Ctrl+b (page up).

Network and Remote Access

SSH Key-Based Authentication

1. Generate a key pair locally: ssh-keygen -t rsa -b 4096
2. Transfer the public key to the server: scp ~/.ssh/id_rsa.pub user@host:~/
3. Append the key to authorized hosts on the server: ``` mkdir -p ~/.ssh cat ~/id_rsa.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys

Secure File Transfer

For large files, rsync is preferred over scp because it supports resuming interrupted transfers.

# SCP transfer
scp -i ~/.ssh/my_key data.tar.gz user@host:/path/

# Rsync transfer (resumable)
rsync --partial --progress -e ssh data.tar.gz user@host:/path/

Negotiating Specific SSH Keys and Ports

ssh -i /path/to/private_key -p 2222 admin@192.168.1.50

Preventing SSH Timeouts

Append keep-alive settings to /etc/ssh/sshd_config:

ClientAliveInterval 60
ClientAliveCountMax 10

Then reload the SSH daemon: sudo systemctl restart ssh

System Configuration

User Management and Sudo

Add a user to the sudo group using usermod:

usermod -aG sudo username

To allow a user to run sudo commands without a password, run sudo visudo and add:

username ALL=(ALL) NOPASSWD:ALL

Increasing Open File Descriptors

Applications may require more than the default 1024 limit. Edit /etc/security/limits.conf:

username soft nofile 65534
username hard nofile 65534

Ensure the user logs out and back in for changes to apply. Verify with ulimit -n.

Resolving /tmp Space Shortages During Compilation

If the default /tmp partition lacks space for intermediate build artifacts, redirect temporary file creation:

mkdir -p /home/username/custom_tmp
export TMPDIR=/home/username/custom_tmp

Disabling Swap Permanently

Comment out the swap entry in /etc/fstab, then run sudo swapoff -a.

Setting Timezone

timedatectl set-timezone Asia/Shanghai

File Permissions

Use symbolic or octal notation to modify permissions. For recursive changes on directories, use -R.

# Symbolic
chmod u+rw script.sh
chmod -R 777 /var/log/app/

# Octal (User=RWE, Group=RWE, Other=RE)
chmod 775 script.sh