Troubleshooting Case: MongoDB Service Fails to Start After Reboot

After a server reboot, mongod failed to start automatically despite being enabled with systemctl enable mongod. The systemd journal reported:

ExecStart=/usr/local/mongodb/bin/mongod -f /etc/mongod.conf (code=exited, status=48)

The MongoDB logs showed:

failed Cannot assign requested address for socket

This indicated a network-related issue. The configuration file contained the correct local IP address, and since this occurred after a reboot, the problem was that the network was not ready when mongod.service started. The solution was to add network dependency to the service file:

After=network-online.target
Wants=network-online.target

Essential systemctl Commands

List all running services:

systemctl list-units --type=service

Reload daemon configuration:

systemctl daemon-reload

Enable a service to start at boot:

systemctl enable <service>

Start, stop, restart, and check status:

systemctl start <service>
systemctl stop <service>
systemctl restart <service>
systemctl status <service>

Service Unit File Directories

Understanding where systemd looks for unit files:

Directory	Purpose
/lib/systemd/system	System default unit files
/usr/lib/systemd/system	User-defined unit files
/etc/systemd/system	Unit files for installed software

The boot-time startup symlinks are stored in:

ls /etc/systemd/system/multi-user.target.wants/

When you run systemctl enable, systemd creates a symlink in this directory pointing to the actual unit file.

Timer Units for Scheduled Tasks

Timers are systemd's solution for scheduled jobs, providing an alternative to cron. Create two paired files in /usr/lib/systemd/system/:

1. A .service file that defines the work to be performed
2. A .timer file that controls when the service runs

Timer Configuration Options

Time-based setings:

• OnActiveSec - Time after the timer activates to trigger execution
• OnBootSec - Time after system boot to trigger execution
• OnStartupSec - Time after systemd manager starts to trigger execution
• OnUnitActiveSec - Time after the last activation of the unit
• OnUnitInactiveSec - Time after the unit became inactive
• OnCalendar - Real-time calendar-based scheduling (like cron)
• AccuracySec - Maximum delay before execution (default 60 seconds). Use AccuracySec=1us for microsecond precision

Additional timer options:

• Unit - The service unit to execute (defaults to the same name with .service extension)
• Persistent - Execute missed runs if the timer was inactive
• WakeSystem - Wake system from suspend when timer fires

Supported Time Formats

usec, us, µs    - microseconds
msec, ms        - milliseconds
seconds, second, sec, s
minutes, minute, min, m
hours, hour, hr, h
days, day, d
weeks, week, w
months, month, M (30.44 days)
years, year, y (365.25 days)

Example: Minute-by-Minute Log Cleanup

clean-var-log.timer:

[Unit]
Description=/var/log/message clean timer
Requires=clean-var-log.service

[Timer]
Unit=clean-var-log.service
OnCalendar=*-*-* *:*:00

[Install]
WantedBy=timers.target

clean-var-log.service:

[Unit]
Description=/var/log/message clean service
Wants=clean-var-log.timer

[Service]
Type=oneshot
ExecStart=/usr/bin/sh /root/script/clean-var-log.sh
ExecStart=/usr/bin/ls /var/log

[Install]
WantedBy=multi-user.target

This executes the cleanup script every minute. Multiple ExecStart directives run sequentially.

Timer Management Commands

systemctl daemon-reload
systemctl enable timerecord.timer
systemctl start timerecord.timer
systemctl stop timerecord.timer
systemctl status timerecord.timer
systemctl restart timerecord.timer
systemctl list-timers --all

# View timer and service logs
journalctl -u timerecord.timer
journalctl -u timerecord.service
journalctl -u timerecord
journalctl -f -u timerecord.timer

Running Multiple Instances of a Service

To run multiple instances of a service (like multiple database processes), create a template unit file with @ in the name: name@.service

ls /usr/lib/systemd/system | grep cspost
cspost.service      # Single instance
cspost@.service     # Template for multiple instances

Start instances by specifying names after the @:

for i in {1..4}; do
    sudo systemctl start cspost@post$i
    sudo systemctl status cspost@post$i
done

Enable all instances for boot:

for i in {1..4}; do
    sudo systemctl enable cspost@post$i
done

Check individual instance status:

systemctl status cspost@post2

Use specifiers like %i within the template file to reference the instance identifier. See systemd.unit(5) for complete specifier documentation.

Cleaning Up Orphaned Unit Cache

After deleting a unit file physically, systemd may still show it with "not-found" status. To clean the cache:

systemctl reset-failed

This removes units marked as missing after a daemon-reload.

Service Resource Limits

Limit memory usage per service in the [Service] section:

MemoryLimit=1G
WorkingDirectory=/path/to/directory

Note: Memory limit implementation varies across distributions (CentOS 7 vs later versions).

Journal Logging Configuration

Configure journald in /etc/systemd/journald.conf:

[Journal]
Storage=persistent    # Disk storage in /var/log/journal
# Storage=volatile    # Memory only in /run/log/journal
# Storage=auto        # Persistent if /var/log/journal exists
# Storage=none       # Discard all logs

Verify configuration:

systemctl restart systemd-journald.service
journalctl --verify

journalctl Command Reference

Basic viewing:

journalctl                    # All logs (current boot only)
journalctl -r                 # Reverse order, use arrow keys
journalctl -k                 # Kernel messages only
journalctl -n 20              # Last 20 lines
journalctl -f                 # Real-time tail

Time-based filtering:

journalctl --since "2012-10-30 18:17:16"
journalctl --since "20 min ago"
journalctl --since yesterday
journalctl --since today
journalctl --since "2015-01-10" --until "2015-01-11 03:00"
journalctl --since 09:00 --until "1 hour ago"

Priority filtering (0-7):

0: emerg    4: warning
1: alert    5: notice
2: crit     6: info
3: err      7: debug

journalctl -p err

Unit-specific logs:

journalctl -u nginx.service
journalctl -u nginx.service -u php-fpm.service
journalctl /usr/lib/systemd/nginx.service

Boot-based filtering:

journalctl -b -0              # Current boot
journalctl -b -1              # Previous boot
journalctl -b -2              # Two boots ago

Process filtering:

journalctl _PID=1
journalctl _UID=0
journalctl _SYSTEMD_UNIT=sshd.service
journalctl _EXE=/usr/sbin/sshd
journalctl _COMM=systemd
# Combine filters
journalctl _SYSTEMD_UNIT=sshd.service _PID=1182

Output formats:

journalctl -o short            # Default syslog format
journalctl -o short-iso       # ISO 8601 timestamps
journalctl -o short-precise   # Microsecond precision
journalctl -o verbose         # All fields
journalctl -o json            # JSON single line
journalctl -o json-pretty     # JSON formatted
journalctl -o cat            # Message text only

Maintenance:

journalctl --vacuum-time=1years
journalctl --vacuum-size=1G
journalctl --disk-usage

Service Configuration File Structure

[Unit] Section

Defines metadata and ordering:

[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.service
Before=other.service
Wants=sshd-keygen.service
Requires=other.service

• Description - Brief service description
• Documentation - Manual page links
• After - Start after listed units
• Before - Start before listed units
• Wants - Weak dependency (continues if dependancy fails)
• Requires - Strong dependency (fails if dependency fails)

[Service] Section

Defines execution behavior:

Environment and execution:

[Service]
EnvironmentFile=/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/sbin/sshd -k
ExecStartPre=/usr/bin/check-config
ExecStartPost=/usr/bin/post-action
ExecStopPost=/usr/bin/cleanup

Prepend - to any command to ignore errors: EnvironmentFile=-/path/to/file

Type values:

Type	Behavior
simple	Default, ExecStart is main process
forking	Parent exitss, child becomes main process
oneshot	Runs once, completes before next service starts
dbus	Waits for D-Bus bus name before continuing
notify	Waits for notification signal before continuing
idle	Waits for all other tasks to complete first

Restart behavior:

[Service]
Type=simple
Restart=on-failure
RestartSec=42s
KillMode=process

KillMode values:

• control-group - Kill all processes in the control group
• process - Kill only the main process
• mixed - SIGTERM to main, SIGKILL to children
• none - No processes killed, only ExecStop runs

Restart values:

• no - Never restart
• on-success - Restart on clean exit (code 0)
• on-failure - Restart on abnormal exit
• on-abnormal - Restart on signal or timeout
• on-abort - Restart on uncaught signal
• on-watchdog - Restart on watchdog timeout
• always - Always restart

For daemons, use on-failure. For services that may intentionally fail, use on-abnormal.

RemainAfterExit=yes - Keeps service active after ExecStart completes (for oneshot services with stop commands).

[Install] Section

Defines boot-time behavior:

[Install]
WantedBy=multi-user.target

WantedBy specifies which Target pulls in this service. Common targets:

• multi-user.target - Command-line multi-user mode
• graphical.target - Graphical interface (depends on multi-user.target)
• timers.target - For timer units

Target Management

Check default target:

systemctl get-default

Switch targets:

sudo systemctl isolate shutdown.target

View target configuration:

systemctl cat multi-user.target

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes

List services in a target:

systemctl list-dependencies multi-user.target

Applying Configuration Changes

After modifying any unit file:

sudo systemctl daemon-reload
sudo systemctl restart <service>

References

• systemd.unit(5)
• systemd.service(5)
• systemd.timer(5)
• systemd.kill(5)
• journalctl(1)