Comprehensive Infrastructure Monitoring with Zabbix

Infrastructure Monitoring Service Overview

Monitoring Service Benefits

  1. Continuous real-time system monitoring
    • Implement alert notifications via email/WeChat/SMS/voice calls
  2. Real-time feedback on current system status
  3. Ensure service reliability and security
  4. Maintain stable and continuous business operations

Essential Monitoring Commands for System Components

CPU: top, htop, glances

Metrics to monitor:

  • us: User space processes (40%)
  • sy: Kernel space processes (40%) - MySQL processes
  • id: Idle time (20%)

Memory: top, htop, free

Metrics to monitor:

  • Available memory percentage
  • Swap space utilization

Disk: df, iotop, glances

yum install iotop
iotop


df -h

Metrics to monitor:

  • Disk usage statistics
  • Disk I/O consumption

Network: iftop, glances

iftop

Metrics to monitor:

  • Network bandwidth utilization

Processes: top, htop, ps, glances

top

Metrics to monitor:

  • Memory consumption per process - Tomcat (Java) applications filling memory (memory overflow) causing service hang (requiring restart)
  • CPU utilization per process - MySQL

Load: w, top, uptime, glances

Metrics to monitor:

  • 10-minute load average (should be less than CPU core count)
  • 15-minute load average

Implementing Server Monitoring with Shell Scripts

Memory monitoring: Check every minute, trigger email alert when available memory drops below 100MB, displaying remaining memory value

#!/bin/bash
while true
do 
  available_mem=$(free -m | awk 'NR==2{print $NF}')  
  if [ $available_mem -lt 100 ]
  then 
     echo $available_mem | mail -s "Memory Alert" admin@example.com
  fi
  sleep 60
done

ab -n 10000 -c 3 http://10.0.0.100/zabbix/index.php

Zabbix Monitoring Service Deployment

Official documentation

https://www.zabbix.com/documentation

Zabbix software components:

  • zabbix-server: Monitoring service core
  • zabbix-agent: Monitoring client
  • zabbix-web: Web interface service
  • php: Dynamic request processing
  • mysql: Database for storing monitoring data
  • zabbix-proxy: Collects agent information and forwards to zabbix-server

Zabbix installation options:

  • Recommended version: 4.0 LTS (Long Term Support)

Step 1: Configure YUM Repository

Using Tsinghua mirror:

http://repo.zabbix.com/zabbix/4.0/rhel/7/x86_64/


wget http://repo.zabbix.com/zabbix/4.0/rhel/7/x86_64/zabbix-release-4.0-1.el7.noarch.rpm
rpm -ivh zabbix-release-4.0-1.el7.noarch.rpm
cat /etc/yum.repos.d/zabbix.repo

Step 2: Install Zabbix Server Components

yum install deltarpm -y
yum install -y zabbix-server-mysql zabbix-web-mysql httpd php mariadb-server

Step 3: Software Configuration

Edit /etc/zabbix/zabbix_server.conf

DBPassword=zabbix

Set timezone in /etc/httpd/conf.d/zabbix.conf

php_value date.timezone Asia/Shanghai

Step 4: Database Configuration

Set MySQL root password

systemctl start mariadb.service 
systemctl status mariadb.service
mysql_secure_installation

Create zabbix database and user

CREATE DATABASE zabbix CHARACTER SET utf8 COLLATE utf8_bin;
GRANT ALL PRIVILEGES ON zabbix.* TO zabbix@localhost IDENTIFIED BY 'zabbix';

Import schema to zabbix database

zcat /usr/share/doc/zabbix-server-mysql-4.0.15/create.sql.gz | mysql -uzabbix -pzabbix zabbix

Step 5: Start Zabbix Services

systemctl start zabbix-server 
systemctl enable zabbix-server

systemctl start httpd
systemctl enable httpd

Step 6: Access Zabbix Web Interface for Initial Configuration

http://10.0.0.7/zabbix/setup.php

Step 7: Login to Zabbix Web Interface

Username: Admin
Password: zabbix

Configure language settings to Chinese if needed

Monitoring a Server Host

You may notice alerts about unmonitored hosts

Adding Hosts to Monitor

Method 1: Monitor the Zabbix server itself

yum install -y zabbix-agent
systemctl start zabbix-agent.service
systemctl enable zabbix-agent.service

Method 2: Monitor other machines

Install zabbix-agent

rpm -ivh https://mirror.tuna.tsinghua.edu.cn/zabbix/zabbix/4.0/rhel/7/x86_64/zabbix-agent-4.0.11-1.el7.x86_64.rpm

Configure zabbix-agent

vim /etc/zabbix/zabbix_agentd.conf
Server=10.0.0.7

Start zabbix-agent

systemctl start zabbix-agent
systemctl enable zabbix-agent

Verify zabbix monitoring connection

netstat -lntup | grep 10050
    tcp      0     0 0.0.0.0:10050           0.0.0.0:*            LISTEN      4509/zabbix_agentd  
    tcp      0     0 :::10050                :::*                 LISTEN      4509/zabbix_agentd

Add host through Zabbix web interface

Implementing Custom Monitoring in Zabbix

Items: Custom metrics to collect host information

Applications: Groups of similar monitoring items for organized viewing

Templates: Collections of items, triggers, and graphs that can be applied to multiple hosts

Actions: Define alert recipients, message formats, and notification methods (email, WeChat, SMS, voice)

Macro definitions:

https://www.zabbix.com/documentation/4.0/en/manual/appendix/macros/supported_by_location

Triggers: Conditional expressions that generate alerts

Graphs: Visual representations combining multiple metrics for analysis

Media Types: Define notification methods

Simple Custom Monitoring Configuration

Requirement: Monitor if nginx service is running

Step 1: Create custom monitoring command

ps -ef | grep -c [n]ginx

Step 2: Configure zabbix-agent

Syntax format:

  • UserParameter=key,command

Method 1: Directly modify zabbix-agent configuration file /etc/zabbix/zabbix_agentd.conf

UserParameter=nginx.status,ps -ef | grep -c [n]ginx

Method 2: Create web_server.conf in /etc/zabbix/zabbix_agentd.d/ directory (recommended)

UserParameter=nginx.status,ps -ef | grep -c [n]ginx

Step 3: Restart zabbix-agent service

systemctl restart zabbix-agent

Step 4: Verify on zabbix-server

yum install -y zabbix-get
zabbix_get -s 10.104.204.21 -k 'nginx.status'

Step 5: Configure in Zabbix web interface

Navigate to create item page

Configure item parameters:

  • Name: Nginx Service Status
  • Key: nginx.status
  • Update interval: 30s
  • Application: Web Services

Verify data collection

Advanced Custom Monitoring Configuraton (Multiple Services)

(1) Create configuration file on zabbix-agent: /etc/zabbix/zabbix_agentd.d/service_status.conf

UserParameter=service.status[*],netstat -lntup | grep -c $1

(2) Restart agent

systemctl restart zabbix-agent

(3) Test from zabbix-server

zabbix_get -s 10.104.204.21 -k 'service.status[22]'

(4) Configure in web interface

Key: service.status[22]

Implementing Zabbix Alerting - Triggers and Actions

Web Interface and Sound Alerts

Step 1: Create Trigger

Expression functions:

  • last(): Latest collected value
  • max(): Maximum value in a time period
  • min(): Minimum value in a time period
  • diff(): Checks if values changed over time
  • change(): Detects value changes
  • avg(): Average value over time period

Step 2: Configure web interface for sound alerts

User profile → Message notifications → Enable frontend message notifications

Test by stopping nginx on agent:

systemctl stop nginx

Email Alerting

Step 1: Create Trigger

Same as web interface alerts

Step 2: Configure Action

Enable default action

Step 3: Configure Email Settings

Set up SMTP server connection (using 163 Mail example)

Step 4: Define Alert Recipients

Add email addresses for notification recipients

Test by stopping nginx and checking received email alert

Test recovery by starting nginx and checking recovery email

Tags: Zabbix monitoring infrastructure Linux system-administration

Posted on Sat, 26 Sep 2026 16:18:26 +0000 by stevenrhiggins

Hot Tags

©2026 Freaks City