Zabbix Overview

Zabbix is a robust, open-source enterprise-class distributed monitoring solution. It provides comprehensive monitoring for networks, servers, applications, and services. While the software is freely available, its development is sustained by a dedicated international team, with commercial support offered as a service. Zabbix's web-based interface allows for centralized management and configuraton of monitoring tasks.

At its core, Zabbix comprises the Zabbix Server and Zabbix Agents. A Zabbix Proxy can also be optionally deployed to distribute the monitoring load. The Zabbix Server can gather data from remote systems and network devices through various methods, including SNMP, Zabbix Agents, and port checks. While Zabbix Server and Proxy run on Linux and Unix-like systems, Zabbix Agents are also compatible with Windows.

Key Zabbix Features

• Comprehensive Monitoring: Supports performance monitoring for servers, network devices, databases, and common protocols like FTP. It includes flexible alerting and detailed reporting with graphing capabilities.
• Automatic Discovery: Capable of automatically discovering network devices and servers based on configured rules. It also supports low-level discovery (LLD) for dynamic, large-scale monitoring of items, often requiring custom scripts.
• Distributed Monitoring: Enables centralized management of distributed monitoring points.
• Extensibility: Offers a powerful API for integration with other systems and custom development. Developing custom plugins and defining unique monitoring items and alert levels is straightforward.
• Data Collection: Supports SNMP (polling and trapping), IPMI, JMX, SSH, and Telnet. Custom checks can be configured with flexible collection frequencies.
• Flexible Triggers: Allows for the definition of complex problem thresholds, known as triggers, based on data collected from the backend database.
• Customizable Alerting: Provides a notification system with customizable escalation, recipients, and media types. Macros can be used for dynamic alert content.
• Real-time Graphing: Visualizes collected monitoring data in real-time graphs.
• Web Monitoring: Simulates user interactions with websites, checking return values and response times.
• API Functionality: The API enables seamless integration with various systems, including mobile clients.

Zabbix Versions

Zabbix offers several stable versions, including 3.0, 2.4, 2.2, and 2.0. For installation in this guide, Zabbix 3.0 is used with CentOS 7 via yum.

Zabbix Advantages and Disadvantages

Advantages:

• Open-source with no software licensing costs.
• Low hardware requirements for the Zabbix Server.
• Broad device support with numerous built-in monitoring templates.
• Supports distributed management and automatic discovery for streamlined monitoring.
• Highly extensible with an open API and easy plugin development.
• Passive monitoring mode reduces server load when dealing with many items.
• API facilitates integration with other systems.

Disadvantages:

• Requires agent installation on monitored hosts.
• Stores significant amounts of data in the database, making the data base a potential bottleneck.

Zabbix Monitoring Principle

Zabbix operates on a Client/Server (C/S) model for data collection and a Browser/Server (B/S) model for web-based configuration and display. Monitored endpoints typically run Zabbix Agents, while network devices can be monitored via SNMP. The Zabbix Server collects data from agents and SNMP devices, storing it in a database (e.g., MySQL, Oracle). This data is then presented through a web interface powered by Apache and PHP.

Zabbix System Requirements

• Server: Zabbix Server requires a LAMP (Linux, Apache, MySQL, PHP) or LNMP stack. Hardware requirements are generally low.
• Agent: Zabbix Agents are available for most common operating systems, including Linux, HPUX, Solaris, Sun, and Windows.
• SNMP: Supports a wide range of network devices using the Simple Network Management Protocol (SNMP).

Zabbix Environment Configuration

1. Environment Details

• Operating System: CentOS Linux release 7.2.1511 (Core)
• Kernel: 3.10.0-327.28.3.el7.x86_64

2. Installing Zabbix Yum Repository

To install Zabbix 3.0 using yum, first add the official Zabbix repository:

[root@localhost ~]# rpm -ivh http://mirrors.aliyun.com/zabbix/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm

This command creates a zabbix.repo file in /etc/yum.repos.d/, enabling yum to find Zabbix packages.

3. Installing Zabbix Packages

Install the necessary Zabbix server, web interface, and database packages. Note that on CentOS 7, MySQL is provided by MariaDB:

[root@localhost ~]# yum install zabbix-server zabbix-web zabbix-server-mysql zabbix-web-mysql mariadb-server mariadb -y
# If the server itself needs monitoring, also install zabbix-agent:
# [root@localhost ~]# yum install zabbix-agent -y

4. Configuring PHP Timezone

Modify the Zabbix Apache configuration file to set the PHP timezone:

[root@localhost ~]# sed -i 's@# php_value date.timezone Europe/Riga@php_value date.timezone Asia/Shanghai@g' /etc/httpd/conf.d/zabbix.conf

Database Setup

1. Starting the Database Service

[root@localhost ~]# systemctl start mariadb

2. Creating Zabbix Database and User

Connect to the MySQL/MariaDB server and create a dedicated database and user for Zabbix:

mysql
create database zabbix character set utf8 collate utf8_bin;
grant all on zabbix.* to zabbix@'localhost' identified by 'your_password';
exit

Replace 'your_password' with a strong password.

3. Importing Zabbix Database Schema

Import the Zabbix database structure. The SQL schema file is typically compressed;

[root@localhost ~]# cd /usr/share/doc/zabbix-server-mysql-3.0.4/ # Adjust version if necessary
[root@localhost zabbix-server-mysql-3.0.4]# zcat create.sql.gz | mysql -uzabbix -p zabbix
# Enter password when prompted

4. Configuring Zabbix Server

Edit the Zabbix server configuration file to specify database connection details:

# vim /etc/zabbix/zabbix_server.conf

DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=your_password

5. Starting Zabbix Server and Apache

[root@localhost ~]# systemctl start zabbix-server
[root@localhost ~]# systemctl start httpd

Ensure port 80 is not in use by another service.

6. Zabbix Web Interface Installation

Access the Zabbix web installer via your browser: http://your_server_ip/zabbix/setup.php

Follow the on-screen instructions, ensuring all prerequisites are met. Configure the database connection, set a name for your Zabbix server, and review the configuration summary before finishing.

Important: Change the default administrator password immediately after the initial setup.

7. Configuring Zabbix Agent

On any host you wish to monitor using Zabbix Agent, configure the agent file:

# vim /etc/zabbix/zabbix_agentd.conf

Server=your_zabbix_server_ip
ServerActive=your_zabbix_server_ip

Replace your_zabbix_server_ip with the actual IP address of your Zabbix server.

[root@linux-node1 ~]# systemctl start zabbix-agent

Verify the agent is listening on port 10050:

[root@localhost ~]# netstat -lntp | grep zabbix

Web Interface Configuration

Navigate to Administration -> Hosts to add a new host to monitor.

If monitoring fails, consult the Zabbix logs located in /var/log/zabbix/.

A green ZBX icon indicates successful monitoring. Since SNMP, JMX, or IPMI are not yet configured, basic agent monitoring will be active.

Access Monitoring -> Graphs to view available graphs for the host. For example, CPU load average can be analyzed. Zabbix performance heavily relies on its database efficiency.

Zabbix Security Settings

1. Default User Credentials

Immediately after setup, change the default Zabbix administrator credentials to enhance security.

Zabbix Menu Explanation

• Dashboard: Customizable overview of monitored data. Add graphs by clicking 'Add screen' and selecting desired items.
• Screens: Aggregate multiple graphs into a single view for large displays.
• Maps: Visualize network topology and infrastructure.
• Status of Zabbix: Provides an overview of server status, host counts (enabled, disabled, templates), item counts, trigger states, active users, and server performance metrics.

Alert Severity Levels

Zabbix defines severity levels for alerts (e.g., Information, Warning, Average, High, Disaster). These levels can be configured to trigger specific actions and notifications.

• Trigger Levels: Define the severity of an alert based on conditions.
• Alert Escalation: Configure rules for escalating alerts if not acknowledged.
• Notification Media: Set up various methods for sending notifications, such as email, SMS, or custom scripts.

Adding a Host and Configuration

1. Host Creation

Zabbix Agents enable custom monitoring beyond SNMP capabilities.

• Monitoring Templates: Zabbix uses templates composed of various monitoring items (e.g., CPU usage, memory usage). Linking a host to a template automatically applies these items. For SNMP monitoring, search for SNMP templates. Some templates may require manual creation.
• IPMI Configuration: If using IPMI, provide the username and password.
• Macros: Define variables (macros) within templates for dynamic values, such as SNMP community strings.
• Host Asset Information: Configure manual or automatic population of host asset details.

After configuring templates and macros, click Add to create the host. A green SNMP status indicates successful connection.

2. Graph Explanation

Navigate to Configuration -> Hosts -> Graphs to view and manage graphs. Graphs are generated from monitoring items. Colors and display options are customizable.

3. Aggregated Screens (Screens) Setup

Create custom screens to group multiple graphs. In Zabbix 3.0, screens are managed under Monitoring -> Screens.

1. Click Create Screen.
2. Define the screen name and layout (e.g., 2x2 grid).
3. Edit the screen and add individual graphs or other elements using Add.

Monitoring Case: Custom Item Configuration

This section details creating a custom item to monitor Nginx's active connections.

Nginx Status Module Configuration

Ensure the Nginx status module is enabled in nginx.conf:

location /status {
    stub_status on;
    access_log off;
    allow 192.168.56.0/24;
    deny all;
}

Restart Nginx and test the status page: http://your_server_ip:8080/status.

Custom Item Setup

1. Zabbix Agent Configuration: Ensure Zabbix Agent is configured to include custom user parameters:

   # vim /etc/zabbix/zabbix_agentd.conf
   Include=/etc/zabbix/zabbix_agentd.d/
   

   Create a new configuration file (e.g., nginx.conf) in this directory:

   # /etc/zabbix/zabbix_agentd.d/nginx.conf
   UserParameter=nginx.active,/usr/bin/curl -s http://192.168.56.11:8080/status|grep Active|awk -F "[ ]"'{print $3}'
   

   Use absolute paths for commands and consider wrapping complex logic in shell scripts.

2. Restart Zabbix Agent:

   [root@linux-node1 ~]# systemctl restart zabbix-agent
   

3. Test with zabbix-get: Install zabbix-get if not present: yum install zabbix-get -y

   [root@linux-node1 ~]# zabbix_get -s 192.168.56.11 -k "nginx.active"
   

   If you encounter connection refused errors, ensure Server= in zabbix_agentd.conf includes the Zabbix server's IP address.

4. Create Item in Zabbix Web UI:

   • Navigate to Configuration -> Hosts -> Items -> Create item.
   • Name: Nginx Active Connections
   • Type: Zabbix agent
   • Key: nginx.active
   • Data type: Numeric (unsigned)
   • Update interval: 60 seconds
   • History storage period: 90d (or as needed)
   • Trend storage period: 365d (or as needed)
   • Application: Create a new application group (e.g., Nginx).

Custom Graphs

1. Go to Configuration -> Hosts -> Graphs -> Create graph.
2. Name: Nginx Connections
3. Width/Height: Set dimensions.
4. Graph type: Line (or desired type).
5. Click Add and select the Nginx Active Connections item.
6. Preview and save the graph.

Access the graph via Monitoring -> Graphs.

Zabbix Alerting Configuration

1. Trigger Creation

Triggers define alerting conditions. To create a trigger for Nginx active connections:

1. Navigate to Configuration -> Hosts -> Triggers -> Create trigger.
2. Name: Nginx High Active Connections
3. Severity: Warning (or desired level).
4. Expression: {YourHostName:nginx.active.last()}>1 This triggers if the last recorded value for nginx.active is greater than 1.

2. Action Configuration

Actions define what happens when a trigger fires.

1. Navigate to Configuration -> Actions -> Create action.
2. Name: Nginx Alert Action
3. Conditions: Set conditions (e.g., Trigger severity >= Warning, Host group = Nginx Servers).
4. Operations: Define actions like sending a message via Media type (e.g., Email).
   • Send to Media: Select user(s) or media type.
   • Message: Customize the subject and body using macros like {HOST.NAME}, {TRIGGER.NAME}, {EVENT.VALUE}.

3. Media Type Configuration (Email Example)

1. Navigate to Administration -> Media types -> Email.
2. Configure SMTP server details, including sender address, authentication credentials, and port.

4. User Media Configuration

1. Navigate to Administration -> Users.
2. Select a user, go to the Media tab, and click Add.
3. Configure the Type (e.g., Email) and Send to address. Ensure the user's configured media matches the Action settings.

Custom Alert Scripts

Zabbix allows external scripts for custom alerting. Place scripts in /usr/lib/zabbix/alertscripts and configure AlertScriptsPath in zabbix_server.conf. Scripts receive alert details as arguments (recipient, subject, body).

Zabbix MySQL Monitoring

Zabbix can monitor MySQL using built-in templates or more advanced solutions like Percona Monitoring Plugins.

Percona Monitoring Plugins Installation

1. Install Percona Repository and Packages:

   [root@linux-node1 ~]# yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
   [root@linux-node1 ~]# yum install percona-zabbix-templates php php-mysql -y
   

   Ensure PHP and php-mysql are installed as Percona plugins use PHP for data collection.

2. Import Zabbix Template: Download the Percona template (.xml file) and import it via Configuration -> Templates -> Import in the Zabbix UI.

3. Configure User Parameters: Copy the userparameter_percona_mysql.conf file to /etc/zabbix/zabbix_agentd.d/.

4. Configure MySQL Credentials: Create a configuration file (e.g., /var/lib/zabbix/percona/scripts/ss_get_mysql_stats.php.cnf) with MySQL user and password for the agent.

   <?php
   $mysql_user = 'zabbix_monitor'; // Use a dedicated monitoring user
   $mysql_pass = 'your_password';
   ?>
   

5. Restart Zabbix Agent:

   [root@linux-node1 ~]# systemctl restart zabbix-agent
   

6. Test Data Retrieval: Use zabbix_get or verify items in the Zabbix UI after linking the template to the MySQL host.

   [root@linux-node1 scripts]# ./get_mysql_stats_wrapper.sh gm # Example key
   

Zabbix Web Monitoring

Zabbix can monitor web scenarios (sequences of HTTP requests).

1. Creating a Web Scenario

1. Navigate to Monitoring -> Web -> Create scenario.
2. Name: Give the scenario a descriptive name.
3. Host: Select the host associated with the web application.
4. Steps: Define the sequence of steps:
   • URL: The target URL.
   • Required string: Text that must be present in the response.
   • Status codes: Expected HTTP status codes (e.g., 200).

2. Trigger Configuration for Web Scenarios

By default, web scenarios may not have associated triggers. Create triggers manually based on scenario status or response times.

• Navigate to Configuration -> Hosts -> Triggers -> Create trigger.
• Expression: Use items like {YourHostName:web.check.failed[scenario_name].last()}>0 to detect failures.

Zabbix High Availability and Distributed Monitoring

Active vs. Passive Mode

Zabbix Agents can operate in passive (server polls agent) or active (agent sends data to server) modes. Active mode is recommended for large deployments (>300 hosts) or when facing network latency issues.

• Configuration (zabbix_agentd.conf):
  • Server: IP of Zabbix server (for passive mode).
  • ServerActive: IP of Zabbix server (for active mode).
  • StartAgents: Number of agent listeners (set to 0 for active-only mode).
  • Hostname: Unique identifier for the host.

Zabbix Proxy

A Zabbix Proxy acts as an intermediary server, collecting data from agents and forwarding it to the Zabbix Server. Proxies are essential for:

• Distributed Environments: Managing geographically dispersed locations or multiple data centers.
• Scalability: Reducing the load on the Zabbix Server by distributing data collection.
• Network Unreliability: Ensuring data is collected even with intermittent network connectivity between agents and the server.

Proxies require thier own database. Installation involves setting up the zabbix-proxy package and configuring its connection to both the Zabbix Server and its local database.

Distributed Monitoring Setup

1. Install Zabbix Proxy: Install zabbix-proxy and its corresponding database package (e.g., zabbix-proxy-mysql).
2. Configure Proxy Database: Create a database and user for the proxy.
3. Import Proxy Schema: Import the proxy schema (schema.sql.gz) into the proxy database.
4. Configure Proxy: Edit /etc/zabbix/zabbix_proxy.conf, specifying Server (Zabbix Server IP), Hostname, and database connection details.
5. Start Proxy Service: systemctl start zabbix-proxy.
6. Configure Zabbix Server: In the Zabbix UI, add the proxy under Administration -> Proxies.
7. Link Hosts to Proxy: When adding or editing hosts, select the appropriate proxy under the Proxy option.

Automation

Zabbix supports several automation methods:

• Auto-Registration: Agents can automatically register with the Zabbix Server based on metadata. Configure HostMetadata and HostMetadataItem in zabbix_agentd.conf and set up Actions in the Zabbix UI to process these registrations.
• Network Discovery: Zabbix can discover network devices and hosts within specified IP ranges. Configure discovery rules and associated actions to automatically add discovered devices.
• Zabbix API: The Zabbix API allows programmatic interaction for managing hosts, items, triggers, and other configuration elements. This enables powerful automation through custom scripts or integration with other management tools.

Using the Zabbix API

1. Authentication: Obtain an authentication token using user.login method via a POST request.
2. Host Management: Use methods like host.create, host.get, and host.delete to manage hosts.
3. Template Management: Retrieve template information using template.get.

Custom Python scripts can simplify API interactions, automating tasks like host provisioning and configuration.