Zabbix: a complexe to start with but very useful when you get the hang of it

Being able to monitor your servers and application is essential but it's no use when you're not monitoring the proper elements or when you're not notified when failures occur.

Zabbix allows you to set up monitoring using pre-built template for well known solutions but also create your own monitoring probes.

It's also very convenient as it can interface with many other platforms such as monitoring solutions (Nagio or ELK ) or ticketing solutions ( Redmine or JIRA for example.

• Official website: Zabbix
• Documentation: Multiplatform installation documentation for Zabbix components
• Git source: Zabbix source code

The official documentation is quite complete and well maintained but has a few limitations also. This article is a quick review and gives a few tips on how to setup Zabbix as a service.

Installing Zabbix 6.0 on Linux Ubuntu 22.04 as a service

For our example we will be installing Zabbix 6.0 which the latest LTS to date coupled with a PostgreSQL database hosted on our dedicated DB server and using Apache2 for the front-end components on a server with the IP 192.168.1.50 on our private network.

The multiplatform installation documentation for Zabbix components is actually quite convenient to get through the installation process so let's follow it:

However, the documentation implies that you are installing Zabbix on a server which is also a DB host.
In this example, the database is hosted elsewhere so the command lines will need to be adapted.

Pre-requisite: Firewall/Network port configuration

If you have a firewall installed and enabled, allow access to HTTP/S ports (443 and 80) as well as Zabbix agent (10050 and 10051).

For example using UFW:

sudo ufw allow proto tcp from any to any port 80,443
sudo ufw allow proto tcp from any to any port 10050,10051

Check that you rules are correct:

sudo ufw status

$ sudo ufw status

Status: active

To                         Action      From
--                         ------      ----
80,443/tcp                 ALLOW       Anywhere
22/tcp                     ALLOW       Anywhere
10050,10051/tcp            ALLOW       Anywhere
5432/tcp                   ALLOW       Anywhere
80,443/tcp (v6)            ALLOW       Anywhere (v6)
22/tcp (v6)                ALLOW       Anywhere (v6)
10050,10051/tcp (v6)       ALLOW       Anywhere (v6)
5432/tcp (v6)              ALLOW       Anywhere (v6)

With this configuration:

• Users can connect to the UI through HTTP (80) or HTTPS (443)
• Zabbix agents can communicate with the server using ports 10050 and 10051
• Zabbix server can connect to its PSQL database using port 5432

And of course port 22 is enable for SSH access.

Installing Zabbix Server as a service

Core service packages

sudo apt update
sudo apt -y upgrade
sudo apt install -y zabbix-server-pgsql zabbix-sql-scripts

Note: "zabbix-sql-scripts" could be installed directly on our DB server as is contains script to setup the Zabbix server database or run DB optimisation and clean-up tasks.
In this example, it will be installed on the Zabbix server as I prefer to have packages grouped per application, keeping the DB server clean and because the scripts can be run remotely.

you should now have Zabbix server configuration file:

/etc/zabbix/zabbix_server.conf

And the SQL scripts in:

/usr/share/zabbix-sql-scripts/

Database setup

Connecting to our dedicated PSQL DB server (IP 192.168.1.22), we can create a Zabbix dedicated database and user:

CREATE USER zabbixadmin with password '7mRmHy4vmJ54MnXjz9M';
CREATE DATABASE zabbix with owner zabbixadmin ;
GRANT ALL PRIVILEGES ON DATABASE zabbix TO zabbixadmin;

Then from our Zabbix server, we can test the remove access (requires installing postgres-client):

psql -h 192.168.1.22 -U zabbixadmin -p 5432 zabbix

If all is OK, we can import the Zabbix DB template but we will not be using the command line provided by Zabbix documentation:

zcat /usr/share/zabbix-sql-scripts/postgresql/server.sql.gz | sudo -u zabbix psql zabbix

This command line implies that PSQL is running on the same server as Zabbix and also that there is a zabbix Linux user:

• our PSQL service is running on another server
• there is no "zabbix" Linux user on our DB server, only a DB user, and we don't want to create one.

The correct command line here:

zcat /usr/share/zabbix-sql-scripts/postgresql/server.sql.gz | sudo psql -U zabbixadmin2  -h 192.168.1.22 -p 5432 zabbix 

This should import the Zabbix database template to our remote PSQL server.

Zabbix server configuration file

Let's update the essentials in our Zabbix Server configuration file by configuring the DB connection information:

sudo vi /etc/zabbix/zabbix_server.conf

Content is at the start of the file:

### Option: DBHost
#       Database host name.
#       If set to localhost, socket is used for MySQL.
#       If set to empty string, socket is used for PostgreSQL.
#       If set to empty string, the Net Service Name connection method is used to connect to Oracle database; also see
#       the TNS_ADMIN environment variable to specify the directory where the tnsnames.ora file is located.
#
# Mandatory: no
# Default:
# DBHost=localhost
DBHost=192.168.1.22

### Option: DBName
#       Database name.
#       If the Net Service Name connection method is used to connect to Oracle database, specify the service name from
#       the tnsnames.ora file or set to empty string; also see the TWO_TASK environment variable if DBName is set to
#       empty string.
#
# Mandatory: yes
# Default:
# DBName=
DBName=zabbix

### Option: DBSchema
#       Schema name. Used for PostgreSQL.
#
# Mandatory: no
# Default:
# DBSchema=

### Option: DBUser
#       Database user.
#
# Mandatory: no
# Default:
# DBUser=
DBUser=zabbixadmin

### Option: DBPassword
#       Database password.
#       Comment this line if no password is used.
#
# Mandatory: no
# Default:
# DBPassword=
DBPassword=7mRmHy4vmJ54MnXjz9M

### Option: DBSocket
#       Path to MySQL socket.
#
# Mandatory: no
# Default:
# DBSocket=

### Option: DBPort
#       Database port when not using local socket.
#       If the Net Service Name connection method is used to connect to Oracle database, the port number from the
#       tnsnames.ora file will be used. The port number set here will be ignored.
#
# Mandatory: no
# Range: 1024-65535
# Default:
# DBPort=
DBPort=5442

And let's start the Zabbix server:

sudo systemctl enable zabbix-server.service
sudo systemctl start zabbix-server.service

And let's check the service is running:

sudo systemctl status zabbix-server.service

The server can know receive and store information from agents but it has no UI.
From this point, we can either interface Zabbix server to push the information to another platform such as Nagios or we install Zabbix dedicated front-end pakcage.

Installing Zabbix front-end

Install the front-end package by running the following commands:

sudo apt install -y zabbix-frontend-php php8.1-pgsql zabbix-apache-conf

Note: "php8.1-pgsql" is used here because I want to run the latest versions available.
In some cases, I have had to downgrade to "php7.4-pgsql" for compatibility reasons with other application and made it work but would not recommend it.

You should now have a configuration file:

/etc/zabbix/web/zabbix.conf.php

Full file example source

Update the necessary parameters:

// Zabbix GUI configuration file.

$DB['TYPE']     = 'POSTGRESQL';
$DB['SERVER']   = '192.168.1.22';
$DB['PORT']     = '5432';
$DB['DATABASE'] = 'zabbix';
$DB['USER']     = 'zabbixadmin';
$DB['PASSWORD'] = '7mRmHy4vmJ54MnXjz9M';

Apache site configuration file example

For this example, we will simply update the default site configuration file which has previously been enabled:

sudo vi /etc/apache2/sites-enabled/000-default.conf

Content:

        ServerAdmin webmaster@localhost
        # DocumentRoot /var/www/html
        DocumentRoot /usr/share/zabbix

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

DocumentRoot is set to the working directory for Zabbix front-end and Apache2 is restarted:

sudo systemctl restart apache2

Note: in my example, I'll be setting Zabbix behing a Traefik service so only need to configure port 80 as Zabbix will be the service that redirects all communications to port 443 (HTTPS).

You should now have a running and accessible UI for Zabbix:

Zabbix agent setup

On each server we want to monitor, an agent must be installed.

Zabbix agent vs agent2

Essentially, Zabbix agent is the legacy agent whereas agent2 is a more recent updated agent with additional features but also some missing features and not supported by all platforms the legacy agents runs on.

Once again, Zabbix documentation gives us a very good descriptions of agent & agent2.

Check out the documentation to select the most appropriate agent for you usage.

In my experience, for a standard monitoring system running on Linux Ubuntu, both agents can be used. The choice between them was not based on the supported features but rather on the Zabbix Templates I wanted to deploy on the Zabbix Server.

In some cases, Agent2 was required meaning that out of 30 servers, 20 would use the legacy Agent and 10 would be using Agent2.

Active or passive mode

Zabbix agents can work in active or passive mode.

For ease of use and configuration, I only used 1 mode at a time because it simplified management but as with Agent/Agent2, some servers can be configured in the passive mode and others in the active mode all while being monitored by the same Zabbix server.

Agents can even use both active and passive modes on the same monitored host.

Agent setup examples

Documentation:

• Agent
• Agent2

Although agents can run active and passive modes concurrently with the default settings, we can't use both Agent and Agent2 concurrently as they will be using the same ports.

Having both agents might not be needed depending on your situation, but if you do:

• select a pair of ports other than 10050/10051 for the 2nd agent running
• make sure that this other pair of ports is open in your Networt/Firewall configuration for both monitored server and Zabbix server
• update the setup in the Zabbix server UI "configuration > host >" for the monitored host(s) concerned and add an agent with the selected port(s)

Here is the basic configuration to set up to have a working system with:

• Zabbix server IP 192.168.1.50
• Application server on which we are installing Zabbix agents. IP 192.168.1.15X

Server=192.168.1.50
ListenPort=10050
ServerActive=192.168.1.50
Hostname=ApplicationServer1

Server=192.168.1.50
ListenPort=10050
ServerActive=192.168.1.50
Hostname=ApplicationServer1

Timeout=30
EnableRemoteCommands=1

Timeout=30
AllowKey=system.run[*]

Every time you update the configuration file, don't forget to restart the service to implement the change(s).

Once the agent is up and running, you should be able to create your hosts through the UI:

1. Once logged in the UI, go to "Configuration > Hosts"
2. Click on "Create Host" and fill in the required fields
3. Link a relevant template such as "Linux by Zabbix agent"
4. Go to "Monitoring" and create or edit an existing dashboard
5. Add a module "Top hosts": it allows you to select the data to be shown from a group of hosts

If the agents are properly set up, the block should automatically update with the requested data:

Tips and tricks

Check communication between agent and server

To run command lines to test communication without the UI, install the Zabbix-get package:
sudo apt install zabbix-get -y

The format of the command line to run from the Zabbix server:

zabbix_get -s [IP of agent to connect] -p [port used] -k "[item Key]"

Example:

zabbix_get -s 192.168.1.150 -p 10050 -k "system.uptime"

Update user password from DB

You forgot your accont password as well as the admin password but you can still acces and update the database.
As a security measure, passwords are hashed by the software code before being saved to the database.

Passwords therefore need to be hashed when updating directly in DB:

update zabbix.users set passwd=md5('mynewpassword') where alias='Admin';

Install new languages for the UI

Zabbix UI languages are based on what is available on the server meaning you must install the Linux language package on the server in order to call it in Zabbix server UI.

To install a new language:

1. Check what is present on the server
   Short version

   locale -a
   

2. Add a new language. Example : en_US.UTF-8

   sudo locale-gen "en_US.UTF-8"
   

3. Reconfigure the server to use that language:

   sudo dpkg-reconfigure locales
   

   And follow UI instructions

4. Restart Zabbix UI:

   sudo systemctl restart apache2.service
   

5. Configure Zabbix to use the language:
   Go to "Administration > GUI > Default language" and update from there.