Before you run anything
This guide is general information about how we approach monitoring. It is not advice for your specific environment. Commands are shown for reference; read each one before running it, and install packages on a system where you are permitted to do so.
Version numbers, package names and download URLs were checked at the review date shown above and move quickly. Follow the official download pages linked at the end rather than copying a version number from here. MDCS accepts no liability for decisions taken on the basis of this guide.
Which tool, for what
The three are routinely presented as competitors. They are not, and choosing between them as though they were is the first mistake.
| Tool | What it is | Reach for it when |
|---|---|---|
| Zabbix | A complete monitoring system: collection, storage, triggers, alerting and a web interface. Templates for common devices, agent and agentless checks, and native SNMP. | The estate is infrastructure — servers, switches, hypervisors, storage, UPS. Anything that will never expose a metrics endpoint. |
| Prometheus | A time-series database that pulls metrics from HTTP endpoints, with a query language and its own alerting component. | The estate is applications and containers, and the things being monitored can be made to expose metrics themselves. |
| Grafana | A visualisation layer. It queries other systems and draws them. It stores no metrics of its own. | You have more than one source and want one place to look at them, or you want dashboards better than the built-in ones. |
The common arrangement
Zabbix for the infrastructure, Prometheus for anything containerised, and Grafana over both so there is one place to look. Zabbix has its own perfectly usable interface, so Grafana is optional — add it when you have a second data source or a dashboard requirement Zabbix cannot meet.
Do not run all three because a guide listed all three
A single Zabbix server monitoring a hundred hosts is a complete, working monitoring system. Adding Prometheus and Grafana to that adds two more things to maintain, upgrade and secure. Add each one when there is a question the existing setup cannot answer.
01 · Zabbix — getting started
Zabbix ships a repository per version and per distribution. Install the repository package first, then the components, then set up the database. Version 7.0 is the current long-term support release; 7.4 is the current standard release.
Use the official repository, not your distribution's
Debian and Ubuntu ship their own Zabbix packages. They lag behind and Zabbix does not support them. The download page generates the exact commands for your combination of version, distribution, database and web server — that is the reliable route, and it is why this guide does not reproduce a fixed URL.
Server, frontend and database — the shape of it
The frontend then answers on /zabbix on that host. The default account is Admin with password zabbix — change it before the machine is reachable by anyone else.
Package names differ by database and web server. The commands above assume MySQL or MariaDB with Apache; PostgreSQL and nginx use different package names, which the download page will give you.
Agents on the machines you want to watch
A reply of 1 means the agent is reachable and the firewall allows it. The agent listens on 10050; the server listens on 10051. Both need to be open in the right direction.
Hostname must match exactly what you enter when adding the host in the frontend. A mismatch is the single most common reason a newly added host stays grey and never collects.
02 · Zabbix — making it useful
A default install with a linked template collects a great deal and tells you very little. These are the changes worth making first.
Recipe: stop the disk-space alert that fires every week
The stock filesystem trigger fires at a fixed percentage. On a 4 TB volume, 20% free is 800 GB and nothing is wrong. On a 40 GB root volume, 20% free is a genuine warning.
Macros can be set at template, host group or host level. Set them on the host group so a new server of the same kind inherits the right threshold rather than the default.
Recipe: alert on trend, not on a single sample
A trigger that fires the moment CPU touches 90% will fire during every backup. One that fires when CPU has averaged above 90% for fifteen minutes describes a problem.
The third is the most useful trigger most estates are missing. It predicts a full disk a day ahead rather than reporting one at 3am.
Recipe: watch a service that has no template
The script runs as the zabbix user. If it needs privileges, grant them narrowly through sudoers for that one command rather than running the agent as root.
Two settings worth changing on day one
Housekeeping. Default history retention keeps a great deal of raw data. On a busy server this becomes the reason the database grows and the frontend slows. Set history to something short — a week or two — and let trends carry the long view.
Action escalation. Configure escalation steps so an unacknowledged problem repeats and then reaches somebody else. Without it, a missed notification is simply missed.
03 · Prometheus — getting started
Prometheus is a single binary with a configuration file. It pulls metrics from endpoints you list; it does not receive pushes, and nothing is installed on the targets except an exporter that publishes a metrics page.
Prometheus itself
node_exporter, on each machine you want to watch
node_exporter publishes on 9100. Every metric it produces is prefixed node_, which makes them easy to find in a query.
Run it as a systemd service under the node_exporter user rather than from a shell, and bind it to an address Prometheus can reach but the internet cannot.
Telling Prometheus what to scrape
Labels applied here follow the metric everywhere. Adding env and role at scrape time is what later allows one alert rule to cover every production web server.
04 · Prometheus — useful queries and rules
PromQL is the reason to use Prometheus and the reason people give up on it. These cover most of what an infrastructure estate needs.
Recipe: the queries worth saving
rate() before avg(), not the other way round. Averaging a counter and then taking the rate produces a number that looks plausible and is wrong.
Recipe: alert rules that survive contact with reality
The for clause is the difference between a monitoring system and a pager that goes off during backups. Nothing should alert without one.
The reload endpoint only works if Prometheus was started with --web.enable-lifecycle. Without that flag it returns 404, which is easily mistaken for a broken URL.
05 · Grafana — getting started
Grafana installs from its own APT or YUM repository. It answers on port 3000, and the first login is admin / admin, which it will ask you to change.
Then add a data source: Prometheus at http://localhost:9090, or Zabbix through the Zabbix plugin, which needs the Zabbix API URL and a Zabbix account with read permission.
Grafana is a web application with credentials and access to your monitoring data. Put it behind TLS and do not publish it to the internet without authentication in front of it.
06 · Grafana — dashboards people actually open
Recipe: import before you build
The dashboard library has thousands of community dashboards. Importing one is a numeric ID, and it is a far better starting point than an empty canvas.
Community dashboards are written against particular exporter versions and metric names. If panels show "No data", the usual cause is a metric name that has changed. Edit the panel and check the query against what your exporter actually publishes.
Recipe: three dashboards, not one
| Dashboard | Answers | Keep it to |
|---|---|---|
| Operations | Is anything broken right now? | One screen. Red or green. No history, no trends, nothing that needs interpretation. |
| Capacity | What will run out, and when? | Trends over weeks and months. Disk growth, memory headroom, connection counts. |
| Service | Is the thing users care about working? | Response time, error rate, throughput. No CPU graphs — nobody outside the team cares about CPU. |
The test for a panel
If somebody looked at it and could not say what they would do differently, it belongs on a different dashboard or on none. A dashboard is not an inventory of available metrics.
Recipe: variables, so one dashboard serves the estate
One dashboard with a host dropdown replaces one dashboard per host, and it stays correct when servers are added.
Thresholds, and where they come from
This is the part that decides whether any of the above is worth having, and it cannot be copied from a guide — including this one.
- Watch for two weeks before setting a threshold. Your normal is not anybody else's. A batch job that takes CPU to 100% every night is normal until it stops happening.
- Set the threshold above the noisiest normal thing you can find. Then tighten it if something is missed. Starting tight and loosening trains people to ignore alerts in the interim.
- Every alert names an action. Where nobody can say what a person should do, it is a metric. Put it on a dashboard.
- Review what fired every month. Anything that fired and needed no action gets removed or its threshold moved. This is the whole discipline, and it is the step that is always skipped.
- Separate warning from critical by response, not by severity. Critical means somebody is woken. Warning means somebody looks tomorrow. If nothing is woken, it is not critical.
The number that tells you whether it is working
Alerts per week, and what proportion required an action. If the second figure is below about half, the system is training people to dismiss it — and the alert that mattered will be dismissed with the rest.
Downloads and add-ons
Checked at the review date at the top of this page. At that date Zabbix was at 7.4 with 7.0 as the long-term support branch — take LTS if you would rather upgrade less often. Grafana was at 13, Prometheus at 3. Versions and package names change; follow the official pages rather than copying a version number from any article, including this one.
Zabbix
Generates exact install commands for your version, distribution, database and web server. The reliable route.
The official package repository. Packages for RHEL and derivatives, Ubuntu, Debian, SUSE and Raspbian.
Source tarballs, official containers, a pre-configured virtual appliance, and agent binaries for non-Linux systems.
Package installation notes, including the cases where distribution packages cause problems.
Prometheus
Precompiled binaries and container images for Prometheus and the officially maintained components.
The catalogue. Databases, message queues, hardware, network devices, UPS units. Official exporters are marked as such — prefer those.
Host metrics for Linux and Unix. The starting point for any infrastructure use, and the source of every node_ metric.
The project's own guide to installing it and configuring Prometheus to scrape it.
Grafana
Current release for every platform, with the APT and YUM repository instructions.
Thousands of community dashboards, imported by ID. Filter by data source first. The single most useful add-on for anyone starting out.
Query Zabbix from Grafana — metrics, triggers, problems and annotations from Zabbix events. Requires Grafana 11.6 or later.
Source, issues and installation options for the Zabbix plugin.
Where to start if you take one link
The Grafana dashboard library. Importing a well-made dashboard for something you already collect teaches more about what good monitoring looks like than any amount of building from scratch.