Install Alertmanager and set up Prometheus alerts

In this article, I will guide you through installing Alertmanager and setting up Prometheus alerts.

What is Alertmanager?

If you don’t know about Alertmanager, it is an indispensable component when using Prometheus. Alertmanager will handle alerts sent from client applications such as the Prometheus server.

An easy to understand explanation is:

• You collect monitor data from endpoints and store it in the Prometheus database.
• You create alert rules in Prometheus, for example: server ram > 70%.
• And you push these alerts to Alertmanager, it will handle duplicate alerts, group alerts and send alerts to the notification channel you want: mail, telegram,…

Install Alertmanager

You can check the latest version of the software at this link.

Download and extract the software.

wget https://github.com/prometheus/alertmanager/releases/download/v0.21.0/alertmanager-0.21.0.linux-amd64.tar.gz
tar -xvzf alertmanager-0.21.0.linux-amd64.tar.gz && rm -f alertmanager-0.21.0.linux-amd64.tar.gz
cd alertmanager-0.21.0.linux-amd64/

Move the binary executable file to the local directory, create a config directory for alertmanager.

mv alertmanager /usr/local/bin/
mkdir -p /etc/alertmanager/data
mv alertmanager.yml /etc/alertmanager/

Create user alertmanager.

useradd -rs /bin/false alertmanager
chown alertmanager:alertmanager /usr/local/bin/alertmanager
chown -R alertmanager:alertmanager /etc/alertmanager/*

Create a systemd file for the alertmanager service.

nano /etc/systemd/system/alertmanager.service

Copy the following content into the file.

[Unit]
Description=Alert Manager Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=alertmanager
Group=alertmanager
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--web.listen-address=":9093" \
--storage.path=/etc/alertmanager/data

SyslogIdentifier=alertmanager
Restart=always

[Install]
WantedBy=multi-user.target

Reload the daemon and start the service.

systemctl daemon-reload
systemctl enable alertmanager.service
systemctl start alertmanager.service

Set up virtual host for Alertmanager

Now you can open a browser and visit IP:9093, for example: http://192.168.1.10:9093.

If you use nginx proxy for Prometheus as I wrote, you can access alertmanager through the domain.

Open the virtual host file of the prometheus domain and add the following block to the file.

location /alertmanager/ {
    proxy_pass   http://127.0.0.1:9093/;
    include /etc/nginx/conf.d/resource/proxypass.inc;
    auth_basic              "Prometheus";
    auth_basic_user_file    /etc/nginx/htpasswd/prometheus;
}

The virtual host file will look like this.

...
server {
    listen 443 ssl http2;
        ...

    location / {
        ...
    }

    ...

    location /alertmanager/ {
        proxy_pass   http://127.0.0.1:9093/;
        include /etc/nginx/conf.d/resource/proxypass.inc;
        auth_basic              "Prometheus";
        auth_basic_user_file    /etc/nginx/htpasswd/prometheus;
    }
}

Reload nginx service.

service nginx reload

Now, you can open a browser and access it through the domain, for example: https://prometheus.example.com/alertmanager.

Create Prometheus rule files

In the Prometheus installation article, I created the /etc/prometheus/rules folder, I will create rule files in this folder.

For example, I will create a rule file to check if ping the host is down.

nano /etc/prometheus/rules/ping.yml

Copy the content below into the file and save.

groups: 
- name: Ping
  rules:
  - alert: Ping Host
    expr: probe_success{job="blackbox_ping"} == 0
    for: 5s
    labels:
      severity: "Critical"
    annotations:
      summary: 'Host {{ $labels.instance }} down more than 5 second.'

Set up Prometheus configuration file

After you have created the rule file, you will now set up Prometheus to push alerts to Alertmanager.

Open the prometheus.yml file.

nano /etc/prometheus/prometheus.yml

You look in the file (paragraph at the top) and edit it like this.

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
    - "/etc/prometheus/rules/ping.yml"

Now, you reload the prometheus and alertmanager services.

service prometheus reload
service alertmanager reload

You can check the rules on the Prometheus interface, select the Alerts menu.

Conclusion

So this article has guided you step by step to successfully install Alertmanager. Create a rule in Prometheus and set it up to push alerts to Alertmanager. In the next article, I will install Telegram Bot to be able to send these alerts to Telegram for you.