Alertmanager sends alerts with Telegram bot

In this article, we will set up Alertmanager to send alerts with Telegram bot. A monitoring system is only truly useful if it can send you alerts. Simply because you cannot look at the monitor screen 24/7/365. And you can’t even see the entire dashboard.

Install Golang on Ubuntu 18

We will use an open source Telegram bot with a repository link here.

Note that this bot uses Golang which has a version greater than go1.10, the repository version of Ubuntu server 18.04 is go1.10.

So, we need to install Golang go1.13 from source. Download the software and unzip.

apt install make -y
wget https://dl.google.com/go/go1.13.3.linux-amd64.tar.gz
tar -xvf go1.13.3.linux-amd64.tar.gz && rm -f go1.13.3.linux-amd64.tar.gz

Export environment variables for Go.

mv go /usr/local
export GOROOT=/usr/local/go

Convert the old Go version file into a backup file. Create a Go file link with the new version.

mv /usr/bin/go /usr/bin/go.bk
ln -s /usr/local/go/bin/go /usr/bin/go

You check the Go version with the following command.

go version

Install Alertmanager Telegram bot

Recommended Reading: Install Alertmanager and set up Prometheus alerts

Now, we will download the Telegram bot source code to the server.

go get github.com/inCaller/prometheus_bot

Move into the source directory and build.

cd go/src/github.com/inCaller/prometheus_bot/
make clean
make

Move the binary executable file locally.

mv prometheus_bot /usr/local/bin/telegrambot

Create telegrambot configuration folder. Copy the necessary configuration files.

mkdir /etc/telegrambot
mv testdata /etc/telegrambot/
cp /etc/telegrambot/testdata/template.tmpl /etc/telegrambot/template.tmpl

Create config.yaml file.

nano /etc/telegrambot/config.yaml

You edit the content as follows.

telegram_token: "Bot-Token"
template_path: "/etc/telegrambot/template.tmpl"
time_zone: "Asia/Ho_Chi_Minh"
split_token: "|"
time_outdata: "02/01/2006 15:04:05"
split_msg_byte: 10000

In the file above, you need to exchange the Bot-Token content with your bot telegram token.

Recommended Reading: How to create Telegram Bot and get chat ID

Create user for telegrambot service.

useradd -rs /bin/false telegrambot
chown telegrambot:telegrambot /usr/local/bin/telegrambot
chown -R telegrambot:telegrambot /etc/telegrambot/*

Create systemd file for telegrambot service.

nano /etc/systemd/system/telegrambot.service

Copy the content below into the file and save.

[Unit]
Description=Prometheus Telegram Bot Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=telegrambot
Group=telegrambot
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/telegrambot \
-c /etc/telegrambot/config.yaml \
-l ":9087" \
-t /etc/telegrambot/template.tmpl

SyslogIdentifier=telegrambot
Restart=always

[Install]
WantedBy=multi-user.target

Reload the daemon and start the telegrambot service. The service will run on port 9087.

systemctl daemon-reload
systemctl enable telegrambot.service
systemctl start telegrambot.service

Set up Alertmanager to send alerts to Telegram bot

Open the alertmanager.yml file.

nano /etc/alertmanager/alertmanager.yml

You set up the file to look like below.

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 5m
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:9087/alert/-123456789'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

In the content there are 2 lines you need to edit.

• repeat_interval: 5m: the time between sending alerts, here I set the alert to be sent every 5 minutes.
• url: ‘http://127.0.0.1:9087/alert/-123456789’: you need to replace 123456789 with chat-ID your Telegram chat room.

Now, reload the services.

service prometheus reload
service alertmanager reload
service telegrambot reload

As a result, you will receive a Telegram message that looks like the following.

Conclusion

So I have guided you step by step to successfully install Alertmanager Telegram bot. Now, you just need to define the necessary rules for Prometheus. Every time a rule matches a monitor event in the system, you will receive a warning message.