Alertmanager sends alerts with Telegram bot

by Daniel Pham
Published: Updated:
This entry is part 8 of 14 in the series Install Prometheus and Grafana on Ubuntu 18

In this article, we will set up Alertmanager to send alerts with Telegram bot. A monitoring system is only truly useful if it can send you alerts. Simply because you cannot look at the monitor screen 24/7/365. And you can’t even see the entire dashboard.

Install Golang on Ubuntu 18

We will use an open source Telegram bot with a repository link here.

Note that this bot uses Golang which has a version greater than go1.10, the repository version of Ubuntu server 18.04 is go1.10.

So, we need to install Golang go1.13 from source. Download the software and unzip.

apt install make -y
wget https://dl.google.com/go/go1.13.3.linux-amd64.tar.gz
tar -xvf go1.13.3.linux-amd64.tar.gz && rm -f go1.13.3.linux-amd64.tar.gz

Export environment variables for Go.

mv go /usr/local
export GOROOT=/usr/local/go

Convert the old Go version file into a backup file. Create a Go file link with the new version.

mv /usr/bin/go /usr/bin/go.bk
ln -s /usr/local/go/bin/go /usr/bin/go

You check the Go version with the following command.

go version

Install Alertmanager Telegram bot

Recommended Reading: Install Alertmanager and set up Prometheus alerts

Now, we will download the Telegram bot source code to the server.

go get github.com/inCaller/prometheus_bot

Move into the source directory and build.

cd go/src/github.com/inCaller/prometheus_bot/
make clean
make

Move the binary executable file locally.

mv prometheus_bot /usr/local/bin/telegrambot

Create telegrambot configuration folder. Copy the necessary configuration files.

mkdir /etc/telegrambot
mv testdata /etc/telegrambot/
cp /etc/telegrambot/testdata/template.tmpl /etc/telegrambot/template.tmpl

Create config.yaml file.

nano /etc/telegrambot/config.yaml

You edit the content as follows.

telegram_token: "Bot-Token"
template_path: "/etc/telegrambot/template.tmpl"
time_zone: "Asia/Ho_Chi_Minh"
split_token: "|"
time_outdata: "02/01/2006 15:04:05"
split_msg_byte: 10000

In the file above, you need to exchange the Bot-Token content with your bot telegram token.

Recommended Reading: How to create Telegram Bot and get chat ID

Create user for telegrambot service.

useradd -rs /bin/false telegrambot
chown telegrambot:telegrambot /usr/local/bin/telegrambot
chown -R telegrambot:telegrambot /etc/telegrambot/*

Create systemd file for telegrambot service.

nano /etc/systemd/system/telegrambot.service

Copy the content below into the file and save.

[Unit]
Description=Prometheus Telegram Bot Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=telegrambot
Group=telegrambot
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/telegrambot \
-c /etc/telegrambot/config.yaml \
-l ":9087" \
-t /etc/telegrambot/template.tmpl

SyslogIdentifier=telegrambot
Restart=always

[Install]
WantedBy=multi-user.target

Reload the daemon and start the telegrambot service. The service will run on port 9087.

systemctl daemon-reload
systemctl enable telegrambot.service
systemctl start telegrambot.service

Set up Alertmanager to send alerts to Telegram bot

Open the alertmanager.yml file.

nano /etc/alertmanager/alertmanager.yml

You set up the file to look like below.

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 5m
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:9087/alert/-123456789'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

In the content there are 2 lines you need to edit.

  • repeat_interval: 5m: the time between sending alerts, here I set the alert to be sent every 5 minutes.
  • url: ‘http://127.0.0.1:9087/alert/-123456789’: you need to replace 123456789 with chat-ID your Telegram chat room.

Now, reload the services.

service prometheus reload
service alertmanager reload
service telegrambot reload

As a result, you will receive a Telegram message that looks like the following.

Alertmanager sends alerts with Telegram bot
Alertmanager sends alerts with telegram bot.

Conclusion

So I have guided you step by step to successfully install Alertmanager Telegram bot. Now, you just need to define the necessary rules for Prometheus. Every time a rule matches a monitor event in the system, you will receive a warning message.

0 0 votes
Article Rating
Series Navigation«« Previous part: Install Alertmanager and set up Prometheus alertsNext part: How to create Telegram Bot and get chat ID »»

You may also like

Subscribe
Notify of
guest
1 Comment
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
sakurai
Guest
3 years ago

Hi anh Dũng,
Nhờ anh chỉ giúp mình file template.tmpl hiện không có ở link
https://github.com/inCaller/prometheus_bot
nên mình không thể cấu hình tiếp được.
cảm ơn anh Dũng nhiều

DevOps Lite is a personal blog specializing in technology with main topics about DevOps, DevSecOps, SRE and System Administrator. Articles are shared for free and contributed to the community.

SUPPORT US

FOLLOW US

Subscribe my Newsletter for new blog posts. Stay updated from your inbox!

© 2021-2024 DevOpsLite.com – All rights reserved.

Please write sources “DevOpsLite.com” when using articles from this website.

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More

2
0
Would love your thoughts, please comment.x
()
x

Adblock Detected

Please support us by disabling your AdBlocker extension from your browsers for our website.