Monitor linux server with Prometheus node exporter

In this article, we will monitor linux server with Prometheus node exporter. If you have installed blackbox exporter before, this article will be a little different.

With blackbox exporter, you can install it right on the prometheus server. But with node exporter, you must install it on the Linux server that needs monitoring. The node exporter here is like an agent monitor.

Install Prometheus node exporter

First, check the latest version of the node exporter at this link. At the time of writing, the latest version is 1.0.1.

We download and unzip the installation package.

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
tar -xzf node_exporter-1.0.1.linux-amd64.tar.gz && rm -f node_exporter-1.0.1.linux-amd64.tar.gz
cd node_exporter-1.0.1.linux-amd64

Move the binary executable file to the local bin directory.

mv node_exporter /usr/local/bin/
cd .. && rm -rf node_exporter-1.0.1.linux-amd64

Create a user for the node exporter service.

useradd -rs /bin/false node_exporter
chown node_exporter:node_exporter /usr/local/bin/node_exporter

Create a systemd file for the node exporter service.

nano /etc/systemd/system/node_exporter.service

Copy the content below into the file.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/node_exporter

SyslogIdentifier=node_exporter
Restart=always

[Install]
WantedBy=default.target

Reload the daemon and start the node exporter service.

systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter

After starting the node_exporter service, it will run on port 9100.

For those who are lazy, you can download the node exporter installation script I wrote for quick installation.

Set up Prometheus configuration

Now, we will set up prometheus to call node_exporter targets on Linux servers.

Open the prometheus configuration file.

nano /etc/prometheus/prometheus.yml

You add the following paragraph to the file.

  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
    - targets:
      - hostname-linux:9100

You replace hostname-linux with the IP address or DNS name of the Linux server you need to monitor.

Next, find the section below and add the rule file path for the linux server.

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
    - "/etc/prometheus/rules/linux.yml"

Warning: You need to open the firewall to allow the prometheus server to access port 9100 on the linux server that needs monitoring.

Set up alert rule in Prometheus

If you don’t know how to set up rules, you should refer to the article about your blackbox exporter.

Recommended Reading: Install Alertmanager and set up Prometheus alerts

You create a rule file containing the necessary warnings for the Linux server.

nano /etc/prometheus/rules/linux.yml

Then you create rules, for example an alert as below. This alert will determine that the node_exporter service on the target linux server is not working. Thereby, alert sends us a notification that the host is down.

You can download the complete rule file for Linux server at this gitlab link.

groups:
- name: Linux Server
  rules:
  - alert: Server Linux Down
    expr: up{job="node_exporter"} == 0
    for: 10s
    labels:
      severity: "Critical"
    annotations:
      Summary: 'Server "{{ $labels.instance }}" down.'

After setting up prometheus, you can reload the service.

systemctl reload prometheus

Set up the Grafana dashboard

My Grafana dashboard is using version ID 11074. You can import ID directly from Grafana.

Or download and import json from the dashboard you saved.

Conclusion

So you have completed the steps to set up a Linux monitor server with Prometheus node exporter. This tool is quite effective, it pushes almost complete Linux resource server information.