Concepts
Grafana is a dashboard application; it displays data in a visual way, i.e. graphs and interactive charts.
Prometheus Scraper sits underneath Grafana and ‘scrapes’ the data from various inputs; making these ‘scrapes’ available to Grafana.
You can think of Prometheus as a sort of butler; collecting data and presenting it.
Node-Exporter is one such input Prometheus collects and makes available to Grafana.
What is Grafana?
Grafana is a multi-platform open source analytics and interactive visualization web application.
I’ve already covered how to set up a Grafana dashboard for collecting my iceshrimp.NET metrics .
This post is about expanding on the data available to Grafana.
What is Prometheus?
Prometheus is a system for collecting and storing metrics as time series data, with a query language and alerting features.
Prometheus collects metrics from targets by scraping metrics HTTP endpoints.
https://prometheus.io/docs/prometheus/latest/getting_started/
I’ve already written about how to get a usable Grafana Dashboard .
Prometheus extends the Dashboard by bringing in extra data sources from e.g. nginx.
What is Node Exporter?
The Prometheus Node Exporter exposes a wide variety of hardware- and kernel-related metrics.
Install Prometheus
Resources
https://prometheus.io/docs/prometheus/latest/installation/
https://devopscube.com/install-configure-prometheus-linux/
Create a prometheus account
I prefer to have separate accounts for distinct processes.
In a Terminal:
sudo useradd --no-create-home --shell /bin/false prometheus
Create directories
Prometheus needs a place to live.
To create and take ownership execute the following in a Terminal:
sudo mkdir /etc/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
Download and Install
cd ~
wget https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-armv7.tar.gz
Extract and rename
tar xfz prometheus-2.53.1.linux-armv7.tar.gz
mv prometheus-2.53.1.linux-armv7/ prometheus/
rm prometheus-2.53.1.linux-armv7.tar.gz
Copy files to the correct location
In a Terminal:
sudo cp prometheus/prometheus /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
Then:
sudo cp prometheus/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/promtool
Then:
sudo cp -r prometheus/consoles /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
Then:
sudo cp -r prometheus/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
Then:
sudo cp prometheus/prometheus.yml /etc/prometheus/prometheus.yml
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
Tidy up
cd ~
sudo rm -R prometheus
Configure ufw
I am only interested in the prometheus metrics and dashboards being visible from inside my network.
To see the current list of applications allowed through the firewall, in a Terminal:
sudo ufw verbose
Then:
sudo ufw allow from server.ip/24 to any port 9090 comment "Prometheus traffic from LAN only"
Create a prometheus service file under systemd
In a Terminal:
sudo nano /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.enable-admin-api
[Install]
WantedBy=multi-user.target
Then:
Reload the systemd daemon:
sudo systemctl daemon-reload
Start:
sudo systemctl start prometheus
Enable the daemon to restart after a reboot:
sudo systemctl enable prometheus
Check its status:
sudo systemctl status prometheus
View
To view the Prometheus homepage and start to interact go to:
http://server.ip:9090
The server needs to have Node Exporter installed to collect all the system metrics and make it available for Prometheus to scrape it.
Admin Notes
To remove (annoying) old jobs in the Grafana drop-down:
https://stackoverflow.com/questions/54704117/how-can-i-delete-old-jobs-from-prometheus https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series
Prometheus API needs enabling in the prometheus.service:
' --web.enable-admin-api'
(Already added to the live version and the example above)
To return existing jobs:
curl http://localhost:9090/api/v1/label/job/values
Example result:
{"status":"success","data":["node","node_explorer_ServerName","node_exporter_ServerName","prometheus"]}
To remove redundant jobs:
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node"}'
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node_explorer_ServerName"}'
Then finish deleting entirely:
curl -X POST 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
To restart:
sudo systemctl restart prometheus
Check its status:
sudo systemctl status prometheus
To check:
curl http://localhost:9090/api/v1/label/job/values
Add Prometheus Node Exporter
Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors.
Resources
https://prometheus.io/docs/guides/node-exporter/
https://devopscube.com/monitor-linux-servers-prometheus-node-exporter/
https://blog.devops.dev/a-step-by-step-guide-to-installing-prometheus-and-grafana-201c66f88b73
https://pimylifeup.com/raspberry-pi-prometheus/
Create a node_exporter account
I’ve already said that I prefer to have separate accounts for distinct processes.
In a Terminal:
sudo useradd -rs /bin/false node_exporter
Download
cd ~
sudo wget -O node_exporter.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-armv7.tar.gz
Extract, copy and chown
tar xvf node_exporter.tar.gz
sudo mv node_exporter-1.8.2.linux-armv7/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
Tidy up
sudo rm -R node_exporter-1.8.2.linux-armv7
sudo rm node_exporter.tar.gz
Configure ufw
Allow access to ’node_exporter’ from LAN only.
To see the current list of applications allowed through the firewall, in a Terminal:
sudo ufw verbose
Then:
sudo ufw allow from server.ip/24 to any port 9100 comment "node_exporter traffic from LAN only"
Create a node_exporter service file under systemd
In a Terminal:
sudo nano /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
Reload the systemd daemon:
sudo systemctl daemon-reload
Start:
sudo systemctl start node_exporter
Enable the daemon to restart after a reboot:
sudo systemctl enable node_exporter
Check its status:
sudo systemctl status node_exporter
To view the metrics homepage go to:
http://server.ip:9100/metrics
Include the node_exporter stats in the Prometheus data
Spacing is ! important !
In a Terminal:
sudo nano /etc/prometheus/prometheus.yml
# ServerName node_exporter job
- job_name: "node_exporter_ServerName"
static_configs:
- targets: ['localhost:9100']
Restart Prometheus
To restart:
sudo systemctl restart prometheus
Check its status:
sudo systemctl status prometheus
To see the new endpoint ’node_exporter_ServerName’ in the targets homepage go to:
http://server.ip:9090/targets
Create the node_exporter dashboard
Resources:
https://devopscube.com/integrate-visualize-prometheus-grafana/
https://grafana.com/grafana/dashboards/?dataSource=prometheus
https://grafana.com/grafana/dashboards/1860-node-exporter-full/
Add Prometheus as a datasource:
Steps:
Connection
Prometheus server URL
http://localhost:9090
Import a dashboard
A pre-made dashboard already exists.
It can be imported and tinkered with.
Steps:
Dashboards
New
Import
https://grafana.com/grafana/dashboards/1860-node-exporter-full
Load
Select datasource
Import
Save
Add nginx stub_status
The ngx_http_stub_status_module module provides access to basic status information.
This module is not built by default, it should be enabled with the
--with-http_stub_status_moduleconfiguration parameter.https://nginx.org/en/docs/http/ngx_http_stub_status_module.html
Resources:
https://github.com/nginxinc/nginx-prometheus-exporter?tab=readme-ov-file#prerequisites
Add the stub_status module:
In a Terminal:
sudo nano /etc/nginx/conf.d/stub_status.conf
server {
listen 8080;
# Optionally: allow access only from localhost
listen localhost:8080;
location /stub_status {
stub_status;
}
}
Check and Reload nginx
To check the config syntax and restart:
sudo nginx -t && sudo systemctl restart nginx
Configure ufw
Allow access to stub_status from LAN only.
To see the current list of applications allowed through the firewall, in a Terminal:
sudo ufw verbose
Then:
sudo ufw allow from server.ip/24 to any port 8080 comment "stub_status traffic from LAN only"
To view the stub_status homepage go to:
http://server.ip:8080/stub_status
Add nginx-prometheus-exporter
NGINX Prometheus exporter makes it possible to monitor NGINX or NGINX Plus using Prometheus.
Create an nginx_exporter account
I’ve already said that I prefer to have separate accounts for distinct processes.
In a Terminal:
sudo useradd -rs /bin/false nginx_exporter
Download
cd ~
wget https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/v1.3.0/nginx-prometheus-exporter_1.3.0_linux_armv7.tar.gz
Extract, copy and chown
tar xvf nginx-prometheus-exporter_1.3.0_linux_armv7.tar.gz
sudo mv nginx-prometheus-exporter /usr/local/bin/nginx_prometheus_exporter
sudo chown nginx_exporter:nginx_exporter /usr/local/bin/nginx_prometheus_exporter
Tidy up
rm nginx-prometheus-exporter_1.3.0_linux_armv7.tar.gz
rm -R manpages
rm -R completions
rm LICENSE
rm README.md
Configure ufw
Allow access to ’nginx_exporter’ from LAN only
To see the current list of applications allowed through the firewall, in a Terminal:
sudo ufw verbose
Then:
sudo ufw allow from server.ip/24 to any port 9113 comment "nginx_exporter traffic from LAN only"
Create an nginx_exporter service file under systemd
In a Terminal:
sudo nano /etc/systemd/system/nginx_exporter.service
[Unit]
Description=Nginx Exporter
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0
[Service]
User=nginx_exporter
Group=nginx_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/nginx_prometheus_exporter \
-nginx.scrape-uri=http://localhost:8080/stub_status
[Install]
WantedBy=multi-user.target
Reload the systemd daemon:
sudo systemctl daemon-reload
Start:
sudo systemctl start nginx_exporter
Enable the daemon to restart after a reboot:
sudo systemctl enable nginx_exporter
Check its status:
sudo systemctl status nginx_exporter
Update the Prometheus yaml
sudo nano /etc/prometheus/prometheus.yml
# ServerName nginx_exporter job
- job_name: "nginx_exporter_ServerName"
static_configs:
- targets: ['localhost:9113']
Restart Prometheus
To restart:
sudo systemctl restart prometheus
Check its status:
sudo systemctl status prometheus
To view the targets homepage go to:
http://server.ip:9090/targets
To see then new endpoint nginx_exporter_ServerName
Create an nginx_exporter dashboard
Official dashboard for NGINX Prometheus exporter for https://github.com/nginxinc/nginx-prometheus-exporter
Resources
https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/grafana/dashboard.json
Use the New Dashboard button and click Import.
Upload dashboard.json or copy and paste the contents of the file in the textbox and click Load.
Set the Prometheus datasource and click Import.
The dashboard will appear.
Note how you can filter the instance label just below the dashboard title (top left corner).
This allows you to filter metrics per instance.
By default, all instances are selected.