Set up Prometheus and connect linode servers together

Tutorial

Go to Linode, open VPC, create a new VPC and add all nodes that you want, restart nodes, then continue this tutorial. This makes sure that all nodes are on the same subnet network.

Download node_exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

Extract

tar xvfz node_exporter-1.8.2.linux-amd64.tar.gz

Create systemd for node_exporter

sudo nano /etc/systemd/system/node_exporter.service

With content

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Restart=always
ExecStart=/root/node_exporter/node_exporter --web.listen-address="0.0.0.0:9100"

[Install]
WantedBy=multi-user.target

 Then make sure to start, auto start, and make sure it's running

sudo systemctl daemon-reload
sudo systemctl start node_exporter.service
sudo systemctl status node_exporter.service
curl http://localhost:9100/metrics

If service did not start, check for existing service on port 9100

sudo ss -tulnp | grep 9100

In that case change port to 9101 or similar in the systemd service file. 

Add ufw rule to allow remote connections to this server on port 9100

 sudo ufw allow from 192.168.1.0/24 to any port 9100
 sudo ufw reload
 sudo ufw status numbered