My home monitoring setup with Grafana and Prometheus
 
 
Go to file
ekzyis d5a784477a Add lightningd target 2022-12-28 08:45:02 +00:00
grafana Fix prometheus connection 2022-12-27 09:34:32 +00:00
prometheus Add lightningd target 2022-12-28 08:45:02 +00:00
screenshots Add README.md 2021-12-10 17:57:27 +01:00
src Add lightningbot process monitoring 2022-10-02 02:53:35 +02:00
timescaledb/docker-entrypoint-initdb.d Update timescaledb config 2022-12-04 02:38:18 +01:00
.env.template pre-commit run --all-files 2022-04-16 20:51:09 +02:00
.gitignore Implement Grafana/Prometheus stack 2022-02-23 03:21:30 +01:00
.pre-commit-config.yaml pip install pre-commit 2022-04-16 20:51:09 +02:00
README.md Update README 2022-12-27 07:22:06 +01:00
docker-compose.yaml Use network_mode host for prometheus 2022-12-27 09:22:32 +00:00
download_dashboards.sh Fix * expanded in dashboard expressions 2022-04-05 15:49:14 +02:00
pyproject.toml pre-commit run --all-files 2022-04-16 20:51:09 +02:00
reload_prom_config.sh Update reload_prom_config 2022-12-27 09:22:32 +00:00
requirements.txt Add tqdm as progress bar for import 2022-05-20 15:38:19 +02:00
upload_dashboards.sh Implement dashboard upload 2022-04-03 17:32:32 +02:00
vps-grafana.service Update systemd unit files 2022-12-04 02:40:13 +01:00
vps-tunnel.service Update systemd unit files 2022-12-04 02:40:13 +01:00

README.md

monitor

Setup

Start Grafana, Prometheus and TimescaleDB:

docker-compose up -d

If first time: Visit http://localhost:3000/ to setup Grafana. Default user and password is admin.

Monitoring

Check scraping configuration in prometheus/prometheus.yml.

You can use cron to make sure scripts run on boot.

System monitoring

Install prometheus-node-exporter: pacman -S prometheus-node-exporter.

Required collectors:

  • stat for node_boot_time_seconds
  • time for node_time_seconds
  • cpu for node_cpu_seconds_total
  • meminfo for node_memory_*
  • filesystem for node_filesystem_*
  • hwmon for node_hwmon_temp_celsius
  • netdev for node_network_*

Therefore, run like this on every host you want to monitor:

prometheus-node-exporter \
  --web.listen-address="10.101.111.x:9100" \
  --collector.disable-defaults \
  --collector.stat \
  --collector.time \
  --collector.cpu \
  --collector.meminfo \
  --collector.filesystem \
  --collector.hwmon \
  --collector.netdev

Latency / bandwidth monitoring

  • ncat -vkl -p 9112 -e '/bin/bash src/latency.sh' (run on localhost)

  • ncat -vkl -p 9113 -e '/bin/bash src/bandwidth.sh' (run on localhost)

Public IP monitoring

Run python src/ip2.py periodically with cron.

Dashboards download / upload

  • Download: bash download_dashboards.sh
  • Upload: bash upload_dashboards.sh

See files for details.