Software

Metrics and timeseries

We use prometheus to scrape endpoints for data. These endpoints are simple text, made available on a given port by an exporter

Prometheus has a query language to query the timeseries.

Logs

We use promtail as the agent to send log data to a loki instance.

loki has a query language to search the logs and return log entries.

An alternative would be elasticsearch as the log database and kibana to make queries into the logs.

Presentation

We use grafana as the dashboard "engine" and to query for different datasources. Prometheus and loki are two datasources. There are others as well.

Testing

1. Download a test vm from here and install it in vmware workstation
2. Connect it to vmnet111 (or the equivante 192.168.111.0/24 subnet)
3. There is an information page on http://192.168.111.20

Unprivileged user is sysuser:sysuser123 and admins user is root:root123

Working with grafana

1. Make a block diagram showing the relationship between node_exporter, prometheus and grafana.
2. Add a grafana dashboard that uses the node_exporter values from prometheus
3. Add a grafana dashboard that uses the logs from loki

4. Set up an extra VM with node_exporter and update prometheus to scrape from it

   This is either a new Linux VM or the openbsd router.

If you have the obsolete OpenBSD 6.6 router, please upgrade to version 7.0. See here