Learning notes
Setting up Prometheus has been pretty self-explanatory but here are some learning notes I've taken along the way.
Vectors
Prometheus docs mention instant-vectors and range-vectors. PromQL functions take different vector types as arguments.
What is less clear is that you can easily convert an instant-vector to a range-vector, for use in functions that take range-vector arguments.
Using range-vector selectors.
For example a networking counter would likely use rate().
rate(collectd_interface_if_octets_0_total{instance="gw1.swehack.local",interface="re1"}[5m])
Grafana data source
Selecting Prometheus as your grafana data source means you're also using PromQL in grafana to query data. So grafana docs are pointless in this case.
What to query for?
Do a manual scrape of any exporter to see what type of metrics it's exporting.
For example;
$ curl -X GET 'http://localhost:9105/metrics'
Which is the port where I run a collectd_exporter process.
Collectd
The collectd_exporter also acts as a collectd server in that all the collectd clients push their metrics to it and it then exposes them through its web interface.