OpenWrt and push Prometheus metrics

This note is about sending Prometheus metrics directly from OpenWRT router to Grafana Cloud.
No intermediate scrape agent or Prometheus instance is needed.

Get the metrics

That is documented well in the grafana blog, and boils down to:

# install exporter package
opkg install prometheus-node-exporter-lua
# and the metrics are exposed at
curl localhost:9100/metrics

Now how to ship them to Grafana Cloud?
Articles for prometheus-node-exporter-lua usually assume that you have a Prometheus instance somewhere in the network, which would scrape the metrics from the router and ship them. But can we do it right from the router, maybe by bash?

Send the metrics

There are 2 formats in which Grafana Cloud is accepting the metrics:

  1. remote-write, which is snappy encoded protobuf messages. That complicates things, as it is hard to implement on bash. And go static binary would be >10Mb in size, which is pretty big for a router built-in memdisk
  2. influx-line-proto, which is simple HTTP plain-text POST.

Let’s see if we can convert prometheus metrics to the latter. Excerpt from Grafana Cloud docs:

We convert the above into the following series in Prometheus:
for each (field, value) in fields: metric_name_field{tags...} value @timestamp
For example: diskio,host=work,name=dm-1 write_bytes=651264i,read_time=29i 1661369405000000000
Will be converted to:
diskio_write_bytes{host="work", name="dm-1"} 651264 1661369405000000000
diskio_read_time{host="work", name="dm-1"} 29 1661369405000000000
The timestamp is optional.

In our case, reverse transformation is needed. So the prometheus metric:
node_network_receive_bytes_total{device="wlan0"} 1061251143
should turn to:
node_network_receive_bytes,instance=openwrt,device=wlan0 total=1061251143
Note that in case of direct send, there is no target-level labels (like instance, job etc) attached from scraping agent, so we have to specifically set such labels, like instance here.

Transformation above could be done via sed -r. But if we also want to follow influx-line-proto specs, and correctly escape quotes in the following metric (notice the comma):
node_openwrt_info{target="ipq807x/generic", board_name="redmi,ax6", id="OpenWrt", model="Redmi AX6", release="SNAPSHOT", system="ARMv8 Processor rev 4"} 1
Then some scripting language would be a better alternative. There is no python, but there is lua already installed for UI purposes. Yes, lua does not have regex (because regex lib size is larger than the whole lua distro) but it has match which should be enough.

Below is the resulting script which work in a loop with interval, getting metrics from 127.0.0.1:9100/metrics, transforming them and sending to specified Grafana Cloud url:

#!/usr/bin/lua
local extlabels = ",instance=ax6" -- additional labels to append
local scrape_url = "http://127.0.0.1:9100/metrics" -- URL to scrape metrics from
local url = "https://prometheus-us-central1.grafana.net/api/v1/push/influx/write" -- Grafana Cloud url
local auth = "1234:xxx" -- Grafana Cloud user:password
local interval = 15

local http = require "socket.http"
local ltn12 = require "ltn12"
local function escape(s) return s:gsub("[, =]", "\\%1") end

local function to_influx_line(line)
    if line:match("^#") then return end
    local name, field, ls, val = line:match("^([%w_]+)_(%w+)({[^}]*}) (.+)$")
    if not name then name, field, val = line:match("^([%w_]+)_(%w+) (.+)$") end
    if not name then return end
    local labels = ""
    if ls then
        for label, v in ls:sub(2, -2):gmatch('([%w_]+)="([^"]*)"') do
            labels = labels .. "," .. label .. "=" .. escape(v)
        end
    end
    return name .. extlabels .. labels .. " " .. field .. "=" .. val .. "\n"
end

local function fetch_and_send_metrics()
    local resp = {}
    local _, code = http.request{url = scrape_url, sink = ltn12.sink.table(resp)}
    if code ~= 200 then return false, "Error fetching metrics: HTTP " .. code .. "\n" end

    local metrics = {}
    for line in table.concat(resp):gmatch("[^\r\n]+") do
        local m = to_influx_line(line)
        if m then table.insert(metrics, m) end
    end
    if #metrics == 0 then return false, "No valid metrics\n" end

    local payload = table.concat(metrics)
    resp = {}
    _, code = http.request{
        url = url, method = "POST",
        headers = {["Authorization"] = "Bearer " .. auth, ["Content-Type"] = "text/plain", ["Content-Length"] = #payload},
        source = ltn12.source.string(payload), sink = ltn12.sink.table(resp)
    }

    if code ~= 200 and code ~= 204 then
        return false, "Error sending: HTTP " .. code .. "\n" .. table.concat(resp) .. "\n"
    end
    return true, string.format("Sent %d metrics at %s\n", #metrics, os.date("%Y-%m-%d %H:%M:%S"))
end

print(string.format("Starting metrics collection with %ds interval...\n", interval))
while true do
    local ok, msg = fetch_and_send_metrics()
    if not ok then print(msg) end
    os.execute("sleep " .. interval)
end

Set your variables at the top, save it as /usr/bin/prometheus-push and mark as executable. Start locally first, to check that data is arriving:

chmod +x /usr/bin/prometheus-push
/usr/bin/prometheus-push
Starting metrics collection with 15s interval...
# stop by multiple Ctrl-C

If everything is fine, the output should stay empty. If there is an error that lua has no support for https, you might need to: opkg install luasec

Service

To have this script automatically run on system start and continue to run after you close ssh connection, let’s mark it as a Service:

cat > /etc/init.d/prometheus-push <<EOF
#!/bin/sh /etc/rc.common

START=99
STOP=10
USE_PROCD=1

PROG=/usr/bin/prometheus-push

start_service() {
    procd_open_instance
    procd_set_param command $PROG
    procd_set_param respawn
    procd_set_param stdout 1
    procd_set_param stderr 1
    procd_close_instance
}
EOF
chmod +x /etc/init.d/prometheus-push
/etc/init.d/prometheus-push enable
/etc/init.d/prometheus-push start
/etc/init.d/prometheus-push status
running
ps | grep prometheus
12364 root      2800 S    {prometheus-node} /usr/bin/lua /usr/bin/prometheus-node-exporter-lua --bind 127.0.0.1 --port 9100
18410 root      6184 S    {prometheus-push} /usr/bin/lua /usr/bin/prometheus-push

You can also start and stop it in UI at System > Startup:

Enjoy your metrics, which only need internet connection to work now!

Related content: