dashboards: add machine inventory and per-role client-up panels #78

Merged
eureka-cpu merged 4 commits from dashboard/multi-validator-client-inventory into main 2026-08-31 19:33:31 +00:00
Collaborator

Closes #55

Extend the overview dashboard to reflect the mixed validator (vero and
vouch) and multi-client ecl fleet instead of vouch-only assumptions:

  • Machine inventory row: three bargauge panels (Validator, Executable,
    Consensus), one colored bar per machine per role. Green/full when a
    client is up, red/short when its scrape target reports down (label
    includes " DOWN" so the specific failure is named inline).
    Machines with no validator role are simply omitted from that panel.
  • Validators/Execution layer/Beacon nodes rows: added " clients
    up" bargauges showing the live split by client type (e.g. Total,
    Vero, Vouch), grouped together under Validators up so all the
    summary "up" panels read together before the detail/timeseries
    panels.
  • Removed the "Active beacon clients" panel, which read Vouch's own
    internal consensusclient_multi_connections gauge and had already
    gone blind on machines running vero instead of vouch; beacon counts
    now come from each beacon client's own up{} series.
  • Removed "Vouch ready", now redundant with Validators up.

Every query and panel was verified live end-to-end (a tunnel to a
running monitor's Mimir, driving grafatui directly) before landing,
including confirming real limitations of grafatui v0.1.7 that shaped
this design: stat panels only ever render one series per panel (hence
bargauge for the client-type splits), table panels are hardcoded to
exactly two columns with no per-label column support, and bar order
in bargauge panels is value-sorted with no name/host sort - which is
why per-role bars live in one panel each rather than a single
unified grid.

Closes #55 Extend the overview dashboard to reflect the mixed validator (vero and vouch) and multi-client ecl fleet instead of vouch-only assumptions: - Machine inventory row: three bargauge panels (Validator, Executable, Consensus), one colored bar per machine per role. Green/full when a client is up, red/short when its scrape target reports down (label includes "<client> DOWN" so the specific failure is named inline). Machines with no validator role are simply omitted from that panel. - Validators/Execution layer/Beacon nodes rows: added "<role> clients up" bargauges showing the live split by client type (e.g. Total, Vero, Vouch), grouped together under Validators up so all the summary "up" panels read together before the detail/timeseries panels. - Removed the "Active beacon clients" panel, which read Vouch's own internal consensusclient_multi_connections gauge and had already gone blind on machines running vero instead of vouch; beacon counts now come from each beacon client's own up{} series. - Removed "Vouch ready", now redundant with Validators up. Every query and panel was verified live end-to-end (a tunnel to a running monitor's Mimir, driving grafatui directly) before landing, including confirming real limitations of grafatui v0.1.7 that shaped this design: stat panels only ever render one series per panel (hence bargauge for the client-type splits), table panels are hardcoded to exactly two columns with no per-label column support, and bar order in bargauge panels is value-sorted with no name/host sort - which is why per-role bars live in one panel each rather than a single unified grid.
eureka-cpu requested review from brianmcgee 2026-07-31 00:22:52 +00:00
eureka-cpu self-assigned this 2026-07-31 00:22:56 +00:00
eureka-cpu force-pushed dashboard/multi-validator-client-inventory from ea0a4f74d0 to 153129dbab 2026-07-31 18:02:20 +00:00 Compare
Author
Collaborator

The dashboard can be viewed from this repo via ssh tunnel. Two buffers, one for the tunnel and one to run the dashboard:

ssh -L 9009:localhost:9009 root@<monitor-ip>
nix run .#grafatui -- \
  --prometheus-url http://localhost:9009/prometheus \
  --grafana-json ./dashboards/overview.json
The dashboard can be viewed from this repo via ssh tunnel. Two buffers, one for the tunnel and one to run the dashboard: ```sh ssh -L 9009:localhost:9009 root@<monitor-ip> ``` ```sh nix run .#grafatui -- \ --prometheus-url http://localhost:9009/prometheus \ --grafana-json ./dashboards/overview.json ```
brianmcgee added 1 commit 2026-08-10 12:44:37 +00:00
dashboards: add machine inventory and per-role client-up panels
buildbot/nix-eval evaluation succeeded
buildbot/nix-build gitea:clan/dvt-lite#checks.default.checks.x86_64-linux.integration-test-erigon-lodestar failed
buildbot/nix-build gitea:clan/dvt-lite#checks.default.checks.x86_64-linux.integration-test-besu-nimbus failed
buildbot/nix-build 2 of 29 attributes failed
2ff7d240de
Extend the overview dashboard to reflect the mixed validator (vero and
vouch) and multi-client ecl fleet instead of vouch-only assumptions:

- Machine inventory row: three bargauge panels (Validator, Executable,
  Consensus), one colored bar per machine per role. Green/full when a
  client is up, red/short when its scrape target reports down (label
  includes "<client> DOWN" so the specific failure is named inline).
  Machines with no validator role are simply omitted from that panel.
- Validators/Execution layer/Beacon nodes rows: added "<role> clients
  up" bargauges showing the live split by client type (e.g. Total,
  Vero, Vouch), grouped together under Validators up so all the
  summary "up" panels read together before the detail/timeseries
  panels.
- Removed the "Active beacon clients" panel, which read Vouch's own
  internal consensusclient_multi_connections gauge and had already
  gone blind on machines running vero instead of vouch; beacon counts
  now come from each beacon client's own up{} series.
- Removed "Vouch ready", now redundant with Validators up.

Every query and panel was verified live end-to-end (a tunnel to a
running monitor's Mimir, driving grafatui directly) before landing,
including confirming real limitations of grafatui v0.1.7 that shaped
this design: stat panels only ever render one series per panel (hence
bargauge for the client-type splits), table panels are hardcoded to
exactly two columns with no per-label column support, and bar order
in bargauge panels is value-sorted with no name/host sort - which is
why per-role bars live in one panel each rather than a single
unified grid.
brianmcgee force-pushed dashboard/multi-validator-client-inventory from 153129dbab to 2ff7d240de 2026-08-10 12:44:37 +00:00 Compare
brianmcgee added 1 commit 2026-08-10 14:43:00 +00:00
monitor: normalise vouch/vero attestation metrics with recording rules
buildbot/nix-eval evaluation succeeded
buildbot/nix-build 29 attributes built
ae6cc08ce9
Vouch and vero expose different attestation metrics, so a mixed fleet
can't be queried uniformly. Add Mimir recording rules projecting both
clients onto shared dvt: series (results, submission timing,
submissions), keyed by host, and rebuild the overview dashboard on
them.

Rules ship via the read-only `local` ruler storage backend so they
deploy atomically with the system. Selectors pin the alloy job because
vero's metric names are unprefixed. Vero's per-beacon-node metrics are
avoided: their own `host` label suppresses the machine label and
collides across machines.

The dashboard now uses Grafana's expanded-row layout so it imports into
web Grafana; grafatui rendering is unchanged. Rules have promtool unit
tests.
brianmcgee added 1 commit 2026-08-10 14:57:37 +00:00
dashboards: add dirk signing row to overview
buildbot/nix-eval evaluation succeeded
buildbot/nix-build gitea:clan/dvt-lite#checks.default.checks.x86_64-linux.integration-test-geth-lighthouse failed
buildbot/nix-build gitea:clan/dvt-lite#checks.default.checks.x86_64-linux.integration-test-erigon-lodestar failed
buildbot/nix-build 2 of 29 attributes failed
10680527b9
Split the sign-requests panel from missed-attestations into successes
and denials/failures: denials are rare and were drowned out by the
succeeded series. An empty denials panel is the healthy state.

Both count panels filter request=~"attestation|proposal|generic"
because dirk's account manager increments the signer counter with
generate/lock/unlock (upstream copy-paste bug in accountmanager.go).
eureka-cpu added 1 commit 2026-08-31 19:09:26 +00:00
Merge branch 'main' into dashboard/multi-validator-client-inventory
buildbot/nix-eval evaluation succeeded
buildbot/nix-build 29 attributes built
ab598e1621
eureka-cpu merged commit 0e19901820 into main 2026-08-31 19:33:31 +00:00
eureka-cpu deleted branch dashboard/multi-validator-client-inventory 2026-08-31 19:33:35 +00:00
Sign in to join this conversation.