Extend the overview dashboard to reflect the mixed validator (vero and
vouch) and multi-client ecl fleet instead of vouch-only assumptions:
Machine inventory row: three bargauge panels (Validator, Executable,
Consensus), one colored bar per machine per role. Green/full when a
client is up, red/short when its scrape target reports down (label
includes " DOWN" so the specific failure is named inline).
Machines with no validator role are simply omitted from that panel.
Validators/Execution layer/Beacon nodes rows: added " clients
up" bargauges showing the live split by client type (e.g. Total,
Vero, Vouch), grouped together under Validators up so all the
summary "up" panels read together before the detail/timeseries
panels.
Removed the "Active beacon clients" panel, which read Vouch's own
internal consensusclient_multi_connections gauge and had already
gone blind on machines running vero instead of vouch; beacon counts
now come from each beacon client's own up{} series.
Removed "Vouch ready", now redundant with Validators up.
Every query and panel was verified live end-to-end (a tunnel to a
running monitor's Mimir, driving grafatui directly) before landing,
including confirming real limitations of grafatui v0.1.7 that shaped
this design: stat panels only ever render one series per panel (hence
bargauge for the client-type splits), table panels are hardcoded to
exactly two columns with no per-label column support, and bar order
in bargauge panels is value-sorted with no name/host sort - which is
why per-role bars live in one panel each rather than a single
unified grid.
Closes #55
Extend the overview dashboard to reflect the mixed validator (vero and
vouch) and multi-client ecl fleet instead of vouch-only assumptions:
- Machine inventory row: three bargauge panels (Validator, Executable,
Consensus), one colored bar per machine per role. Green/full when a
client is up, red/short when its scrape target reports down (label
includes "<client> DOWN" so the specific failure is named inline).
Machines with no validator role are simply omitted from that panel.
- Validators/Execution layer/Beacon nodes rows: added "<role> clients
up" bargauges showing the live split by client type (e.g. Total,
Vero, Vouch), grouped together under Validators up so all the
summary "up" panels read together before the detail/timeseries
panels.
- Removed the "Active beacon clients" panel, which read Vouch's own
internal consensusclient_multi_connections gauge and had already
gone blind on machines running vero instead of vouch; beacon counts
now come from each beacon client's own up{} series.
- Removed "Vouch ready", now redundant with Validators up.
Every query and panel was verified live end-to-end (a tunnel to a
running monitor's Mimir, driving grafatui directly) before landing,
including confirming real limitations of grafatui v0.1.7 that shaped
this design: stat panels only ever render one series per panel (hence
bargauge for the client-type splits), table panels are hardcoded to
exactly two columns with no per-label column support, and bar order
in bargauge panels is value-sorted with no name/host sort - which is
why per-role bars live in one panel each rather than a single
unified grid.
eureka-cpu
requested review from brianmcgee 2026-07-31 00:22:52 +00:00
eureka-cpu
self-assigned this 2026-07-31 00:22:56 +00:00
The dashboard can be viewed from this repo via ssh tunnel. Two buffers, one for the tunnel and one to run the dashboard:
ssh -L 9009:localhost:9009 root@<monitor-ip>
nix run .#grafatui -- \
--prometheus-url http://localhost:9009/prometheus \
--grafana-json ./dashboards/overview.json
The dashboard can be viewed from this repo via ssh tunnel. Two buffers, one for the tunnel and one to run the dashboard:
```sh
ssh -L 9009:localhost:9009 root@<monitor-ip>
```
```sh
nix run .#grafatui -- \
--prometheus-url http://localhost:9009/prometheus \
--grafana-json ./dashboards/overview.json
```
Extend the overview dashboard to reflect the mixed validator (vero and
vouch) and multi-client ecl fleet instead of vouch-only assumptions:
- Machine inventory row: three bargauge panels (Validator, Executable,
Consensus), one colored bar per machine per role. Green/full when a
client is up, red/short when its scrape target reports down (label
includes "<client> DOWN" so the specific failure is named inline).
Machines with no validator role are simply omitted from that panel.
- Validators/Execution layer/Beacon nodes rows: added "<role> clients
up" bargauges showing the live split by client type (e.g. Total,
Vero, Vouch), grouped together under Validators up so all the
summary "up" panels read together before the detail/timeseries
panels.
- Removed the "Active beacon clients" panel, which read Vouch's own
internal consensusclient_multi_connections gauge and had already
gone blind on machines running vero instead of vouch; beacon counts
now come from each beacon client's own up{} series.
- Removed "Vouch ready", now redundant with Validators up.
Every query and panel was verified live end-to-end (a tunnel to a
running monitor's Mimir, driving grafatui directly) before landing,
including confirming real limitations of grafatui v0.1.7 that shaped
this design: stat panels only ever render one series per panel (hence
bargauge for the client-type splits), table panels are hardcoded to
exactly two columns with no per-label column support, and bar order
in bargauge panels is value-sorted with no name/host sort - which is
why per-role bars live in one panel each rather than a single
unified grid.
Vouch and vero expose different attestation metrics, so a mixed fleet
can't be queried uniformly. Add Mimir recording rules projecting both
clients onto shared dvt: series (results, submission timing,
submissions), keyed by host, and rebuild the overview dashboard on
them.
Rules ship via the read-only `local` ruler storage backend so they
deploy atomically with the system. Selectors pin the alloy job because
vero's metric names are unprefixed. Vero's per-beacon-node metrics are
avoided: their own `host` label suppresses the machine label and
collides across machines.
The dashboard now uses Grafana's expanded-row layout so it imports into
web Grafana; grafatui rendering is unchanged. Rules have promtool unit
tests.
Split the sign-requests panel from missed-attestations into successes
and denials/failures: denials are rare and were drowned out by the
succeeded series. An empty denials panel is the healthy state.
Both count panels filter request=~"attestation|proposal|generic"
because dirk's account manager increments the signer counter with
generate/lock/unlock (upstream copy-paste bug in accountmanager.go).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #55
Extend the overview dashboard to reflect the mixed validator (vero and
vouch) and multi-client ecl fleet instead of vouch-only assumptions:
Consensus), one colored bar per machine per role. Green/full when a
client is up, red/short when its scrape target reports down (label
includes " DOWN" so the specific failure is named inline).
Machines with no validator role are simply omitted from that panel.
up" bargauges showing the live split by client type (e.g. Total,
Vero, Vouch), grouped together under Validators up so all the
summary "up" panels read together before the detail/timeseries
panels.
internal consensusclient_multi_connections gauge and had already
gone blind on machines running vero instead of vouch; beacon counts
now come from each beacon client's own up{} series.
Every query and panel was verified live end-to-end (a tunnel to a
running monitor's Mimir, driving grafatui directly) before landing,
including confirming real limitations of grafatui v0.1.7 that shaped
this design: stat panels only ever render one series per panel (hence
bargauge for the client-type splits), table panels are hardcoded to
exactly two columns with no per-label column support, and bar order
in bargauge panels is value-sorted with no name/host sort - which is
why per-role bars live in one panel each rather than a single
unified grid.
ea0a4f74d0to153129dbabThe dashboard can be viewed from this repo via ssh tunnel. Two buffers, one for the tunnel and one to run the dashboard:
Extend the overview dashboard to reflect the mixed validator (vero and vouch) and multi-client ecl fleet instead of vouch-only assumptions: - Machine inventory row: three bargauge panels (Validator, Executable, Consensus), one colored bar per machine per role. Green/full when a client is up, red/short when its scrape target reports down (label includes "<client> DOWN" so the specific failure is named inline). Machines with no validator role are simply omitted from that panel. - Validators/Execution layer/Beacon nodes rows: added "<role> clients up" bargauges showing the live split by client type (e.g. Total, Vero, Vouch), grouped together under Validators up so all the summary "up" panels read together before the detail/timeseries panels. - Removed the "Active beacon clients" panel, which read Vouch's own internal consensusclient_multi_connections gauge and had already gone blind on machines running vero instead of vouch; beacon counts now come from each beacon client's own up{} series. - Removed "Vouch ready", now redundant with Validators up. Every query and panel was verified live end-to-end (a tunnel to a running monitor's Mimir, driving grafatui directly) before landing, including confirming real limitations of grafatui v0.1.7 that shaped this design: stat panels only ever render one series per panel (hence bargauge for the client-type splits), table panels are hardcoded to exactly two columns with no per-label column support, and bar order in bargauge panels is value-sorted with no name/host sort - which is why per-role bars live in one panel each rather than a single unified grid.153129dbabto2ff7d240de