Summary
The user is facing issues with the Grafana User Dashboard, where all statistics are empty except for one plot. They are questioning the functionality and maintenance of the dashboard, suspecting a problem with their Prometheus configuration. They mention using flyte-binary, v1.12.0, and have followed the relevant instructions, but despite Grafana recognizing their projects/domains, the data remains empty after testing both successful and failed workflows. The user notes that the dashboard used to work in April but has since taken down the cluster running Prometheus/Grafana/Flyte and currently lacks a working example. They recall potential updates to Grafana templates and are uncertain if these updates are reflected in the Grafana marketplace or if there have been changes in Prometheus metrics from Flyte. They reference a recent Grafana-related pull request that may indicate changes.
charlie
Thanks for the work on this <@U04H6UUE78B>! I've tested out the updates and left a few small comments on the pull request :raised_hands:
david.espejo
<@U06RTQ8FEP4> <@U07H4UM6PFW> and everyone else interested: would you be able to <https://github.com/flyteorg/flyte/blob/6ef3a755bbbda12536aeb65066aa1ecc1526014c/deployment/stats/prometheus/flyteuser.dashboard.json|try this update> to the Grafana user dashboard and share your feedback https://github.com/flyteorg/flyte/pull/5703|here?
Please keep in mind some metrics there come from kube-state-metrics(eg. CPU/memory usage and ResourceQuota), which I installed using the https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/|kube-prometheus-stack (not the only way to enable it).
Thanks!