v2.10.2

Prometheus EC2 service-discovery credential-rotation fix (#53): refresh-ec2-credentials.sh now restarts prometheus alongside cloudwatch-exporter and grafana on IMDS key rotation.

v2.10.1 — RES Node List Owner-column fix

v2.10.1 — RES tag correction

Patch on top of v2.10.

Fixed

• RES Node List: removed the always-empty Owner column. v2.10 sourced an
  Owner column from a res:Owner instance tag, but RES does not tag VDI
  instances with the desktop owner — the owner is held in RES's database
  (idea_session_owner), not as an EC2 tag (confirmed against the
  RES source). The column would have been
  permanently blank. The list now uses Project (res:Project, which RES
  propagates to every project instance) and Desktop (the instance Name
  tag) for attribution. The res_owner relabel was dropped from
  prometheus/prometheus-res.yml.

No other changes. RES monitoring remains fully opt-in.

v2.10 — Amazon RES VDI monitoring + GPU clock fix

v2.10 — Amazon RES VDI monitoring

Monitor Amazon Research and Engineering Studio (RES)
virtual desktops on the same monitoring stack you already run on a
ParallelCluster or PCS head node — useful for rightsizing (e.g. spotting an
overprovisioned, idle-GPU g5.48xlarge). Plus a GPU clock-unit fix.

No breaking changes. RES support is fully opt-in and inert unless
RES_ENVIRONMENT_NAME is set — existing deployments are unaffected.

Added — RES desktop monitoring (#52)

• RES desktops are treated as compute nodes (node_exporter + dcgm-exporter on
  GPU desktops) and discovered automatically by their res:EnvironmentName tag.
• RES Node List and RES Node Details dashboards, mirroring the Compute
  equivalents, with click-through. The list shows Owner / Project / Instance
  Type and a GPU% column so idle-GPU rightsizing candidates stand out.
• New opt-in res_instances Prometheus scrape job (prometheus/prometheus-res.yml),
  appended and templated by the installer only when RES_ENVIRONMENT_NAME is set.
• installer/platform/res.sh: post-install.sh can run on a RES desktop as a
  compute node, detected via RES_ENVIRONMENT_NAME or the res:EnvironmentName
  IMDS instance tag. Wire it into your RES project launch script — see the
  "Monitoring Amazon RES desktops" section in the README.

Fixed

• GPU Details: SM / Memory clock graphs showed Hz instead of GHz. The two
  timeseries clock panels applied Grafana's hertz unit to the raw DCGM value,
  which is reported in MHz, so ~1980 MHz rendered as ~2 kHz. They now scale by
  1e6 before the unit, matching the already-correct stat panels.

Validation

The RES data path was validated live on AWS: EC2 service discovery by the
res:EnvironmentName tag, relabeling to instance_name="RES" with
owner/project/desktop labels, the dashboard PromQL, the details drill-down, and
the IMDS detection path. The GPU% column (DCGM) and the installer on a real
RES-ready AMI are best confirmed in your RES environment during rollout.

Full changelog: see CHANGELOG.md.

v2.9.1 — Configurable dcgm-exporter image

What's new

Makes the dcgm-exporter image configurable via the DCGM_EXPORTER_IMAGE environment variable, defaulting to the existing safe pin. This lets deployments on newer GPUs that need DCGM ≥ 4.4.0 — e.g. B300 (p6-b300) — run a capable build without forking. No breaking changes.

Contributed by @DaisukeMiyamoto, validated on real B300 and B200 hardware.

How it works

• compose/compute.gpu.yml: image: ${DCGM_EXPORTER_IMAGE:-nvcr.io/nvidia/k8s/dcgm-exporter:4.2.0-4.1.0-ubuntu22.04} — the default stays inline, so a manual docker compose up still works.
• installer/install.sh exports DCGM_EXPORTER_IMAGE (empty when unset → the inline default applies) before the GPU docker compose up, so an operator value set in launch-template user data flows through.

Usage (newer GPUs)

Set the override in the compute node's user data, by digest to bypass the Docker 29.x OCI-index tag-pull failure (#47):

export DCGM_EXPORTER_IMAGE=nvcr.io/nvidia/k8s/dcgm-exporter@sha256:<digest>

Leave it unset to keep the default 4.2.0 pin (covers up to B200).

Backward compatibility

Unset → existing 4.2.0 behavior, unchanged. Verified on B200 with the default and B300 with a DCGM 4.5.2 digest override (312 DCGM_FI_* series flowing, including the DCGM_FI_PROF_* profiling set the dashboards consume).

Closes #50 via #51.

Full Changelog: v2.9...v2.9.1

v2.9 — Grafana 11 → 13 upgrade

What's new

Grafana 11 → 13 upgrade — the deferred major bump from the v2.8 dependency refresh, handled as its own release given the two-major jump. Validated end-to-end on a live ParallelCluster (Grafana 13.0.2 running, all dashboards provision and render, both Prometheus and CloudWatch datasources work, CloudWatch Logs queries return data).

Changed

• Grafana 11.2.2 → 13.0.2.

Fixed (pre-existing installer bug, found during validation)

• Platform dashboards were never deployed on ParallelCluster. The installer's dashboard-deploy step copied from grafana/dashboards/${PLATFORM}, but PLATFORM is parallelcluster while the subdir is named pcluster. The copy silently no-op'd (suppressed stderr) and the follow-up rm -rf then deleted pcluster/, so logs.json (Cluster Logs) and head-node-details.json (HeadNode Details) were lost on every clean PC install. PCS was unaffected (PLATFORM=pcs matched pcs/). Now maps parallelcluster→pcluster. If you deployed v2.7/v2.8 on ParallelCluster and the Cluster Logs / HeadNode Details dashboards were missing, this is why — v2.9 fixes it.

Compatibility audit (clean)

• No Angular panels (removed in v12); all dashboards use modern React panel types at schema v39.
• No legacy alerting (removed in v11/v12) — this project ships no alert rules.
• Datasources are file-provisioned and referenced by name, so the v13 removal of deprecated numeric-id data source APIs and the RBAC/Terraform data-source-permission changes do not apply.
• Grafana GF_* env vars and Cognito OAuth keys unchanged across 11→13.

Full Changelog: v2.8...v2.9

v2.8 — Dependency refresh

What's new

Dependency refresh — bumps the low-risk pinned components to current upstream releases. No functional changes. Validated end-to-end on a live Amazon Linux 2023 instance (Docker Compose v5.1.4 installs and runs; all three bumped images pull and return 200 on their readiness/metrics endpoints).

Changed

Component	From	To
Prometheus	v3.1.0	v3.12.0
node_exporter	v1.9.0	v1.11.1
Pushgateway	v1.11.2	v1.11.3
Docker Compose plugin	v2.29.7	v5.1.4

• Docker Compose v5.1.4 is installed as a binary on Amazon Linux 2 / 2023; Ubuntu and RHEL continue to use the distro docker-compose-plugin package.
• README Components table synced; dropped stale "v2" wording from the Compose label and installer comments.

Held back deliberately

• Grafana stays at 11.2.2 — the 11 → 13 jump is two majors and needs a dedicated dashboard/datasource compatibility pass before bumping.
• DCGM exporter stays at 4.2.0-4.1.0-ubuntu22.04 — newer tags aren't pullable on Docker 29.x (#47); making the image configurable is tracked in #50.

Full Changelog: v2.7...v2.8

v2.7 — EFA fabric metrics & Cluster Logs dashboard

What's new

Closes the two main gaps identified against the AWS HPC blog's observability_for_pcs solution. Validated end-to-end on a live ParallelCluster (hpc8a.96xlarge + g6e.16xlarge, both EFA-enabled).

EFA fabric metrics

• New efa-metrics.sh textfile collector reads the EFA hw_counters/ directory — RDMA read/write bytes, SRD retransmits, work-request errors — which node_exporter's infiniband collector does not expose. Surfaced as node_amazonefa_* (matching the upstream awsome-distributed-training EFA exporter names) via a systemd timer on compute and head/login nodes. No-op on non-EFA instances.
• Compute Node Details: four new EFA panels (RDMA Read/Write Throughput, SRD Retransmitted Packets, Work-Request Errors). The existing bandwidth/packet panels were also migrated from node_infiniband_* to node_amazonefa_* — on real EFA hardware the infiniband collector emits nothing, so those panels previously stayed empty on EFA instances.

Cluster Logs dashboard (ParallelCluster)

• New searchable CloudWatch Logs dashboard: slurmctld, slurmd, clustermgtd, computemgtd, cfn-init, cloud-init, plus an all-streams panel, with a free-text filter. Replaces the old logs.json.disabled stub.
• The installer pins the log group name + ARN (from /etc/chef/dna.json + region + account), and the Grafana container now mounts the host-refreshed AWS credentials so the CloudWatch datasource works under Imds.Secured=true. Default window is 12h so bootstrap logs are visible.

Fixes surfaced during hardware validation

• Dashboard provisioning dedup: the installer now removes both platform subdirs after copying to root. Previously the active platform's subdir was left in place, so dashboards existed twice → duplicate UID → Grafana refused to save any provisioned dashboard.
• CloudWatch credential refresh: refresh-ec2-credentials.sh restarts the cloudwatch-exporter and Grafana when IMDS rotates the role credentials, so the long-running CloudWatch consumers don't fail with expired tokens (~6h) and blank the Storage/Logs dashboards.
• EFS metric names corrected in storage.json (iobytes/iolimit, not io_bytes/io_limit).
• FSx utilization panels documented: CPUUtilization / FileServerDiskThroughputUtilization are only published by Persistent-2 (and some Persistent-1 SSD) Lustre deployments — not Scratch/HDD.
• CloudWatch datasource region is now substituted (was an invalid hardcoded us-east).

Full Changelog: v2.6.5...v2.7

v2.6.5 — GPU monitoring fixes for PCS on Docker 29.x

What's new

GPU monitoring fixes for AWS PCS on Docker 29.x, contributed and validated on real PCS GPU hardware by @DaisukeMiyamoto. Completes issue #47.

Fixes

• dcgm-exporter not pullable on Docker 29.x (compose/compute.gpu.yml) — Repinned 4.5.2-4.8.1-ubuntu22.04 to 4.2.0-4.1.0-ubuntu22.04. From tag 4.2.3 onward NVCR publishes an OCI image index with attestation/SBOM manifests that Docker 29.x cannot pull (error from registry: Incorrect Repository Format), leaving GPU compute nodes with no dcgm-exporter and empty GPU dashboards. 4.2.0-4.1.0 is the newest tag whose manifest is a plain Docker manifest-list v2; it pulls cleanly on Docker 29.x and emits the same DCGM_FI_DEV_* fields the dashboards use.
• Misleading GPU Health throttle tiles (grafana/dashboards/gpu-health.json) — Removed the thermal-throttle and power-throttle summary tiles. They counted rate(DCGM_FI_DEV_*_VIOLATION[1h]) > 0, but those are monotonic accumulated-throttle-time counters, so any GPU that has ever throttled reads non-zero — healthy H100s were flagged red essentially always (16/16 on an idle cluster). The per-node throttle time-series panels below- Misleading GPU Health throttle tiles (grafana/dashboards/gpsummary-tile false-positive (also #47) was fixed in v2.6.4 (count_over_timetomax_over_time`).

Credits

Thanks to @DaisukeMiyamoto for the fixes and the real-hardware validation.

Full Changelog: v2.6.4...v2.6.5

v2.6.4 — PCS reliability fixes

What's new

Two AWS PCS reliability fixes, contributed and validated on real PCS hardware by @DaisukeMiyamoto.

Fixes

• Login Node dashboard HTTP 422 (prometheus/prometheus-pcs.yml) — A PCS-managed login node carries the aws:pcs:cluster-id tag, so it was discovered by both the static login_node job and EC2 service discovery, producing duplicate node_uname_info series and breaking the dashboard group_left joins with many-to-many matching not allowed. Node-type selection now keys on the existing monitoring-role tag and drops monitoring-role=login from EC2 SD, so the login node is scraped only once. This also frees the EC2 Name tag for arbitrary operator use. (#45 via #46)
• First-boot "Stale file handle" race — The installer extracted, chowned, and seded the monitoring tree under the shared /home filesystem (FSx/NFS on PCS). When the login node and compute nodes bootstrapped concurrently they clobbered each other's inodes, intermittently failing with error reading input file: Stale file handle and leaving nodes unmonitored. The tree now installs on node-local - **First-boot "Stale file handle" race** — The installer extracted, chown
  Fully backward compatible with Amazon Linux 2023 and ParallelCluster. Nodes without a monitoring-role tag are still treated as compute, and legacy Name=Compute nodes resolve to the same dashboard value.

Credits

Thanks to @DaisukeMiyamoto for both fixes and the thorough real-hardware validation.

Full Changelog: v2.6.3...v2.6.4

v2.6.3 — Ubuntu PCS AMI support

What's new

Adds support for running the monitoring stack on Ubuntu-based AWS PCS AMIs (validated on the PCS Ubuntu 24.04 DLAMI Base). The installer previously assumed an Amazon Linux 2023 environment (ec2-user), which broke on Ubuntu where the OS user is ubuntu.

Fixes

• installer/install.sh — Remove an illegal top-level local declaration in the PCS login-node branch. Under Ubuntu's bash with set -euo pipefail this raised local: can only be used in a function and aborted the install.
• installer/platform/pcs.sh — Auto-detect the platform user: ubuntu when /home/ubuntu exists and the ubuntu user is present, otherwise the existing ec2-user default.
• post-install.sh — Apply the same auto-detection for CLUSTER_USER so the tarball is extracted into the correct home directory.

Backward compatibility

Fully backward compatible with Amazon Linux 2023: when /home/ubuntu is absent, the ec2-user default is preserved. No changes to the ParallelCluster (pcluster) code path.

Credits

Thanks to @DaisukeMiyamoto for the report and the fix (#43, #44).

Full Changelog: v2.6.2...v2.6.3