Update templates for external monitoring by mattkjames7 · Pull Request #208 · memgraph/helm-charts

mattkjames7 · 2026-03-17T17:38:20Z

Summary

This PR adds end-to-end remote observability support for both memgraph and memgraph-high-availability Helm charts by introducing:

optional vmagentRemote support for Prometheus remote_write metrics export,
optional vectorRemote sidecars for shipping Memgraph logs to Loki-compatible endpoints,
chart value/template updates and README examples for configuring auth, labels, and namespaces.

It also improves HA behavior by adding safer cluster setup hook/retry handling and forcing mg-exporter rollout on relevant config changes.

Validation

Adds a new CI workflow: .github/workflows/test-remote-monitoring.yaml
Provisions a remote monitoring stack (scripts/remote-monitoring-stack.yaml)
Verifies both metrics and logs ingestion with scripts/verify-remote-monitoring.sh for:
- HA chart deployment
- standalone chart deployment

Notes

Includes minor formatting cleanup in charts/memgraph-lab/values.yaml.

as51340 · 2026-04-09T10:53:05Z

Also, make sure to open a PR on docs for this 😄

Adding monitoring of metrics and logs to S3 for CI runs. Related infra PRs: - memgraph/infra#455 - memgraph/infra#452 Related on helm-charts: - memgraph/helm-charts#208 Related on prometheus-exporter - memgraph/prometheus-exporter#31 Table showing example `CLUSTER_ID`, `CLUSTER_ENV` and `SERVICE_NAME` for each monitored job: | Workflow | Job (`github.job`) | Step | CLUSTER_ID | CLUSTER_ENV | SERVICE_NAME | Backend | |---|---|---|---|---|---|---| | reusable_stress_tests.yaml | `stress_test_standalone` | Run stress tests (native standalone) | `17834421029-stress_test_standalone` | `ci-stress-tests` | `stress-native-standalone` *(default from test_name)* | Docker | | reusable_stress_tests.yaml | `stress_test_native_ha` | Run stress tests (native HA) | `17834421029-stress_test_native_ha` | `ci-stress-tests` | `stress-native-ha` | Docker | | reusable_stress_tests.yaml | `stress_test_docker_ha` | Run stress tests (docker HA) | `17834421029-stress_test_docker_ha` | `ci-stress-tests` | `stress-docker-ha` | Docker | | reusable_stress_tests.yaml | `stress_test_eks_ha` | Run stress tests (EKS HA) | `17834421029-stress_test_eks_ha` | `ci-stress-tests` | `stress-eks-ha` | Docker + `kube-prometheus-stack` within the cluster | | reusable_package_mage.yaml | `BuildAndPush` | Run end-to-end tests | `17834421029-BuildAndPush` | `ci-mage-<arch>` | `mage-e2e` *(default)* | Docker | | reusable_package_mage.yaml | `BuildAndPush` | Run End-to-end correctness tests | `17834421029-BuildAndPush` | `ci-mage-<arch>` | `mage-e2e-correctness` *(default)* | Docker | | reusable_package_mage.yaml | `BuildAndPush` | Run End-to-end migration tests | `17834421029-BuildAndPush` | `ci-mage-<arch>` | `mage-e2e-migration` *(default)* | Docker | | diff_debug.yaml | `core` | Run drivers tests | `17834421029-core` | `ci-diff-debug` | `drivers` *(default)* | Docker | | diff_debug.yaml | `core` | Run HA driver tests | `17834421029-core` | `ci-diff-debug` | `drivers-high-availability` *(default)* | Docker | | diff_debug.yaml | `integration` | Run integration tests | `17834421029-integration` | `ci-diff-debug` | `integration` *(default)* | Docker | | diff_release.yaml | `core` | Run GQL Behave tests | `17834421029-core` | `ci-diff-release` | `gql-behave` *(default)* | Docker | | diff_release.yaml | `stress` | Run stress test (plain) | `17834421029-stress` | `ci-diff-release` | `stress-plain` *(default)* | Docker | | diff_release.yaml | `stress` | Run stress test (SSL) | `17834421029-stress` | `ci-diff-release` | `stress-ssl` *(default)* | Docker | | diff_release.yaml | `stress` | Run durability test (plain) | `17834421029-stress` | `ci-diff-release` | `durability` *(default)* | Docker | | diff_release.yaml | `stress` | Run durability test (large) | `17834421029-stress` | `ci-diff-release` | `durability-large` *(default)* | Docker | | reusable_release_tests.yaml | `debug_build` | Run drivers tests | `17834421029-debug_build` | `ci-release-tests` | `drivers` *(default)* | Docker | | reusable_release_tests.yaml | `debug_build` | Run HA driver tests | `17834421029-debug_build` | `ci-release-tests` | `drivers-high-availability` *(default)* | Docker | | reusable_release_tests.yaml | `debug_build` | Run cppcheck and clang-format | `17834421029-debug_build` | `ci-release-tests` | `cppcheck-and-clang-format` *(default)* | Docker | | reusable_release_tests.yaml | `debug_integration_test` | Run integration tests | `17834421029-debug_integration_test` | `ci-release-tests` | `integration` *(default)* | Docker | | reusable_release_tests.yaml | `malloc_build` | Run GQL Behave tests | `17834421029-malloc_build` | `ci-release-tests` | `gql-behave` *(default)* | Docker | | reusable_release_tests.yaml | `release_build` | Run GQL Behave tests | `17834421029-release_build` | `ci-release-tests` | `gql-behave` *(default)* | Docker | | reusable_release_tests.yaml | `release_durability_stress_tests` | Run stress test (plain) | `17834421029-release_durability_stress_tests` | `ci-release-tests` | `stress-plain` *(default)* | Docker | | reusable_release_tests.yaml | `release_durability_stress_tests` | Run stress test (SSL) | `17834421029-release_durability_stress_tests` | `ci-release-tests` | `stress-ssl` *(default)* | Docker | | reusable_release_tests.yaml | `release_durability_stress_tests` | Run durability test (plain) | `17834421029-release_durability_stress_tests` | `ci-release-tests` | `durability` *(default)* | Docker | | reusable_release_tests.yaml | `release_durability_stress_tests` | Run durability test (large) | `17834421029-release_durability_stress_tests` | `ci-release-tests` | `durability-large` *(default)* | Docker | | reusable_release_tests.yaml | `stress_test_large` | Run stress test (large) | `17834421029-stress_test_large` | `ci-release-tests` | `stress-large` *(default)* | Docker | | reusable_release_tests.yaml | `release_jepsen_test` | Start Jepsen monitoring (habank) | `17834421029-release_jepsen_test` | `ci-release-tests` | `jepsen-habank` | Docker (jepsen script) | | reusable_stress_jepsen.yaml | `ha-test` | Start Jepsen monitoring (workload) | `17834421029-ha-test` | `ci-stress-jepsen` | `jepsen-ha-mt` or `jepsen-hacreate` | Docker (jepsen script) | | diff_jepsen.yaml | `core` | Start Jepsen monitoring (habank) | `17834421029-core` | `ci-jepsen` | `jepsen-habank` | Docker (jepsen script) | Added monitoring to the following CI Jobs/tests: - [x] Diff and Release build and test workflows - [x] Release / Core / GQL Behave - [x] Release / Stress / - [x] stress-plain - [x] stress-ssl - [x] durability - [x] durability-large - [x] Debug / Core / - [x] drivers - [x] drivers-high-availability - [x] Debug / Integration - [x] Jepsen - [x] habank - [x] Stress test large workflow - [x] Stress tests - [x] Native standalone - [x] Native HA - [x] Docker HA - [x] EKS HA - [x] Jepsen stress tests - [x] ISSU - [x] MAGE - [x] E2E - [x] E2E Correctness - [x] E2E Migration TODO (next PR): - Add automated core dump and stack trace uploads to S3 + log entry with links to both. - Monitor E2E tests (currently broken, but should work with OpenTelemetry endpoint being added in memgraph#3911 )  --- > [!NOTE] > **Medium Risk** > Touches core CI execution paths (GitHub workflows, `mgbuild.sh`, and stress/Jepsen/EKS deployment scripts) and introduces new Docker Compose-based monitoring sidecars, which could cause CI flakes or failures if endpoints/targets are misconfigured. > > **Overview** > Adds optional **remote monitoring of CI runs** (metrics + websocket logs) across multiple GitHub workflows by passing new `mgbuild.sh` flags (`--enable-monitoring`, `--monitoring-host`, `--cluster-id`, `--cluster-env`, optional `--service-name`) and wiring in monitoring-related secrets. > > Extends `release/package/mgbuild.sh` to start/stop a new Docker Compose monitoring stack around `test-memgraph`/`test-mage` runs, auto-resolve HA monitoring targets (native/docker/EKS), validate required monitoring args, and handle cleanup/trap chaining. > > Introduces `tools/ci/monitoring/` (compose files + `up.sh`/`down.sh` + templates + Jepsen helper) and updates stress/HA deployment scripts to emit `MEMGRAPH_METRICS_TARGETS`/`MEMGRAPH_LOG_WS_TARGETS` and, for EKS, create external monitoring services; ISSU upgrade test now clones helm-charts and can enable remote write/log shipping via Helm values when monitoring creds are provided. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit da0e4ec. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

mattkjames7 added 4 commits March 1, 2026 14:43

update templates to add loki

0c3cf65

Merge branch 'main' into add-loki

57842a7

broken

5935f5c

fixed for standalone again

444e29e

mattkjames7 mentioned this pull request Mar 18, 2026

Monitoring CI memgraph/memgraph#3890

Merged

25 tasks

mattkjames7 added 23 commits March 18, 2026 17:25

fixed loglevel missing

89e370b

more fixes

548371c

use cluster-setup.yaml from main

9fc425c

Merge branch 'main' into merge-add-loki

8d882be

set hook wait

472de3a

added change to exporter for reloading when config changes

4bed668

update readme

8517410

added monitoring workflow

aeb5c08

update workflow

3a0c2c6

more workflow fixes

c49308b

linting

643d7a4

Merge branch 'main' into add-loki

47716c7

linting

336693e

udpate workflow

fbfe6db

fix HA launch

32ea06f

update workflow

4925293

update example in readme

9343ab9

separate yaml from CI workflow

160d913

update script to fix pre-commit failure

5307113

separate out verification script

2a503ce

udpate workflow

d53e60d

fix workflow

43f3353

fix verification script

5912f3f

mattkjames7 self-assigned this Apr 8, 2026

mattkjames7 added the enhancement label Apr 8, 2026

mattkjames7 added 4 commits April 8, 2026 14:58

addresss pr comment

3212ae4

addresss pr comment

c632add

addresss pr comment

b21c521

addresss pr comment

be733c8

mattkjames7 marked this pull request as ready for review April 8, 2026 14:23

mattkjames7 requested review from antejavor, as51340 and katarinasupe as code owners April 8, 2026 14:23

as51340 requested changes Apr 9, 2026

View reviewed changes

as51340 reviewed Apr 9, 2026

View reviewed changes

Comment thread charts/memgraph-high-availability/values.yaml

as51340 added memgraph-ha Memgraph HA chart issue memgraph Memgraph standalone issue labels Apr 9, 2026

mattkjames7 added 2 commits April 9, 2026 10:29

use localhost for websocket IP

5578fe4

remove hook-weight

614a6ff

as51340 reviewed Apr 9, 2026

View reviewed changes

Comment thread charts/memgraph-high-availability/templates/_helpers.tpl Outdated

mattkjames7 added 5 commits April 9, 2026 11:13

move websocket stuff to values

910fd93

remove vmagent service

8b7f71f

move websocket args to values for standalone

6cd3459

comments

61240db

more comments about duplicate scraping

f46a4ab

mattkjames7 added 2 commits April 9, 2026 12:01

replaced nginx with vmauth to represent production setup

5170d31

wait for port

c5352c3

mattkjames7 mentioned this pull request Apr 9, 2026

Add remote monitoring to helm-charts memgraph/documentation#1583

Merged

11 tasks

mattkjames7 requested a review from as51340 April 9, 2026 11:48

as51340 approved these changes Apr 9, 2026

View reviewed changes

Comment thread charts/memgraph-high-availability/templates/cluster-setup.yaml Outdated

Comment thread charts/memgraph-high-availability/values.yaml Outdated

mattkjames7 added 2 commits April 9, 2026 15:20

remove hook-failed

3f3608f

do not enable monitoring by default

73f08a4

mattkjames7 merged commit e5fd382 into main Apr 10, 2026
7 checks passed

Navigation Menu

Uh oh!

Update templates for external monitoring - #208

Update templates for external monitoring#208
mattkjames7 merged 47 commits into
mainfrom
add-loki

mattkjames7 commented Mar 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

as51340 commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mattkjames7 commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

as51340 commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mattkjames7 commented Mar 17, 2026 •

edited

Loading