Continuation of #1293

[tspearconquest]

Description

Hello,

In follow-up to #1293, I have upgraded the bundle chart to version 6.0.30 and enabled the metrics described in the previous issue again, and am experiencing the same issue with dropped metrics as before with the exception that I now see the following different error message in the logs.

time="2026-01-07T23:57:33Z" level=info msg="Skipped 6 metric families with unsupported OpenMetrics types: [gotk_resource_info gotk_resource_info gotk_resource_info gotk_resource_info gotk_resource_info gotk_resource_info]"
time="2026-01-07T23:57:33Z" level=error msg="Failed while trying to parse metrics: reading text format failed: text format parsing error in line 1545: second HELP line for metric name \"gotk_resource_info\""

Please review the previous issue for important context. When I port forward to the KSM pod and curl the metrics endpoint I can see the additional HELP lines. This seems to happen because of how the metrics for Flux CD are required to be configured.

# HELP gotk_resource_info The current state of a Flux HelmRelease resource.
# TYPE gotk_resource_info info
# HELP gotk_resource_info The current state of a Flux Kustomization resource.
# TYPE gotk_resource_info info
# HELP gotk_resource_info The current state of a Flux GitRepository resource.
# TYPE gotk_resource_info info
# HELP gotk_resource_info The current state of a Flux HelmChart resource.
# TYPE gotk_resource_info info
# HELP gotk_resource_info The current state of a Flux HelmRepository resource.
# TYPE gotk_resource_info info
# HELP gotk_resource_info The current state of a Flux OCIRepository resource.
# TYPE gotk_resource_info info

This happens because there is a gotk_resource_info metric configured for multiple different custom resource kinds in the cluster.

Here are 2 of the metrics from above as I've defined them in the helm chart values, so that you can reproduce this (note, you'll need to install flux and deploy a couple of flux custom resources (GitRepository and HelmChart resources) - but once that is done, you can add this to your nri-bundle helm values.yaml and do a helm upgrade to get these metrics in your KSM pod. Then port forward localhost:8080 to the KSM pod on port 8080, and run:

curl https://localhost:8080/metrics > metrics.txt

nri-bundle helm chart values.yaml snippet:

kube-state-metrics:
  enabled: true
  rbac:
    extraRules:
      - apiGroups:
          - source.toolkit.fluxcd.io
          - kustomize.toolkit.fluxcd.io
          - helm.toolkit.fluxcd.io
          - notification.toolkit.fluxcd.io
          - image.toolkit.fluxcd.io
        resources:
          - gitrepositories
          - buckets
          - helmrepositories
          - helmcharts
          - ocirepositories
          - kustomizations
          - helmreleases
          - alerts
          - providers
          - receivers
          - imagerepositories
          - imagepolicies
          - imageupdateautomations
        verbs: [ "list", "watch" ]
  customResourceState:
    enabled: true
    config:
      spec:
        resources:
          - groupVersionKind:
              group: source.toolkit.fluxcd.io
              version: v1
              kind: GitRepository
            metricNamePrefix: gotk
            metrics:
              - name: "resource_info"
                help: "The current state of a Flux GitRepository resource."
                each:
                  type: Info
                  info:
                    labelsFromPath:
                      name: [ metadata, name ]
                labelsFromPath:
                  exported_namespace: [ metadata, namespace ]
                  ready: [ status, conditions, "[type=Ready]", status ]
                  suspended: [ spec, suspend ]
                  revision: [ status, artifact, revision ]
                  url: [ spec, url ]
          - groupVersionKind:
              group: source.toolkit.fluxcd.io
              version: v1
              kind: HelmRepository
            metricNamePrefix: gotk
            metrics:
              - name: "resource_info"
                help: "The current state of a Flux HelmRepository resource."
                each:
                  type: Info
                  info:
                    labelsFromPath:
                      name: [ metadata, name ]
                labelsFromPath:
                  exported_namespace: [ metadata, namespace ]
                  ready: [ status, conditions, "[type=Ready]", status ]
                  suspended: [ spec, suspend ]
                  revision: [ status, artifact, revision ]
                  url: [ spec, url ]

Acceptance Criteria

1. The following NRQL query in the dashboard returns a value other than "no data": SELECT latest(timestamp) FROM K8sReplicasetSample WHERE clusterName = 'my-cluster-name'.
2. The Kubernetes integration logs no errors.
3. The dashboard in the attached screenshot has data where there is currently showing just zeroes.

Dashboard link: https://onenr.io/07j9ppvrARO

Additional context

The gotk_resource_info metric outputs different data depending on which kind of resource the metrics are being produced for. This error did not start occurring in the integration until #1337 was implemented, however I see that this error comes from the prometheus common expfmt library (the text parsing method), so I suspect that the help line was intended to be filtered by #1337 but for some reason it was missed. Unfortunately I'm not strong enough with Golang to draft a fix for this.

Estimates

Please provide initial t-shirt size. S = 1-3 days, M = 3-5 days (1 week), L = 1-2 weeks (1 sprint)

[Newrelic-Boomi]

https://new-relic.atlassian.net/browse/NR-515039

[dbudziwojskiNR]

Hi @tspearconquest! Thanks for letting us know. We're working on this and will be in touch with updates. Thank you!