Helm v4 server-side apply can conflict with kube-webhook-certgen-managed caBundle

[0tarof]

Description

When certManager.enabled=false and customTLSCertificate=false, the nri-metadata-injection chart renders an empty clientConfig.caBundle in the MutatingWebhookConfiguration:

webhooks:
  - name: metadata-injection.newrelic.com
    clientConfig:
      service:
        ...
      caBundle: ""

The chart also runs kube-webhook-certgen hook jobs. After install/upgrade, the patch hook writes the actual CA bundle into the same webhooks[].clientConfig.caBundle field.

With Helm v4 server-side apply, this can produce a field ownership conflict on a later upgrade because Helm applies the rendered empty field while kube-webhook-certgen owns the live caBundle field.

The failure looks like:

UPGRADE FAILED: conflict occurred while applying object ... admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
Apply failed with 1 conflict: conflict with "kube-webhook-certgen" using admissionregistration.k8s.io/v1: .webhooks[name="metadata-injection.newrelic.com"].clientConfig.caBundle

Expected Behavior

Helm upgrades should not conflict with kube-webhook-certgen over clientConfig.caBundle.

Ideally, only one actor should manage this field:

• kube-webhook-certgen for the default chart-managed certificate flow;
• cert-manager/cainjector when certManager.enabled=true;
• an explicit custom TLS/CA management path when customTLSCertificate=true.

Troubleshooting or NR Diag results

I reproduced the field ownership conflict with a minimal server-side apply example:

1. Create a MutatingWebhookConfiguration with caBundle: "" using field manager helm.
2. Update the same caBundle field using field manager kube-webhook-certgen.
3. Apply the original Helm-owned manifest again.

The second Helm apply fails with a managedFields conflict on:

.webhooks[name="..."].clientConfig.caBundle

I also reproduced the behavior with the actual chart using Helm v4.

Steps to Reproduce

One reproduced path:

1. Use Helm v4.

2. Install an affected chart version with default certificate settings:

   helm upgrade --install nri-metadata-injection \
     newrelic-metadata-injection/nri-metadata-injection \
     --version 4.34.2 \
     --namespace nr-test \
     --create-namespace \
     --set cluster=test

3. Confirm the live MutatingWebhookConfiguration has a non-empty clientConfig.caBundle after the hook job runs.

4. Run another Helm v4 upgrade of a chart that still renders caBundle: "".

   helm upgrade nri-metadata-injection \
     newrelic-metadata-injection/nri-metadata-injection \
     --version 4.34.2 \
     --namespace nr-test \
     --set cluster=test

The upgrade can fail with the clientConfig.caBundle field ownership conflict shown above.

Your Environment

Original failing environment:

• Kubernetes: Amazon EKS 1.33
• Chart path: installed via nri-bundle
• nri-bundle upgrade path observed around 7.0.12 -> 7.0.16
• nri-metadata-injection chart version observed in the live object: 4.34.2
• certManager.enabled=false
• customTLSCertificate=false
• Certificate hook: kube-webhook-certgen
• Helm: v4 through the deployment tooling
• Deployment tooling detail: the EKS 1.33 environment used @aws-cdk/lambda-layer-kubectl-v33 2.2.x; this package line bundled Helm v3 through 2.1.0 and Helm v4 from 2.2.0.

The same chart worked in other environments that were still using Helm v3-based deployment tooling, so this appears to be triggered by the combination of this chart behavior and Helm v4 server-side apply.

Reproduction and candidate-fix validation environment

I also reproduced and tested this locally with:

• Helm v4.2.0
• Helm v3.21.1 for compatibility comparison
• kind v0.31.0
• Kubernetes:
  • kindest/node:v1.32.0
  • kindest/node:v1.33.0
  • kindest/node:v1.34.0
• Chart versions checked around nri-metadata-injection 4.34.x

Local matrix summary:

Kubernetes	Helm path	Result
v1.32.0	Helm v4 old chart -> old chart	failed with expected caBundle conflict
v1.33.0	Helm v4 old chart -> old chart	failed with expected caBundle conflict
v1.34.0	Helm v4 old chart -> old chart	failed with expected caBundle conflict
v1.32.0	Helm v4 old chart -> candidate chart omitting certgen-managed caBundle	succeeded
v1.33.0	Helm v4 old chart -> candidate chart omitting certgen-managed caBundle	succeeded
v1.34.0	Helm v4 old chart -> candidate chart omitting certgen-managed caBundle	succeeded
v1.32.0	Helm v3 old chart -> candidate chart omitting certgen-managed caBundle	succeeded; caBundle present after hook
v1.33.0	Helm v3 old chart -> candidate chart omitting certgen-managed caBundle	succeeded; caBundle present after hook
v1.34.0	Helm v3 old chart -> candidate chart omitting certgen-managed caBundle	succeeded; caBundle present after hook

Additional context

This looks like a chart field-ownership issue exposed by Helm v4 server-side apply.

The straightforward fix would be to stop rendering clientConfig.caBundle when another actor is expected to populate it.

At first glance, that is a very small diff:

diff --git a/charts/nri-metadata-injection/templates/admission-webhooks/mutatingWebhookConfiguration.yaml b/charts/nri-metadata-injection/templates/admission-webhooks/mutatingWebhookConfiguration.yaml
@@
       name: {{ include "newrelic.common.naming.fullname" . }}
       namespace: {{ .Release.Namespace }}
       path: "/mutate"
-{{- if not .Values.certManager.enabled }}
-    caBundle: ""
-{{- end }}
   rules:

However, Helm v3 upgrade behavior makes the migration story more subtle. Applying this clean change to every path may not be migration-safe for existing users.

Observed compatibility concern:

Path	Clean fix behavior	Concern
Default kube-webhook-certgen path	Helm stops rendering caBundle	Appears safe in my testing. On the first Helm v3 migration, Helm may temporarily remove the live caBundle, but the post-upgrade hook restores it. Future upgrades should be better because Helm no longer manages the field.
certManager.enabled=true	No change	Appears safe. This path already omits caBundle and lets cert-manager/cainjector manage it.
customTLSCertificate=true	Helm stops rendering caBundle	Potentially risky. Helm v3 may remove the live caBundle during the first migration, and there is no kube-webhook-certgen hook in this mode to restore it. Existing users may need to manually patch it again.

Because of that, a narrower compatibility-preserving change may be safer:

diff --git a/charts/nri-metadata-injection/templates/admission-webhooks/mutatingWebhookConfiguration.yaml b/charts/nri-metadata-injection/templates/admission-webhooks/mutatingWebhookConfiguration.yaml
@@
       name: {{ include "newrelic.common.naming.fullname" . }}
       namespace: {{ .Release.Namespace }}
       path: "/mutate"
-{{- if not .Values.certManager.enabled }}
+{{- if and (not .Values.certManager.enabled) .Values.customTLSCertificate }}
     caBundle: ""
 {{- end }}
   rules:

This would:

• certManager.enabled=true: already omits caBundle, so cert-manager/cainjector owns it.
• certManager.enabled=false and customTLSCertificate=false: omit caBundle, so kube-webhook-certgen owns it.
• customTLSCertificate=true: keep the existing rendered caBundle: "" behavior, so this issue does not change the custom TLS migration behavior.

This is not as clean as removing caBundle entirely, because it keeps the custom TLS behavior that may already be questionable. But it avoids changing the highest-risk path while fixing the default kube-webhook-certgen ownership conflict.

Another possible long-term fix would be to add an explicit value for custom TLS CA management, for example rendering caBundle only when a non-empty user-provided CA bundle value is set. That would make ownership explicit, but it is a larger compatibility/design change.

Questions:

• Should this chart render clientConfig.caBundle at all when kube-webhook-certgen is responsible for patching it?
• For customTLSCertificate=true, should the chart keep the current behavior, add an explicit caBundle value, or use another migration-safe mechanism?
• Would maintainers prefer a narrow fix for the default kube-webhook-certgen path first, and a separate issue/PR for custom TLS CA ownership?

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5): Unknown
Suggested T-Shirt size (S, M, L, XL, Unknown): Unknown

[Newrelic-Boomi]

https://new-relic.atlassian.net/browse/NR-578695