Goldmane Calico Cloud emitter: include Cloud tigera-operator-signer in CA trust bundle

[denislemire]

Description

On Calico OSS clusters connected to Calico Cloud via ManagementClusterConnection, Goldmane cannot sustain flow uploads to Guardian after Calico 3.32. The cluster shows Connected in Calico Cloud but Service Graph / flow data stays empty.

After fixing calico-system tier egress (#4804), Goldmane reaches Guardian but the flow emitter fails TLS verification:

Error emitting flows to https://guardian.calico-system.svc.cluster.local:443/api/v1/flows/bulk:
tls: failed to verify certificate: x509: certificate signed by unknown authority
(possibly because of "crypto/rsa: verification error" while trying to verify
candidate authority certificate "tigera-operator-signer")

Root cause

pkg/render/goldmane/component.go (v1.42.0) sets PUSH_URL when ManagementClusterConnection != nil:

{Name: "PUSH_URL", Value: fmt.Sprintf("%s/api/v1/flows/bulk", guardianSvc)},

It sets CA_CERT_PATH to the operator local goldmane-ca-bundle ConfigMap (TrustedCertBundle). It does not include the Calico Cloud tigera-operator-signer delivered in Secret tigera-voltron-linseed-certs-public (tigera-operator namespace).

Goldmane uses the same CA_CERT_PATH for gRPC server trust and the HTTP flow emitter (goldmane/pkg/daemon/daemon.go v3.32.0).

The Guardian Service maps port 443 → pod port 8080 (Voltron tunnel). Flow bulk POSTs terminate TLS with the Cloud tigera-linseed leaf (issuer: Cloud tigera-operator-signer), not the local guardian-key-pair material.

CA comparison (SHA-256 fingerprint prefix)

Material	Subject CN	Fingerprint
Local goldmane-ca-bundle	tigera-operator-signer	03:AE:C3:19:…
Cloud CA in tigera-voltron-linseed-certs-public (2nd PEM in tls.crt)	tigera-operator-signer	5B:58:B7:69:…

Same CN, different keys. A combined bundle (local + Cloud operator-signer) verifies both Felix→Goldmane gRPC (local mTLS) and the emitter upload path.

Reproduction

1. Calico OSS 3.32.0 + operator v1.42.0, Goldmane + ManagementClusterConnection applied.
2. Ensure Goldmane egress to Guardian and Kubernetes API (see guardian and goldmane network policies missing egress DNS rules #4804 and comment there).
3. kubectl logs -n calico-system deployment/goldmane — TLS errors above.
4. kubectl get configmap goldmane-ca-bundle -n calico-system -o jsonpath='{.data.tigera-ca-bundle\.crt}' | awk '/BEGIN CERTIFICATE/{n++} END{print n}' — 1 cert (local only).

Approaches tested

Approach	Result
Re-apply ManagementClusterConnection	No change
SERVER_NAME=guardian.calico-system.svc.cluster.local only	TLS verify errors may pause; uploads still fail with EOF; operator reverts Deployment patches on reconcile
CLIENT_CERT_PATH / CLIENT_KEY_PATH from tigera-managed-cluster-connection	No sustained uploads
Merge Cloud CA into goldmane-ca-bundle → tigera-ca-bundle.crt	Successfully emitted flows after retries in Goldmane logs

Expected behavior

When ManagementClusterConnection is configured, operator should populate Goldmane trusted CA material with a combined bundle: local tigera-operator-signer plus the Cloud tigera-operator-signer from tigera-voltron-linseed-certs-public.

Workaround (until operator fix)

1. Patch ConfigMap goldmane-ca-bundle to append the Cloud tigera-operator-signer (2nd certificate in tigera-voltron-linseed-certs-public / tls.crt).
2. Because the operator may revert the ConfigMap on reconcile, run a small reconciler (Deployment + RBAC) on an interval (~45s) to re-merge when only one cert is present. The reconciler pod needs calico-system tier egress to the Kubernetes API.

Verification:

kubectl logs -n calico-system deployment/goldmane | grep 'Successfully emitted'
kubectl get configmap goldmane-ca-bundle -n calico-system \
  -o jsonpath='{.data.tigera-ca-bundle\.crt}' \
  | awk '/BEGIN CERTIFICATE/{n++} END{print n " certs"}'  # expect 2

Full write-up with verification steps and policy context: https://gist.github.com/denislemire/72591aa927c1e1dcee544a4d26953ef6

Environment

• Calico OSS 3.32.0, Tigera Operator v1.42.0
• Calico Cloud managed cluster connection (free tier)
• Private multi-node Kubernetes cluster

Related

• guardian and goldmane network policies missing egress DNS rules #4804 — guardian/goldmane missing calico-system tier egress (OSS path)

[denislemire]

Follow-up (2026-06-11) — confirmed fix and workaround

(Supersedes the earlier comment above; issue body has also been updated.)

Further testing confirms the blocker is CA trust pairing, not SERVER_NAME alone.

Root cause (confirmed)

• Goldmane CA_CERT_PATH points at local goldmane-ca-bundle → local tigera-operator-signer (03:AE:C3:19:…).
• Calico Cloud ships a different tigera-operator-signer in tigera-voltron-linseed-certs-public (5B:58:B7:69:…). Same CN, different key.
• Guardian Service 443 → pod :8080 (Voltron tunnel). Flow bulk POSTs present the Cloud tigera-linseed leaf, which chains to the Cloud operator-signer.

Dead ends

Attempt	Result
SERVER_NAME=guardian.calico-system.svc.cluster.local only	Verify errors may stop briefly → sustained EOF on POST
Client cert/key from tigera-managed-cluster-connection	No sustained uploads
Re-apply connection manifest	No change

Fix that works

Append the Cloud tigera-operator-signer (2nd PEM in tigera-voltron-linseed-certs-public / tls.crt) to goldmane-ca-bundle key tigera-ca-bundle.crt. Goldmane then logs:

Successfully emitted flows after retries. bucket=emitter.bucketKey{...} retries=4

openssl verify of the tigera-linseed leaf against the merged bundle returns OK.

Workaround detail

The operator reverts goldmane-ca-bundle on reconcile, so a small reconciler Deployment (RBAC: get/patch ConfigMap goldmane-ca-bundle, get Secret tigera-voltron-linseed-certs-public) re-merges on an interval (~45s). That pod also needs calico-system tier egress to the kubernetes Service (same default-deny tier as Goldmane).

Suggested operator fix

When ManagementClusterConnection is set, wire a combined CA bundle into Goldmane emitter trust (local + Cloud operator-signer) so Felix→Goldmane gRPC continues to work.

Full report: https://gist.github.com/denislemire/72591aa927c1e1dcee544a4d26953ef6

Happy to retest once a build includes the combined bundle.