[TT-17431] `opentelemetry.headers.fromSecret` still renders the secret in plaintext (#363 not resolved)

[cShirley14]

PR #375 closed #363, but the secret is still exposed in the rendered manifest.

Security impact

A secret (e.g. an OTel auth/Bearer token) is written in plaintext into the Deployment, readable by anyone with get on the Deployment.

Mechanism

The env var is emitted in components/tyk-gateway/templates/deployment-gw-repset.yaml as an inline value::

- name: TYK_GW_OPENTELEMETRY_HEADERS
  value: "{{ include "otel-headers" . }}"

#375 (files changed) only touched the otel-headers helper in components/tyk-gateway/templates/_helpers.tpl and the values.yaml comments. It left that inline value: site untouched, and the helper now does a lookup + b64dec at render time:

{{- $secret := lookup "v1" "Secret" $.Release.Namespace $value.fromSecret.name -}}
{{- $secretValue := index $secret.data $value.fromSecret.key | b64dec -}}
{{- $headersList = append $headersList (printf "%s:%s" $key $secretValue) -}}

b64dec-ing a secret into a string that is then inlined as value: is the exposure. The result is visible in kubectl get deploy -o yaml and helm get manifest.

Reproduce (no cluster needed)

A direct-string header skips lookup and renders under helm template, proving the consumption site inlines the header string:

helm template tyk tyk-helm/tyk-data-plane \
  --set tyk-gateway.gateway.opentelemetry.enabled=true \
  --set tyk-gateway.gateway.opentelemetry.headers.Authorization="Bearer supersecrettoken" \
  | grep -A1 TYK_GW_OPENTELEMETRY_HEADERS

Output:

- name: TYK_GW_OPENTELEMETRY_HEADERS
  value: "Authorization:Bearer supersecrettoken"

fromSecret feeds the same helper and the same inline value: (see Mechanism), with the secret lookup-ed and b64dec-ed in first.

Expected

A reference, not a value:

- name: TYK_GW_OPENTELEMETRY_HEADERS
  valueFrom:
    secretKeyRef:
      name: otel-auth-example
      key: authorization

Why it's not a one-line swap

TYK_GW_OPENTELEMETRY_HEADERS is a single env var holding a joined key:value,key:value string, and K8s can't build one env var from multiple secretKeyRefs. The fix: when a header uses fromSecret, emit valueFrom and have that secret key hold the full formatted header string. Remove the lookup/b64dec path.

Environment

tyk-data-plane chart (latest); applies to any chart embedding the tyk-gateway otel-headers helper. The helm template repro above was run on Helm 4.2.0.

References

• Configuring OpenTelemetry via tyk-stack Helm chart exposes Authorization token #363 (unresolved core)
• [TT-13680] feat: support reading headers from Kubernetes Secrets #375 (the change that missed deployment-gw-repset.yaml)

[probelabs]

Hi @cShirley14,

Thank you for this incredibly detailed and well-analyzed security report. Your clear explanation and reproducible example are much appreciated.

Summary of the Issue

You've correctly identified that when using tyk-gateway.gateway.opentelemetry.headers.fromSecret, the secret value is decoded and rendered directly as a plaintext value in the TYK_GW_OPENTELEMETRY_HEADERS environment variable. This action exposes sensitive information, such as authentication tokens, to any user or system with read access to the gateway's Deployment manifest.

Root Cause Analysis

Your analysis is spot on. The root cause is the combination of two factors:

1. The otel-headers helper template performs a lookup and b64dec at Helm render time.
2. The gateway Deployment template consumes this resolved value directly via the value field, rather than using a valueFrom.secretKeyRef reference.

Proposed Solution

We agree with your assessment that a simple switch to valueFrom is complicated by the fact that TYK_GW_OPENTELEMETRY_HEADERS is a single environment variable that can be composed of multiple headers.

Your proposed solution seems to be the most practical and secure path forward:

• When a header uses fromSecret, the template should render a valueFrom.secretKeyRef block.
• The referenced secret key must contain the entire, pre-formatted header string (e.g., Authorization: Bearer my-secret-token).
• This correctly avoids exposing the secret in the manifest while working within the limitations of Kubernetes environment variable definitions.

This would be a breaking change for users currently using the fromSecret feature, and the documentation would need to be updated accordingly to guide users on how to structure their secrets for this new approach.

The team will review this report to validate the findings and prioritize a fix.

References

• components/tyk-gateway/templates/deployment-gw-repset.yaml – The Deployment template that inlines the secret value.
• components/tyk-gateway/templates/_helpers.tpl – The helper template containing the lookup logic.

---

Powered by Visor from Probelabs

💡 TIP: You can chat with Visor using /visor ask <your question>

[nipr-jdoenges]

To avoid a breaking change for users currently using the fromSecret feature, the chart could continue to calculate the otel-headers value the same way, and then store that value into yet another Secret created and managed by the chart and referenced by the Deployment spec.

Or just offer a toggle to opt-in to the new behavior and leave the old behavior unchanged.

[cShirley14]

/visor ask

To avoid a breaking change for users currently using the fromSecret feature, the chart could continue to calculate the otel-headers value the same way, and then store that value into yet another Secret created and managed by the chart and referenced by the Deployment spec.

Or just offer a toggle to opt-in to the new behavior and leave the old behavior unchanged.

I completely agree; these approaches can co-exist to provide a more secure, native option.

[probelabs]

---

Powered by Visor from Probelabs

Last updated: 2026-06-04T13:37:12.581Z | Triggered by: cShirley14 | Commit: 55e16f0

💡 TIP: You can chat with Visor using /visor ask <your question>

[cShirley14]

/visor ask Did you see the solution above? Both approaches can co-exist.

[probelabs]

---

Powered by Visor from Probelabs

Last updated: 2026-06-04T13:41:33.908Z | Triggered by: cShirley14 | Commit: 55e16f0

💡 TIP: You can chat with Visor using /visor ask <your question>