gardener-extension-otelcol

The gardener-extension-otelcol repo provides a Gardener Extension for an OpenTelemetry Collector, which runs in the shoot control-plane namespace and forwards observability signals for control-plane components to a remote OpenTelemetry Collector receiver.

Warning

This extension is in early development state. Do not use it in a production environment.

Requirements

Go 1.25.x or later
GNU Make
Docker for local development
Gardener Local Setup for local development

Code structure

The project repo uses the following code structure.

Package	Description
`cmd`	Command-line application of the extension
`pkg/admission`	Implementations for the Gardener extension admission `Validator` and `Mutator` interfaces
`pkg/apis`	Extension API types, e.g. configuration spec, etc.
`pkg/actuator`	Implementations for the Gardener Extension Actuator interfaces
`pkg/controller`	Utility wrappers for creating Kubernetes reconcilers for Gardener Actuators
`pkg/imagevector`	Image vector for container images
`pkg/heartbeat`	Utility wrappers for creating heartbeat reconcilers for Gardener extensions
`pkg/metrics`	Metrics emitted by the extension
`pkg/mgr`	Utility wrappers for creating `controller-runtime` managers using functional options API
`pkg/version`	Version metadata information about the extension
`internal/tools`	Go-based tools used for testing and linting the project
`charts`	Helm charts for deploying the extension
`examples`	Example Kubernetes resources, which can be used in a dev environment
`test`	Various files (e.g. schemas, CRDs, etc.), used during testing

Usage

You can enable the extension for a Gardener Shoot cluster by updating the .spec.extensions of your shoot manifest.

The following example shoot manifest snippet enables the extension and configures the OpenTelemetry Collector to emit the signals for the shoot control-plane components via the Debug Exporter. The configuration is exporter-oriented: spec.targets is a list of exporter destinations. Each target pairs a self-contained exporter with the signals it receives (metrics, logs and/or events). When we talk about events we mean Kubernetes Events. Technically an event is a log signal — from OpenTelemetry's point of view it travels through a logs pipeline, so the configured backend must be capable of receiving logs. Gardener exposes it as a separate signal for convenience, so events can be routed and filtered independently of other logs. Each (target, signal) pair becomes one collector pipeline, so a target can fan out several signals to one destination, and a signal can fan out to several targets. A signal is enabled if at least one target lists it. An exporter enables one or more transports (otlp_http, otlp_grpc, debug); a target's signal pipelines fan out to all of its enabled transports, so a single target can send to several destinations at once. Enabling the debug transport makes the target write to the collector's own logs.

...

spec:
  extensions:
    - type: otelcol
      providerConfig:
        apiVersion: otelcol.extensions.gardener.cloud/v1alpha1
        kind: CollectorConfig
        spec:
          targets:
            - exporter:
                debug:
                  verbosity: basic  # basic, normal or detailed
              signals: [metrics, logs, events]

This configuration however is only useful while developing or troubleshooting an issue with the collector, because signals are not actually forwarded to a remote OpenTelemetry Collector receiver.

The following configuration snippet enables the extension for a shoot and configures it to forward the signals of the control-plane components to a remote collector. A single target can serve several signals at once via its signals list. Here the target uses the OTLP HTTP exporter.

...

spec:
  extensions:
    - type: otelcol
      providerConfig:
        apiVersion: otelcol.extensions.gardener.cloud/v1alpha1
        kind: CollectorConfig
        spec:
          targets:
            - exporter:
                otlp_http:
                  endpoint: "https://opentelemetry-receiver.example.org"
              signals: [metrics, logs, events]

The following example snippet expands on the previous one by adding TLS configuration settings and Bearer token authentication with the remote collector.

...

spec:
  extensions:
    - type: otelcol
      providerConfig:
        apiVersion: otelcol.extensions.gardener.cloud/v1alpha1
        kind: CollectorConfig
        spec:
          targets:
            - exporter:
                otlp_http:
                  endpoint: "https://opentelemetry-receiver.example.org"
                  token:
                    resourceRef:
                      name: otelcol-bearer-token
                      dataKey: token
                  tls:
                    ca:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: ca.crt
                    cert:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: client.crt
                    key:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: client.key
              signals: [metrics, logs, events]
  resources:
  - name: otelcol-bearer-token
    resourceRef:
      apiVersion: v1
      kind: Secret
      name: my-otelcol-bearer-token
  - name: otelcol-tls
    resourceRef:
      apiVersion: v1
      kind: Secret
      name: my-otelcol-tls

In order to provide the otelcol-tls and otelcol-bearer-token secrets from the example above to the extension, you should first create the respective secrets in the shoot project namespace, which can then be referenced via Gardener Referenced Resources.

This example snippet enables the extension to forward the signals of the control-plane components to a remote collector using the OTLP gRPC exporter.

  extensions:
    - type: otelcol
      providerConfig:
        apiVersion: otelcol.extensions.gardener.cloud/v1alpha1
        kind: CollectorConfig
        spec:
          targets:
            - exporter:
                otlp_grpc:
                  endpoint: "https://opentelemetry-receiver.default.svc.cluster.local:4317"
                  token:
                    resourceRef:
                      name: otelcol-bearer-token
                      dataKey: token
                  tls:
                    ca:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: ca.crt
                    cert:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: client.crt
                    key:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: client.key
              signals: [metrics, logs, events]

For additional configuration settings, which can be provided to the extension, please make sure to check the OTel Extension API spec documentation.

Fan a signal out to multiple destinations

Because a signal is enabled by listing it in a target's signals section, the same signal can appear in several targets and thus be exported to several destinations at once, each with an independent exporter. A single target can also fan out to several transports at once — list more than one of otlp_http/otlp_grpc/debug under its exporter. This example sends metrics to a primary HTTP collector, a secondary gRPC collector, and additionally mirrors them to a debug exporter.

  extensions:
    - type: otelcol
      providerConfig:
        apiVersion: otelcol.extensions.gardener.cloud/v1alpha1
        kind: CollectorConfig
        spec:
          targets:
            # Primary: a remote collector over HTTP.
            - exporter:
                otlp_http:
                  endpoint: "https://opentelemetry-receiver.example.org"
                  token:
                    resourceRef:
                      name: otelcol-bearer-token
                      dataKey: token
                  tls:
                    ca:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: ca.crt
                    cert:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: client.crt
                    key:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: client.key
              signals: [metrics]
            # Secondary: a different collector over gRPC, additionally mirrored
            # to the collector's own logs via the debug transport.
            - exporter:
                otlp_grpc:
                  endpoint: "https://backup-receiver.example.org:4317"
                  token:
                    resourceRef:
                      name: otelcol-bearer-token
                      dataKey: token
                  tls:
                    ca:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: ca.crt
                    cert:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: client.crt
                    key:
                      resourceRef:
                        name: otelcol-tls
                        dataKey: client.key
                debug:
                  verbosity: basic
              signals: [metrics]

Selecting signals

The extension can restrict which telemetry the collector processes by enabling or disabling whole signals through the targets' signals.

Select which signals are collected

A signal (metrics, logs, events) is collected if at least one target lists it in its signals. Only enabled signals get a pipeline; a signal listed by no target is not collected or exported.

spec:
  extensions:
    - type: otelcol
      providerConfig:
        apiVersion: otelcol.extensions.gardener.cloud/v1alpha1
        kind: CollectorConfig
        spec:
          targets:
            # Only collect and export metrics; the other pipelines are not created.
            - exporter:
                otlp_http:
                  endpoint: "https://opentelemetry-receiver.example.org"
              signals: [metrics]

Development

In order to build a binary of the extension, you can use the following command.

make build

The resulting binary can be found in bin/extension.

In order to build a Docker image of the extension, you can use the following command.

make docker-build

Run the following command to get usage info about the available Makefile targets.

make help

For local development of the gardener-extension-otelcol it is recommended that you setup a development Gardener environment.

Please refer to the next sections for more information about deploying and testing the extension in a Gardener development environment.

Development Environment with Gardener Operator

The extension can also be deployed via the Gardener Operator.

In order to start a local development environment with the Gardener Operator, please refer to the following documentations.

In summary, these are the steps you need to follow in order to start a local development environment with the Gardener Operator, however, please make sure that you read the documents above for additional details.

make kind-up gardener-up

Before you continue with the next steps, make sure that you configure your KUBECONFIG to point to the kubeconfig file of the cluster, which runs the Gardener Operator.

There will be two kubeconfig files created for you, after the dev environment has been created.

Path	Description
`/path/to/gardener/dev-setup/kubeconfigs/runtime/kubeconfig`	The runtime cluster (`gardener-operator` runs in it)
`/path/to/gardener/dev-setup/kubeconfigs/virtual-garden/kubeconfig`	The virtual garden cluster

Throughout this document we will refer to the kubeconfigs for runtime and virtual clusters as $KUBECONFIG_RUNTIME and $KUBECONFIG_VIRTUAL respectively.

Before deploying the extension we need to target the runtime cluster, since this is where the extension resources for gardener-operator reside.

export KUBECONFIG=$KUBECONFIG_RUNTIME

In order to deploy the extension, execute the following command.

make deploy-operator

The deploy-operator target takes care of the following.

Builds a Docker image of the extension
Loads the image into the kind cluster nodes
Packages the Helm charts and pushes them to the local registry
Deploys the Extension (from group operator.gardener.cloud/v1alpha1) to the runtime cluster

Verify that we have successfully created the Extension (from group operator.gardener.cloud/v1alpha1) resource.

$ kubectl --kubeconfig $KUBECONFIG_RUNTIME get extop otelcol
NAME      INSTALLED   REQUIRED RUNTIME   REQUIRED VIRTUAL   AGE
otelcol   True        False              False              13s

Verify that the respective ControllerRegistration and ControllerDeployment resources have been created by the gardener-operator in the virtual garden cluster.

$ kubectl --kubeconfig $KUBECONFIG_VIRTUAL get controllerregistrations,controllerdeployments otelcol
NAME                                                 RESOURCES           AGE
controllerregistration.core.gardener.cloud/otelcol   Extension/otelcol   42s

NAME                                               AGE
controllerdeployment.core.gardener.cloud/otelcol   42s

Finally, we can create an example shoot with our extension enabled. The examples/shoot.yaml file provides a ready-to-use shoot manifest with the extension enabled and configured.

The provided example shoot references secrets from the project namespace, which are used to configure the TLS settings between the exporter and a local dev receiver, running in the default namespace.

The following commands will create the TLS secrets and a dev shoot, configured with the extension.

kubectl --kubeconfig $KUBECONFIG_VIRTUAL apply -f examples/secret-tls.yaml
kubectl --kubeconfig $KUBECONFIG_VIRTUAL apply -f examples/secret-bearer-token.yaml
kubectl --kubeconfig $KUBECONFIG_VIRTUAL apply -f examples/shoot.yaml

After the dev shoot cluster is created, use the following command to create a dev OpenTelemetry receiver in the default namespace.

kubectl --kubeconfig $KUBECONFIG_RUNTIME apply -f examples/opentelemetry-receiver.yaml

If you have an already existing and running shoot, for which you want to enable the extension, simply follow the instructions from the previous sections in order to enable and configure the extension manually.

Once we create the shoot cluster, gardenlet will start deploying our gardener-extension-otelcol, since it is required by our shoot.

Verify that the extension has been successfully installed by checking the corresponding ControllerInstallation resource for our extension.

$ kubectl --kubeconfig $KUBECONFIG_VIRTUAL get controllerinstallations
NAME                      REGISTRATION        SEED    VALID   INSTALLED   HEALTHY   PROGRESSING   AGE
otelcol-8rvmn             otelcol             local   True    True        True      False         64s

After your shoot cluster has been successfully created and reconciled, verify that the extension is healthy.

$ kubectl --kubeconfig $KUBECONFIG_RUNTIME --namespace shoot--local--local get extensions otelcol
NAME                INSTALLED   REQUIRED RUNTIME   REQUIRED VIRTUAL   AGE
otelcol             True        False              True               13m

Verify that the ManagedResource created by the extension is healthy as well.

$ kubectl --kubeconfig $KUBECONFIG_RUNTIME --namespace shoot--local--local get managedresource external-otelcol
NAME               CLASS   APPLIED   HEALTHY   PROGRESSING   AGE
external-otelcol   seed    True      True      False         6m20s

After successful reconciliation we should see the following OpenTelemetry collectors in the shoot control-plane namespace.

$ kubectl --kubeconfig $KUBECONFIG_RUNTIME --namespace shoot--local--local get otelcol external-otelcol
NAME                      MODE          VERSION   READY   AGE     IMAGE                                                                                                                          MANAGEMENT
external-otelcol          statefulset   0.141.0   1/1     6m45s   europe-docker.pkg.dev/gardener-project/releases/3rd/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.141.0   managed

We should also see that the Collector and Target Allocator are running and healthy.

$ kubectl --kubeconfig $KUBECONFIG_RUNTIME --namespace shoot--local--local get sts external-otelcol-collector
NAME                         READY   AGE
external-otelcol-collector   1/1     3m30s

$ kubectl --kubeconfig $KUBECONFIG_RUNTIME --namespace shoot--local--local get deployment external-otelcol-targetallocator
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
external-otelcol-targetallocator   1/1     1            1           3m38s

In order to trigger reconciliation of the extension you can annotate the extension resource.

kubectl --kubeconfig $KUBECONFIG_RUNTIME --namespace shoot--local--local annotate extensions otelcol gardener.cloud/operation=reconcile

In order to delete the dev shoot, TLS secrets and dev OpenTelemetry receiver you can run the following commands.

kubectl --kubeconfig $KUBECONFIG_VIRTUAL --namespace garden-local annotate shoot local confirmation.gardener.cloud/deletion=true --overwrite
kubectl --kubeconfig $KUBECONFIG_VIRTUAL delete -f examples/shoot.yaml --ignore-not-found=true --wait=false
kubectl --kubeconfig $KUBECONFIG_RUNTIME delete -f examples/opentelemetry-receiver.yaml --ignore-not-found=true --wait=false
kubectl --kubeconfig $KUBECONFIG_VIRTUAL delete -f examples/secret-tls.yaml --ignore-not-found=true --wait=false
kubectl --kubeconfig $KUBECONFIG_VIRTUAL delete -f examples/secret-bearer-token.yaml --ignore-not-found=true --wait=false

Troubleshooting

This section provides some hints related to troubleshooting the OpenTelemetry Collector, which is managed by the Gardener extension.

Check the official OpenTelemetry Troubleshooting Guides

Make sure that you check the following official OpenTelemetry documentation:

Check the logs of the OpenTelemetry Collector and Target Allocator

Check the logs of the deployment/external-otelcol-targetallocator and statefulset/external-otelcol-collector, e.g.

kubectl --namespace shoot--local--local logs -f deployments/external-otelcol-targetallocator
kubectl --namespace shoot--local--local logs -f statefulset/external-otelcol-collector

Verify that there are `ServiceMonitors` in the shoot control-plane namespace

The Target Allocator deployed by the extension is configured to discover ServiceMonitor resources with the following labels:

prometheus=shoot

Confirm that ServiceMonitors with these labels exist in the shoot control-plane namespace, e.g.

kubectl --namespace shoot--local--local get servicemonitors -l prometheus=shoot

Check the configuration of the Collector and Target Allocator

The Target Allocator and Collector configmaps are labeled with observability.gardener.cloud/app=external-otelcol. Check and confirm that the configuration settings in these configmaps are correct.

$ kubectl --namespace shoot--local--local get cm -l observability.gardener.cloud/app=external-otelcol
NAME                                      DATA   AGE
external-otelcol-collector-c30d03f4       1      13m
external-otelcol-targetallocator-config   1      13m

Verify that the Target Allocator discovers scrape targets

The communication between the Target Allocator and the Collector happens over mTLS, so we will need the client certificate of the collector, in order to confirm that the Target Allocator has discovered targets for scraping.

First, get the secret which contains the client certificate used by the Collector, e.g.

kubectl --namespace shoot--local--local get secret -l name=otelcol-collector-client

Save the client TLS secret locally:

mkdir client-cert
for k in tls.key tls.crt; do
  kubectl --namespace shoot--local--local get secret -l name=otelcol-collector-client -o yaml | \
    yq ".items[0].data.\"${k}\"" | base64 -d > "./client-cert/${k}"
done

Next, we need to port-forward the Target Allocator service locally.

kubectl --namespace shoot--local--local port-forward service/external-otelcol-targetallocator-https 8443:443

Now we can query the Target Allocator and review the jobs and the scrape targets it can dispatch to collectors.

curl -k --cert client-cert/tls.crt --key client-cert/tls.key -X GET 'https://localhost:8443/jobs' | jq '.'

The following command will query the Target Allocator for the scrape configs.

curl -k --cert client-cert/tls.crt --key client-cert/tls.key -X GET 'https://localhost:8443/scrape_configs' | jq '.'

In addition to the API paths served by the Target Allocator you can also inspect the configuration via /debug endpoints using your browser.

In order to do that we can use mitmproxy. Keep in mind that mitmpoxy expects to find the key and certificate in a single PEM-encoded file.

cat client-cert/tls.crt client-cert/tls.key > client-cert/mitmproxy.pem

Now we can start the mitmproxy.

mitmproxy -k \
          --listen-port=8080 \
          --set client_certs=client-cert/mitmproxy.pem \
          --mode upstream:https://localhost:8443

Open up your browser at http://localhost:8080/ in order to view the Target Collector jobs, scrape configs, and assigned collectors.

Tests

In order to run the tests use the command below:

make test

In order to test the Helm chart and the manifests provided by it you can run the following command.

make check-helm

In order to test the example resources from the examples/ directory you can run the following command.

make check-examples

Documentation

Make sure to check the following documents for more information about Gardener Extensions and the available extensions API.

Contributing

gardener-extension-otelcol is hosted on Github.

Please contribute by reporting issues, suggesting features or by sending patches using pull requests.

License

This project is Open Source and licensed under Apache License 2.0.

Name		Name	Last commit message	Last commit date
.github		.github
charts		charts
cmd/extension		cmd/extension
docs/api-reference		docs/api-reference
examples		examples
hack		hack
images		images
internal/tools		internal/tools
pkg		pkg
test		test
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
.golangci.yaml		.golangci.yaml
CODEOWNERS		CODEOWNERS
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
OWNERS		OWNERS
OWNERS_ALIASES		OWNERS_ALIASES
README.md		README.md
VERSION		VERSION
api-ref-docs.yaml		api-ref-docs.yaml
go.mod		go.mod
go.sum		go.sum
renovate.json5		renovate.json5

Navigation Menu

Uh oh!

Repository files navigation

gardener-extension-otelcol

Requirements

Code structure

Usage

Fan a signal out to multiple destinations

Selecting signals

Select which signals are collected

Development

Development Environment with Gardener Operator

Troubleshooting

Check the official OpenTelemetry Troubleshooting Guides

Check the logs of the OpenTelemetry Collector and Target Allocator

Verify that there are `ServiceMonitors` in the shoot control-plane namespace

Check the configuration of the Collector and Target Allocator

Verify that the Target Allocator discovers scrape targets

Tests

Documentation

Contributing

License

About

Releases

Packages

Used by

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

gardener-extension-otelcol

Requirements

Code structure

Usage

Fan a signal out to multiple destinations

Selecting signals

Select which signals are collected

Development

Development Environment with Gardener Operator

Troubleshooting

Check the official OpenTelemetry Troubleshooting Guides

Check the logs of the OpenTelemetry Collector and Target Allocator

Verify that there are ServiceMonitors in the shoot control-plane namespace

Check the configuration of the Collector and Target Allocator

Verify that the Target Allocator discovers scrape targets

Tests

Documentation

Contributing

License

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

Verify that there are `ServiceMonitors` in the shoot control-plane namespace