Keksposé

Expose your Kafka cluster outside your Minikube, Kind, or Docker Desktop Kubernetes clusters.

What is Keksposé?

Strimzi makes it easy to run Apache Kafka on Kubernetes. But using the Kafka cluster from applications running outside your Kubernetes cluster can be challenging. Strimzi tries to make it as easy as possible and supports several different ways how to expose the Kafka cluster:

• Using Load Balancers
• Using Node Ports
• Using the Kubernetes Nginx Ingress controller

When running in production-grade Kubernetes environments, these usually work fine. But when using a local Kubernetes cluster such as Minikube, Kind, or Docker Desktop, these mechanisms are often not well-supported:

• The local clusters often lack proper support for load balancers
• They often run inside additional virtual machines or containers and use complex networking so that node ports do not work out of the box
• While Ingress usually works, the installation of the right Ingress controller and its configuration is often complicated

One of the ways how applications are often exposed from local clusters is using port forwarding with kubectl port-foward. But using port forwarding with Apache Kafka is not so simple because of its custom discovery protocol. You need to forward ports for each broker in the Kafka cluster. And you need to make sure your advertised hosts and ports are correctly configured to the local address. Keksposé makes it possible to use port forwarding with Apache Kafka by proxying the Kafka protocol and adjusting the advertised hosts and ports to the forwarded port addresses.

How does it work?

By default, Keksposé finds a listener without TLS encryption on your Strimzi-based Apache Kafka clusters and exposes it. It creates a port-forward for each of the Kafka brokers in your cluster. But it stands in the middle between the Kafka clients and the Kafka brokers and changes the advertised hosts and ports to the local addresses of the forwarded ports. Your Kafka clients can then connect to the forwarded ports and through Keksposé to the Kafka cluster to send and receive messages.

If you use --allow-insecure-tls, Keksposé can also use TLS-encrypted Kafka listeners. In that mode, Keksposé establishes TLS to the upstream brokers, terminates it inside the proxy, rewrites the Kafka protocol responses as usual, and still exposes a plain local TCP stream to your client applications. Certificate verification is disabled in this mode, so it is intended only for development and debugging. mTLS authentication is not supported.

flowchart LR
    subgraph Local Computer
        direction TB
        A[Kafka Clients]
        B[Keksposé]
        subgraph Keksposé
            direction TB
            B[[Updating advertised addresses]]
        end
    end

    subgraph Kubernetes
        direction LR
        D[Kafka Cluster]
    end
    
    A <--> |Sending / Receiving messages| B
    B <--> |Port-Forwarding| D

Loading

Kekspose is written in Golang using the Strimzi Go APIs. That allows it to provide native binaries to make it easier to run Keksposé.

Note: For the previous version based on the Kroxylicious proxy, see the 0.6.x release branch and use one of the 0.6.x releases.

How to use Keksposé?

Installation

You can download one of the release binaries from one of the releases and use it.

You can also install Keksposé using Homebrew:

brew tap scholzj/tap
brew install kekspose

Configuration

Keksposé supports several parameters that can be used to configure it:

Option	Description	Default Value
--help / -h	Help
--kubeconfig	Path to the kubeconfig file to use for Kubernetes API requests.
--context	Name of the Kubernetes context to use from the kubeconfig file.
--namespace / -n	Namespace of the Kafka cluster. This is also the namespace where the Keksposé proxy will be deployed. Defaults to the namespace from your Kubernetes configuration.
--cluster-name / -c	Name of the Kafka cluster.	my-cluster
--listener-name/ -l	Name of the listener that should be exposed. If not set, Keksposé will try to find a suitable listener on its own.
--starting-port / -p	The starting port number. This port number will be used for the bootstrap connection and will be used as the basis to calculate the per-broker ports.	50000
--allow-unready	Allow connecting to Kafka clusters even when the Kafka resource is not marked as Ready.	false
--allow-insecure-tls	Allow using TLS-encrypted Kafka listeners with certificate verification disabled. Keksposé will terminate TLS upstream and still expose a plaintext local stream.	false
--verbose / -v	Enables verbose logging (can be repeated: -v, -vv, -vvv).
--log-api	Restrict RPC logging to these Kafka APIs (comma-separated names, e.g. Metadata,Produce). Default: all APIs. Requires -v.	all APIs
--trace-api	Decode and log full message bodies only for these Kafka APIs (comma-separated names, e.g. Metadata). Default: all logged APIs. Requires -vv.	all APIs

If you are using the Keksposé binary, you can pass the options from the command line.

Using TLS-encrypted listeners

By default, Keksposé uses only listeners with tls: false. If you want to use a TLS-encrypted listener, add --allow-insecure-tls.

When --allow-insecure-tls is set:

• Automatic listener selection includes listeners with tls: true.
• An explicitly selected TLS listener is accepted as well.
• Keksposé connects to the Kafka brokers using TLS, but disables certificate verification.
• Your local client still connects to Keksposé over a plaintext TCP connection.
• The encrypted listener might use SASL authentication, but mTLS authentication is not supported.

This mode is meant for local development only.

Debugging Kafka clients

In the verbose mode (-v or --verbose), Keksposé will log high level information about the request and responses it is forwarding. For example:

-> request node=2000 api=Fetch apiKey=1 apiVersion=17 correlationId=694 clientId=console-consumer bodySize=26
<- response node=2000 api=Fetch apiKey=1 apiVersion=17 correlationId=694 clientId=console-consumer bodySize=13

You can further increase the verbosity by using an extra verbose mode -vv. This will in addition dump the decoded content of the request and response bodies for every Kafka API.

When the full RPC stream is too noisy, you can narrow it down:

• --log-api Metadata,Produce logs only the listed APIs (at -v).
• --trace-api Metadata keeps the one-line summaries for everything but only decodes and dumps the bodies of the listed APIs (at -vv), so you pay the decoding cost only for the APIs you care about.

Frequently Asked Questions

What Strimzi versions does Keksposé support?

Keksposé 0.9.0 and newer supports only the Strimzi v1 CRD API version. That means it is compatible with Strimzi 0.49.0 and newer. Older Keksposé versions support Strimzi v1beta2 API and can be used also with Strimzi 0.51 and older.

Does Keksposé support Kafka clusters with authentication?

Keksposé supports Kafka clusters with SASL-based authentication, such as SCRAM-SHA or OAuth.

By default, it uses listeners without TLS encryption. If needed for local development, you can also use TLS-encrypted listeners with --allow-insecure-tls. In that mode, Keksposé disables certificate verification for the upstream broker connection.

Does Keksposé support TLS-encrypted Kafka listeners?

Yes, with --allow-insecure-tls.

When this option is used, Keksposé can automatically select a TLS-encrypted listener or use one specified through --listener-name. It will connect to the brokers using TLS and terminate it inside the proxy, but it does not verify the broker certificates.

This option is intended for development use only.

What happens when I scale my Kafka cluster?

You need to restart Keksposé after scaling up your Kafka cluster or changing the IDs of the Apache Kafka nodes.

Does Keksposé support KRaft-based Apache Kafka clusters?

Keksposé supports a Kraft-based Apache Kafka cluster. However, it currently exposes the broker nodes only. If you have any need / use-cases for exposing the controller nodes as well, feel free to open an issue.

What access rights do I need to run Keksposé?

Running Keksposé requires the following access rights to your Kubernetes cluster:

• Reading the Kafka and KafkaNodePool Strimzi resources from the selected namespace
• Needs to be able to forward ports from the proxy Pod

The recent Keksposé versions do not need the access rights to create or delete Pods in the selected namespace.

Does Keksposé work only with local Kubernetes clusters?

Keksposé was designed with a focus on local Kubernetes environments such as Minikube, Kind, or Docker Desktop. But it can be used with any Kubernetes cluster regardless of how and where you run it.

Can the same Kafka cluster be exposed to multiple users in parallel?

Any user who has access to the Kubernetes cluster and the necessary rights can expose it. The same cluster can be exposed in parallel by multiple users.

What does the name Keksposé mean?

It is a combination of several words: Kafka, Expose ... and Keks (biscuit) because everyone likes them 😉.