Skip to content

Commit 513b9e4

Browse files
authored
Document how to live-update mTLS client certificates per SDK (#5032)
* Document how to live-update mTLS client certificates per SDK Each SDK's client-connection page shows how to rotate an API key on a running Worker with no downtime, but none covered mTLS client certificates. Per-SDK mechanism differs and is now documented where it exists: - Go: tls.Config.GetClientCertificate callback (no reconnect needed) - Java: gRPC AdvancedTlsX509KeyManager (no reconnect needed) - Python, .NET, Ruby: connect a new client with the new cert and assign it to the running Worker (client/Client property setter) - TypeScript: same pattern via worker.connection, available since @temporalio/worker 1.15.0 - PHP: noted as not yet supported, requires a Worker restart Also tidies the bare/miscased external reference link in cloud-access-control.mdx into a proper citation. Addresses #3694 * Note that Rust doesn't yet support live mTLS cert rotation Verified against the Rust SDK's public temporal_sdk::Worker API and sdk-core: TlsOptions is read once into a static connection, and while sdk-core gained a replace_client primitive in 2024 (used internally by the other language SDKs' hot-swap support), it isn't exposed on the Rust SDK's own public Worker type. Confirmed as a real, currently-open gap via sdk-rust#1338. * Update mTLS client certificate rotation instructions Clarify the process for rotating mTLS client certificates in the Rust SDK documentation.
1 parent e577862 commit 513b9e4

9 files changed

Lines changed: 187 additions & 3 deletions

File tree

docs/best-practices/cloud-access-control.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,9 +38,9 @@ The high-level end-to-end rotation process is:
3838
5. **Remove old credentials**: Remove old certificates and API keys from your secrets provider after confirming successful migration
3939

4040
This approach ensures near-zero-downtime rotation and prevents authentication failures that could impact running workflows. For specific guidance to rotate mTLS certificates and API keys, see:
41-
- https://docs.temporal.io/cloud/certificates#manage-certificates
42-
- https://docs.temporal.io/cloud/api-keys#rotate-an-api-key
43-
- https://github.com/temporal-sa/temporal-Worker-cert-rotation
41+
- [How to add, update, and remove certificates in a Temporal Cloud Namespace](/cloud/certificates#manage-certificates)
42+
- [Rotate an API key](/cloud/api-keys#rotate-an-api-key)
43+
- Per-SDK code for rotating a Worker's mTLS client certificate without a restart is documented on each SDK's Temporal Client page, in the "Connect to Temporal Cloud" section (for example, [Go](/develop/go/client/temporal-client#connect-to-temporal-cloud)) — Go, Java, Python, .NET, Ruby, and TypeScript are all supported; PHP and Rust are not yet. The [temporal-worker-cert-rotation](https://github.com/temporal-sa/temporal-worker-cert-rotation) reference implementation walks through automating this with cert-manager on Kubernetes.
4444

4545
For mutual TLS (mTLS) implementations, using Let's Encrypt is not recommended, as it is designed primarily for public-facing services and lacks support for internal certificate requirements.
4646

docs/develop/dotnet/client/temporal-client.mdx

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -460,6 +460,42 @@ To update an API key, update the value of `ApiKey` on the existing client connec
460460
myClient.Connection.ApiKey = myKeyUpdated;
461461
```
462462

463+
To connect using mTLS instead of an API key, provide the client certificate and private key on `Tls`:
464+
465+
```csharp
466+
var myClient = TemporalClient.ConnectAsync(new(<endpoint>)
467+
{
468+
Namespace = "<namespace_id>.<account_id>",
469+
Tls = new()
470+
{
471+
ClientCert = File.ReadAllBytes("client-cert.pem"),
472+
ClientPrivateKey = File.ReadAllBytes("client-private-key.pem"),
473+
},
474+
});
475+
```
476+
477+
Unlike `ApiKey`, `TlsOptions` is not mutable on an existing connection — the certificate bytes are fixed once the
478+
connection is established. To rotate an mTLS client certificate without restarting your Worker, connect a new client
479+
with the new certificate and assign it to the running `TemporalWorker`'s `Client` property:
480+
481+
```csharp
482+
var newClient = await TemporalClient.ConnectAsync(new(<endpoint>)
483+
{
484+
Namespace = "<namespace_id>.<account_id>",
485+
Tls = new()
486+
{
487+
ClientCert = File.ReadAllBytes("client-cert-new.pem"),
488+
ClientPrivateKey = File.ReadAllBytes("client-private-key-new.pem"),
489+
},
490+
});
491+
492+
// worker is the TemporalWorker instance already running against the old client
493+
worker.Client = newClient;
494+
```
495+
496+
Setting `Client` replaces the connection the Worker uses for subsequent calls to the Temporal Service (Workflow Task
497+
completion, Activity Heartbeats, and so on); calls already in flight on the old client are not interrupted.
498+
463499
</TabItem>
464500

465501
</Tabs>

docs/develop/go/client/temporal-client.mdx

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -421,6 +421,38 @@ creds := client.NewAPIKeyDynamicCredentials(
421421
myKey = myKeyUpdated
422422
```
423423

424+
To rotate an mTLS client certificate without restarting your Worker, set `GetClientCertificate` on the `tls.Config`
425+
instead of setting `Certificates` directly. Go's standard library `crypto/tls` package calls this function on every new
426+
connection, so it always picks up the current certificate:
427+
428+
```go
429+
clientCertPath := "/path/to/client.crt"
430+
clientKeyPath := "/path/to/client.key"
431+
432+
clientOptions := client.Options{
433+
HostPort: <endpoint>,
434+
Namespace: <namespace_id>.<account_id>,
435+
ConnectionOptions: client.ConnectionOptions{
436+
TLS: &tls.Config{
437+
GetClientCertificate: func(*tls.CertificateRequestInfo) (*tls.Certificate, error) {
438+
cert, err := tls.LoadX509KeyPair(clientCertPath, clientKeyPath)
439+
if err != nil {
440+
return nil, err
441+
}
442+
return &cert, nil
443+
},
444+
},
445+
},
446+
}
447+
c, err := client.Dial(clientOptions)
448+
```
449+
450+
Rotate the certificate by overwriting the files at `clientCertPath` and `clientKeyPath`; the next time the connection
451+
re-establishes (for example, after Temporal Cloud's periodic connection recycling), `GetClientCertificate` reads the new
452+
files. This works because `ConnectionOptions.TLS` is a real `*tls.Config`, so any option `crypto/tls` supports is
453+
available. See this [reference implementation](https://github.com/temporal-sa/temporal-worker-cert-rotation) for a full
454+
walkthrough, including automating certificate issuance with cert-manager on Kubernetes.
455+
424456
You can use a combination of environment variables, configuration files, and code to set connection options. For
425457
example, you can load a base configuration from environment variables or a configuration file, and then override
426458
specific options in code.

docs/develop/java/client/temporal-client.mdx

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -600,6 +600,39 @@ the new Client:
600600
```
601601
<!--SNIPEND-->
602602

603+
To rotate an mTLS client certificate without restarting your Worker, build the `SslContext` with gRPC's
604+
[`AdvancedTlsX509KeyManager`](https://grpc.github.io/grpc-java/javadoc/io/grpc/util/AdvancedTlsX509KeyManager.html)
605+
instead of `SimpleSslContextBuilder`. `updateIdentityCredentialsFromFile` schedules a periodic reread of the certificate
606+
and key files, so the same `SslContext` keeps serving fresh credentials for the life of the process:
607+
608+
```java
609+
String tlsCertPath = "/path/to/tls.crt";
610+
String tlsKeyPath = "/path/to/tls.key"; // PKCS8 format
611+
612+
// Reread the certificate and key files every 5 minutes.
613+
AdvancedTlsX509KeyManager keyManager = new AdvancedTlsX509KeyManager();
614+
keyManager.updateIdentityCredentialsFromFile(
615+
new File(tlsKeyPath), new File(tlsCertPath),
616+
5, TimeUnit.MINUTES,
617+
Executors.newSingleThreadScheduledExecutor());
618+
619+
SslContextBuilder sslContextBuilder = SslContextBuilder.forClient();
620+
GrpcSslContexts.configure(sslContextBuilder);
621+
SslContext sslContext = sslContextBuilder.keyManager(keyManager).build();
622+
623+
WorkflowServiceStubsOptions stubsOptions = WorkflowServiceStubsOptions
624+
.newBuilder()
625+
.setSslContext(sslContext)
626+
.setTarget(gRPCEndpoint)
627+
.build();
628+
WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newServiceStubs(stubsOptions);
629+
```
630+
631+
Rotate the certificate by overwriting the files at `tlsCertPath` and `tlsKeyPath`; `keyManager` picks up the new files on
632+
its next scheduled check, and the Worker keeps running against the same `client`/`serviceStub`. See this
633+
[reference implementation](https://github.com/temporal-sa/temporal-worker-cert-rotation) (written for the Go SDK, but the
634+
"What if I am not using the Go SDK?" section covers this Java approach) for a full walkthrough.
635+
603636
</TabItem>
604637
</Tabs>
605638

docs/develop/php/client/temporal-client.mdx

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,13 @@ $serviceClient = \Temporal\Client\GRPC\ServiceClient::createSSL(/*...*/)
129129
->withAuthKey('your-api-key');
130130
```
131131

132+
Rotating an mTLS client certificate without restarting the Worker isn't currently supported by the PHP SDK or RoadRunner
133+
— the certificate configured on `ServiceClient::createSSL()` or in RoadRunner's `tls:` block is fixed for the life of
134+
the process. To rotate a certificate, stage the new certificate alongside the old one on your Temporal Cloud Namespace,
135+
then restart your Worker (RoadRunner process) with the new certificate before removing the old one. See
136+
[Update certificates using Temporal Cloud UI/tcld](/cloud/certificates#manage-certificates) for the zero-downtime
137+
staging sequence.
138+
132139
## How to start a Workflow Execution {/* #start-workflow-execution */}
133140

134141
[Workflow Execution](/workflow-execution) semantics rely on several parameters—that is, to start a Workflow Execution you must supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific contextual data, and Workflow Function parameters.

docs/develop/python/client/temporal-client.mdx

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -470,6 +470,33 @@ async def main():
470470
For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see
471471
[Temporal Customization Samples](https://github.com/temporalio/samples-server).
472472

473+
Unlike an API key, `TLSConfig` is static: the certificate bytes you pass to `Client.connect` are fixed for the lifetime
474+
of that `Client`. To rotate an mTLS client certificate without restarting your Worker, connect a new `Client` with the
475+
new certificate, then assign it to the running `Worker`'s `client` property:
476+
477+
```python
478+
with open("client-cert-new.pem", "rb") as f:
479+
client_cert = f.read()
480+
with open("client-private-key-new.pem", "rb") as f:
481+
client_private_key = f.read()
482+
483+
new_client = await Client.connect(
484+
"your-custom-namespace.tmprl.cloud:7233",
485+
namespace="<your-custom-namespace>.<account-id>",
486+
tls=TLSConfig(
487+
client_cert=client_cert,
488+
client_private_key=client_private_key,
489+
),
490+
)
491+
492+
# worker is the Worker instance already running against the old client
493+
worker.client = new_client
494+
```
495+
496+
The Worker starts using `new_client` for subsequent calls to the Temporal Service (Workflow Task completion, Activity
497+
Heartbeats, and so on); calls already in flight on the old client finish normally. The new client must use the same
498+
`Runtime` as the Worker's current client.
499+
473500
</TabItem>
474501

475502
</Tabs>

docs/develop/ruby/client/temporal-client.mdx

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -416,6 +416,27 @@ client = Temporalio::Client.connect(
416416
For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see
417417
[Temporal Customization Samples](https://github.com/temporalio/samples-server).
418418

419+
`TLSOptions` is static: the certificate you pass to `Client.connect` is fixed for the lifetime of that client. To rotate
420+
an mTLS client certificate without restarting your Worker, connect a new client with the new certificate, then assign
421+
it to the running `Worker`'s `client`:
422+
423+
```ruby
424+
new_client = Temporalio::Client.connect(
425+
'<endpoint>', # Endpoint
426+
'<namespace_id>.<account_id>', # Namespace
427+
tls: Temporalio::Client::Connection::TLSOptions.new(
428+
client_cert: File.read('my-client-cert-new.pem'),
429+
client_private_key: File.read('my-client-key-new.pem')
430+
)
431+
)
432+
433+
# my_worker is the Worker instance already running against the old client
434+
my_worker.client = new_client
435+
```
436+
437+
The Worker starts using `new_client` for subsequent calls to the Temporal Service; calls already in flight on the old
438+
client finish normally.
439+
419440
</TabItem>
420441

421442
</Tabs>

docs/develop/rust/client/temporal-client.mdx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,11 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
335335

336336
</Tabs>
337337

338+
Rotating an mTLS client certificate without restarting the Worker isn't currently supported by the Rust SDK — the
339+
certificate in `TlsOptions` is read once and baked into the connection at `Connection::connect`.To rotate a certificate, stage the new certificate alongside the old one on your Temporal Cloud Namespace, then restart your Worker with the new certificate before removing the old one; see
340+
[Update certificates using Temporal Cloud UI/tcld](/cloud/certificates#manage-certificates) for the zero-downtime
341+
staging sequence.
342+
338343
## Start a Workflow Execution {/* #start-workflow-execution */}
339344

340345
To start a Workflow Execution, supply:

docs/develop/typescript/client/temporal-client.mdx

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -598,6 +598,29 @@ const worker = await Worker.create({
598598
});
599599
```
600600

601+
`@temporalio/worker` v1.15.0 and later support replacing a running Worker's connection, which lets you rotate an mTLS
602+
client certificate without restarting the Worker. gRPC's TLS credentials don't support dynamic certs, so instead of
603+
updating the existing `NativeConnection` you create a new one with the new certificate and assign it to `worker.connection`:
604+
605+
```ts
606+
import { readFileSync } from 'fs';
607+
608+
const newConnection = await NativeConnection.connect({
609+
address: <endpoint>,
610+
tls: {
611+
clientCertPair: {
612+
crt: readFileSync('client-cert-new.pem'),
613+
key: readFileSync('client-key-new.pem'),
614+
},
615+
},
616+
});
617+
618+
worker.connection = newConnection;
619+
```
620+
621+
The Worker starts using `newConnection` for subsequent calls to the Temporal Service; calls already in flight on the old
622+
connection finish normally.
623+
601624
</TabItem>
602625

603626
</Tabs>

0 commit comments

Comments
 (0)