Skip to content

Commit 6d02663

Browse files
jackkleemanjackkleeman
committed
Split out minio docs and document network isolation
1 parent a785b0c commit 6d02663

2 files changed

Lines changed: 169 additions & 140 deletions

File tree

‎README.md‎

Lines changed: 17 additions & 140 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,19 @@ The operator introduces two Custom Resource Definitions (CRDs): `RestateCluster`
5050

5151
The `RestateCluster` CRD defines a Restate cluster. The operator watches for these objects and creates the necessary Kubernetes resources, such as `StatefulSet`, `Service`, and `NetworkPolicy` objects in a new namespace that matches the `RestateCluster` name.
5252

53+
*By default, the operator enforces network isolation on the cluster, allowing only the following*:
54+
1. Peer to peer traffic between Restate pods
55+
2. Traffic from the operator to Restate pods
56+
3. Egress traffic to the public internet
57+
4. Egress traffic to coredns
58+
4. Egress traffic to pods in any namespace labelled with `allow.restate.dev/<cluster-name>`
59+
60+
*All other traffic is denied by default.*
61+
62+
The default behaviour can be disabled with `spec.security.disableNetworkPolicies: true`.
63+
Alternatively, you can add new allowed inbound callers of the Restate ports with `spec.security.networkPeers.{ingress,admin,metrics}`, which are arrays of [`NetworkPolicyPeer`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#networkpolicypeer-v1-networking-k8s-io).
64+
You can allow new outbound destinations by adding to the `spec.security.networkEgressRules` list, which is an array of [`NetworkPolicyEgressRule`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#networkpolicyegressrule-v1-networking-k8s-io).
65+
5366
**NOTE**: Each cluster is created in its own namespace (enforced by the operator). Do not create the namespace manually,
5467
and do not use the same namespace as the operator is in.
5568

@@ -69,6 +82,8 @@ spec:
6982
storageRequestBytes: 2147483648 # 2 GiB
7083
```
7184
85+
For the full schema as a [Pkl](https://pkl-lang.org/) template see [`crd/RestateCluster.pkl`](./crd/RestateCluster.pkl).
86+
7287
More examples are available just below the spec that follows.
7388

7489
#### Spec Fields
@@ -269,146 +284,8 @@ spec:
269284
# aws-allow-http = true
270285
```
271286

272-
#### MinIO Configuration Example
273-
274-
To configure a `RestateCluster` with a self-hosted S3-compatible object store like [MinIO](https://min.io/), you can point the server to your MinIO instance. For security, it's best to create a dedicated service account with credentials scoped only to the buckets Restate needs.
275-
276-
##### 1. Create a Scoped Access Key & Buckets
277-
278-
First, we'll define a policy, create the buckets, and then create a service account with a new access key that is restricted by that policy.
279-
280-
The following commands use the `mc` client to set up your MinIO instance. They assume you have port-forwarded your MinIO service.
281-
282-
```bash
283-
# Forward the MinIO service to your local machine
284-
kubectl port-forward --namespace storage svc/minio 9000:443
285-
286-
# Alias your MinIO deployment for easier access
287-
# Use https:// and --insecure if connecting to a port-forwarded TLS port (like 443)
288-
mc alias set local-minio https://localhost:9000 YOUR_ADMIN_ACCESS_KEY YOUR_ADMIN_SECRET_KEY --insecure
289-
290-
# Create the buckets
291-
mc mb --insecure local-minio/restate-metadata
292-
mc mb --insecure local-minio/restate-snapshots
293-
294-
# Add the new policy to MinIO
295-
296-
cat <<EOF | mc admin policy add local-minio restate-s3-policy
297-
{
298-
"Version": "2012-10-17",
299-
"Statement": [
300-
{
301-
"Effect": "Allow",
302-
"Action": [
303-
"s3:GetBucketLocation",
304-
"s3:ListBucket"
305-
],
306-
"Resource": [
307-
"arn:aws:s3:::restate-metadata",
308-
"arn:aws:s3:::restate-snapshots"
309-
]
310-
},
311-
{
312-
"Effect": "Allow",
313-
"Action": [
314-
"s3:PutObject",
315-
"s3:GetObject",
316-
"s3:DeleteObject",
317-
"s3:ListMultipartUploadParts",
318-
"s3:AbortMultipartUpload"
319-
],
320-
"Resource": [
321-
"arn:aws:s3:::restate-metadata/*",
322-
"arn:aws:s3:::restate-snapshots/*"
323-
]
324-
}
325-
]
326-
}
327-
EOF
328-
329-
# Create a new service account for the Restate application
330-
# This command will output a new AccessKey and SecretKey.
331-
mc admin service-account add local-minio restate
332-
333-
# Attach the policy to the new service account
334-
mc admin policy set local-minio restate-s3-policy service-account=restate
335-
```
336-
337-
When you run `mc admin service-account add`, it will output a new `AccessKey` and `SecretKey`. **Save these securely**, as you will use them in the next step.
338-
339-
##### 2. Create a Kubernetes Secret
340-
341-
Next, create a Kubernetes `Secret` containing the new, **scoped credentials** you just generated for the `restate` service account.
342-
343-
**Important**: This secret must be created in the namespace where the cluster will run, which is the same as the `metadata.name` of your `RestateCluster`. For this example, the namespace is `restate-with-minio`.
344-
345-
```yaml
346-
apiVersion: v1
347-
kind: Secret
348-
metadata:
349-
name: minio-credentials
350-
namespace: restate-with-minio
351-
type: Opaque
352-
stringData:
353-
# Use the keys generated from the 'mc admin service-account add' command
354-
AWS_ACCESS_KEY_ID: YOUR_NEW_RESTATE_ACCESS_KEY
355-
AWS_SECRET_ACCESS_KEY: YOUR_NEW_RESTATE_SECRET_KEY
356-
```
357-
358-
##### 3. Configure the `RestateCluster`
359-
360-
Finally, define your `RestateCluster` resource. This manifest is the same as before, but it will now use the Kubernetes secret containing the limited-permission keys.
361-
362-
```yaml
363-
apiVersion: restate.dev/v1
364-
kind: RestateCluster
365-
metadata:
366-
name: restate-minio-test
367-
spec:
368-
compute:
369-
replicas: 3
370-
image: restatedev/restate:1.4
371-
env:
372-
- name: AWS_ACCESS_KEY_ID
373-
valueFrom:
374-
secretKeyRef:
375-
name: minio-credentials
376-
key: AWS_ACCESS_KEY_ID
377-
- name: AWS_SECRET_ACCESS_KEY
378-
valueFrom:
379-
secretKeyRef:
380-
name: minio-credentials
381-
key: AWS_SECRET_ACCESS_KEY
382-
storage:
383-
storageRequestBytes: 2147483648 # 2 GiB
384-
config: |
385-
roles = [ "worker", "admin", "log-server", "http-ingress" ]
386-
auto-provision = true
387-
default-num-partitions = 128
388-
default-replication = 2
389-
390-
[metadata-client]
391-
type = "object-store"
392-
path = "s3://restate-metadata/metadata"
393-
aws-endpoint-url = "http://minio.minio-namespace.svc.cluster.local:9000"
394-
aws-allow-http = true
395-
aws-region = "local"
396-
397-
[bifrost]
398-
default-provider = "replicated"
399-
400-
[worker.snapshots]
401-
destination = "s3://restate-snapshots/snapshots"
402-
snapshot-interval-num-records = 10000
403-
aws-endpoint-url = "http://minio.minio-namespace.svc.cluster.local:9000"
404-
aws-allow-http = true
405-
aws-region = "local"
406-
```
407-
408-
> **A Note on AWS Credentials**: You might notice we are using standard AWS environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) directly, rather than the Restate-specific format like `RESTATE_WORKER__SNAPSHOTS_AWS_ACCESS_KEY_ID`. This is because Restate uses the underlying AWS SDK, which automatically and conventionally discovers credentials from these standard environment variables. However, if you are using Minio for snapshots and also need to use AWS Lambda services from your Restate cluster, then you may need different AWS credentials for different components.
409-
410-
For the full schema as a [Pkl](https://pkl-lang.org/) template see [`crd/RestateCluster.pkl`](./crd/RestateCluster.pkl).
411-
287+
#### MinIO example
288+
See [docs/minio.md](./docs/minio.md)
412289

413290
### `RestateDeployment`
414291

‎docs/minio.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# MinIO Configuration Example
2+
3+
To configure a `RestateCluster` with a self-hosted S3-compatible object store like [MinIO](https://min.io/), you can point the server to your MinIO instance. For security, it's best to create a dedicated service account with credentials scoped only to the buckets Restate needs.
4+
5+
## 1. Create a Scoped Access Key & Buckets
6+
7+
First, we'll define a policy, create the buckets, and then create a service account with a new access key that is restricted by that policy.
8+
9+
The following commands use the `mc` client to set up your MinIO instance. They assume you have port-forwarded your MinIO service.
10+
11+
```bash
12+
# Forward the MinIO service to your local machine
13+
kubectl port-forward --namespace storage svc/minio 9000:443
14+
15+
# Alias your MinIO deployment for easier access
16+
# Use https:// and --insecure if connecting to a port-forwarded TLS port (like 443)
17+
mc alias set local-minio https://localhost:9000 YOUR_ADMIN_ACCESS_KEY YOUR_ADMIN_SECRET_KEY --insecure
18+
19+
# Create the buckets
20+
mc mb --insecure local-minio/restate-metadata
21+
mc mb --insecure local-minio/restate-snapshots
22+
23+
# Add the new policy to MinIO
24+
25+
cat <<EOF | mc admin policy add local-minio restate-s3-policy
26+
{
27+
"Version": "2012-10-17",
28+
"Statement": [
29+
{
30+
"Effect": "Allow",
31+
"Action": [
32+
"s3:GetBucketLocation",
33+
"s3:ListBucket"
34+
],
35+
"Resource": [
36+
"arn:aws:s3:::restate-metadata",
37+
"arn:aws:s3:::restate-snapshots"
38+
]
39+
},
40+
{
41+
"Effect": "Allow",
42+
"Action": [
43+
"s3:PutObject",
44+
"s3:GetObject",
45+
"s3:DeleteObject",
46+
"s3:ListMultipartUploadParts",
47+
"s3:AbortMultipartUpload"
48+
],
49+
"Resource": [
50+
"arn:aws:s3:::restate-metadata/*",
51+
"arn:aws:s3:::restate-snapshots/*"
52+
]
53+
}
54+
]
55+
}
56+
EOF
57+
58+
# Create a new service account for the Restate application
59+
# This command will output a new AccessKey and SecretKey.
60+
mc admin service-account add local-minio restate
61+
62+
# Attach the policy to the new service account
63+
mc admin policy set local-minio restate-s3-policy service-account=restate
64+
```
65+
66+
When you run `mc admin service-account add`, it will output a new `AccessKey` and `SecretKey`. **Save these securely**, as you will use them in the next step.
67+
68+
### 2. Create a Kubernetes Secret
69+
70+
Next, create a Kubernetes `Secret` containing the new, **scoped credentials** you just generated for the `restate` service account.
71+
72+
**Important**: This secret must be created in the namespace where the cluster will run, which is the same as the `metadata.name` of your `RestateCluster`. For this example, the namespace is `restate-with-minio`.
73+
74+
```yaml
75+
apiVersion: v1
76+
kind: Secret
77+
metadata:
78+
name: minio-credentials
79+
namespace: restate-with-minio
80+
type: Opaque
81+
stringData:
82+
# Use the keys generated from the 'mc admin service-account add' command
83+
access-key: YOUR_NEW_RESTATE_ACCESS_KEY
84+
secret-key: YOUR_NEW_RESTATE_SECRET_KEY
85+
```
86+
87+
##### 3. Configure the `RestateCluster`
88+
89+
Finally, define your `RestateCluster` resource. This manifest is the same as before, but it will now use the Kubernetes secret containing the limited-permission keys.
90+
Additionally, we add the MinIO namespace to the allowed list of egress peers.
91+
92+
```yaml
93+
apiVersion: restate.dev/v1
94+
kind: RestateCluster
95+
metadata:
96+
name: restate-minio-test
97+
spec:
98+
compute:
99+
replicas: 3
100+
image: restatedev/restate:1.4
101+
env:
102+
- name: RESTATE_METADATA_CLIENT__AWS_ACCESS_KEY_ID
103+
valueFrom:
104+
secretKeyRef:
105+
name: minio-credentials
106+
key: access-key
107+
- name: RESTATE_METADATA_CLIENT__AWS_SECRET_ACCESS_KEY
108+
valueFrom:
109+
secretKeyRef:
110+
name: minio-credentials
111+
key: secret-key
112+
- name: RESTATE_WORKER__SNAPSHOTS__AWS_ACCESS_KEY_ID
113+
valueFrom:
114+
secretKeyRef:
115+
name: minio-credentials
116+
key: access-key
117+
- name: RESTATE_WORKER__SNAPSHOTS__AWS_SECRET_ACCESS_KEY
118+
valueFrom:
119+
secretKeyRef:
120+
name: minio-credentials
121+
key: secret-key
122+
security:
123+
networkEgressRules:
124+
- to:
125+
- namespaceSelector:
126+
matchLabels:
127+
kubernetes.io/metadata.name: minio-namespace
128+
storage:
129+
storageRequestBytes: 2147483648 # 2 GiB
130+
config: |
131+
roles = [ "worker", "admin", "log-server", "http-ingress" ]
132+
auto-provision = true
133+
default-num-partitions = 128
134+
default-replication = 2
135+
136+
[metadata-client]
137+
type = "object-store"
138+
path = "s3://restate-metadata/metadata"
139+
aws-endpoint-url = "http://minio.minio-namespace.svc.cluster.local:9000"
140+
aws-allow-http = true
141+
aws-region = "local"
142+
143+
[bifrost]
144+
default-provider = "replicated"
145+
146+
[worker.snapshots]
147+
destination = "s3://restate-snapshots/snapshots"
148+
snapshot-interval-num-records = 10000
149+
aws-endpoint-url = "http://minio.minio-namespace.svc.cluster.local:9000"
150+
aws-allow-http = true
151+
aws-region = "local"
152+
```

0 commit comments

Comments
 (0)