Skip to content

Commit 37f267c

Browse files
FelixTJDietrichFelixTJDietrich
committed
feat(server): validate production configuration on startup
1 parent 5728244 commit 37f267c

31 files changed

Lines changed: 1181 additions & 50 deletions

‎.changeset/redacted-configuration-readiness.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"hephaestus": minor
3+
---
4+
5+
Production startup now reports all catalogued production configuration problems together without exposing configured values, and instance administrators can inspect redacted deployment and runtime readiness facts through the API. New self-hosted installations generate and preserve internal secrets with `setup.sh`. **Operators:** validate production settings against the configuration readiness guide before upgrading; the process now refuses to start when a catalogued required setting is missing or invalid.

‎MIGRATION.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,22 @@ review runs and no feedback is prepared about them.
8282
**Operator action after upgrading**: open **Practices → Review → When and where** and confirm the
8383
**People** and **Repositories** counts. Existing members need no action. If expected contributors are
8484
missing, follow [Who counts as a person](https://ls1intum.github.io/Hephaestus/admin/practice-review#who-counts-as-a-person).
85+
86+
#### 🔴 Production configuration is validated before startup
87+
88+
**Affected**: deployments that activate the `prod` Spring profile and have a missing, malformed, or
89+
role-inconsistent required setting.
90+
91+
Production processes now validate the applicable requirements in the configuration readiness
92+
catalogue together and refuse to start until every reported error is resolved. The failure report
93+
identifies properties and documentation but never includes configured values.
94+
95+
**Action**: before upgrading, compare every production role's settings with the
96+
[configuration readiness guide](https://ls1intum.github.io/Hephaestus/admin/configuration-readiness).
97+
Run a staging process with the production profile and correct every required setting it reports.
98+
After the server role starts, an instance administrator can inspect the redacted facts through
99+
`GET /api/admin/configuration-readiness`.
100+
85101
#### 🔴 Practice area API names are replaced by practice group names
86102

87103
**Affected**: anything calling the application API directly. The generated Hephaestus web client is

‎docker/self-host/.env.example‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Hephaestus self-hosted — environment
33
# =============================================================================
44
#
5-
# cp .env.example .env # then fill in every REQUIRED value below
5+
# ./setup.sh # generate internal secrets, then fill in external values
66
# docker compose up -d
77
#
88
# Full guide (read it first): https://ls1intum.github.io/Hephaestus/admin/install
@@ -29,23 +29,18 @@ ACME_EMAIL=
2929
# --- Secrets (REQUIRED — generate once, then never change) -------------------
3030

3131
# Database password. Applied only when the data volume is first initialized.
32-
# Generate: openssl rand -hex 16
3332
POSTGRES_PASSWORD=
3433

3534
# AES-256 key encrypting credentials at rest (and sealing the JWT signing key).
36-
# EXACTLY 32 bytes — i.e. 32 ASCII characters (a non-ASCII character costs more than one byte,
37-
# and the key is rejected at boot). Losing or changing it makes every stored provider
35+
# EXACTLY 32 printable, non-space ASCII characters. Losing or changing it makes every stored provider
3836
# token unreadable — treat it like the database itself and back it up.
39-
# Generate: openssl rand -base64 24 | cut -c1-32
4037
HEPHAESTUS_SECURITY_ENCRYPTION_KEY=
4138

4239
# Base64-encoded 32-byte AES key sealing the short-lived OAuth state cookies.
43-
# Generate: openssl rand -base64 32
4440
HEPHAESTUS_AUTH_STATE_COOKIE_KEY=
4541

4642
# Shared secret verifying inbound GitHub/GitLab webhooks (min 32 chars).
4743
# You will enter this same value on the GitHub side — see the install guide.
48-
# Generate: openssl rand -hex 32
4944
WEBHOOK_SECRET=
5045

5146
# --- Login (at least one provider REQUIRED, or nobody can sign in) -----------

‎docker/self-host/compose.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Hephaestus — the one blessed self-hosted deployment
33
# =============================================================================
44
#
5-
# cp .env.example .env # fill in the values
5+
# ./setup.sh # generate internal secrets, then fill in external values
66
# docker compose up -d
77
#
88
# This file composes the same service definitions the maintainers deploy

‎docker/self-host/setup.sh‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
#!/bin/sh
2+
3+
set -eu
4+
umask 077
5+
6+
directory=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
7+
environment_file="$directory/.env"
8+
example_file="$directory/.env.example"
9+
10+
command -v openssl >/dev/null 2>&1 || {
11+
printf '%s\n' "openssl is required to generate installation secrets." >&2
12+
exit 1
13+
}
14+
15+
if [ -L "$environment_file" ]; then
16+
printf '%s\n' "Refusing to write through symlink $environment_file." >&2
17+
exit 1
18+
fi
19+
20+
working_file=$(mktemp "$directory/.env.setup.XXXXXX")
21+
generated_keys=
22+
trap 'rm -f "$working_file"' EXIT HUP INT TERM
23+
24+
if [ -f "$environment_file" ]; then
25+
cp "$environment_file" "$working_file"
26+
else
27+
cp "$example_file" "$working_file"
28+
fi
29+
chmod 600 "$working_file"
30+
31+
for key in POSTGRES_PASSWORD HEPHAESTUS_SECURITY_ENCRYPTION_KEY HEPHAESTUS_AUTH_STATE_COOKIE_KEY WEBHOOK_SECRET; do
32+
if [ "$(grep -c "^${key}=" "$working_file")" -gt 1 ]; then
33+
printf 'Refusing duplicate %s assignments in %s.\n' "$key" "$environment_file" >&2
34+
exit 1
35+
fi
36+
done
37+
38+
set_if_empty() {
39+
key=$1
40+
format=$2
41+
if grep -q "^${key}=." "$working_file"; then
42+
return
43+
fi
44+
case "$format" in
45+
hex16) value=$(openssl rand -hex 16) || return 1 ;;
46+
hex32) value=$(openssl rand -hex 32) || return 1 ;;
47+
base64) value=$(openssl rand -base64 32 | tr -d '\n') || return 1 ;;
48+
esac
49+
if ! grep -q "^${key}=" "$working_file"; then
50+
printf '%s=%s\n' "$key" "$value" >> "$working_file"
51+
generated_keys="$generated_keys $key"
52+
elif grep -q "^${key}=$" "$working_file"; then
53+
temporary_file=$(mktemp "$directory/.env.value.XXXXXX")
54+
while IFS= read -r line; do
55+
if [ "$line" = "$key=" ]; then
56+
printf '%s=%s\n' "$key" "$value"
57+
else
58+
printf '%s\n' "$line"
59+
fi
60+
done < "$working_file" > "$temporary_file"
61+
chmod 600 "$temporary_file"
62+
mv "$temporary_file" "$working_file"
63+
generated_keys="$generated_keys $key"
64+
fi
65+
}
66+
67+
set_if_empty POSTGRES_PASSWORD hex16
68+
set_if_empty HEPHAESTUS_SECURITY_ENCRYPTION_KEY hex16
69+
set_if_empty HEPHAESTUS_AUTH_STATE_COOKIE_KEY base64
70+
set_if_empty WEBHOOK_SECRET hex32
71+
72+
mv "$working_file" "$environment_file"
73+
74+
for key in $generated_keys; do
75+
printf 'Generated %s.\n' "$key"
76+
done
77+
78+
printf '\nConfiguration written to %s. Generated values were not printed.\n' "$environment_file"
79+
printf '%s\n' 'Set APP_HOSTNAME, ACME_EMAIL, one OAuth provider, and HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS before starting Hephaestus.'

‎docker/self-host/setup.test.sh‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
#!/bin/sh
2+
3+
set -eu
4+
5+
source_directory=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
6+
temporary_directory=$(mktemp -d)
7+
trap 'rm -rf "$temporary_directory"' EXIT HUP INT TERM
8+
9+
new_fixture() {
10+
fixture=$(mktemp -d "$temporary_directory/fixture.XXXXXX")
11+
cp "$source_directory/setup.sh" "$source_directory/.env.example" "$fixture/"
12+
}
13+
14+
new_fixture
15+
output=$($fixture/setup.sh)
16+
environment_file="$fixture/.env"
17+
grep -Eq '^POSTGRES_PASSWORD=[0-9a-f]{32}$' "$environment_file"
18+
grep -Eq '^HEPHAESTUS_SECURITY_ENCRYPTION_KEY=[0-9a-f]{32}$' "$environment_file"
19+
grep -Eq '^HEPHAESTUS_AUTH_STATE_COOKIE_KEY=[A-Za-z0-9+/]{43}=$' "$environment_file"
20+
grep -Eq '^WEBHOOK_SECRET=[0-9a-f]{64}$' "$environment_file"
21+
[ "$(stat -c '%a' "$environment_file")" = 600 ]
22+
while IFS= read -r secret; do
23+
case "$output" in
24+
*"$secret"*) printf '%s\n' "setup printed a generated secret" >&2; exit 1 ;;
25+
esac
26+
done <<EOF_SECRETS
27+
$(grep -E '^(POSTGRES_PASSWORD|HEPHAESTUS_SECURITY_ENCRYPTION_KEY|HEPHAESTUS_AUTH_STATE_COOKIE_KEY|WEBHOOK_SECRET)=' "$environment_file" | cut -d= -f2-)
28+
EOF_SECRETS
29+
cp "$environment_file" "$fixture/first.env"
30+
$fixture/setup.sh >/dev/null
31+
cmp "$fixture/first.env" "$environment_file"
32+
33+
new_fixture
34+
sed -e 's/^POSTGRES_PASSWORD=$/POSTGRES_PASSWORD=preserved/' -e '/^WEBHOOK_SECRET=$/d' "$fixture/.env.example" > "$fixture/.env"
35+
$fixture/setup.sh >/dev/null
36+
grep -q '^POSTGRES_PASSWORD=preserved$' "$fixture/.env"
37+
grep -Eq '^WEBHOOK_SECRET=[0-9a-f]{64}$' "$fixture/.env"
38+
39+
new_fixture
40+
printf '%s\n' 'POSTGRES_PASSWORD=duplicate' >> "$fixture/.env.example"
41+
if $fixture/setup.sh >/dev/null 2>&1; then
42+
printf '%s\n' "setup accepted a duplicate managed setting" >&2
43+
exit 1
44+
fi
45+
[ ! -e "$fixture/.env" ]
46+
47+
new_fixture
48+
cp "$fixture/.env.example" "$fixture/.env"
49+
cp "$fixture/.env" "$fixture/before.env"
50+
mkdir "$fixture/bin"
51+
cat > "$fixture/bin/openssl" <<'EOF_OPENSSL'
52+
#!/bin/sh
53+
exit 1
54+
EOF_OPENSSL
55+
chmod +x "$fixture/bin/openssl"
56+
if PATH="$fixture/bin:$PATH" $fixture/setup.sh >/dev/null 2>&1; then
57+
printf '%s\n' "setup accepted a failed secret generator" >&2
58+
exit 1
59+
fi
60+
cmp "$fixture/before.env" "$fixture/.env"
61+
62+
new_fixture
63+
printf '%s' unchanged > "$fixture/target"
64+
ln -s "$fixture/target" "$fixture/.env"
65+
if $fixture/setup.sh >/dev/null 2>&1; then
66+
printf '%s\n' "setup accepted an environment symlink" >&2
67+
exit 1
68+
fi
69+
[ "$(cat "$fixture/target")" = unchanged ]
70+
71+
printf '%s\n' "Self-host setup tests passed."

‎docs/admin/configuration-readiness.mdx‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
---
2+
title: Configuration readiness
3+
---
4+
5+
# Configuration readiness
6+
7+
The production profile validates deployment settings during startup. The server-role endpoint
8+
`GET /api/admin/configuration-readiness` returns those facts plus checks that need runtime state. It
9+
requires the `app_admin` authority and is available only when boot-fatal checks pass. Diagnostics
10+
never contain configured values.
11+
12+
The table names variables inside the application container. In the supported self-host stack,
13+
Compose maps `POSTGRES_PASSWORD` to `DATABASE_PASSWORD` and owns the runtime-role values.
14+
15+
| Setting | Application environment variable |
16+
| --- | --- |
17+
| `spring.datasource.url` | `DATABASE_URL` |
18+
| `spring.datasource.username` | `DATABASE_USERNAME` |
19+
| `spring.datasource.password` | `DATABASE_PASSWORD` |
20+
| `hephaestus.runtime.server.enabled` | `HEPHAESTUS_RUNTIME_SERVER_ENABLED` |
21+
| `hephaestus.runtime.worker.enabled` | `HEPHAESTUS_RUNTIME_WORKER_ENABLED` |
22+
| `hephaestus.runtime.webhook.enabled` | `HEPHAESTUS_RUNTIME_WEBHOOK_ENABLED` |
23+
| `hephaestus.host-url` | `APPLICATION_HOST_URL` |
24+
| `hephaestus.security.encryption-key` | `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` |
25+
| `hephaestus.webhook.secret` | `WEBHOOK_SECRET` |
26+
| `hephaestus.sync.nats.enabled` | `NATS_ENABLED` |
27+
| `hephaestus.sync.nats.server` | `NATS_SERVER` |
28+
| `hephaestus.auth.state-cookie-key` | `HEPHAESTUS_AUTH_STATE_COOKIE_KEY` |
29+
| `hephaestus.llm.egress.allow-loopback` | `HEPHAESTUS_LLM_EGRESS_ALLOW_LOOPBACK` |
30+
| `hephaestus.agent.image.require-digest` | `HEPHAESTUS_AGENT_IMAGE_REQUIRE_DIGEST` |
31+
| `hephaestus.agent.image.reference` | `HEPHAESTUS_AGENT_IMAGE_REFERENCE` |
32+
| `hephaestus.sandbox.container-runtime` | `SANDBOX_CONTAINER_RUNTIME` |
33+
| `hephaestus.sentry.dsn` | `SENTRY_DSN` |
34+
35+
## Runtime roles {#runtime-roles}
36+
37+
Enable at least one of `hephaestus.runtime.server.enabled`, `worker.enabled`, or `webhook.enabled`;
38+
each accepts only `true` or `false`.
39+
The supported split topology enables only webhook on the webhook process, only worker on a remote
40+
worker, and server (optionally with a colocated worker) on the application process.
41+
42+
## Database {#database}
43+
44+
Every role uses PostgreSQL. `DATABASE_URL` must be a PostgreSQL URL; the production profile adds the
45+
`jdbc:` prefix. Supply a non-empty username and password. This syntax check does not replace the
46+
connection and migration health checks performed by Spring Boot and Liquibase.
47+
48+
## Credential encryption {#credential-encryption}
49+
50+
Set `hephaestus.security.encryption-key` to exactly 32 printable, non-space ASCII characters and keep it
51+
with the database backup. The supported self-host setup generates it with `openssl rand -hex 16`. Do
52+
not change it on an existing installation.
53+
54+
## External URL {#external-url}
55+
56+
Set `hephaestus.host-url` to the public HTTPS origin, without credentials, a path other than `/`, a
57+
query, or a fragment.
58+
59+
## Webhooks {#webhooks}
60+
61+
Server and webhook roles require `hephaestus.webhook.secret` with at least 32 printable, non-space ASCII
62+
characters. The supported self-host setup generates an independent value; never reuse another
63+
application key.
64+
65+
## NATS {#nats}
66+
67+
Server and webhook roles require NATS and an explicit `nats://` or `tls://` URI with a host, an optional
68+
valid port, and no query, fragment, or non-root path. A worker-only process must disable NATS because
69+
its job queue is PostgreSQL-backed. This check validates syntax and role consistency, not
70+
authentication, connectivity, or JetStream health.
71+
72+
## Login {#login}
73+
74+
The server role requires a Base64-encoded 32-byte `hephaestus.auth.state-cookie-key` and an enabled
75+
GitHub or GitLab sign-in provider in the database-backed provider catalogue. Environment provider
76+
entries are seeds, not the readiness authority. Slack and Outline are link-only providers and do not
77+
satisfy sign-in readiness. Worker and webhook roles do not load login providers.
78+
79+
## LLM proxy {#llm-proxy}
80+
81+
Worker roles must leave `hephaestus.llm.egress.allow-loopback=false`. Provider credentials and model
82+
configuration are database-backed runtime configuration and are not deployment settings.
83+
84+
## Agent image {#agent-image}
85+
86+
Worker roles require digest enforcement and a SHA-256-pinned `hephaestus.agent.image.reference`. See
87+
[Agent image digests](./agent-image-digests.md).
88+
89+
## Sandbox isolation {#sandbox-isolation}
90+
91+
Set `SANDBOX_CONTAINER_RUNTIME=runsc` on workers after
92+
[installing and configuring gVisor](https://gvisor.dev/docs/user_guide/install/) on the host. This
93+
recommendation is non-fatal.
94+
95+
## Optional observability {#optional-observability}
96+
97+
Sentry is optional. When configured, `hephaestus.sentry.dsn` must use HTTPS. The fact is classified
98+
`OPTIONAL` and never prevents startup.

‎docs/admin/install.mdx‎

Lines changed: 18 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,8 @@ The stack you get:
3232
- **AI practice review adds real memory**: each concurrent review sandbox may use up to
3333
4 GiB. The default caps it at 1 concurrent sandbox; raise
3434
`SANDBOX_MAX_CONCURRENT` only with RAM to match.
35-
- 64-bit Linux with **Docker Engine ≥ 24** and **Docker Compose ≥ 2.24.4** (`docker compose version`), plus `git`.
35+
- 64-bit Linux with **Docker Engine ≥ 24** and **Docker Compose ≥ 2.24.4** (`docker compose version`),
36+
plus `git` and `openssl`.
3637
- A **DNS A record** for your hostname pointing at the host, with ports **80 and 443**
3738
reachable from the internet (Let's Encrypt HTTP-01, OAuth callbacks, webhooks).
3839
- Outbound HTTPS to `ghcr.io` and `docker.io` (images — NATS and Traefik come from Docker Hub),
@@ -52,28 +53,28 @@ sudo git clone --depth 1 --branch "v$VERSION" \
5253
https://github.com/ls1intum/Hephaestus.git /opt/hephaestus
5354
sudo chown -R "$USER" /opt/hephaestus
5455
cd /opt/hephaestus/docker/self-host
55-
cp .env.example .env
56+
./setup.sh
5657
```
5758

58-
Everything below runs from `/opt/hephaestus/docker/self-host`. Once step 2 has filled in every
59-
required value, `docker compose config` renders the effective configuration — it is the quickest way
60-
to confirm a variable landed where you expected. Before then it exits with the first
61-
`required variable … is missing a value`, which is also how every other `docker compose` subcommand
62-
behaves in this directory; see [Troubleshooting](#troubleshooting).
59+
`setup.sh` creates `.env` with mode `0600` and generates the database password, credential-encryption
60+
key, OAuth state-cookie key, and webhook secret. It does not replace non-empty values and never prints
61+
a secret.
62+
63+
Everything below runs from `/opt/hephaestus/docker/self-host`. After step 2 supplies the remaining
64+
required values, `docker compose config` renders the effective configuration. Before then it exits
65+
at the first missing required variable, as does every other `docker compose` subcommand in this
66+
directory; see [Troubleshooting](#troubleshooting).
6367

6468
## 2. Configure `.env`
6569

66-
Open `.env` and fill in every **REQUIRED** value. The short version:
70+
Open `.env` and fill in the remaining **REQUIRED** values. `setup.sh` manages the four internal
71+
secrets listed above; do not replace them manually.
6772

6873
| Variable | What / how |
6974
| --- | --- |
7075
| `APP_HOSTNAME` | Your public hostname, e.g. `hephaestus.example.com` |
7176
| `IMAGE_TAG` | The same `$VERSION` you fetched above — an exact release, never `latest` |
7277
| `ACME_EMAIL` | Let's Encrypt expiry notices |
73-
| `POSTGRES_PASSWORD` | `openssl rand -hex 16` |
74-
| `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` | `openssl rand -base64 24 \| cut -c1-32` — exactly 32 chars (see the warning below) |
75-
| `HEPHAESTUS_AUTH_STATE_COOKIE_KEY` | `openssl rand -base64 32` |
76-
| `WEBHOOK_SECRET` | `openssl rand -hex 32` — you'll enter the same value on GitHub later |
7778
| `GH_OAUTH_CLIENT_ID` / `_SECRET` | From step 3 |
7879
| `HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS` | From step 4 — set it **before** first boot |
7980

@@ -83,6 +84,11 @@ signing key. If you lose or change it, those tokens become unreadable and all se
8384
invalidate. Store it as carefully as the database itself, and never change it after first boot.
8485
:::
8586

87+
The other generated values also have lifecycle consequences: changing the database password does not
88+
update an initialized PostgreSQL volume, rotating the state-cookie key invalidates OAuth flows already
89+
in progress, and rotating the webhook secret requires the same change at every configured provider.
90+
Back up `.env` with the database as described in [Backup & Restore](./backup-restore).
91+
8692
## 3. Create the GitHub OAuth App (login — mandatory)
8793

8894
Without a login provider the instance boots but shows **no sign-in button**. GitHub login

‎docs/sidebars.admin.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ const sidebars: SidebarsConfig = {
2020
{ type: "doc", id: "production-setup", label: "Integrations & Reference Deployment" },
2121
{ type: "doc", id: "compatibility-policy", label: "Compatibility Policy" },
2222
{ type: "doc", id: "runtime-roles", label: "Runtime Roles" },
23+
{ type: "doc", id: "configuration-readiness", label: "Configuration Readiness" },
2324
{ type: "doc", id: "agent-image-digests", label: "Agent image digests" },
2425
{ type: "doc", id: "buildpacks-cds-decision", label: "Server image build (Buildpacks + CDS)" },
2526
{ type: "doc", id: "legal-pages", label: "Legal Pages" },

0 commit comments

Comments
 (0)