Skip to content

Commit 9865ff5

Browse files
FelixTJDietrichFelixTJDietrich
committed
feat(server): validate production configuration on startup
1 parent 66042a3 commit 9865ff5

34 files changed

Lines changed: 1267 additions & 56 deletions

‎.changeset/redacted-configuration-readiness.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
"hephaestus": minor
3+
---
4+
5+
Production startup now reports all catalogued production configuration problems together without
6+
exposing configured values, and instance administrators can inspect redacted deployment and runtime
7+
readiness facts through the API. New self-hosted installations generate and preserve internal secrets
8+
with `setup.sh`. **Operators:** validate production settings against the configuration readiness guide
9+
before upgrading; the process now refuses to start when a catalogued required setting is missing or
10+
invalid.

‎AGENTS.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,11 @@ adding a rule there enables it nowhere. `webapp/AGENTS.md` § Linting has the re
9595
Holds wherever TypeScript is written here, the Bun agent trees and `scripts/**` included.
9696
`webapp/AGENTS.md` wins over it inside the SPA.
9797

98+
Prefer typed Bun/TypeScript for repository automation, validation, and tests. Keep shell only at a
99+
real runtime boundary where Bun is unavailable, such as an end-user bootstrap that must run before
100+
the application toolchain is installed; keep that boundary POSIX-compatible and move its substantive
101+
test orchestration into TypeScript.
102+
98103
- Separate import groups with blank lines wherever their relative evaluation order must not change; oxfmt sorts within each group.
99104
- **A leading `_` marks what the language or a tool reads that way** — an intentionally unused
100105
binding, a server field name (`_id`), a runtime global. It never marks something private, which

‎MIGRATION.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,22 @@ review runs and no feedback is prepared about them.
8282
**Operator action after upgrading**: open **Practices → Review → When and where** and confirm the
8383
**People** and **Repositories** counts. Existing members need no action. If expected contributors are
8484
missing, follow [Who counts as a person](https://ls1intum.github.io/Hephaestus/admin/practice-review#who-counts-as-a-person).
85+
86+
#### 🔴 Production configuration is validated before startup
87+
88+
**Affected**: deployments that activate the `prod` Spring profile and have a missing, malformed, or
89+
role-inconsistent required setting.
90+
91+
Production processes now validate the applicable requirements in the configuration readiness
92+
catalogue together and refuse to start until every reported error is resolved. The failure report
93+
identifies properties and documentation but never includes configured values.
94+
95+
**Action**: before upgrading, compare every production role's settings with the
96+
[configuration readiness guide](https://ls1intum.github.io/Hephaestus/admin/configuration-readiness).
97+
Run a staging process with the production profile and correct every required setting it reports.
98+
After the server role starts, an instance administrator can inspect the redacted facts through
99+
`GET /api/admin/configuration-readiness`.
100+
85101
#### 🔴 Practice area API names are replaced by practice group names
86102

87103
**Affected**: anything calling the application API directly. The generated Hephaestus web client is

‎docker/self-host/.env.example‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Hephaestus self-hosted — environment
33
# =============================================================================
44
#
5-
# cp .env.example .env # then fill in every REQUIRED value below
5+
# ./setup.sh # generate internal secrets, then fill in external values
66
# docker compose up -d
77
#
88
# Full guide (read it first): https://ls1intum.github.io/Hephaestus/admin/install
@@ -29,23 +29,18 @@ ACME_EMAIL=
2929
# --- Secrets (REQUIRED — generate once, then never change) -------------------
3030

3131
# Database password. Applied only when the data volume is first initialized.
32-
# Generate: openssl rand -hex 16
3332
POSTGRES_PASSWORD=
3433

3534
# AES-256 key encrypting credentials at rest (and sealing the JWT signing key).
36-
# EXACTLY 32 bytes — i.e. 32 ASCII characters (a non-ASCII character costs more than one byte,
37-
# and the key is rejected at boot). Losing or changing it makes every stored provider
35+
# EXACTLY 32 printable, non-space ASCII characters. Losing or changing it makes every stored provider
3836
# token unreadable — treat it like the database itself and back it up.
39-
# Generate: openssl rand -base64 24 | cut -c1-32
4037
HEPHAESTUS_SECURITY_ENCRYPTION_KEY=
4138

4239
# Base64-encoded 32-byte AES key sealing the short-lived OAuth state cookies.
43-
# Generate: openssl rand -base64 32
4440
HEPHAESTUS_AUTH_STATE_COOKIE_KEY=
4541

4642
# Shared secret verifying inbound GitHub/GitLab webhooks (min 32 chars).
4743
# You will enter this same value on the GitHub side — see the install guide.
48-
# Generate: openssl rand -hex 32
4944
WEBHOOK_SECRET=
5045

5146
# --- Login (at least one provider REQUIRED, or nobody can sign in) -----------

‎docker/self-host/compose.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Hephaestus — the one blessed self-hosted deployment
33
# =============================================================================
44
#
5-
# cp .env.example .env # fill in the values
5+
# ./setup.sh # generate internal secrets, then fill in external values
66
# docker compose up -d
77
#
88
# This file composes the same service definitions the maintainers deploy

‎docker/self-host/setup.sh‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
#!/bin/sh
2+
3+
set -eu
4+
umask 077
5+
6+
directory=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
7+
environment_file="$directory/.env"
8+
example_file="$directory/.env.example"
9+
10+
command -v openssl >/dev/null 2>&1 || {
11+
printf '%s\n' "openssl is required to generate installation secrets." >&2
12+
exit 1
13+
}
14+
15+
if [ -L "$environment_file" ]; then
16+
printf '%s\n' "Refusing to write through symlink $environment_file." >&2
17+
exit 1
18+
fi
19+
20+
working_file=$(mktemp "$directory/.env.setup.XXXXXX")
21+
generated_keys=
22+
trap 'rm -f "$working_file"' EXIT HUP INT TERM
23+
24+
if [ -f "$environment_file" ]; then
25+
cp "$environment_file" "$working_file"
26+
else
27+
cp "$example_file" "$working_file"
28+
fi
29+
chmod 600 "$working_file"
30+
31+
for key in POSTGRES_PASSWORD HEPHAESTUS_SECURITY_ENCRYPTION_KEY HEPHAESTUS_AUTH_STATE_COOKIE_KEY WEBHOOK_SECRET; do
32+
if [ "$(grep -c "^${key}=" "$working_file")" -gt 1 ]; then
33+
printf 'Refusing duplicate %s assignments in %s.\n' "$key" "$environment_file" >&2
34+
exit 1
35+
fi
36+
done
37+
38+
set_if_empty() {
39+
key=$1
40+
format=$2
41+
if grep -q "^${key}=." "$working_file"; then
42+
return
43+
fi
44+
case "$format" in
45+
hex16) value=$(openssl rand -hex 16) || return 1 ;;
46+
hex32) value=$(openssl rand -hex 32) || return 1 ;;
47+
base64) value=$(openssl rand -base64 32 | tr -d '\n') || return 1 ;;
48+
esac
49+
if ! grep -q "^${key}=" "$working_file"; then
50+
printf '%s=%s\n' "$key" "$value" >> "$working_file"
51+
generated_keys="$generated_keys $key"
52+
elif grep -q "^${key}=$" "$working_file"; then
53+
temporary_file=$(mktemp "$directory/.env.value.XXXXXX")
54+
while IFS= read -r line; do
55+
if [ "$line" = "$key=" ]; then
56+
printf '%s=%s\n' "$key" "$value"
57+
else
58+
printf '%s\n' "$line"
59+
fi
60+
done < "$working_file" > "$temporary_file"
61+
chmod 600 "$temporary_file"
62+
mv "$temporary_file" "$working_file"
63+
generated_keys="$generated_keys $key"
64+
fi
65+
}
66+
67+
set_if_empty POSTGRES_PASSWORD hex16
68+
set_if_empty HEPHAESTUS_SECURITY_ENCRYPTION_KEY hex16
69+
set_if_empty HEPHAESTUS_AUTH_STATE_COOKIE_KEY base64
70+
set_if_empty WEBHOOK_SECRET hex32
71+
72+
mv "$working_file" "$environment_file"
73+
74+
for key in $generated_keys; do
75+
printf 'Generated %s.\n' "$key"
76+
done
77+
78+
printf '\nConfiguration written to %s. Generated values were not printed.\n' "$environment_file"
79+
printf '%s\n' 'Set APP_HOSTNAME, ACME_EMAIL, one OAuth provider, and HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS before starting Hephaestus.'

‎docs/admin/configuration-readiness.mdx‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
---
2+
title: Configuration readiness
3+
---
4+
5+
# Configuration readiness
6+
7+
The production profile validates deployment settings during startup. The server-role endpoint
8+
`GET /api/admin/configuration-readiness` returns those facts plus checks that need runtime state. It
9+
requires the `app_admin` authority and is available only when boot-fatal checks pass. Diagnostics
10+
never contain configured values.
11+
12+
Each fact has a stable `id`, the affected configuration `subject`, applicable runtime `roles`, a
13+
`requirement`, a `status`, an explanation, and a documentation link. Requirements are `REQUIRED`,
14+
`RECOMMENDED`, or `OPTIONAL`. Status is one of:
15+
16+
| Status | Meaning |
17+
| --- | --- |
18+
| `SATISFIED` | The applicable check passed. |
19+
| `ACTION_REQUIRED` | An applicable check failed. During startup, required deployment facts with this status prevent startup. |
20+
| `NOT_CONFIGURED` | An optional setting is absent. |
21+
| `NOT_APPLICABLE` | The check does not apply to this process's runtime roles. |
22+
23+
The table names variables inside the application container. In the supported self-host stack,
24+
Compose maps `POSTGRES_PASSWORD` to `DATABASE_PASSWORD` and owns the runtime-role values.
25+
26+
| Setting | Application environment variable |
27+
| --- | --- |
28+
| `spring.datasource.url` | `DATABASE_URL` |
29+
| `spring.datasource.username` | `DATABASE_USERNAME` |
30+
| `spring.datasource.password` | `DATABASE_PASSWORD` |
31+
| `hephaestus.runtime.server.enabled` | `HEPHAESTUS_RUNTIME_SERVER_ENABLED` |
32+
| `hephaestus.runtime.worker.enabled` | `HEPHAESTUS_RUNTIME_WORKER_ENABLED` |
33+
| `hephaestus.runtime.webhook.enabled` | `HEPHAESTUS_RUNTIME_WEBHOOK_ENABLED` |
34+
| `hephaestus.host-url` | `APPLICATION_HOST_URL` |
35+
| `hephaestus.security.encryption-key` | `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` |
36+
| `hephaestus.webhook.secret` | `WEBHOOK_SECRET` |
37+
| `hephaestus.sync.nats.enabled` | `NATS_ENABLED` |
38+
| `hephaestus.sync.nats.server` | `NATS_SERVER` |
39+
| `hephaestus.auth.state-cookie-key` | `HEPHAESTUS_AUTH_STATE_COOKIE_KEY` |
40+
| `hephaestus.llm.egress.allow-loopback` | `HEPHAESTUS_LLM_EGRESS_ALLOW_LOOPBACK` |
41+
| `hephaestus.agent.image.require-digest` | `HEPHAESTUS_AGENT_IMAGE_REQUIRE_DIGEST` |
42+
| `hephaestus.agent.image.reference` | `HEPHAESTUS_AGENT_IMAGE_REFERENCE` |
43+
| `hephaestus.sandbox.container-runtime` | `SANDBOX_CONTAINER_RUNTIME` |
44+
| `hephaestus.sentry.dsn` | `SENTRY_DSN` |
45+
46+
## Runtime roles
47+
48+
Enable at least one of `hephaestus.runtime.server.enabled`, `worker.enabled`, or `webhook.enabled`;
49+
each accepts only `true` or `false`.
50+
The supported split topology enables only webhook on the webhook process, only worker on a remote
51+
worker, and server (optionally with a colocated worker) on the application process.
52+
53+
## Database
54+
55+
Every role uses PostgreSQL. `DATABASE_URL` must be a PostgreSQL URL; the production profile adds the
56+
`jdbc:` prefix. Supply a non-empty username and password. This syntax check does not replace the
57+
connection and migration health checks performed by Spring Boot and Liquibase.
58+
59+
## Credential encryption
60+
61+
Set `hephaestus.security.encryption-key` to exactly 32 printable, non-space ASCII characters and keep it
62+
with the database backup. The supported self-host setup generates it. Do not change it on an existing
63+
installation.
64+
65+
## External URL
66+
67+
Set `hephaestus.host-url` to the public HTTPS origin, without credentials, a path other than `/`, a
68+
query, or a fragment.
69+
70+
## Webhooks
71+
72+
Server and webhook roles require `hephaestus.webhook.secret` with at least 32 printable, non-space ASCII
73+
characters. The supported self-host setup generates an independent value; never reuse another
74+
application key.
75+
76+
## NATS
77+
78+
Server and webhook roles require NATS and an explicit `nats://` or `tls://` URI with a host, an optional
79+
valid port, and no query, fragment, or non-root path. A worker-only process must disable NATS because
80+
its job queue is PostgreSQL-backed. This check validates syntax and role consistency, not
81+
authentication, connectivity, or JetStream health.
82+
83+
## Login
84+
85+
The server role requires a Base64-encoded 32-byte `hephaestus.auth.state-cookie-key` and an enabled
86+
GitHub or GitLab sign-in provider in the database-backed provider catalogue. Environment provider
87+
entries are seeds, not the readiness authority. Slack and Outline are link-only providers and do not
88+
satisfy sign-in readiness. Worker and webhook roles do not load login providers.
89+
90+
## LLM proxy
91+
92+
Worker roles must leave `hephaestus.llm.egress.allow-loopback=false`. Provider credentials and model
93+
configuration are database-backed runtime configuration and are not deployment settings.
94+
95+
## Agent image
96+
97+
Worker roles require digest enforcement and a SHA-256-pinned `hephaestus.agent.image.reference`. See
98+
[Agent image digests](./agent-image-digests.md).
99+
100+
## Sandbox isolation
101+
102+
Set `SANDBOX_CONTAINER_RUNTIME=runsc` on workers after
103+
[installing and configuring gVisor](https://gvisor.dev/docs/user_guide/install/) on the host. This
104+
recommendation is non-fatal.
105+
106+
## Optional observability
107+
108+
Sentry is optional. When configured, `hephaestus.sentry.dsn` must use HTTPS. The fact is classified
109+
`OPTIONAL` and never prevents startup.

‎docs/admin/install.mdx‎

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,8 @@ The stack you get:
3232
- **AI practice review adds real memory**: each concurrent review sandbox may use up to
3333
4 GiB. The default caps it at 1 concurrent sandbox; raise
3434
`SANDBOX_MAX_CONCURRENT` only with RAM to match.
35-
- 64-bit Linux with **Docker Engine ≥ 24** and **Docker Compose ≥ 2.24.4** (`docker compose version`), plus `git`.
35+
- 64-bit Linux with **Docker Engine ≥ 24** and **Docker Compose ≥ 2.24.4** (`docker compose version`),
36+
plus `git` and `openssl`.
3637
- A **DNS A record** for your hostname pointing at the host, with ports **80 and 443**
3738
reachable from the internet (Let's Encrypt HTTP-01, OAuth callbacks, webhooks).
3839
- Outbound HTTPS to `ghcr.io` and `docker.io` (images — NATS and Traefik come from Docker Hub),
@@ -52,28 +53,28 @@ sudo git clone --depth 1 --branch "v$VERSION" \
5253
https://github.com/ls1intum/Hephaestus.git /opt/hephaestus
5354
sudo chown -R "$USER" /opt/hephaestus
5455
cd /opt/hephaestus/docker/self-host
55-
cp .env.example .env
56+
./setup.sh
5657
```
5758

58-
Everything below runs from `/opt/hephaestus/docker/self-host`. Once step 2 has filled in every
59-
required value, `docker compose config` renders the effective configuration — it is the quickest way
60-
to confirm a variable landed where you expected. Before then it exits with the first
61-
`required variable … is missing a value`, which is also how every other `docker compose` subcommand
62-
behaves in this directory; see [Troubleshooting](#troubleshooting).
59+
`setup.sh` creates or updates `.env` with mode `0600` and generates the database password,
60+
credential-encryption key, OAuth state-cookie key, and webhook secret. It does not replace non-empty
61+
values and never prints a secret.
62+
63+
Everything below runs from `/opt/hephaestus/docker/self-host`. After step 2 supplies the remaining
64+
required values, `docker compose config` renders the effective configuration. Before then it exits
65+
at the first missing required variable, as does every other `docker compose` subcommand in this
66+
directory; see [Troubleshooting](#troubleshooting).
6367

6468
## 2. Configure `.env`
6569

66-
Open `.env` and fill in every **REQUIRED** value. The short version:
70+
Open `.env` and fill in the remaining **REQUIRED** values. `setup.sh` manages the four internal
71+
secrets listed above; do not replace them manually.
6772

6873
| Variable | What / how |
6974
| --- | --- |
7075
| `APP_HOSTNAME` | Your public hostname, e.g. `hephaestus.example.com` |
7176
| `IMAGE_TAG` | The same `$VERSION` you fetched above — an exact release, never `latest` |
7277
| `ACME_EMAIL` | Let's Encrypt expiry notices |
73-
| `POSTGRES_PASSWORD` | `openssl rand -hex 16` |
74-
| `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` | `openssl rand -base64 24 \| cut -c1-32` — exactly 32 chars (see the warning below) |
75-
| `HEPHAESTUS_AUTH_STATE_COOKIE_KEY` | `openssl rand -base64 32` |
76-
| `WEBHOOK_SECRET` | `openssl rand -hex 32` — you'll enter the same value on GitHub later |
7778
| `GH_OAUTH_CLIENT_ID` / `_SECRET` | From step 3 |
7879
| `HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS` | From step 4 — set it **before** first boot |
7980

@@ -83,6 +84,11 @@ signing key. If you lose or change it, those tokens become unreadable and all se
8384
invalidate. Store it as carefully as the database itself, and never change it after first boot.
8485
:::
8586

87+
The other generated values also have lifecycle consequences: changing the database password does not
88+
update an initialized PostgreSQL volume, rotating the state-cookie key invalidates OAuth flows already
89+
in progress, and rotating the webhook secret requires the same change at every configured provider.
90+
Back up `.env` with the database as described in [Backup & Restore](./backup-restore).
91+
8692
## 3. Create the GitHub OAuth App (login — mandatory)
8793

8894
Without a login provider the instance boots but shows **no sign-in button**. GitHub login
@@ -203,7 +209,7 @@ a backup instead. Set up [Backup & Restore](./backup-restore) **before** you hav
203209
| Symptom | Cause / fix |
204210
| --- | --- |
205211
| `application-server` exits: *ProxyTrustGuard* | `HEPHAESTUS_TRUSTED_PROXIES` is blank. The compose default is non-blank, so this means you overrode it to empty — unset your override, or set it to match `reverse-proxy`'s IP if you changed the network. |
206-
| Exits: encryption-key length error | `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` must be exactly 32 characters — regenerate with the command above. |
212+
| Exits: encryption-key length error | `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` must be exactly 32 printable, non-space ASCII characters. Restore the original key for an existing database. On a new installation, clear the invalid value and rerun `./setup.sh`. |
207213
| Login page has no sign-in button | No login provider configured: both `GH_OAUTH_CLIENT_ID` **and** `_SECRET` must be non-empty (a half-filled pair is skipped and logged at ERROR). |
208214
| OAuth redirect loop / cookie never sticks | A proxy in front of Traefik is injecting a `Domain=` attribute on cookies, which browsers reject for `__Host-` cookies. Serve the stack directly on ports 80/443. |
209215
| Signed in but not admin | `HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS` didn't match. On the account's **first** login the server logs (INFO) the exact `provider subject username` it saw — copy that into the allowlist and restart. (For an account that already existed, check that log line from its first sign-in, or use the numeric id.) |

‎docs/admin/production-setup.mdx‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -34,11 +34,13 @@ consists of:
3434

3535
## Base environment variables
3636

37-
The required secrets — `HEPHAESTUS_SECURITY_ENCRYPTION_KEY`, `HEPHAESTUS_AUTH_STATE_COOKIE_KEY`,
38-
`WEBHOOK_SECRET`, `POSTGRES_PASSWORD`, the GitHub OAuth pair, and
39-
`HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS` — are documented once, on the
40-
[Install guide](./install#2-configure-env), along with how to generate each and the warning about
41-
never changing the encryption key. Everything there applies to this stack too.
37+
The required base values — `HEPHAESTUS_SECURITY_ENCRYPTION_KEY`,
38+
`HEPHAESTUS_AUTH_STATE_COOKIE_KEY`, `WEBHOOK_SECRET`, `POSTGRES_PASSWORD`, the GitHub OAuth pair, and
39+
`HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS` — are documented once in the
40+
[Install guide](./install#2-configure-env), along with their lifecycle constraints. The self-host
41+
installer generates its internal secrets; operators of the reference deployment must provision
42+
equivalent values through its protected secret-management system. The warning about never changing
43+
the encryption key applies to both stacks.
4244

4345
These variables exist only in the reference deployment, so they live here:
4446

‎docs/sidebars.admin.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ const sidebars: SidebarsConfig = {
2020
{ type: "doc", id: "production-setup", label: "Integrations & Reference Deployment" },
2121
{ type: "doc", id: "compatibility-policy", label: "Compatibility Policy" },
2222
{ type: "doc", id: "runtime-roles", label: "Runtime Roles" },
23+
{ type: "doc", id: "configuration-readiness", label: "Configuration Readiness" },
2324
{ type: "doc", id: "agent-image-digests", label: "Agent image digests" },
2425
{ type: "doc", id: "buildpacks-cds-decision", label: "Server image build (Buildpacks + CDS)" },
2526
{ type: "doc", id: "legal-pages", label: "Legal Pages" },

0 commit comments

Comments
 (0)