Skip to content

fix(ci): repair the Coolify preview deployment and make previews opt-in - #1348

Merged
FelixTJDietrich merged 1 commit into
mainfrom
fix/deploy-preview-topology
Jul 13, 2026
Merged

fix(ci): repair the Coolify preview deployment and make previews opt-in#1348
FelixTJDietrich merged 1 commit into
mainfrom
fix/deploy-preview-topology

Conversation

@FelixTJDietrich

@FelixTJDietrich FelixTJDietrich commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Description

The Coolify preview deployment has been broken for months. docker/preview/* was never updated for the native-auth cutover (#1317) or the Java webhook receiver (#1300, #1306), and had drifted far enough that no stack could boot — the app resource sat at exited:unhealthy while shared infra still ran six-month-old Keycloak, webhook-ingest and postfix containers.

This restores it end to end, and makes previews opt-in rather than automatic. Every item below is one independently fatal bug, found by deploying to hephaestus-test.aet.cit.tum.de and reading the failures.

Why nothing could boot

  • ${VAR:?Required} became the literal string Required. Coolify seeds its own env store from these placeholders and keeps the error message as the value. The stack therefore ran with HEPHAESTUS_AUTH_STATE_COOKIE_KEY=Required, which base64-decodes to 6 bytes; AuthSecurityConfig requires 32 and aborts the boot. All :? defaults are gone.
  • HEPHAESTUS_SECURITY_ENCRYPTION_KEY was never set. JwtSigningKeySealer fail-fasts without it in prod.
  • webhook-server had no datasource. application-prod.yml binds spring.datasource.url to jdbc:${DATABASE_URL} with no default and application-webhook.yml keeps JPA active, so an unset value failed placeholder resolution before the context started.
  • The server image was built from a Dockerfile that does not exist. build: context: ./server has never been valid — the image comes from Paketo buildpacks (spring-boot:build-image), which is also what supplies the /workspace/health-check binary the healthchecks invoke. Every build died with failed to read dockerfile. It is now pulled from GHCR.

Why login could not have worked

application-prod.yml pins hephaestus.auth.api-base-path=/api, which is correct for production, where Traefik strips that prefix. Nothing strips it here, so every OAuth callback was built as https://<api-host>/api/login/oauth2/code/<provider> and 404'd. The prefix is now cleared for this stack.

Separately, the trusted-proxies regex reached the container double-escaped (Coolify re-escapes backslashes when it writes the stack .env), so \. arrived as \\. — which as a Java regex matches a literal backslash and therefore never matches an IP. Tomcat silently distrusted Traefik, ignored X-Forwarded-Proto, and built every redirect_uri as http://, which the IdP rejects. The value must be written without backslashes, and there is an inline comment saying so.

Preview isolation

  • hostname: postgres let previews reach each other's databases. Coolify attaches every stack to the shared coolify network, where a fixed hostname made every preview's Postgres answer to the same postgres name. Services are now addressed through the per-deploy SERVICE_NAME_* aliases.
  • The NATS durable consumer name was shared, so the base deploy and every preview competed for the same JetStream consumer. It is now per-deploy.
  • sync.nats.server and agent.nats.server both default to ${NATS_SERVER}. They are now pinned separately, so pointing the integration consumer at another environment's NATS cannot also submit this stack's agent jobs into that environment's AGENT queue for its workers to execute.

Volume ownership

Named volumes are created root:root but the buildpack image runs as cnb (1002:1001), so GitRepositoryManager could not mkdir under /data/git-repos (every checkout failed) and Tomcat's AccessLogValve could not open /var/log/hephaestus/access — which was never mounted here at all. A volume-init step now hands both mounts to the runtime user. It has to live in the compose file: previews get fresh root-owned volumes on every deploy, so a one-off chown would silently regress.

Routing

Coolify generates the Traefik routers itself from the per-service domains, so the stack is served as:

SPA https://<domain>
API https://api.<domain>
Webhooks https://webhooks.<domain>/webhooks/{github,gitlab,slack}

Two constraints are worth recording, because both silently produce a working-looking but wrong deployment:

  1. Hand-written Traefik labels do not work. Coolify does not interpolate ${VARS} inside labels:, so the rule reaches Traefik with a literal ${APP_HOSTNAME} host and never matches.
  2. The receiver needs a path-less host. Coolify appends a strip-prefix middleware whenever a service domain carries a path, and that is a per-application UI toggle. The receiver owns the full /webhooks/{kind} path, so a stripped /webhooks 404s every delivery; a path-less domain is never stripped.

The API service is named appserver, not application-server, and the name is load-bearing — see the inline comment.

Preview lifecycle: opt in, then stay current

Coolify's preview switch is all-or-nothing: enabled, it deploys every opened PR. We don't want a stack per PR, but we do want the previews that exist to stay current. So the switch is now off, a preview is created from the Coolify UI when a PR actually needs one, and the Preview / Coolify job refreshes it on each push. Coolify's deploy API only accepts a pull request that already has a preview, so a PR without one is a no-op:

notice: PR #1348 has no preview deployment (Pull request 1348 not found for this resource.)

The job is configured entirely through repository variables (COOLIFY_URL, COOLIFY_APP_UUID, COOLIFY_PROJECT_UUID, COOLIFY_ENVIRONMENT_UUID) plus a COOLIFY_API_TOKEN secret scoped to Coolify's deploy ability. No instance identifiers are hardcoded, so a fork — or anyone running their own Coolify — skips the job cleanly instead of posting links into someone else's instance. Fork PRs are skipped by design: a preview runs with the instance's real credentials.

How to test

Deployed and verified on hephaestus-test.aet.cit.tum.de (Coolify): 5/5 containers healthy, 0 ERROR lines in the app log.

SPA       https://hephaestus.felixdietrich.com               200
API       https://api.hephaestus.felixdietrich.com           200 (readiness UP)
webhooks  POST /webhooks/github  no signature   → 401
                                 valid HMAC     → 202
                                 bad signature  → 401
OAuth     redirect_uri = https://api.hephaestus.felixdietrich.com/login/oauth2/code/github

GitHub and GitLab (gitlab.lrz.de, scope read_user) both resolve to the correct authorize URL, and Liquibase migrates cleanly against the existing database. The Preview / Coolify job on this PR exercises the no-preview path shown above and passes.

Notes for reviewers

  • Title scope. This is fix(ci), not fix(docker): nothing in the shipped application changed — only preview deployment config and the preview job — and docker is a release-triggering scope in .releaserc, so it would cut a patch release and publish new production images for an unchanged app. (Separately: the PR template's header comment lists docker under "NO RELEASE" while .releaserc does not. The template is wrong; follow-up.)
  • Not verified end to end: the LLM/mentor path, and Slack. A default AgentConfig is seeded, but exercising the mentor needs a real workspace and PR.
  • Previews pull the server image rather than building it. The webapp still builds from source, so a preview always exercises the PR's frontend; set IMAGE_TAG=pr-<id> to exercise its server too (that tag exists only when the PR touched server/).
  • Unrelated latent bug, not fixed here: changeset 1772284124265-7 does RESTART WITH (MAX(id) + 1), which Postgres rejects when the max id is negative. GitLab-synced milestones carry hashed negative ids, so this fails on any database that has them (it did on the test box). GREATEST(…, 1) would make it safe, but the changeset is very likely already applied on staging and prod, and editing it would break their Liquibase checksums — so it needs its own follow-up.

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@FelixTJDietrich, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 23 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1aeaad2f-08d8-480e-be28-c6447d19472d

📥 Commits

Reviewing files that changed from the base of the PR and between f1def40 and e061f99.

📒 Files selected for processing (3)
  • .github/workflows/cicd.yml
  • docker/preview/compose.app.yaml
  • docker/preview/compose.shared-infra.yaml
📝 Walkthrough

Walkthrough

The preview Compose stack adopts Coolify service naming, isolates database and runtime volumes per deployment, initializes volume permissions, updates appserver/webapp dependencies, and revises shared webhook-server environment handling and documentation.

Changes

Coolify preview deployment

Layer / File(s) Summary
Preview service naming and database isolation
docker/preview/compose.app.yaml
Preview database, seed-loader, appserver, and integration settings use Coolify-generated service names and FQDNs instead of hardcoded or required preview-domain values.
Volume initialization and service startup
docker/preview/compose.app.yaml
A volume-init service prepares Git repository and Tomcat log volumes; appserver and webapp startup dependencies use completion and health conditions.
Shared webhook and infrastructure configuration
docker/preview/compose.shared-infra.yaml
Webhook-server defaults and documentation are updated, healthcheck behavior is documented, and NATS JetStream formatting changes without changing its settings.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

  • ls1intum/Hephaestus#1325: Related webhook path handling and trusted-proxy/external-URL configuration for preview deployments.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title captures the main change: repairing the Coolify preview deployment; the opt-in wording is extra but not misleading.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/deploy-preview-topology

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
docker/preview/compose.app.yaml (1)

359-362: 🔒 Security & Privacy | 🔵 Trivial

Docker-socket RW mount grants host-level control from the preview container.

Mounting /var/run/docker.sock read-write into appserver gives that container the ability to spawn arbitrary privileged containers on the host — effectively root-equivalent access. The comment notes "prod does the same," so this isn't a new pattern introduced here, but previews are the environment most likely to run less-trusted/experimental code (PR branches), so it's worth confirming this tradeoff is intentional for preview too, not just inherited from prod without re-evaluation.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docker/preview/compose.app.yaml` around lines 359 - 362, Review the appserver
volume configuration and explicitly confirm whether the read-write
/var/run/docker.sock mount is intentional for preview deployments. If Docker
host control is not required, remove the mount or replace it with a
least-privilege alternative; otherwise document and preserve the deliberate
security tradeoff rather than inheriting it implicitly from production.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docker/preview/compose.app.yaml`:
- Around line 351-358: Update WEBHOOK_EXTERNAL_URL in the preview compose
configuration to default to the shared-infra webhook-server host rather than the
undefined PREVIEW_DOMAIN-based preview origin, while preserving explicit
WEBHOOK_EXTERNAL_URL overrides.

---

Nitpick comments:
In `@docker/preview/compose.app.yaml`:
- Around line 359-362: Review the appserver volume configuration and explicitly
confirm whether the read-write /var/run/docker.sock mount is intentional for
preview deployments. If Docker host control is not required, remove the mount or
replace it with a least-privilege alternative; otherwise document and preserve
the deliberate security tradeoff rather than inheriting it implicitly from
production.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 27932fb4-91ff-4d51-9f5b-4836c654c4ad

📥 Commits

Reviewing files that changed from the base of the PR and between 9ab3bd6 and f1def40.

📒 Files selected for processing (2)
  • docker/preview/compose.app.yaml
  • docker/preview/compose.shared-infra.yaml

Comment thread docker/preview/compose.app.yaml Outdated
Comment on lines 351 to 358
# Webhook auto-registration. Points at the shared-infra receiver's own host, not this
# deploy's origin: webhook-server lives in the shared-infra stack and is routed there.
# Bare origin — GitLabWebhookService appends /webhooks/gitlab itself.
WEBHOOK_SECRET: ${WEBHOOK_SECRET:-}
WEBHOOK_EXTERNAL_URL: https://${PREVIEW_DOMAIN:?Required}
WEBHOOK_EXTERNAL_URL: ${WEBHOOK_EXTERNAL_URL:-https://${PREVIEW_DOMAIN}}
# thc healthcheck target: actuator liveness on the management port.
THC_PORT: "8080"
THC_PATH: /actuator/health/liveness

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map the relevant compose files and search for the variable names involved.
git ls-files 'docker/preview/*.yaml' 'docker/preview/*.yml' | sort
printf '\n--- SEARCH ---\n'
rg -n '\b(PREVIEW_DOMAIN|WEBHOOK_EXTERNAL_URL|SERVICE_FQDN_|SERVICE_NAME_)' docker/preview -S

Repository: ls1intum/Hephaestus

Length of output: 2937


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- PREVIEW_DOMAIN SEARCH ---'
rg -n '\bPREVIEW_DOMAIN\b' docker/preview -S

printf '\n%s\n' '--- SHARED-INFRA COMPOSE OUTLINE ---'
ast-grep outline docker/preview/compose.shared-infra.yaml --view expanded || true

printf '\n%s\n' '--- APP COMPOSE AROUND WEBHOOK LINES ---'
sed -n '330,365p' docker/preview/compose.app.yaml

printf '\n%s\n' '--- SHARED-INFRA AROUND WEBHOOK/FQDN LINES ---'
sed -n '1,220p' docker/preview/compose.shared-infra.yaml

Repository: ls1intum/Hephaestus

Length of output: 10174


🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n '\bWEBHOOK_(EXTERNAL_URL|SERVER|HOST|URL)|webhooks\.' docker/preview README.md -S

Repository: ls1intum/Hephaestus

Length of output: 553


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- PREVIEW_* / DOMAIN SEARCH ---'
rg -n '\bPREVIEW_[A-Z0-9_]+\b|\bBASE_DOMAIN\b|\bWEBHOOK_EXTERNAL_URL\b|\bSERVICE_FQDN_[A-Z0-9_]+\b' . -S

printf '\n%s\n' '--- COMPOSE FILE LOCATIONS ---'
git ls-files 'docker/preview/*.yaml' 'docker/preview/*.yml' 'docker/preview/*.md' | sort

Repository: ls1intum/Hephaestus

Length of output: 3620


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- docker/compose.core.yaml around WEBHOOK_EXTERNAL_URL ---'
sed -n '1,80p' docker/compose.core.yaml

printf '\n%s\n' '--- docker/compose.app.yaml around WEBHOOK_EXTERNAL_URL ---'
sed -n '140,175p' docker/compose.app.yaml

printf '\n%s\n' '--- server application.yml around webhook external-url ---'
sed -n '476,492p' server/src/main/resources/application.yml

printf '\n%s\n' '--- preview compose.app.yaml around APP_FQDN and webhook block ---'
sed -n '88,100p;348,358p' docker/preview/compose.app.yaml

Repository: ls1intum/Hephaestus

Length of output: 10606


WEBHOOK_EXTERNAL_URL falls back to an undefined variable here

PREVIEW_DOMAIN isn’t defined anywhere in the preview compose files, so previews without an explicit override resolve this to an invalid URL and webhook auto-registration fails. Point this at the shared-infra webhook-server host instead of the preview origin.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docker/preview/compose.app.yaml` around lines 351 - 358, Update
WEBHOOK_EXTERNAL_URL in the preview compose configuration to default to the
shared-infra webhook-server host rather than the undefined PREVIEW_DOMAIN-based
preview origin, while preserving explicit WEBHOOK_EXTERNAL_URL overrides.

@FelixTJDietrich
FelixTJDietrich force-pushed the fix/deploy-preview-topology branch 4 times, most recently from b15e392 to 1f74782 Compare July 13, 2026 21:03
@FelixTJDietrich FelixTJDietrich changed the title fix(docker): repair the Coolify preview deployment fix(ci): repair the Coolify preview deployment and make previews opt-in Jul 13, 2026
docker/preview/* was never updated for the native-auth cutover (#1317) or the Java
webhook receiver (#1300, #1306), and had drifted far enough that no stack could boot.

Previews are now opt-in: Coolify's all-or-nothing preview switch stays off, and the
Preview / Coolify job refreshes only the previews that already exist.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@FelixTJDietrich
FelixTJDietrich force-pushed the fix/deploy-preview-topology branch from 1f74782 to e061f99 Compare July 13, 2026 21:08
@FelixTJDietrich
FelixTJDietrich merged commit 2e0e153 into main Jul 13, 2026
42 checks passed
@FelixTJDietrich
FelixTJDietrich deleted the fix/deploy-preview-topology branch July 13, 2026 21:24
@github-actions

Copy link
Copy Markdown
Contributor

📚 Documentation Preview

Preview has been removed (PR closed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant