Merge pull request #43 from yangchuansheng/codex/llmgateway-runtime-contract

fix(sealos): harden multi-service web runtime smoke

Author: yangchuansheng

skills/docker-to-sealos/SKILL.md

@@ -32,8 +32,9 @@ Extract from Docker Compose/docs:
 - ports, dependencies, service communication
 - env vars and secret usage
 - startup-time validation rules for bootstrap credentials, API keys, salts, secrets, and feature flags
+- multi-service web roles: browser entry, REST API, OpenAI/API gateway, docs, workers, and one-shot jobs
 - resource limits/requests and health checks
-- if official Kubernetes installation docs/manifests are available, also extract app-runtime behavior from them (bootstrap admin fields, external endpoint/protocol assumptions, health probes, startup/init flow)
+- if official Kubernetes installation docs/manifests are available, also extract app-runtime behavior from them (bootstrap admin fields, external endpoint/protocol assumptions, health probes, startup/init flow, migration ordering)
 
 ### Step 2: Infer metadata
 
@@ -65,6 +66,8 @@ Apply field-level mappings from `references/conversion-mappings.md`, including:
 - for DB URL/DSN envs (for example `*_DATABASE_URL`, `*_DB_URL`), when Kubeblocks `endpoint` is host:port, inject `host`/`port`/`username`/`password` via approved `secretKeyRef` envs and compose the final URL with `$(VAR)` expansion
 - edge gateway normalization: when Compose includes Traefik-like edge proxy plus business services, skip the proxy workload and expose business services via Sealos Ingress directly
 - TLS offload normalization for Sealos Ingress: when a business service exposes both 80 and 443, drop 443 from workload/service ports and remove in-container TLS certificate mounts (for example `/etc/nginx/ssl`, `/etc/ssl`, `/certs`) unless official Kubernetes docs explicitly require HTTPS backend-to-service traffic
+- multi-service web normalization: expose the verified browser entry in the App resource, expose API/gateway/docs only when they are intended public surfaces, and keep workers private with no Service/Ingress
+- URL topology: browser-facing env vars must use public HTTPS URLs, while server-to-server env vars must use Kubernetes Service FQDNs unless the app explicitly requires public callbacks
 - prefer `scripts/compose_to_template.py --kompose-mode always` as deterministic conversion entrypoint (require `kompose` for reproducible workload shaping)
 - when official Kubernetes installation docs/manifests exist, perform a dual-source merge: use Compose as baseline topology, then align app-runtime semantics with official Kubernetes guidance
 
@@ -77,6 +80,8 @@ Apply field-level mappings from `references/conversion-mappings.md`, including:
 - For PostgreSQL custom databases (non-`postgres`), the init Job must wait for PostgreSQL readiness before execution and create the target database idempotently.
 - Critical application compatibility objects must be verified in live database state. Use idempotent initContainer self-healing for compatibility views, legacy tables/views, indexes, extensions, search paths, and bootstrap state that the app requires on every cold start.
 - One-shot init Jobs may create initial databases or seed state, but app startup gates must verify the final database objects directly. Treat TTL-expired Jobs as historical evidence and rely on database state for acceptance.
+- Worker, gateway, and background services that depend on app migrations must wait for the required tables, migration markers, or app-specific readiness objects, not only for the database port.
+- Redis readiness probes or initContainers must tolerate authenticated Redis responses such as `NOAUTH` or `Authentication required` when credentials are not needed for readiness.
 - PostgreSQL bootstrap shell must use safe quoting patterns. Prefer shell-level existence checks plus simple SQL statements when possible. Use single-quoted heredocs or SQL files for psql variable interpolation, and avoid PL/pgSQL `DO $$` blocks in inline shell commands when a guard query can express the same logic.
 - Do not use `psql -c "..."` for `:'var'` variable interpolation. Use `psql -v name=value <<'SQL' ... :'name' ... SQL` or pass already-safe literal SQL.
 
@@ -121,6 +126,7 @@ If validation fails, fix template/rules/examples first.
 - App resource `spec.displayType` must be `normal`.
 - App resource `spec.type` must be `link`.
 - App resource `spec.data.url` must be the browser entry URL that succeeds from a fresh Sealos launch. For apps with safe-path, setup-path, or entrance-path behavior, verify the configured path and root path, then choose the URL that supports login or first-run setup without hidden prior navigation.
+- SSR/Next.js/React server apps must not use a path that renders a server-side exception as the App URL or HTTP probe. Treat visible `Application error`, `server-side exception`, `Internal Server Error`, or `Unhandled Runtime Error` text as a failed entry path even if the HTTP status is 2xx/3xx.
 - Never use `spec.template` in App resource.
 - `cloud.sealos.io/app-deploy-manager` label value must equal resource `metadata.name`.
 - `metadata.labels.app` label value must equal resource `metadata.name` for managed app workloads.
@@ -170,6 +176,7 @@ If validation fails, fix template/rules/examples first.
 - Database connection/bootstrap may use Kubeblocks-provided secrets, and reserved Kubeblocks database secret names must not be redefined by custom `Secret` resources.
 - Env vars must be declared before referenced (for example password before URL composition).
 - Follow official app env var naming; do not invent prefixes.
+- For split frontend/API/gateway apps, keep public browser URLs and internal service URLs separate. Frontend/browser callback variables use `https://${{ defaults.<host> }}.${{ SEALOS_CLOUD_DOMAIN }}`; backend-to-backend variables use `http://<service>.$(SEALOS_NAMESPACE).svc.cluster.local:<port>` or the fully rendered Service FQDN.
 - When the application requires its public URL configured via a file-based config system (e.g., node-config `config/default.json`, PHP config files), create a ConfigMap containing the config file with the public URL set to `https://${{ defaults.app_host }}.${{ SEALOS_CLOUD_DOMAIN }}`, and mount it to the application's config directory. The ConfigMap must follow standard naming and label conventions.
 - For PostgreSQL custom databases (non-`postgres`), include `${{ defaults.app_name }}-pg-init` Job and implement startup-safe/idempotent creation logic (readiness wait + existence check before create).
 - For application-specific database compatibility, include an initContainer or startup gate that idempotently creates or repairs required views, aliases, indexes, extensions, privileges, role search paths, and legacy compatibility objects before the business container starts.

skills/docker-to-sealos/references/must-rules-map.yaml

@@ -60,6 +60,10 @@ must_rules:
     enforcement:
       type: manual
       note: "Requires live Sealos App URL smoke from a fresh browser/session."
+  - must: "SSR/Next.js/React server apps must not use a path that renders a server-side exception as the App URL or HTTP probe. Treat visible `Application error`, `server-side exception`, `Internal Server Error`, or `Unhandled Runtime Error` text as a failed entry path even if the HTTP status is 2xx/3xx."
+    enforcement:
+      type: manual
+      note: "Requires HTTP body/runtime smoke inspection, covered by sealos-live-smoke failure signals."
   - must: "Never use `spec.template` in App resource."
     enforcement:
       type: rule
@@ -208,6 +212,10 @@ must_rules:
     enforcement:
       type: manual
       note: "Requires per-application canonical env dictionary from upstream docs."
+  - must: "For split frontend/API/gateway apps, keep public browser URLs and internal service URLs separate. Frontend/browser callback variables use `https://${{ defaults.<host> }}.${{ SEALOS_CLOUD_DOMAIN }}`; backend-to-backend variables use `http://<service>.$(SEALOS_NAMESPACE).svc.cluster.local:<port>` or the fully rendered Service FQDN."
+    enforcement:
+      type: manual
+      note: "Requires app-specific URL topology review across frontend, API, gateway, and worker env vars."
   - must: "When the application requires its public URL configured via a file-based config system (e.g., node-config `config/default.json`, PHP config files), create a ConfigMap containing the config file with the public URL set to `https://${{ defaults.app_host }}.${{ SEALOS_CLOUD_DOMAIN }}`, and mount it to the application's config directory. The ConfigMap must follow standard naming and label conventions."
     enforcement:
       type: manual

skills/docker-to-sealos/references/sealos-specs.md

@@ -150,6 +150,38 @@ spec:
 
 **Do NOT add any other fields.** Fields like `menuData`, `nameColor`, `template`, etc. do not exist in the App CRD and will cause `strict decoding error: unknown field` on apply.
 
+### Multi-Service Web Runtime Contract
+
+For split-service web apps, classify each component before writing resources:
+
+- **Browser entry**: dashboard, console, setup UI, or docs page opened by users.
+- **REST API**: authenticated application or dashboard API.
+- **Protocol gateway**: OpenAI-compatible, webhook, or SDK-facing API surface.
+- **Docs/static service**: optional public documentation.
+- **Worker**: background processor with no public Service/Ingress unless upstream explicitly exposes one.
+
+The App CRD `spec.data.url` must point to the browser entry URL that works from a fresh Sealos launch. Test the root path and any login/setup/entrance path from upstream docs or source. Choose the path that loads without prior navigation and reaches login, registration, or setup.
+
+For SSR/Next.js/React server apps, HTTP status is not enough. Treat the entry as failed when the response body contains visible runtime failure text such as:
+
+- `Application error`
+- `server-side exception`
+- `Internal Server Error`
+- `Unhandled Runtime Error`
+- Next.js runtime digest markers such as `NEXT_`
+
+These failures invalidate the App URL and any HTTP probe path even when the HTTP status is 2xx or 3xx.
+
+Use `nginx.ingress.kubernetes.io/app-root` only after the target path has been verified as a stable fresh-session entry. It must not hide a broken root path, missing public URL configuration, or wrong backend service routing.
+
+Public and internal URLs must stay separate:
+
+- Browser-facing env vars use public HTTPS URLs: `https://${{ defaults.<host> }}.${{ SEALOS_CLOUD_DOMAIN }}`.
+- Backend-to-backend env vars use Kubernetes Service DNS names, for example `http://<service>.${{ SEALOS_NAMESPACE }}.svc.cluster.local:80`.
+- Use public URLs for server-side callbacks only when upstream explicitly requires external callback or OAuth-style redirect behavior.
+
+When workers, protocol gateways, or background services depend on database migrations, gate startup on app-specific database state such as migration markers or required tables, not merely on the database port.
+
 ## Defaults and Inputs Configuration Specification
 
 ### Basic Principles

skills/sealos-deploy/modules/pipeline.md

@@ -1083,6 +1083,7 @@ Acceptance checklist:
 - Service endpoints are populated.
 - The actual App URL loads from a fresh session.
 - Login-gated apps complete setup/login and one authenticated action.
+- SSR/browser failure text such as `Application error`, `server-side exception`, `Internal Server Error`, and `Unhandled Runtime Error` is absent from smoke responses.
 - Recent logs are clear of recurring startup, migration, bootstrap, and access-control failures.
 - Main business containers keep `command`/`args` short and close to the official entrypoint; repeated file preparation, permission repair, database bootstrap, or compatibility self-healing belongs in initContainers, Jobs, or ConfigMap scripts.
 - Shell wrappers in main containers end with `exec <final-process>` so signal handling remains correct.

skills/sealos-deploy/references/live-smoke-playbooks.md

@@ -16,7 +16,8 @@ Run this pass after Phase 6 for every deployment unless the user explicitly asks
 3. Visit the actual App URL exactly as Sealos launches it. Test the root path and the configured App URL path when the app uses an entrance or safe-path mechanism.
 4. For login-gated web apps, complete registration or login, confirm a token/session, and open at least one authenticated page or API route.
 5. Scan recent logs after login. Treat recurring application errors as deployment failures even when all Pods are Running.
-6. Inventory the full footprint before cleanup or handoff:
+6. Treat visible SSR/browser failure text such as `Application error`, `server-side exception`, `Internal Server Error`, and `Unhandled Runtime Error` as failed smoke even when HTTP returns 2xx/3xx.
+7. Inventory the full footprint before cleanup or handoff:
    - `instances.app.sealos.io`
    - `apps.app.sealos.io`
    - Deployments/StatefulSets/CronJobs/Jobs
@@ -68,6 +69,30 @@ Bootstrap quoting guidance:
 - Use `psql -tAc "SELECT ..."` plus guarded `psql -c` or single-quoted heredocs for idempotent object creation.
 - Use `psql -v name=value` variable interpolation inside heredocs for sensitive SQL values such as passwords. Do not rely on `psql -c "ALTER ROLE ... :'var'"`; psql colon variables are not expanded in that form.
 
+## LLM Gateway / Multi-Service SSR Web App
+
+Use this playbook for split-service apps with a dashboard, REST API, protocol gateway, docs service, and workers.
+
+Component checks:
+
+- Dashboard/browser entry: visit both root and the App resource URL path from a fresh session.
+- API service: check readiness and recent logs before and after login or signup.
+- Gateway/API protocol service: check readiness and recent logs; do not require provider credentials for basic startup unless upstream requires them.
+- Worker: inspect logs after API migrations complete and after one authenticated dashboard action.
+- Docs/static service: verify it serves a page if exposed publicly.
+
+Runtime acceptance:
+
+- The App URL reaches login, signup, or setup without `Application error: a server-side exception has occurred`.
+- For login-gated apps, complete signup or login, then open at least one authenticated dashboard page.
+- Recent dashboard, API, gateway, and worker logs are clear of recurring SSR, migration, auth/session, and service-to-service URL errors.
+- Gateway or worker pods that depend on database migrations wait for required tables or migration markers, not only PostgreSQL readiness.
+- Public browser URLs and internal service URLs are not mixed: browser-facing config uses public HTTPS hosts, while backend-to-backend config uses Kubernetes Service DNS.
+
+Debug loop:
+
+If the App URL shows `Application error: a server-side exception has occurred`, read dashboard/API logs first, then verify App URL path, public URL env vars, API backend URL, migration completion gates, and Redis/PostgreSQL readiness behavior before reporting success.
+
 ## Generic Login-Gated Web App
 
 Minimum smoke:

skills/sealos-deploy/scripts/sealos-live-smoke.mjs

@@ -39,7 +39,10 @@ function fail(message, extra = {}) {
 }
 
 function joinUrl(base, path) {
-  return new URL(path || "/", base).toString();
+  if (path === undefined || path === null || path === "") {
+    return new URL(base).toString();
+  }
+  return new URL(path, base).toString();
 }
 
 function redact(value) {
@@ -74,6 +77,33 @@ function redactJson(value, key = "") {
   return Object.fromEntries(Object.entries(value).map(([entryKey, entry]) => [entryKey, redactJson(entry, entryKey)]));
 }
 
+const FAILURE_SIGNAL_PATTERNS = [
+  {
+    id: "application-error",
+    pattern: /Application error/i,
+  },
+  {
+    id: "server-side-exception",
+    pattern: /server-side exception/i,
+  },
+  {
+    id: "internal-server-error",
+    pattern: /Internal Server Error/i,
+  },
+  {
+    id: "unhandled-runtime-error",
+    pattern: /Unhandled Runtime Error/i,
+  },
+  {
+    id: "next-runtime-digest",
+    pattern: /\bNEXT_[A-Z0-9_]+\b/i,
+  },
+];
+
+function detectFailureSignals(text) {
+  return FAILURE_SIGNAL_PATTERNS.filter(({ pattern }) => pattern.test(text)).map(({ id }) => id);
+}
+
 async function requestStep(name, url, options = {}) {
   const startedAt = Date.now();
   const response = await fetch(url, {
@@ -95,16 +125,18 @@ async function requestStep(name, url, options = {}) {
     }
   }
   const safeJson = redactJson(json);
+  const failureSignals = detectFailureSignals(text);
   return {
     name,
     url,
     status: response.status,
-    ok: response.status >= 200 && response.status < 400,
+    ok: response.status >= 200 && response.status < 400 && failureSignals.length === 0,
     elapsedMs: Date.now() - startedAt,
     contentType,
     location: response.headers.get("location"),
     setCookie: response.headers.get("set-cookie") ? "<present>" : null,
     bodyPreview: safeJson ? JSON.stringify(safeJson).slice(0, 240) : text.slice(0, 240),
+    failureSignals,
     json: safeJson,
     rawJson: json,
   };
@@ -155,7 +187,7 @@ if (!baseUrl) {
 
 const steps = [];
 try {
-  steps.push(await requestStep("root", joinUrl(baseUrl, args.rootPath || "/")));
+  steps.push(await requestStep("root", joinUrl(baseUrl, args.rootPath)));
 
   if (args.captchaPath) {
     steps.push(await requestStep("captcha", joinUrl(baseUrl, args.captchaPath)));