feat(versioning): Figma-shaped diagram version history on Redis Stack with live collaboration (#657)

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Author: FelixTJDietrich

docker/compose.app.yml

@@ -66,6 +66,7 @@ services:
       - PORT=8000
       - WS_PORT=4444
       - REDIS_URL=redis://db:6379
+      - OWNER_SECRET=${OWNER_SECRET:?OWNER_SECRET must be set}
     networks:
       - apollon-network
     read_only: true

docker/compose.db.yml

@@ -1,14 +1,22 @@
 services:
   db:
-    image: redis:7-alpine
+    # Stack image is required: the server's RedisJSON gate refuses to boot
+    # without the ReJSON module. `redis:7-alpine` ships without it and would
+    # silently break versioning on a fresh deploy.
+    image: redis/redis-stack-server:7.4.0-v8
     restart: unless-stopped
     expose:
       - "6379"
     volumes:
       - dbdata:/data
     networks:
       - apollon-network
-    command: redis-server --appendonly yes --appendfsync everysec
+    command:
+      - redis-stack-server
+      - --appendonly
+      - "yes"
+      - --appendfsync
+      - everysec
     healthcheck:
       test: ["CMD", "redis-cli", "ping"]
       interval: 10s

docker/compose.local.db.yml

@@ -1,10 +1,16 @@
 services:
   db:
-    image: redis:7-alpine
+    image: redis/redis-stack-server:7.4.0-v8
     ports:
       - "${REDIS_PORT:-6379}:6379"
     volumes:
       - dbdata:/data
+    command:
+      - redis-stack-server
+      - --appendonly
+      - "yes"
+      - --appendfsync
+      - everysec
 
 volumes:
   dbdata:

docker/compose.local.yml

@@ -22,18 +22,25 @@ services:
       - PORT=8000
       - WS_PORT=4444
       - REDIS_URL=redis://db:6379
+      - OWNER_SECRET=${OWNER_SECRET:-change-me-in-production}
     depends_on:
       db:
         condition: service_healthy
     networks:
       - apollon-local
 
   db:
-    image: redis:7-alpine
+    image: redis/redis-stack-server:7.4.0-v8
     ports:
       - "6379:6379"
     volumes:
       - dbdata:/data
+    command:
+      - redis-stack-server
+      - --appendonly
+      - "yes"
+      - --appendfsync
+      - everysec
     networks:
       - apollon-local
     healthcheck:

docs/admin/legal-pages.md

@@ -67,7 +67,7 @@ Apollon's privacy notice must cover, at minimum:
 
 1. **Controller** (Art. 4(7) GDPR) — your legal identity and contact.
 2. **DPO** — name or office, per Art. 37 GDPR.
-3. **Processings** — list each, with purpose, legal basis (cite GDPR article + any national law), data categories, retention. For an out-of-the-box Apollon deployment this minimally includes: Redis-stored diagram content (120-day TTL) and WebSocket relay metadata (ephemeral). Operational events (startups, cert renewals, crit-level errors) contain no personal data by design — nginx runs with `access_log off;` and `error_log … crit;`, Traefik with `--accesslog=false`, and Express silent in production. If your fork re-enables per-request access logging at any layer, add that as an additional processing and document its retention.
+3. **Processings** — list each, with purpose, legal basis (cite GDPR article + any national law), data categories, retention. For an out-of-the-box Apollon deployment this minimally includes: Redis-stored diagram content (120-day TTL), Redis-stored version-history snapshots and metadata (same 120-day TTL, scoped per diagram, automatic FIFO eviction at 50 versions), and WebSocket relay metadata (ephemeral). Operational events (startups, cert renewals, crit-level errors) contain no personal data by design — nginx runs with `access_log off;` and `error_log … crit;`, Traefik with `--accesslog=false`, and Express silent in production. If your fork re-enables per-request access logging at any layer, add that as an additional processing and document its retention.
 4. **Recipients** — who else sees the data; "none" is a valid answer.
 5. **Third-country transfers** — state "none" if you host in the EU; otherwise cite adequacy (Art. 45 GDPR) or safeguards (Art. 46 GDPR).
 6. **Retention** — concrete time frames per category.

docs/admin/operations.md

@@ -0,0 +1,114 @@
+# Operations
+
+## Required infrastructure
+
+- **Redis Stack** (`redis/redis-stack-server:7.4.0-v0` is the pinned
+  test image; production `compose.db.yml` uses the same tag) — provides
+  RedisJSON ≥ 2.0, required for HEAD storage. Plain `redis:7-alpine` /
+  `redis:8-alpine` do not bundle the JSON module.
+- **Node.js ≥ 22.14**, **npm ≥ 11.1**.
+
+The server checks `MODULE LIST` at startup and asserts ReJSON is loaded;
+on missing module the boot fails closed (process exits non-zero) rather
+than serving traffic with broken versioning.
+
+## Environment variables
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `HOST` | `localhost` | Express bind host |
+| `PORT` | `8000` | Express HTTP port |
+| `WS_PORT` | `4444` | WebSocket relay port |
+| `CORS_ORIGIN` | unset | Allowed origin (omit to allow all) |
+| `REDIS_URL` | `redis://localhost:6379` | Redis connection |
+| `OWNER_SECRET` | dev placeholder | HMAC secret for soft owner cookie |
+| `MAX_VERSIONS_PER_DIAGRAM` | `50` | FIFO cap on version history |
+| `MAX_SNAPSHOT_BYTES` | `5242880` | Body parser hard limit (5 MiB) |
+| `MAX_DESCRIPTION_LENGTH` | `240` | Version description char limit |
+| `MAX_NAME_LENGTH` | `80` | Version name char limit |
+| `DIAGRAM_TTL_SECONDS` | `10368000` | HEAD TTL (120 d) |
+| `VERSION_TTL_SECONDS` | `10454400` | Version body+meta TTL (121 d) |
+| `AUTO_VERSION_INTERVAL_SECONDS` | `1800` | Marker TTL = auto-version cadence (30 min) |
+
+## Durability
+
+Redis is configured with:
+- `appendonly yes` — AOF on
+- `appendfsync everysec` — at most ~1s of writes can be lost on a hard
+  crash. Applies uniformly to autosave PUTs and snapshot/restore commits;
+  there is no per-route fsync gate.
+
+For a stricter durability story, deploy a Redis replica and set
+`min-replicas-to-write 1` plus `min-replicas-max-lag 10`. The application
+does not call `WAIT` — that command requires configured replicas to
+provide any guarantee, and we deliberately don't claim a guarantee that
+isn't real.
+
+Recommended additional ops:
+- RDB BGSAVE every 6 hours
+- Off-site backup of the `dbdata` volume nightly
+
+## Single-replica WebSocket
+
+The relay server is in-memory (`Map<diagramId, Set<WebSocket>>`). It is
+**not multi-replica safe** — clients on different replicas cannot see each
+other. Production deploys must:
+
+1. Run a single relay replica behind a sticky load balancer, or
+2. Move to a Redis Pub/Sub adapter (deferred; tracked separately).
+
+## Redis Functions lifecycle
+
+The server boot-loads one Lua library named `apollon` containing three
+functions: `commit_snapshot`, `restore_version`, and
+`list_versions_before`. Verify with `redis-cli FUNCTION LIST` (the library
+name `apollon` should appear). Idempotent across restarts; the library
+survives Redis restart because Redis Functions are persisted to AOF/RDB.
+
+## Migration: legacy STRING → RedisJSON
+
+Pre-existing `diagram:{<id>}` keys stored as plain JSON STRINGs (legacy
+format) are upgraded by:
+
+```bash
+npm run migrate:string-to-json --workspace=@tumaet/server
+```
+
+The script:
+- Scans `diagram:*`
+- Skips any key that's already RedisJSON-typed
+- Reads STRING bodies, JSON.SETs them back, preserves TTL
+- Refuses to leave any STRING-typed `diagram:*` HEAD keys behind
+  (boot-gate equivalent)
+
+Idempotent. Safe to re-run.
+
+## Observability
+
+The server emits JSON logs via Pino. Every version event is structured:
+
+```json
+{
+  "level": 30,
+  "event": "version.created",
+  "diagramId": "...",
+  "versionId": "01JKQ...",
+  "kind": "user",
+  "evictedVersionIds": [],
+  "librarySchemaVersion": "4.4.0",
+  "requestId": "...",
+  "msg": "version.created"
+}
+```
+
+`x-request-id` is propagated to all error response bodies — quote it when
+filing support tickets.
+
+A Prometheus `/metrics` endpoint is **not** included in this release.
+Add `prom-client` and a `/metrics` route if a downstream SRE consumer
+requires it.
+
+## See also
+
+- [Version history (developer doc)](../development/versioning.md)
+- [Runbook](runbook.md)

docs/admin/runbook.md

@@ -0,0 +1,116 @@
+# Apollon Standalone runbook
+
+On-call reference for the most likely production issues. Pair with
+`docs/development/versioning.md` for architecture context.
+
+## Health & quick triage
+
+```bash
+# Liveness — Express up + Redis ping reachable.
+curl -fsS https://<host>/health
+# Returns 200 {"status":"ok"} or 503 on Redis trouble.
+```
+
+The server logs are JSON via Pino; correlate by `requestId`:
+
+```bash
+# Find every event for a given request:
+journalctl -u apollon-standalone -o cat | grep '"requestId":"<id>"'
+```
+
+## Stuck or missing snapshot
+
+Symptom: user clicked "Save version," POST returned 5xx, the version
+doesn't appear after refresh.
+
+```bash
+# Inspect the diagram family.
+redis-cli SCAN 0 MATCH 'diagram:{<id>}*' COUNT 200
+redis-cli ZRANGE 'diagram:{<id>}:versions' 0 -1 WITHSCORES
+redis-cli HGETALL 'diagram:{<id>}:meta'
+```
+
+If a version body exists at `diagram:{<id>}:version:<vid>` but is not in
+the ZSET index, **delete the orphan**:
+
+```bash
+redis-cli DEL 'diagram:{<id>}:version:<vid>' 'diagram:{<id>}:version:<vid>:meta'
+```
+
+If a ZSET entry has no body, drop it from the index:
+
+```bash
+redis-cli ZREM 'diagram:{<id>}:versions' '<vid>'
+```
+
+## Corrupt RedisJSON HEAD
+
+Symptom: GET `/api/diagrams/:id` returns 500 INTERNAL with a JSON parse error
+in the logs.
+
+```bash
+redis-cli JSON.GET 'diagram:{<id>}' '$'
+```
+
+If the JSON is truly corrupt and there is no recoverable HEAD:
+
+1. List versions: `redis-cli ZRANGE 'diagram:{<id>}:versions' 0 -1`
+2. Pick the most recent version with `ZREVRANGE`.
+3. Restore it via the API: `POST /api/diagrams/:id/versions/<vid>/restore`
+
+## 503 spike
+
+Symptom: `/health` flips to 503; client requests fail with
+`REDIS_UNAVAILABLE`.
+
+1. `systemctl status apollon-standalone` — is Express up?
+2. `systemctl status redis-stack-server` — is Redis up?
+3. `redis-cli INFO replication` — is the primary writable?
+4. `redis-cli INFO persistence` — AOF rewrite stuck?
+5. `redis-cli FUNCTION LIST` — is the `apollon` library loaded? If not,
+   restart the server (boot re-loads idempotently).
+
+## OWNER_SECRET rotation
+
+Rotating `OWNER_SECRET` invalidates **all** soft-ownership claims for
+existing diagrams. Users will see the confirm-twice friction prompt on
+destructive operations until they re-create the diagram.
+
+Steps:
+1. Generate a new secret: `openssl rand -hex 32`
+2. Update the deployment env (`OWNER_SECRET=<new>`) and restart the server.
+3. Communicate the rotation to active users — destructive ops will now
+   show "Heads up — this diagram was originally created in a different
+   browser. Continue?"
+
+## Backup posture
+
+Apollon Standalone holds only ephemeral diagram data with a 120-day TTL.
+Users export-as-JSON for permanent local copies. AOF (`appendfsync everysec`)
+is enabled, so up to ~1s of writes can be lost on a hard crash. Snapshot
+and restore endpoints do not call `WAIT` — see `operations.md → Durability`
+for the rationale.
+
+## Function reload
+
+The server boot-loads the apollon Lua library via `FUNCTION LOAD REPLACE`.
+Verify with:
+
+```bash
+redis-cli FUNCTION LIST
+# Expected library name: apollon
+# Expected functions:    commit_snapshot, restore_version, list_versions_before
+```
+
+If the library is missing, restart the server (boot-load is idempotent).
+
+## Clearing all version data
+
+For GDPR right-to-erasure on a single diagram:
+
+```bash
+# Delete the entire family (HEAD + meta + versions + bodies).
+redis-cli --scan --pattern 'diagram:{<id>}*' | xargs -r redis-cli DEL
+```
+
+The webapp's `DELETE /api/diagrams/:id` does this automatically.