feat(server): complete privacy retention and observability (#1784)

Author: FelixTJDietrich

.changeset/calm-privacy-jobs.md

@@ -0,0 +1,9 @@
+---
+"hephaestus": minor
+---
+
+Operators can now alert on privacy-job outcomes: account erasure, export generation, export expiry and LLM usage retention each publish success, failure and affected-row counters. A retention pass that runs out of its time budget with rows still expired reports `incomplete` rather than success, so a sweep that never catches up is visible.
+
+LLM usage accounting is no longer kept indefinitely — rows become eligible for deletion 400 days after they were recorded by default, and a daily sweep removes them in batches. The public privacy statement and the record of processing document the window.
+
+**Operators:** the first sweep after upgrade begins deleting usage rows older than the window, and further sweeps continue until the backlog is gone. If your accounting obligations require longer, set `HEPHAESTUS_LLM_USAGE_RETENTION` before upgrading.

.migration/calm-privacy-jobs.md

@@ -0,0 +1,17 @@
+#### 🔴 LLM usage accounting older than the retention window is deleted after upgrade
+
+**Affected**: every deployment that has recorded LLM usage for longer than the window — 400 days
+unless you set your own — and any operator whose commercial or tax retention obligations cover that
+accounting data.
+
+**Before**: rows in the LLM usage ledger — per-run token counts and cost, attributed to a workspace —
+were kept indefinitely.
+
+**After**: a daily sweep deletes usage older than the configured window, default 400 days. The
+deletion is irreversible. Each pass deletes in batches for at most five minutes, so the first sweep
+after upgrade begins clearing the historical backlog and later sweeps finish it; a pass that stops
+with rows still expired reports the `incomplete` privacy-job outcome.
+
+**Migration**: if your accounting obligations require a longer window, set
+`HEPHAESTUS_LLM_USAGE_RETENTION` (an ISO-8601 duration, for example `P3650D`) before deploying this
+release. No action is needed to keep the default.

docker/.env.example

@@ -335,6 +335,10 @@ AGENT_ENABLED=false
 # simply never captured. See docs/contributor/artifact-source-contract.mdx.
 HEPHAESTUS_FABRIC_GC_RETENTION_DAYS=30
 
+# How long LLM usage accounting (token counts and cost) is kept before the nightly sweep deletes it.
+# ISO-8601 duration. Raise it if your accounting obligations run longer; deletion is irreversible.
+HEPHAESTUS_LLM_USAGE_RETENTION=P400D
+
 # Poll interval and claim batch size for the PostgreSQL-backed job queue (defaults shown).
 # AGENT_POLL_INTERVAL=1s
 # AGENT_CLAIM_BATCH_SIZE=5

docker/compose.app.yaml

@@ -245,6 +245,10 @@ services:
       # The reports and the FX fetcher are server-role, so this belongs on this pod.
       HEPHAESTUS_LLM_DISPLAY_CURRENCY: ${HEPHAESTUS_LLM_DISPLAY_CURRENCY:-}
       HEPHAESTUS_LLM_FX_DAILY_URL: ${HEPHAESTUS_LLM_FX_DAILY_URL:-https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml}
+      # Storage limitation on the spend ledger. The nightly sweep is server-role, so this pod only.
+      # Raise it where a commercial or tax retention obligation runs past the default; deletion is
+      # irreversible.
+      HEPHAESTUS_LLM_USAGE_RETENTION: ${HEPHAESTUS_LLM_USAGE_RETENTION:-P400D}
       # SSRF guard. Set on BOTH this pod and the worker or the two disagree: this pod decides whether
       # a loopback provider URL may be SAVED, the worker's proxy decides whether it may be DIALLED.
       # Leave false in production.

docs/admin/dsms/README.md

@@ -19,6 +19,7 @@ A different operator forking Hephaestus must amend, before opening their deploym
 | [`dpia-prescreen.md`](./dpia-prescreen.md) | Art. 35 pre-screen. Records the pending controller/DPO determination, safeguards, and change freeze. |
 | [`processor-checklist.md`](./processor-checklist.md) | Art. 28 checklist. Per-processor AVV status; LRZ-as-separate-controller analysis. |
 | [`artifact-source-governance.md`](./artifact-source-governance.md) | Approval, minimization, processor-egress, retention, and erasure gate for every AI-readable source. |
+| [`personal-data-map.md`](./personal-data-map.md) | Personal-data stores, export coverage, erasure paths, residual retention, and their verification. |
 
 The live imprint and privacy pages are at https://hephaestus.aet.cit.tum.de/imprint and https://hephaestus.aet.cit.tum.de/privacy. Markdown source: [`webapp/public/legal/profiles/tumaet/`](https://github.com/ls1intum/Hephaestus/tree/main/webapp/public/legal/profiles/tumaet).
 

docs/admin/dsms/personal-data-map.md

@@ -0,0 +1,49 @@
+---
+title: Personal-data map and erasure verification
+description: The authoritative map from personal-data stores to export, retention, and erasure controls.
+---
+
+# Personal-data map and erasure verification
+
+This inventory maps each personal-data store to its removal control and executable evidence. Add a
+store or derived copy here, with an erasure or retention test, before it receives personal data:
+`PersonalDataMapArchTest` fails the build when a database table is named in neither this map nor the
+test's own list of tables that hold no personal data, and when a source or test cited below has
+moved.
+Workspace purge applies storage limitation when a workspace loses its purpose; a verified
+person-erasure request follows the separate account or operator path.
+
+| Store / copy | Data and access path | Workspace removal | Person erasure | Residual retention and evidence |
+|---|---|---|---|---|
+| PostgreSQL account, identity and authentication rows — `account`, `account_feature`, `account_export`, `identity_link`, `issued_jwt`, `user_preferences`, `consent_decision`, `auth_event`, `config_audit_event` | Sign-in identity; self-service export plus operator supplement | Account-global rows remain | 48-hour cooldown, then account purge | Audit rows detach the actor and expire after 365 days; exports expire after 48 hours. `server/application/src/test/java/de/tum/cit/aet/hephaestus/core/auth/AccountHardDeleteSweeperIntegrationTest.java` and `server/application/src/test/java/de/tum/cit/aet/hephaestus/core/auth/export/AccountExportRetentionIntegrationTest.java`. |
+| PostgreSQL product feedback and surveys — `product_feedback`, `product_survey_submission` | Data supplied by the account holder; operator supplement | Purge contributors remove workspace-owned rows | Account purge removes account-owned rows | `server/application/src/test/java/de/tum/cit/aet/hephaestus/workspace/WorkspacePurgeIntegrationTest.java` and account-purge coverage above. |
+| PostgreSQL SCM mirror — `user`, `organization`, `organization_membership`, `team`, `team_membership`, `team_repository_permission`, `repository`, `repository_collaborator`, `pull_request`, `pull_request_requested_reviewers`, `pull_request_review`, `pull_request_review_comment`, `pull_request_review_thread`, `issue`, `issue_assignee`, `issue_comment`, `milestone`, `discussion`, `discussion_comment`, `git_commit`, `commit_contributor`, `commit_file_change`, `project`, `project_item`, `project_field_value`, `project_status_update` | GitHub/GitLab profiles and authored work; source-provider export plus operator supplement | Last repository/connection removal and workspace purge erase the active mirror | Verified source-account matching; shared authorship may be pseudonymised where an exception applies | `server/application/src/test/java/de/tum/cit/aet/hephaestus/workspace/ScmWorkspaceErasureIntegrationTest.java`. |
+| PostgreSQL Slack mirror — `slack_message`, `slack_thread`, `slack_monitored_channel`, `slack_participant_consent`, `slack_channel_consent_event`, `mentor_slack_thread` | Messages, identities and attribution; operator supplement | Disconnect and workspace purge | Slack opt-out and verified operator path | `server/application/src/test/java/de/tum/cit/aet/hephaestus/integration/slack/retention/SlackRetentionErasureIntegrationTest.java` and `server/application/src/test/java/de/tum/cit/aet/hephaestus/integration/slack/interactivity/SlackAppHomeOptOutErasureIntegrationTest.java`. |
+| PostgreSQL Outline mirror — `outline_document`, `outline_collection`, `outline_document_event` | Documents and attribution; operator supplement | Collection removal, disconnect and workspace purge | Verified operator path | `server/application/src/test/java/de/tum/cit/aet/hephaestus/integration/outline/lifecycle/OutlineWorkspacePurgeAdapterIntegrationTest.java`. |
+| PostgreSQL mentor, observations, feedback, recognition and activity read models — `chat_thread`, `chat_message`, `chat_message_vote`, `observation`, `reaction`, `feedback`, `feedback_observation`, `feedback_dispatch`, `feedback_placement`, `feedback_approval`, `delivery_policy_evaluation`, `activity_event`, `user_achievement`, `workspace_membership` | Conversations and derived judgements; operator supplement | Purge contributors remove each workspace-owned family | Account or source identity is removed; conversation-derived feedback is erased | `server/application/src/test/java/de/tum/cit/aet/hephaestus/workspace/WorkspacePurgeIntegrationTest.java` exercises the assembled purge chain. |
+| PostgreSQL review-scope selection — `practice_review_person_target` | Which people a workspace administrator selected for practice review; operator supplement | Removed with the workspace: the row cascades from the `workspace_membership` row the purge deletes (`sfk_practice_review_person_target_membership`, `ON DELETE CASCADE`) | Removed with the membership | No independent retention window. `server/application/src/test/java/de/tum/cit/aet/hephaestus/workspace/WorkspacePurgeIntegrationTest.java` proves the membership deletion the cascade follows. |
+| PostgreSQL operator-action and operational-history records — `connection_audit`, `sync_job`, `artifact_signal`, `review_backfill_run`, `review_sweep_schedule` | Which account requested, configured or was selected for a workspace operation, and which artifacts a review campaign covered; operator supplement | **Retained.** Purge transitions connections to `UNINSTALLED` and marks the workspace `PURGED` instead of deleting either row, so neither a purge contributor nor the `ON DELETE CASCADE` on `workspace` reaches these tables | The stored account reference is a login snapshot without a foreign key, or a nullable `user` reference cleared by the database (`sfk_artifact_signal_requested_by`, `ON DELETE SET NULL`); either way it resolves to nothing once the referenced row is gone | `sync_job` keeps only the newest 50 rows per connection, pruned on each insert, and stops changing once the connection is uninstalled. The other four have no retention window and are removed only through the operator-executed erasure path. `server/application/src/main/java/de/tum/cit/aet/hephaestus/integration/core/connection/ConnectionPurgeContributor.java` and `server/application/src/main/java/de/tum/cit/aet/hephaestus/integration/core/sync/SyncJobRepository.java`. |
+| PostgreSQL instance-administration records — `instance_settings`, `instance_llm_settings`, `product_survey` | Which administrator last engaged the instance brake or changed instance LLM policy, kept as a login snapshot (`silent_mode_changed_by`, `updated_by`), and which account authored a survey (`created_by_account_id`); operator supplement | Instance-scoped: workspace purge does not reach them, and a survey aimed at one workspace keeps its definition | The record of who acted deliberately survives account deletion. `silent_mode_changed_by` and `updated_by` are login snapshots with no foreign key, and an account purge does not clear them. `product_survey.created_by_account_id` is a foreign key (`fk_product_survey_creator`, `ON DELETE SET NULL`), but an account purge keeps the account row as a tombstone rather than deleting it, so the reference survives and resolves to a tombstone that carries no personal data | No independent window. A settings row carries only the administrator of the most recent change and the next change overwrites them; a survey carries its author id for as long as the survey exists; the account it points at is a tombstone once that account is purged. `server/application/src/main/java/de/tum/cit/aet/hephaestus/core/settings/InstanceSettings.java`, `server/application/src/main/java/de/tum/cit/aet/hephaestus/agent/catalog/InstanceLlmSettings.java`, `server/application/src/main/java/de/tum/cit/aet/hephaestus/productfeedback/Survey.java`, `server/application/src/main/java/de/tum/cit/aet/hephaestus/core/auth/AccountPurger.java`. |
+| PostgreSQL `agent_job`, manifests, evidence references and diagnostics | Review inputs, outputs and provenance; operator supplement | `AgentWorkspacePurgeAdapter` deletes workspace jobs before workspace configuration | No independent subject lookup; rows leave with workspace purge or retention | `server/application/src/test/java/de/tum/cit/aet/hephaestus/agent/AgentWorkspacePurgeIntegrationTest.java`, `server/application/src/test/java/de/tum/cit/aet/hephaestus/agent/job/AgentJobRetentionServiceTest.java`, and `server/application/src/test/java/de/tum/cit/aet/hephaestus/agent/job/AgentJobRetentionObservationIntegrationTest.java` cover purge, 14-day diagnostics and 90-day rows. |
+| PostgreSQL `llm_usage_event` | Token/cost accounting with workspace and opaque source identifier; monthly workspace/admin reports | Retained for accounting until its retention sweep | The identifier is not a foreign key; deleting a source row leaves nothing for it to resolve | The window and its operator control are in the [record of processing](./record-of-processing.md). `server/application/src/test/java/de/tum/cit/aet/hephaestus/agent/usage/LlmUsageRetentionIntegrationTest.java` proves the cutoff across workspaces; `server/application/src/test/java/de/tum/cit/aet/hephaestus/agent/usage/LlmUsageRetentionSweeperTest.java` verifies job outcomes. |
+| NATS JetStream | Buffered webhook, message and document payloads; not directly included in self-service export | No selective deletion | No selective deletion | Slack/Outline expire after 72 hours; GitHub/GitLab after 180 days; byte ceilings can shorten both. Stream configuration tests enforce these bounds. |
+| PostgreSQL evidence bytes | Bounded review input snapshots; operator supplement | SQL authority is erased with its owning rows | Removed with owning rows | Diagnostic payloads expire after 14 days and job provenance after 90 days; evidence retention tests are linked in the `agent_job` row. |
+| Replay directories and content-addressed blobs | Working copies and immutable blobs; not directly included in self-service export | No selective deletion | No selective deletion | Replay directories and unreferenced blobs become eligible after 30 days. [`artifact-source-governance.md`](./artifact-source-governance.md) owns this control. |
+| Configured LLM provider | Prompts and responses for enabled purposes; provider/operator access process | A completed request cannot be retracted | Provider process | Deployment-specific provider terms must define the bound before processing starts; see the [processor checklist](./processor-checklist.md). |
+| GitHub/GitLab/Slack delivery destination | Posted feedback; source-provider export | Not silently crawled or rewritten | Best-effort correction/removal through the operator path | The provider controls its copy and audit history. |
+| Application metrics | Aggregate operational counts without account, workspace, export or source labels | Not applicable | No subject-level series exists | Backend retention is deployment-specific; operators must record it. |
+| Container logs and support bundles | Operational output; logging policy excludes raw request bodies and secrets, but operators must assess exported bundles | No selective deletion | No selective deletion | Shipped container logs rotate by size; support-bundle retention is deployment-specific. |
+| PostgreSQL and filesystem backups | A copy of the backed-up personal-data corpus | No selective deletion inside a backup | Expiry of the backup copy | The application does not configure off-host backups. Operators who add them must document and enforce their retention; see [backup and restore](../backup-restore.mdx). |
+
+## Operator verification
+
+Each erasure, export and retention job publishes a bounded success/failure counter and an
+affected-row counter; the names, labels and cardinality guarantee are in the
+[observability contract](../observability.mdx). Alert configuration remains deployment-owned: alert
+on `outcome="failure"`, on `outcome="incomplete"` — a retention pass that ran out of its time budget
+with expired rows still in place — and on a missing successful daily `llm_usage_retention` run for
+more than 48 hours. Broader alert routing is tracked in
+[#1369](https://github.com/hephaestus-build/Hephaestus/issues/1369).
+
+Review this map with the [record of processing](./record-of-processing.md),
+[artifact-source governance](./artifact-source-governance.md), and the public privacy statement.