Legacy-conversion review fixes: portal attribution, preserved gate, clone flag

pr-review findings on the drop-legacy migration:
- Converted rows are attributed to their primary tag slug (one form
  config per distinct slug; 'legacy' only for untagged comments):
  gallery membership keys on portal_id, so the catch-all portal emptied
  every legacy tag gallery (PlanGallery via documents/list?tags=), and
  moderation authority landed on a portal no team administers. Tags now
  follow the new [portal, *others] convention.
- The old public gate is preserved exactly: score-flagged rows without
  an explicit APPROVED convert as hidden AND nsfw — the old gate
  excluded them from public view entirely, and blurred-but-fetchable
  would have retroactively published testimony no human ever approved.
- New submissions.map_is_clone column (backfilled true for pre-existing
  submitted rows, false for conversions/drafts): takedown demotes
  draft_status ONLY for clone-backed entries — converted rows reference
  LIVE user documents, and hiding one used to scratch a real person's
  working map. Auto-collect modes inherit the correct behavior for free.
- Converted content values are BTRIMmed (exact-match place/state/zip
  filters).
- stress_test/seed.py no longer deletes from the dropped tables (its
  cleanup path was dead on arrival); unused or_ import removed; agent
  guides repointed from app/comments to app/submissions; stale
  review_tags wording in TokenScope corrected.
- Verified on a seeded probe DB: per-portal attribution, tag order,
  hidden/nsfw matrix, id offsets, and the map_is_clone backfill all
  checked row-by-row; fresh-DB upgrade head green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: nofurtherinformation

.agents/WAGTAIL-CUTOVER-FOLLOWUPS.md

@@ -20,7 +20,7 @@ the Districtr Management System requirements:
   GroupApprovalTask(admin) on the page tree and Gallery snippets — partners
   submit for moderation; publish permissions removed from non-admins.
 - **Locales** pre-provisioned (content.0006).
-- Backend: `has_document` filter on `GET /api/comments/admin/list`.
+- Backend: `has_document` filter on `GET /api/comments/admin/list (module dropped later in the stack; submissions replaced it)`.
 
 **⏳ DEFERRED — Partner report generation** (decided 2026-08-05: spec later).
 The feature list's "Generate a report" has no implementation anywhere. Open

.agents/skills/project/learn-backend/SKILL.md

@@ -24,7 +24,7 @@ Backend conventions for FastAPI + SQLModel/SQLAlchemy services, request dependen
 - `backend/app/core/models.py`
 - `backend/app/utils.py`
 - `backend/app/assignments/assignments.py`
-- `backend/app/comments/*`
+- `backend/app/submissions/* (public testimony; app/district_notes.py for zone notes)`
 - `backend/app/cms/*`
 - `backend/app/save_share/*`
 - `backend/app/exports/*`

.agents/skills/project/learn-cms-moderation/SKILL.md

@@ -20,9 +20,9 @@ CMS authoring/review flows, TipTap extension expectations, and moderation/review
 - `app/src/app/store/cmsFormStore.ts`
 - `backend/app/cms/main.py`
 - `backend/app/cms/models.py`
-- `backend/app/comments/main.py`
-- `backend/app/comments/models.py`
-- `backend/app/comments/moderation.py`
+- `backend/app/submissions/main.py`
+- `backend/app/submissions/models.py`
+- `backend/app/submissions/moderation.py`
 
 ## Hard Invariants
 - CMS content authoring and publishing must preserve role/scope boundaries.
@@ -33,7 +33,7 @@ CMS authoring/review flows, TipTap extension expectations, and moderation/review
 
 ## Preferred Patterns
 - Reuse existing API handlers/store actions for CMS CRUD + publish flows.
-- Keep moderation and review behavior centralized in backend comments module.
+- Keep moderation behavior centralized in the backend submissions module.
 - Keep admin filtering behavior aligned with backend query params.
 - Add tests when changing moderation/review behavior.
 
@@ -51,7 +51,7 @@ CMS authoring/review flows, TipTap extension expectations, and moderation/review
 5. Confirm admin review filters still map to backend behavior.
 
 ## Validation Commands
-- `cd backend && pytest -v tests/test_comments.py tests/test_cms.py`
+- `cd backend && pytest -v tests/test_submissions.py tests/test_cms.py`
 - `cd app && bun run build`
 
 ## See Also

backend/app/alembic/versions/d8f1b52c96e3_drop_legacy_comment_tables.py

@@ -6,19 +6,33 @@
 copied into district_notes by b3d9f47a25c1.
 
 Form comments are real data (dev has live testimony, e.g. the TN workshop),
-so before dropping, every legacy form comment becomes a submission under a
-catch-all 'legacy' form config:
+so before dropping, every legacy form comment becomes a submission:
 
-- tags are preserved verbatim, so tag-filtered galleries keep showing them;
-  only the per-portal admin queue groups them under 'legacy' (portal_id is
-  ON UPDATE CASCADE — re-attribute later with a plain UPDATE if wanted).
+- PORTAL ATTRIBUTION: gallery membership and moderation authority both key
+  on portal_id, so each comment is attributed to its primary tag slug (the
+  earliest-created tag == the portal page's slug in the legacy flow); a
+  form config row is created per distinct slug. Untagged comments fall back
+  to a catch-all 'legacy' config. All created configs get admin_teams='{}'
+  (no team information exists here): they are moderatable only by
+  review:review-all holders until an admin grants teams in the CMS — and
+  they DO accept new public submissions (the portals are real pages), which
+  is intended.
+- tags follow the new-submission convention: [portal_id, *other slugs].
 - map attachments keep their LIVE document reference (legacy behavior);
-  clone-at-submission applies only to new submissions.
-- moderation maps to the new bits preserving what the old public gate
-  showed: hidden = anything REJECTED (comment, commenter, or a tag);
-  nsfw = any moderation score >= 0.2 without an APPROVED override.
+  map_is_clone=false marks them so takedown never demotes a real user's
+  working map (this migration also adds that column).
+- moderation preserves the OLD public gate exactly: hidden = anything
+  REJECTED (comment, commenter, or a tag) OR anything score-flagged
+  (>= 0.2) without an explicit APPROVED override — the old gate excluded
+  score-flagged rows from public view entirely, and converting them to
+  merely-blurred would retroactively publish testimony no human ever
+  approved. nsfw carries the score flag so an admin who un-hides one still
+  gets the blur. Reviewers can unhide false positives from the queue.
 - submission ids are the legacy comment ids offset past MAX(submissions.id),
   so a deploy where pr10..13 already collected new submissions can't collide.
+- negative-zone rows (skipped by b3d9f47a25c1 as never-renderable) are in
+  neither district_notes nor submissions and are dropped with the tables —
+  a deliberate one-way loss.
 
 Downgrade recreates the tables (final shape as of 0db008690d60 + da39a3ee5e6b)
 empty — it does NOT reverse the conversion (converted rows simply remain in
@@ -90,18 +104,30 @@ def _timestamps():
 
 
 def _convert_legacy_form_comments(bind) -> None:
-    # Catch-all portal config, only when there is anything to migrate.
+    # One config per portal the legacy data references: the comment's primary
+    # (earliest) tag slug IS the portal page slug in the legacy flow, and
+    # gallery membership + moderation authority both key on portal_id — a
+    # catch-all portal would empty every legacy tag gallery. Untagged
+    # comments fall back to 'legacy'.
     bind.execute(
         sa.text(
             f"""
             INSERT INTO comments.form_configs
                 (portal_id, name, fields, required_fields,
                  require_email_confirm, admin_teams)
-            SELECT 'legacy', 'Legacy submissions', {_ALL_FIELDS},
-                   '{{}}', false, '{{}}'
-            WHERE EXISTS (
-                SELECT 1 FROM comments.comment c {_FORM_COMMENT_FILTER}
-            )
+            SELECT DISTINCT
+                   COALESCE(t.primary_slug, 'legacy'),
+                   COALESCE(t.primary_slug, 'Legacy submissions'),
+                   {_ALL_FIELDS}, '{{}}'::varchar(64)[], false,
+                   '{{}}'::varchar(255)[]
+            FROM comments.comment c
+            LEFT JOIN LATERAL (
+                SELECT (array_agg(tg.slug ORDER BY tg.id))[1] AS primary_slug
+                FROM comments.comment_tag ct
+                JOIN comments.tag tg ON tg.id = ct.tag_id
+                WHERE ct.comment_id = c.id
+            ) t ON true
+            {_FORM_COMMENT_FILTER}
             ON CONFLICT (portal_id) DO NOTHING
             """
         )
@@ -122,7 +148,8 @@ def _convert_legacy_form_comments(bind) -> None:
                     c.moderation_score AS c_score,
                     cm.review_status::text AS m_status,
                     cm.moderation_score AS m_score,
-                    t.slugs, t.rejected_tag, t.score_flagged_tag, t.max_tag_score
+                    t.slugs, t.rejected_tag, t.score_flagged_tag, t.max_tag_score,
+                    COALESCE((t.slugs)[1], 'legacy') AS portal
                 FROM comments.comment c
                 LEFT JOIN comments.document_comment dc ON dc.comment_id = c.id
                 LEFT JOIN comments.commenter cm ON cm.id = c.commenter_id
@@ -141,35 +168,46 @@ def _convert_legacy_form_comments(bind) -> None:
                 ) t ON true
                 LEFT JOIN document.document d ON d.document_id = dc.document_id
                 WHERE dc.zone IS NULL
+            ),
+            score_flags AS (
+                SELECT e.*,
+                    COALESCE(
+                        (e.c_score >= 0.2 AND e.c_status IS DISTINCT FROM 'APPROVED')
+                        OR (e.m_score >= 0.2 AND e.m_status IS DISTINCT FROM 'APPROVED')
+                        OR e.score_flagged_tag,
+                        false
+                    ) AS score_flagged,
+                    COALESCE(
+                        e.c_status = 'REJECTED'
+                        OR e.m_status = 'REJECTED'
+                        OR e.rejected_tag,
+                        false
+                    ) AS was_rejected
+                FROM enriched e
             )
             INSERT INTO comments.submissions
                 (id, portal_id, map_public_id, tags, status, submitted_at,
-                 nsfw, hidden, flagged, moderation_score,
+                 nsfw, hidden, flagged, moderation_score, map_is_clone,
                  created_at, updated_at)
             SELECT
                 e.id + :offset,
-                'legacy',
+                e.portal,
                 e.public_id,
-                COALESCE(e.slugs, '{}'),
+                ARRAY[e.portal]::varchar(255)[]
+                    || array_remove(COALESCE(e.slugs, '{}'), e.portal),
                 'submitted',
                 e.created_at,
-                COALESCE(
-                    (e.c_score >= 0.2 AND e.c_status IS DISTINCT FROM 'APPROVED')
-                    OR (e.m_score >= 0.2 AND e.m_status IS DISTINCT FROM 'APPROVED')
-                    OR e.score_flagged_tag,
-                    false
-                ),
-                COALESCE(
-                    e.c_status = 'REJECTED'
-                    OR e.m_status = 'REJECTED'
-                    OR e.rejected_tag,
-                    false
-                ),
+                e.score_flagged,
+                -- The OLD public gate fully excluded score-flagged rows that
+                -- no human APPROVED; blurred-but-fetchable would widen their
+                -- exposure retroactively, so they convert as hidden too.
+                e.was_rejected OR e.score_flagged,
                 e.review_flagged,
                 GREATEST(e.c_score, e.m_score, e.max_tag_score),
+                false,
                 e.created_at,
                 e.updated_at
-            FROM enriched e
+            FROM score_flags e
             """
         ),
         {"offset": offset},
@@ -179,7 +217,7 @@ def _convert_legacy_form_comments(bind) -> None:
         sa.text(
             f"""
             INSERT INTO comments.submissions_content (submission_id, field, value)
-            SELECT c.id + :offset, f.field, LEFT(f.value, 5000)
+            SELECT c.id + :offset, f.field, LEFT(BTRIM(f.value), 5000)
             FROM comments.comment c
             LEFT JOIN comments.commenter cm ON cm.id = c.commenter_id
             CROSS JOIN LATERAL (VALUES
@@ -214,6 +252,27 @@ def _convert_legacy_form_comments(bind) -> None:
 
 
 def upgrade() -> None:
+    # Whether the submission's map is a submission-owned frozen clone (new
+    # prompt/form submissions) or a live reference to the author's working
+    # document (drafts, converted legacy rows, later auto-collect modes).
+    # Takedown may only demote the draft_status of CLONES.
+    op.add_column(
+        "submissions",
+        sa.Column(
+            "map_is_clone",
+            sa.Boolean(),
+            nullable=False,
+            server_default="false",
+        ),
+        schema="comments",
+    )
+    # Every pre-existing submitted row was created by clone-at-submission.
+    op.execute(
+        sa.text(
+            "UPDATE comments.submissions SET map_is_clone = true "
+            "WHERE status = 'submitted'"
+        )
+    )
     _convert_legacy_form_comments(op.get_bind())
     op.drop_table("document_comment", schema="comments")
     op.drop_table("comment_tag", schema="comments")
@@ -227,6 +286,7 @@ def upgrade() -> None:
 
 
 def downgrade() -> None:
+    op.drop_column("submissions", "map_is_clone", schema="comments")
     # Recreates the schema only; rows are unrecoverable without a backup.
     op.execute(
         sa.text(

backend/app/core/security.py

@@ -29,7 +29,8 @@ class TokenScope:
     delete_all_content = "delete:delete-all"
 
     review_content = "create:content_review"
-    # Explicit bypass of per-reviewer tag scoping (the `review_tags` claim).
+    # Explicit bypass of per-reviewer scoping (the `teams` claim; formerly
+    # the review_tags claim).
     # Deliberately separate from read:read-all: the *-all read/update/delete
     # scopes govern access across CMS authorship boundaries, while this one
     # widens moderation reach.

backend/app/main.py

@@ -19,7 +19,7 @@
     DataError,
     OperationalError,
 )
-from sqlalchemy import text, or_
+from sqlalchemy import text
 from sqlalchemy.types import Integer
 from sqlmodel import Session, String, select, true, update, col, literal
 from sqlalchemy.sql import and_, exists

backend/app/submissions/main.py

@@ -289,6 +289,7 @@ async def create_submission(
         tags=_normalized_tags(data.tags, config.portal_id),
         status=SubmissionStatus.submitted,
         submitted_at=datetime.now(timezone.utc),
+        map_is_clone=map_public_id is not None,
     )
     session.add(submission)
     session.flush()
@@ -345,6 +346,7 @@ async def finalize_submission(
     clone = clone_document_for_submission(session, source)
 
     submission.map_public_id = clone.public_id
+    submission.map_is_clone = True
     submission.tags = _normalized_tags(data.tags, config.portal_id)
     submission.status = SubmissionStatus.submitted
     submission.submitted_at = datetime.now(timezone.utc)
@@ -598,17 +600,19 @@ async def set_submission_hidden(
 ):
     """Hard takedown/restore for spam and abuse. Resolves the flag report.
 
-    For submitted entries the map is a frozen clone that exists ONLY as a
-    gallery entry, so takedown also demotes its draft_status: without that,
+    For submitted CLONE-backed entries the map exists ONLY as a gallery
+    entry, so takedown also demotes its draft_status: without that,
     the abusive map stays fetchable at its enumerable public_id even while
     hidden from every listing. Restore puts it back to ready_to_share (the
-    status every clone has by construction).
+    status every clone has by construction). Live-referenced maps (converted
+    legacy rows, drafts, auto-collect modes) are NEVER demoted — that would
+    mutate a real user's working document.
     """
     submission = _get_submission_for_admin(submission_pk, auth_result, session)
     submission.hidden = body.hidden
     submission.flagged = False
     session.add(submission)
-    if submission.status == SubmissionStatus.submitted:
+    if submission.status == SubmissionStatus.submitted and submission.map_is_clone:
         session.execute(
             text(
                 """

backend/app/submissions/models.py

@@ -173,6 +173,15 @@ class Submission(TimeStampMixin, SQLModel, table=True):
             Boolean, nullable=False, default=False, server_default="false"
         ),
     )
+    # True when the map is a submission-owned frozen clone; false for live
+    # references (drafts, converted legacy rows, auto-collect modes).
+    # Takedown may only demote the draft_status of clones.
+    map_is_clone: bool = Field(
+        default=False,
+        sa_column=Column(
+            Boolean, nullable=False, default=False, server_default="false"
+        ),
+    )
     flagged: bool = Field(
         default=False,
         sa_column=Column(

backend/stress_test/seed.py

@@ -164,17 +164,6 @@ def delete_documents(session: Session, document_ids: list[str]) -> int:
         conn = session.connection()
         params = {"ids": chunk}
         id_filter = "document_id = ANY(CAST(:ids AS uuid[]))"
-        conn.execute(
-            text(
-                f"""WITH removed AS (
-                    DELETE FROM comments.document_comment WHERE {id_filter}
-                    RETURNING comment_id
-                )
-                DELETE FROM comments.comment
-                WHERE id IN (SELECT comment_id FROM removed)"""
-            ),
-            params,
-        )
         for table in (
             "document.district_unions",
             "document.map_document_user_session",