|
| 1 | +# Wagtail Cutover — Remaining Work & Follow-ups |
| 2 | + |
| 3 | +### Session 2026-08-05 — all administration moved into the Wagtail UI |
| 4 | + |
| 5 | +Shipped (branch commits `0836caa7`, `5e283044`, + this one), reviewed against |
| 6 | +the Districtr Management System requirements: |
| 7 | + |
| 8 | +- **`cms/moderation` app**: comment review, district-comment review, map |
| 9 | + submissions (comments with an attached plan; approve into team galleries as |
| 10 | + draft revisions), and the under-construction toggle are now Wagtail views |
| 11 | + calling the FastAPI admin endpoints with a per-user minted JWT |
| 12 | + (`authapi.serializers.mint_user_access_token`). The Next.js `/admin` tree |
| 13 | + and the external "Comment review" menu link (`FRONTEND_URL`) are gone. |
| 14 | +- **Groups consolidated** (authapi.0007): `partner` (pages own-content, |
| 15 | + galleries drafts, comment/submission moderation), `super_partner` |
| 16 | + (+ compose map, upload overlay, module/overlay editing; GPKG import stays |
| 17 | + admin-only via `datastore.add_gerrydbtable`), `admin`. `editor`/`reviewer` |
| 18 | + deleted; members migrated to partner. |
| 19 | +- **Review//approve** (content.0007): "Admin approval" Workflow + |
| 20 | + GroupApprovalTask(admin) on the page tree and Gallery snippets — partners |
| 21 | + submit for moderation; publish permissions removed from non-admins. |
| 22 | +- **Locales** pre-provisioned (content.0006). |
| 23 | +- Backend: `has_document` filter on `GET /api/comments/admin/list`. |
| 24 | + |
| 25 | +**⏳ DEFERRED — Partner report generation** (decided 2026-08-05: spec later). |
| 26 | +The feature list's "Generate a report" has no implementation anywhere. Open |
| 27 | +questions before building: CSV export vs formatted summary; scope (per |
| 28 | +portal/tag? per team?); which fields (comments, commenters, attached plan |
| 29 | +ids, counts). `bd` is still broken locally (`issue_prefix` missing), so this |
| 30 | +note is the tracking record. |
| 31 | + |
| 32 | +Status as of 2026-06-11, branch `wagtail-cutover` (20 commits ahead of `dev`, all |
| 33 | +quality gates green: backend 306 passed, cms 175 tests, frontend build clean). |
| 34 | +All 13 cutover workstreams and all 10 verified code-review findings are fixed |
| 35 | +and committed. This file tracks what was **deliberately deferred** — pick it up |
| 36 | +in a future session. Plan context: `~/.claude/plans/review-these-conversation-notes-federated-mitten.md` |
| 37 | +(local to Dylan's machine) and the project memory `wagtail-cutover-project`. |
| 38 | + |
| 39 | +### Follow-up session 2026-06-11 — cleared the decision-free backlog |
| 40 | + |
| 41 | +All items below marked **✅ DONE** were fixed and verified this session |
| 42 | +(backend 306 passed, cms non-menu tests pass, frontend build + pre-commit |
| 43 | +clean). Items marked **⏳ NEEDS DECISION** or **⏳ DEFERRED** remain. Details |
| 44 | +inline. The flaky `TestCommenterEndpoint` fix was a real fragility: |
| 45 | +`create_commenter_db`/`create_tag_db` returned `Model.model_construct(...)` |
| 46 | +instances that bypass SQLAlchemy instrumentation and fail FastAPI response |
| 47 | +serialization until the ORM mappers are configured by an unrelated test — |
| 48 | +replaced with real `Model(**row._asdict())` instances. |
| 49 | + |
| 50 | +--- |
| 51 | + |
| 52 | +## 1. Product decisions — RESOLVED 2026-08-04 (session with Dylan) |
| 53 | + |
| 54 | +### 1.1 ✅ Editor scope: **own-content-only** (+ teams for manual control) |
| 55 | +Shipped: `content/0004_editor_own_content_only` revokes the editor group's |
| 56 | +tree-wide `change_page`; editors keep `add_page` (Wagtail's owner model grants |
| 57 | +edit on owned pages) + `publish_page` (applies only to editable, i.e. own, |
| 58 | +pages). `migrate_tiptap --owners "auth0|xx=email,..."` sets `Page.owner` from |
| 59 | +the legacy `author` column (Auth0 subjects — the sub→email mapping must be |
| 60 | +supplied at cutover; there are two distinct authors in the data). |
| 61 | + |
| 62 | +### 1.2 ✅ group_only galleries: **enforced via Teams** |
| 63 | +Shipped: the JWT carries a `map_groups` claim (slugs across the user's teams, |
| 64 | +minted in [cms/authapi/serializers.py](../cms/authapi/serializers.py)); |
| 65 | +[cms/galleries/api.py](../cms/galleries/api.py) requires the gallery's |
| 66 | +`map_group` slug in that claim, or the `admin` role. A merely-valid login no |
| 67 | +longer opens group_only galleries. Scoping unit confirmed as `MapGroup`. |
| 68 | + |
| 69 | +> **Superseded 2026-08-05 (districtr_v2-i06): Team is the tenant now.** |
| 70 | +> MapGroup reverted to a pure listing facet. Team gained a `slug` |
| 71 | +> (authapi.0008), the JWT claim is `teams` (team slugs), Gallery has a |
| 72 | +> required real `team` FK (galleries.0003 — also closes the |
| 73 | +> silently-inaccessible NULL-group gallery bug, districtr_v2-rqz), and |
| 74 | +> teams own map modules directly via TeamDistrictrMap (existing |
| 75 | +> TeamMapGroup ownership auto-expanded per map at migration). Scoping |
| 76 | +> engine: authapi/teams.py (`team_ids_for_user` / `team_slugs_for_user`). |
| 77 | +
|
| 78 | +### 1.3 Refresh-token security posture (still open, low priority) |
| 79 | +`BLACKLIST_AFTER_ROTATION` was turned **off** |
| 80 | +([cms/config/settings/base.py](../cms/config/settings/base.py)) because Next.js |
| 81 | +RSCs cannot persist rotated cookies — single-use tokens deterministically |
| 82 | +bricked admin sessions. Trade-off: a stolen refresh token stays valid until its |
| 83 | +own 14-day expiry. If tighter security is wanted later: implement a reuse |
| 84 | +grace-window serializer (accept the previous token for ~60s after rotation), |
| 85 | +or make middleware the *only* refresher and re-enable blacklisting. |
| 86 | + |
| 87 | +--- |
| 88 | + |
| 89 | +## 2. Functional follow-ups (small, well-scoped) |
| 90 | + |
| 91 | +| Item | Where | Notes | |
| 92 | +|---|---|---| |
| 93 | +| ✅ **DECIDED 2026-08-04: leave as-is** — District comments for tag-scoped reviewers | [backend/app/comments/main.py](../backend/app/comments/main.py) | Blanket 403 stays: scoped reviewers moderate community comments only; full reviewers/admins handle district comments. Menu link already hidden for scoped reviewers. | |
| 94 | +| ✅ **DONE** — `/places` "N map modules" count | [app/src/app/(static)/places/page.tsx](../app/src/app/(static)/places/page.tsx) | Restored: card shows `N map module(s)` from the `districtr_map_slugs` the list endpoint returns. | |
| 95 | +| ✅ **DONE** — GET `/auth/logout` CSRF | [app/src/app/auth/logout/route.ts](../app/src/app/auth/logout/route.ts) | Guarded with the Fetch-Metadata `Sec-Fetch-Site` header — an explicit `cross-site` GET bounces home WITHOUT signing out; same-origin/same-site/direct nav still log out. Chose this over the auto-submit-form approach: lower risk, no coupling to NextAuth CSRF internals, no redirect flash. | |
| 96 | +| ⏳ **DEFERRED** (long-term) — PermissionGuard reads raw JWT client-side | [app/src/app/admin/components/PermissionGuard.tsx](../app/src/app/admin/components/PermissionGuard.tsx) | Now base64url-safe via shared `decodeJwtPayload`, but long-term the access token shouldn't need to reach the client at all — pass roles/scopes as typed session fields and keep the token server-side. Larger auth-session refactor; left as-is. | |
| 97 | +| ✅ **DONE** — Flaky `TestCommenterEndpoint` (3 tests) | backend/tests/test_comments.py | Root cause: `create_commenter_db`/`create_tag_db` returned `model_construct(...)` instances bypassing ORM instrumentation, failing FastAPI response serialization until mappers were configured by an unrelated test. Fixed by returning real `Model(**row._asdict())` instances. Now passes in isolation, subset, and full suite. | |
| 98 | + |
| 99 | +--- |
| 100 | + |
| 101 | +## 3. Performance — ✅ ALL DONE this session |
| 102 | + |
| 103 | +- ✅ **`content_detail` fetches every language's full body** to compute |
| 104 | + `available_languages`, then serializes one |
| 105 | + ([cms/content/api.py](../cms/content/api.py)). Fixed: `values_list('locale__language_code')` |
| 106 | + for the language set + a single full fetch of the chosen page only. |
| 107 | +- ✅ **`content_list` pulls full `body` columns for up to 100 rows** to emit a |
| 108 | + link list (backs `/tags`, `/places`, homepage PlaceMap). Fixed: `.defer('body')`. |
| 109 | +- ✅ **`/api/galleries/` list has no offset/limit clamp** — fixed: mirrors |
| 110 | + `content_list`'s `MAX_PAGE_SIZE` (100) with `offset`/`limit` query params. |
| 111 | +- ✅ **Token mint runs 4 queries where 2 suffice** (groups + assignments queried |
| 112 | + twice across `get_token`/`scopes_for_user`). Fixed: groups + assignments |
| 113 | + queried once in `get_token` and passed into `scopes_for_user` via new |
| 114 | + `group_names` / `has_review_assignments` kwargs. |
| 115 | + |
| 116 | +--- |
| 117 | + |
| 118 | +## 4. Infrastructure / consistency |
| 119 | + |
| 120 | +- ✅ **DONE** — **S3-vs-R2 client divergence** (decided: prod is **AWS S3**). |
| 121 | + All three sites now share one S3 contract: `get_s3_client` honors |
| 122 | + `AWS_S3_ENDPOINT` (optional S3-compatible host) and the R2 `ACCOUNT_ID` |
| 123 | + branching is gone (backend + `cms/datastore/services.py` + |
| 124 | + `cms/config/settings/production.py`). The `R2_BUCKET_NAME`-misnomer TODO is |
| 125 | + resolved: backend reads the bucket through a new `Settings.s3_bucket` |
| 126 | + property (`R2_BUCKET_NAME or AWS_S3_BUCKET`, matching cms `GPKG_BUCKET`); the |
| 127 | + legacy secret name is kept (it's live) but documented. The dead `ACCOUNT_ID` |
| 128 | + / `R2_ACCOUNT_ID` fields were removed (`extra="ignore"` makes a stale prod |
| 129 | + secret harmless). NOTE: backend now reads `AWS_S3_ENDPOINT` where it didn't |
| 130 | + before — if a prod `ACCOUNT_ID`/R2 secret is still set it is now ignored, so |
| 131 | + confirm prod is genuinely on S3 before/at cutover. |
| 132 | +- ✅ **DONE** — **Permission-grant migration boilerplate ×3** (`datastore/0002`, |
| 133 | + `galleries/0002`, `authapi/0003`). Extracted `core/migration_utils.py` |
| 134 | + (`ensure_permissions` + `model_permissions`); all three migrations now use it. |
| 135 | + The `create_permissions` fresh-DB footgun lives in one documented place. |
| 136 | +- ⏳ **DEFERRED** — **Datastore admin tool views** ([cms/datastore/views.py](../cms/datastore/views.py)): |
| 137 | + four hand-rolled form views with permission declared twice (view decorator + |
| 138 | + menu item kwarg). A small shared FormView base would make the fifth tool |
| 139 | + safe to add by construction. Left as-is — pure refactor, lower value. |
| 140 | +- ✅ **DONE (partial)** — **cms fetch wrappers** ([app/src/app/utils/api/cmsContent.ts](../app/src/app/utils/api/cmsContent.ts)): |
| 141 | + the three per-function try/fetch/null blocks now share one `cmsFetch<T>()`. |
| 142 | + CMS URL resolution across three places (`auth.ts`, `admin/config.ts`, |
| 143 | + `cmsContent.ts`) NOT consolidated — `auth.ts` is the critical NextAuth path |
| 144 | + with server-only semantics; left untouched to avoid risk. |
| 145 | +- ⏳ **DEFERRED** — **Dev JWT keys are ephemeral per process** |
| 146 | + ([cms/config/settings/dev.py](../cms/config/settings/dev.py)): fine for the |
| 147 | + single-process compose runserver, but `manage.py shell` mints tokens the |
| 148 | + server won't verify. Run `manage.py generate_jwt_keys` and pin them in |
| 149 | + `cms/.env.docker` if this bites. Also: `KidTokenBackend` mirrors SimpleJWT |
| 150 | + internals — re-check on any SimpleJWT upgrade. |
| 151 | +- **`bd` issue tracker is broken locally** (`pending schema migrations alter |
| 152 | + pre-existing dirty tables`) — no beads issues were filed for any of this |
| 153 | + branch; repair bd and backfill if the team wants tracker history. |
| 154 | + |
| 155 | +--- |
| 156 | + |
| 157 | +## 5. Cutover-day checklist (operational) |
| 158 | + |
| 159 | +> **AWS-first (2026-08-05, Fly is being deprecated):** the CMS now has a full |
| 160 | +> AWS home — `infra/cms.ts` (Fargate service + cms-migrate release task, host |
| 161 | +> rule `cms.districtr.org` / `cms.dev.districtr.org`, cert SAN + DNS record in |
| 162 | +> the `dnsRecords` output) and `.github/workflows/deploy-cms.yml`. The backend |
| 163 | +> and frontend task envs are already pointed at the CMS issuer (no Auth0 config |
| 164 | +> remains in `infra/`). S3 auth uses the task role |
| 165 | +> (`AWS_USE_DEFAULT_CREDENTIALS`); ALB health checks hit `/healthz` |
| 166 | +> (host-validation-exempt middleware). If cutover happens on AWS, step 2 below |
| 167 | +> replaces Fly secrets with Pulumi config. |
| 168 | +
|
| 169 | +1. DB snapshot. |
| 170 | +2. Secrets. **AWS**: `pulumi config set --secret` per stack — |
| 171 | + `djangoSecretKey`, `jwtSigningKey`/`jwtVerifyingKey` |
| 172 | + (`manage.py generate_jwt_keys`), `authSecret`, `resendApiKey` |
| 173 | + (+ verify the Resend sending domain); create the `cms.*.districtr.org` |
| 174 | + DNS records from the `dnsRecords` output. |
| 175 | + **Fly (only if cutover precedes the AWS migration)** — **cms**: |
| 176 | + `JWT_SIGNING_KEY`/`JWT_VERIFYING_KEY`, `DJANGO_SECRET_KEY`, |
| 177 | + `RESEND_API_KEY`, DB + storage creds; |
| 178 | + **api**: `AUTH_JWKS_URL=https://districtr-v2-cms.fly.dev/.well-known/jwks.json`, |
| 179 | + `AUTH_ISSUER`, `AUTH_AUDIENCE`; **frontend**: `AUTH_SECRET`, `CMS_URL`. |
| 180 | +3. Staging rehearsal on the `-dev` Fly apps first (full sequence below, plus a |
| 181 | + backend `alembic revision --autogenerate` afterward proving an empty diff). |
| 182 | +4. Merge → CI deploys api/app/cms (release commands run both migration systems). |
| 183 | +5. `manage.py migrate_tiptap --dry-run` → review report → real run **with |
| 184 | + `--owners "auth0|<sub>=<email>,..."`** (map the two legacy author subjects |
| 185 | + to provisioned users so their pages stay editable under own-content-only). |
| 186 | +5b. In the Wagtail admin, add a "Static pages" index page under Home |
| 187 | + (StaticPage type, new 2026-08-04): static site pages migrate into the CMS |
| 188 | + one at a time — delete the hardcoded Next.js route, publish a StaticPage |
| 189 | + with the same slug (the `/[slug]` catch-all serves it). |
| 190 | +6. `manage.py provision_users users.csv` (CSV: email,name,group — groups are |
| 191 | + now `admin`/`partner`/`super_partner`, consolidated 2026-08-05 by |
| 192 | + authapi.0007) — sends password-setup emails. |
| 193 | +7. Smoke: Wagtail login, edit+publish a page, comment moderation in the |
| 194 | + Wagtail admin (Comment review menu → /admin/moderation/comments/, moved |
| 195 | + in-CMS 2026-08-05; the Next.js /admin pages are gone), under-construction |
| 196 | + toggle (Settings → Frontend settings), thumbnail regen, gallery publish, |
| 197 | + compose-map dry call. |
| 198 | +8. **Disable** (don't delete) the Auth0 tenant; delete after two quiet weeks. |
| 199 | +9. One month post-cutover: ship the migration renaming `cms.tags_content` / |
| 200 | + `cms.places_content` → `*_legacy` (they must survive until then — |
| 201 | + `migrate_tiptap` reads them; alembic's `include_object` already ignores the |
| 202 | + `cms` schema). |
0 commit comments