Skip to content

feat(modmail): migration script - #325

Merged
didinele merged 2 commits into
mainfrom
feat/modmail-migration
Aug 10, 2026
Merged

feat(modmail): migration script#325
didinele merged 2 commits into
mainfrom
feat/modmail-migration

Conversation

@didinele

@didinele didinele commented Aug 10, 2026

Copy link
Copy Markdown
Member

Closes #157

@vercel

vercel Bot commented Aug 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
chatsift-website Ready Ready Preview Aug 10, 2026 9:45am

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 320b5d55-992a-408d-a570-8210c579c4dc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Added a one-off legacy ModMail migration with dry-run, live, and verification modes. The migration maps legacy records into the current schema, validates conflicts, remaps relationships, reports results, and supports migrated DM-origin threads whose channels may be unavailable.

Changes

Legacy ModMail migration

Layer / File(s) Summary
Migration contract and execution setup
packages/private/db/src/scripts/migrateLegacyModmail.ts, package.json, packages/private/db/tsconfig.json, docs/roadmap/01-architecture.md, docs/roadmap/06-modmail-port.md
Defines migration modes, source and target row mappings, identity allocation, command execution, compiler support, and documented migration behavior.
Legacy data transfer
packages/private/db/src/scripts/migrateLegacyModmail.ts
Migrates guild settings, snippets, blocks, alerts, threads, messages, and scheduled closes in batches while remapping relationships and assigning DM origins.
Migration safeguards and verification
packages/private/db/src/scripts/migrateLegacyModmail.ts, docs/roadmap/06-modmail-port.md
Adds preflight checks, conflict handling, transactional dry runs, statistics, environment validation, read-only verification, and exit-status handling.
Migrated thread runtime support
services/api/src/routes/modmail/threads/util.ts
Treats Discord 403 and 404 responses as unavailable tag channels and returns an empty applied-tag list.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant migrateLegacyModmail
  participant LegacyDatabase
  participant TargetDatabase
  Operator->>migrateLegacyModmail: Select migration or verification mode
  migrateLegacyModmail->>LegacyDatabase: Read and validate legacy ModMail data
  migrateLegacyModmail->>TargetDatabase: Write mapped records or compare migrated data
  TargetDatabase-->>migrateLegacyModmail: Return migration or verification results
  migrateLegacyModmail-->>Operator: Print statistics and exit status
Loading

Possibly related issues

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: adding a ModMail migration script.
Description check ✅ Passed The description references issue #157, which is related to the ModMail migration changes.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/modmail-migration

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
packages/private/db/src/scripts/migrateLegacyModmail.ts (2)

999-1005: 🩺 Stability & Availability | 🔵 Trivial

Confirm the target's transaction timeouts before the live run.

The whole migration runs inside one target transaction while the script round-trips to the legacy database between writes. If the target enforces idle_in_transaction_session_timeout or statement_timeout, a large ThreadMessage table can abort the run part way through. Consider setting both to 0 for this session at the start of the transaction, and record the expected wall-clock duration in the cutover runbook.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/private/db/src/scripts/migrateLegacyModmail.ts` around lines 999 -
1005, At the start of the target transaction callback in the live migration
path, configure the transaction session with both
idle_in_transaction_session_timeout and statement_timeout set to 0 before
calling runMigration; preserve the dry-run rollback behavior. Also document the
migration’s expected wall-clock duration in the cutover runbook.

863-876: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Also report target threads that have no legacy counterpart.

The loop iterates legacyPerThread only. A migrated thread present in the target but absent from legacy is never reported. That case arises after a partially applied run, which is exactly when an operator reaches for --verify. Add a reverse pass over targetPerThread.

♻️ Proposed addition
 	const mismatched: string[] = [];
 	for (const [modThreadId, expected] of legacyPerThread) {
 		const actual = targetPerThread.get(modThreadId);
 		if (actual !== expected) {
 			mismatched.push(`${modThreadId} (legacy=${expected} target=${actual ?? 'missing'})`);
 		}
 	}
+
+	for (const modThreadId of targetPerThread.keys()) {
+		if (!legacyPerThread.has(modThreadId)) {
+			mismatched.push(`${modThreadId} (legacy=missing target=${targetPerThread.get(modThreadId)})`);
+		}
+	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/private/db/src/scripts/migrateLegacyModmail.ts` around lines 863 -
876, Extend the verification logic around legacyPerThread and targetPerThread to
also iterate targetPerThread and append any thread IDs absent from
legacyPerThread to mismatched, including the target message count in the
diagnostic. Keep the existing mismatch reporting and success behavior, ensuring
extra target-only threads make verification fail.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/roadmap/06-modmail-port.md`:
- Line 122: Update the GuildSettings-to-guild_settings mapping entry to state
that mod_forum_id is never written by the migration script for any guild, rather
than instructing operators to confirm the legacy channel type during the run.
Reference item 1 for the rationale and required manual handling of forum-channel
setup.

In `@packages/private/db/src/scripts/migrateLegacyModmail.ts`:
- Around line 659-661: Guard the collidingSnippets query in
packages/private/db/src/scripts/migrateLegacyModmail.ts:659-661 by moving it
inside the existing guildIds.length > 0 block. At
packages/private/db/src/scripts/migrateLegacyModmail.ts:751-756, guard the
guildIds query/use at line 731 similarly and short-circuit with
report('snippet_updates', 0, 0) when both migratedLegacyIds and
migratedTargetIds are empty.

---

Nitpick comments:
In `@packages/private/db/src/scripts/migrateLegacyModmail.ts`:
- Around line 999-1005: At the start of the target transaction callback in the
live migration path, configure the transaction session with both
idle_in_transaction_session_timeout and statement_timeout set to 0 before
calling runMigration; preserve the dry-run rollback behavior. Also document the
migration’s expected wall-clock duration in the cutover runbook.
- Around line 863-876: Extend the verification logic around legacyPerThread and
targetPerThread to also iterate targetPerThread and append any thread IDs absent
from legacyPerThread to mismatched, including the target message count in the
diagnostic. Keep the existing mismatch reporting and success behavior, ensuring
extra target-only threads make verification fail.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 58f1c620-f9ef-42d7-aa9c-26fe300bf7b8

📥 Commits

Reviewing files that changed from the base of the PR and between bfa4395 and 8cc4ac7.

📒 Files selected for processing (6)
  • docs/roadmap/01-architecture.md
  • docs/roadmap/06-modmail-port.md
  • package.json
  • packages/private/db/src/scripts/migrateLegacyModmail.ts
  • packages/private/db/tsconfig.json
  • services/api/src/routes/modmail/threads/util.ts

Comment thread docs/roadmap/06-modmail-port.md Outdated
Comment thread packages/private/db/src/scripts/migrateLegacyModmail.ts
Comment thread packages/private/db/src/scripts/migrateLegacyModmail.ts
@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown

Code review

Found 3 high-signal issues (validated independently against the actual source):

1. postgres.ISql is not a real exported type — build-breaking

* every helper to know which one it was handed.
*/
type Executor = postgres.ISql;
// Big enough that the round-trip count stays trivial for a dataset this size, small enough that a

postgres.js (postgres@^3.4.9) exports Sql, TransactionSql, ReservedSql — there is no ISql. The sibling file packages/private/db/src/index.ts already uses the correct name for the identical purpose: export type Database = postgres.Sql;. This fails to compile with TS2694 ("Namespace has no exported member 'ISql'"), which breaks turbo run build --filter=@chatsift/db and therefore the migrate:legacy-modmail script entirely.

Suggested fix: type Executor = postgres.Sql; (matches the doc comment's stated intent, since TransactionSql extends Sql).

2. IS_PRODUCTION parsing silently diverges from the shared z.stringbool() schema

function resolveTargetUrl(): string {
// Mirrors `@chatsift/backend-core`'s `createDatabase()` without importing it (see the file header
// for why). `z.stringbool()` is what parses IS_PRODUCTION there; this is the same accepted set.
const isProduction = ['true', '1', 'yes', 'on'].includes((IS_PRODUCTION ?? '').toLowerCase());
const url = isProduction ? DATABASE_URL_PROD : DATABASE_URL_DEV;

The comment on L950-951 claims this mirrors z.stringbool() (used for the same IS_PRODUCTION var in @chatsift/backend-core's shared env schema), but it doesn't:

  • zod v4's stringbool() truthy set is ["true", "1", "yes", "on", "y", "enabled"] — this script's array omits "y" and "enabled".
  • z.stringbool() throws on an unrecognized value; this script's .includes(...) silently evaluates to false (dev) instead.

Failure scenario: an operator sets IS_PRODUCTION=y or IS_PRODUCTION=enabled (both valid everywhere else in the stack). Every other service resolves to production config, but this script silently resolves isProduction = false and targets DATABASE_URL_DEV — with --live committing real inserts there, no error or warning either way.

3. --verify guild-scoping mismatch produces false FAIL results

const guildIds = (
await legacy<{ guildId: string }[]>`
SELECT "guildId" AS guild_id FROM "GuildSettings"
UNION
SELECT DISTINCT "guildId" AS guild_id FROM "Thread"
`
).map((row) => row.guildId);
console.log('\nRow counts');
// Note which tables are *self-cancelling* under an ON CONFLICT DO NOTHING skip and which are not.
// `guild_settings`/`snippets`/`blocks`/`thread_open_alerts` skip a legacy row precisely because a
// target row with the same key already exists, so the guild-scoped target count still matches the
// legacy one -- a skip is invisible here, correctly. `snippet_updates` is the one exception: the
// updates belonging to a skipped snippet have no counterpart at all, so it is reconciled separately
// below against only those snippets that actually landed.
const [legacyCounts] = await legacy<[Record<string, string>]>`
SELECT
(SELECT COUNT(*) FROM "GuildSettings") AS guild_settings,
(SELECT COUNT(*) FROM "Snippet") AS snippets,
(SELECT COUNT(*) FROM "Block") AS blocks,
(SELECT COUNT(*) FROM "ThreadOpenAlert") AS thread_open_alerts,
(SELECT COUNT(*) FROM "Thread") AS threads,
(SELECT COUNT(*) FROM "ThreadMessage") AS thread_messages,
(SELECT COUNT(*) FROM "ThreadReplyAlert") AS thread_reply_alerts,
(SELECT COUNT(*) FROM "ScheduledThreadClose") AS scheduled_thread_closes
`;
const [targetCounts] = await target<[Record<string, string>]>`
SELECT
(SELECT COUNT(*) FROM guild_settings WHERE guild_id = ANY(${guildIds})) AS guild_settings,
(SELECT COUNT(*) FROM snippets WHERE guild_id = ANY(${guildIds})) AS snippets,
(SELECT COUNT(*) FROM blocks WHERE guild_id = ANY(${guildIds})) AS blocks,
(SELECT COUNT(*) FROM thread_open_alerts WHERE guild_id = ANY(${guildIds})) AS thread_open_alerts,
(SELECT COUNT(*) FROM threads WHERE origin = 'dm' AND user_channel_id IS NULL) AS threads,
(SELECT COUNT(*) FROM thread_messages m JOIN threads t ON t.id = m.thread_id WHERE t.origin = 'dm' AND t.user_channel_id IS NULL) AS thread_messages,
(SELECT COUNT(*) FROM thread_reply_alerts a JOIN threads t ON t.id = a.thread_id WHERE t.origin = 'dm' AND t.user_channel_id IS NULL) AS thread_reply_alerts,
(SELECT COUNT(*) FROM scheduled_thread_closes c JOIN threads t ON t.id = c.thread_id WHERE t.origin = 'dm' AND t.user_channel_id IS NULL) AS scheduled_thread_closes
`;

guildIds (L786-792) is derived only from GuildSettings ∪ Thread. But migrateBlocks and migrateThreadOpenAlerts migrate every legacy row unconditionally, with no guild scoping tied to GuildSettings/Thread. The legacy-side counts (L802-812) are unfiltered COUNT(*), while the target-side counts (L814-824) filter WHERE guild_id = ANY(${guildIds}).

A legacy guild that has Block/ThreadOpenAlert rows but no GuildSettings row and no Thread row (nothing in the legacy or target schema prevents this — neither table has an FK to GuildSettings/Thread) gets counted on the legacy side but excluded from the target-side count, so runVerify reports FAIL blocks / FAIL thread_open_alerts on an otherwise-correct migration. This undermines the reconciliation tool the cutover runbook is meant to rely on.

Fix would need both sides symmetric — either scope the legacy counts by the same guildIds, or widen guildIds to also union in Block/ThreadOpenAlert (and Snippet) guild ids.


Note: posted as a single consolidated comment rather than per-line inline comments — the inline-comment tool this review normally uses wasn't available in this run's environment.

@didinele
didinele merged commit aba621e into main Aug 10, 2026
5 of 6 checks passed
@didinele
didinele deleted the feat/modmail-migration branch August 10, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Write ModMail old-to-new migration script (Thread/ThreadMessage map close to 1:1)

1 participant