perf(backend): materialize virtual collection membership

[gantoine]

Description

Fixes #3881.

virtual_collections was a view that unnested five JSON arrays per rom on every read, so both the collections list and opening a single collection scanned the whole library. The second case is the surprising one: the collection name is computed inside the view, so nothing can be narrowed down early and a collection with 5 roms costs the same as one with 5,000. #3826 sped up roms_metadata with stored generated_* columns but left this view deriving on read; this gives it the same treatment.

Membership now lives in a real table, virtual_collection_roms, with one narrow row per (type, name, rom_id) plus that rom's cover paths. It is backfilled once from the generated_* columns and kept in sync by triggers on roms (virtual_collection_roms_ai/_au on MariaDB/MySQL, virtual_collection_roms_aiu calling romm_sync_virtual_collection_roms() on PostgreSQL); rom deletions ride the foreign key's cascade. No application hook and no background job, same as 0098. The update trigger short-circuits unless a tracked column actually changed, so ordinary rom writes cost nothing extra.

virtual_collections is redefined as a plain aggregate over that table, so the VirtualCollection model, the response schemas and the API shape are unchanged. Filtering roms by a virtual collection is now an indexed subquery instead of an IN list of every member id.

Behavior changes worth calling out

• The list response now carries at most 5 covers per virtual collection (matching SMART_COLLECTION_MAX_COVERS). Aggregating every member's cover was megabytes per request on a large library, and both v1 and v2 slice to 4.
• PostgreSQL descriptions now read Autogenerated <name> collection, like MariaDB's. They previously interpolated the type on PostgreSQL and the name on MariaDB.
• The one-time backfill during the migration pays the old derive-once cost, once, instead of on every request.

Performance

Synthetic 50k-rom library on MariaDB 11.3, old view recreated side by side for comparison:

Query	Before	After
Collections list	1.42s	0.128s
Single collection	0.111s	0.0006s

The single-collection lookup is an index lookup on (type, name) on both engines (confirmed with EXPLAIN), so it no longer scales with library size. The synthetic metadata blobs here are far smaller than the ~1 GB roms table in the report, so real-world absolute numbers should improve much more than these.

Equivalence was checked against the old view on the same dataset: identical set of collections (2848 on both) and identical rom_ids on sampled rows.

Testing

• Full backend suite on MariaDB 11.3: 2278 passed.
• Full backend suite on PostgreSQL 16: 2273 passed, 5 pre-existing failures unrelated to this change (local-timezone assertions in test_saves.py/test_collection.py comparing a non-UTC local zone against UTC; CI runs UTC).
• alembic upgrade / downgrade / re-upgrade verified on both engines.
• New tests cover membership following metadata edits, rom deletion, cover capping, and rom filtering by virtual collection.

Checklist

• I've tested the changes locally
• I've updated relevant comments
• I've assigned reviewers for this PR
• I've added unit tests that cover the changes

AI assistance disclosure

This change was written with AI assistance (Claude Code). The design (trigger-maintained membership table, capped covers), implementation, migration, tests and benchmarks were produced with the agent and reviewed by me; all timings and the old/new equivalence check were run against local MariaDB and PostgreSQL instances.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[greptile-apps]

Greptile Summary

This PR materializes virtual collection membership to speed up collection reads. The main changes are:

• Adds a trigger-maintained membership table for MariaDB, MySQL, and PostgreSQL.
• Rebuilds the virtual collections view over indexed membership rows.
• Fetches at most five cover paths per collection.
• Filters ROMs with an indexed membership subquery.
• Adds tests for synchronization, deletion, covers, and filtering.

Confidence Score: 5/5

This looks mergeable with a small data-integrity hardening change.

• The backfill and triggers use the same generated metadata columns.
• The new query and cover paths match the materialized schema.
• Names that differ only after 400 characters are currently merged.

backend/alembic/versions/0100_virtual_collection_roms.py

Important Files Changed

Filename	Overview
backend/alembic/versions/0100_virtual_collection_roms.py	Creates and synchronizes materialized membership, but truncating names can merge distinct provider values.
backend/handler/database/collections_handler.py	Adds capped cover hydration and an indexed membership-ID query.
backend/handler/database/roms_handler.py	Replaces in-memory membership filtering with a database subquery.
backend/models/collection.py	Maps membership rows and changes cover lists to handler-populated attributes.
backend/tests/handler/database/test_collections_handler.py	Covers membership updates, deletion cascades, cover limits, and ROM filtering.

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
backend/alembic/versions/0100_virtual_collection_roms.py:74
**Truncation Merges Distinct Collections**

When two provider values share their first 400 characters, `LEFT(j.value, 400)` stores both under the same primary-key name. Their ROM memberships are then combined into one virtual collection, so callers receive incorrect collection names and membership instead of two distinct collections.

_{Reviews (1): Last reviewed commit: "perf(backend): materialize virtual colle..." | Re-trigger Greptile}

[greptile-apps]

Truncation Merges Distinct Collections

When two provider values share their first 400 characters, LEFT(j.value, 400) stores both under the same primary-key name. Their ROM memberships are then combined into one virtual collection, so callers receive incorrect collection names and membership instead of two distinct collections.

Prompt To Fix With AI

This is a comment left during a code review.
Path: backend/alembic/versions/0100_virtual_collection_roms.py
Line: 74

Comment:
**Truncation Merges Distinct Collections**

When two provider values share their first 400 characters, `LEFT(j.value, 400)` stores both under the same primary-key name. Their ROM memberships are then combined into one virtual collection, so callers receive incorrect collection names and membership instead of two distinct collections.

How can I resolve this? If you propose a fix, please make it concise.