fix(db): don't use STR_TO_DATE in a generated column (MariaDB 12.3)

[gantoine]

Description

Migration 0098_generated_metadata_columns parsed the gamelist YYYYMMDDThhmmss date with STR_TO_DATE inside a GENERATED ALWAYS AS ... STORED column. MariaDB treats STR_TO_DATE as non-deterministic (it reads lc_time_names) and forbids it there, so the migration aborts with error 1901 and RomM never starts.

The regex guard already pins that value to a fixed-width YYYYMMDDThhmmss, so no parsing function is needed: the string is reshaped into a canonical YYYY-MM-DD hh:mm:ss literal with SUBSTRING/CONCAT and handed to the existing TIMESTAMPDIFF(SECOND, '1970-01-01 00:00:00', ...). Stored values and their session-time-zone independence are unchanged. PostgreSQL is untouched.

Why this only hit some users

The restriction is only enforced starting in MariaDB 12.3, which is why the alpha migrated fine for most people. Tested against the original expression:

MariaDB	Original STR_TO_DATE vcol
11.3.2, 11.4.12, 11.8.8	accepted
12.0.2, 12.1.2, 12.2.2	accepted
12.3.2	rejected, error 1901

Heads-up for anyone already on 5.1.0-alpha.1

The check runs every time MariaDB opens the table, not just at DDL time. So a roms table created by the original 0098 on MariaDB < 12.3 keeps working on that server, but becomes completely unopenable the moment MariaDB itself is upgraded to 12.3+:

SELECT * FROM roms;                 -> ERROR 1901
INSERT INTO roms ...;               -> ERROR 1901
ALTER TABLE roms DROP COLUMN ...;   -> ERROR 1901
CHECK TABLE roms;                   -> Corrupt

Even ALTER and DROP COLUMN fail, so it cannot be repaired in place once the server is on 12.3. This PR fixes fresh installs only; existing databases need the manual fix below (or a backup restore). We deliberately skipped shipping a repair migration since the release is in alpha and users have backups.

MANUAL FIX for existing databases (click to expand)

Take a backup first. This only applies to MariaDB/MySQL; PostgreSQL is unaffected.

Case 1 - still on MariaDB < 12.3 (RomM is running fine). Do this before you ever upgrade MariaDB. Run against your RomM database:

ALTER TABLE roms MODIFY COLUMN generated_first_release_date BIGINT GENERATED ALWAYS AS (
CASE
    WHEN JSON_CONTAINS_PATH(manual_metadata, 'one', '$.first_release_date') AND JSON_UNQUOTE(JSON_EXTRACT(manual_metadata, '$.first_release_date')) NOT IN ('null', 'None', '0', '0.0') AND JSON_UNQUOTE(JSON_EXTRACT(manual_metadata, '$.first_release_date')) REGEXP '^[0-9]+$' THEN CAST(JSON_UNQUOTE(JSON_EXTRACT(manual_metadata, '$.first_release_date')) AS SIGNED)
    WHEN JSON_CONTAINS_PATH(igdb_metadata, 'one', '$.first_release_date') AND JSON_UNQUOTE(JSON_EXTRACT(igdb_metadata, '$.first_release_date')) NOT IN ('null', 'None', '0', '0.0') AND JSON_UNQUOTE(JSON_EXTRACT(igdb_metadata, '$.first_release_date')) REGEXP '^[0-9]+$' THEN CAST(JSON_UNQUOTE(JSON_EXTRACT(igdb_metadata, '$.first_release_date')) AS SIGNED) * 1000
    WHEN JSON_CONTAINS_PATH(ss_metadata, 'one', '$.first_release_date') AND JSON_UNQUOTE(JSON_EXTRACT(ss_metadata, '$.first_release_date')) NOT IN ('null', 'None', '0', '0.0') AND JSON_UNQUOTE(JSON_EXTRACT(ss_metadata, '$.first_release_date')) REGEXP '^[0-9]+$' THEN CAST(JSON_UNQUOTE(JSON_EXTRACT(ss_metadata, '$.first_release_date')) AS SIGNED) * 1000
    WHEN JSON_CONTAINS_PATH(ra_metadata, 'one', '$.first_release_date') AND JSON_UNQUOTE(JSON_EXTRACT(ra_metadata, '$.first_release_date')) NOT IN ('null', 'None', '0', '0.0') AND JSON_UNQUOTE(JSON_EXTRACT(ra_metadata, '$.first_release_date')) REGEXP '^[0-9]+$' THEN CAST(JSON_UNQUOTE(JSON_EXTRACT(ra_metadata, '$.first_release_date')) AS SIGNED) * 1000
    WHEN JSON_CONTAINS_PATH(launchbox_metadata, 'one', '$.first_release_date') AND JSON_UNQUOTE(JSON_EXTRACT(launchbox_metadata, '$.first_release_date')) NOT IN ('null', 'None', '0', '0.0') AND JSON_UNQUOTE(JSON_EXTRACT(launchbox_metadata, '$.first_release_date')) REGEXP '^[0-9]+$' THEN CAST(JSON_UNQUOTE(JSON_EXTRACT(launchbox_metadata, '$.first_release_date')) AS SIGNED) * 1000
    WHEN JSON_CONTAINS_PATH(flashpoint_metadata, 'one', '$.first_release_date') AND JSON_UNQUOTE(JSON_EXTRACT(flashpoint_metadata, '$.first_release_date')) NOT IN ('null', 'None', '0', '0.0') AND JSON_UNQUOTE(JSON_EXTRACT(flashpoint_metadata, '$.first_release_date')) REGEXP '^[0-9]+$' THEN CAST(JSON_UNQUOTE(JSON_EXTRACT(flashpoint_metadata, '$.first_release_date')) AS SIGNED) * 1000
    WHEN JSON_CONTAINS_PATH(gamelist_metadata, 'one', '$.first_release_date') AND JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')) NOT IN ('null', 'None', '0', '0.0') AND JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')) REGEXP '^[0-9]{8}T[0-9]{6}$' AND CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 1, 4) AS SIGNED) >= 1 AND CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 5, 2) AS SIGNED) BETWEEN 1 AND 12 AND CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 7, 2) AS SIGNED) BETWEEN 1 AND (CASE CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 5, 2) AS SIGNED) WHEN 2 THEN IF(((CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 1, 4) AS SIGNED) % 4 = 0 AND CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 1, 4) AS SIGNED) % 100 != 0) OR CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 1, 4) AS SIGNED) % 400 = 0), 29, 28) WHEN 4 THEN 30 WHEN 6 THEN 30 WHEN 9 THEN 30 WHEN 11 THEN 30 ELSE 31 END) AND CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 10, 2) AS SIGNED) <= 23 AND CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 12, 2) AS SIGNED) <= 59 AND CAST(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 14, 2) AS SIGNED) <= 59 THEN TIMESTAMPDIFF(SECOND, '1970-01-01 00:00:00', CONCAT(SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 1, 4), '-', SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 5, 2), '-', SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 7, 2), ' ', SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 10, 2), ':', SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 12, 2), ':', SUBSTRING(JSON_UNQUOTE(JSON_EXTRACT(gamelist_metadata, '$.first_release_date')), 14, 2))) * 1000
    ELSE NULL END
) STORED;

With Docker Compose:

docker compose exec -T <db-service> mariadb -uroot -p<password> <romm-db> < fix.sql

Verify with CHECK TABLE roms; - it should report OK.

This ALTER is a full table rebuild - plan for it

Changing a STORED generated column's expression can only run with ALGORITHM=COPY; INSTANT, NOCOPY and INPLACE are all rejected by MariaDB, so there is no online variant and the table is write-locked for the duration. Cost scales with the total size of roms (the metadata JSON blobs dominate), because every generated column is re-evaluated for every row.

Measured on MariaDB 11.3.2 in a container: ~4s for 131k rows / 47MB, ~35s for 1M rows / 273MB. A large library with big metadata blobs can take considerably longer.

Things that do not help (measured, no meaningful difference): SET GLOBAL innodb_flush_log_at_trx_commit=0, SET SESSION sql_log_bin=0, or dropping idx_roms_generated_first_release_date beforehand. It is CPU-bound on JSON re-evaluation, not IO-bound.

What actually helps:

• Stop RomM first, leaving only the database up. A running RomM holds metadata locks and open transactions, which can stall the ALTER before it even starts copying, and its writes will block once it does.
• Watch it progress from a second shell so you can tell it is working rather than hung:

SELECT ID, STAGE, MAX_STAGE, ROUND(PROGRESS,1) AS pct_in_stage,
       TIME_MS DIV 1000 AS secs, STATE
FROM information_schema.PROCESSLIST
WHERE INFO LIKE 'ALTER TABLE roms%';

You should see copy to tmp table with pct_in_stage climbing. Let it finish - interrupting it leaves the original table intact but wastes the work.

Case 2 - already upgraded to MariaDB 12.3+ and RomM won't start. The table cannot be altered in this state, so roll MariaDB back first:

1. Stop RomM and MariaDB.
2. Pin the image back to your previous version (e.g. mariadb:12.2) and start only the database.
3. Run the ALTER TABLE from Case 1.
4. Confirm CHECK TABLE roms; reports OK.
5. Put the MariaDB image back to 12.3+ and start everything.

Restoring the pre-upgrade backup and applying Case 1 before re-upgrading works equally well.

Case 3 - never successfully ran 0098 (the migration aborted). Nothing to do; the ALTER fails as a unit, so no partial columns are left behind. Just update to a build containing this PR.

Checklist

• I've tested the changes locally
• I've updated relevant comments
• I've assigned reviewers for this PR
• I've added unit tests that cover the changes

Testing

Verified against real MariaDB containers, not just static review:

• Repro'd error 1901 on 12.3.2 with the original expression; the fixed expression applies cleanly on 11.3.2 and 12.3.2.
• Full generated-column ALTER TABLE from the migration applies with no other expression rejected.
• Values unchanged and session-time-zone independent - rows inserted under +05:30, -08:00 and +09:00: 19980521T134500 -> 895758300000 and 20240229T235959 -> 1709251199000, both matching Python's UTC computation. Malformed and null gamelist dates -> NULL; the integer branches (igdb x1000, manual x1) are unaffected.
• Confirmed the code-review finding: the 8+6 digit regex admits calendar-invalid values, and under the default STRICT_TRANS_TABLES a single pre-existing 20201301T000000 row aborts the migration ALTER with Incorrect datetime value. CAST(... AS DATETIME) (which would yield NULL) is itself barred from a generated column, so the components are range-checked in the CASE guard instead.
• Invalid dates now yield NULL with no error on MariaDB 12.3.2 and MySQL 8.4.10: month 13, day 30 in February, day 31 in April, Feb 29 in a non-leap year, Feb 29 in 1900 (century non-leap), day 0, all-zeroes, hour 25, minute 60. Leap-year handling verified both ways: 2020/2024/2000 Feb 29 accepted, 2021/1900 rejected.
• The migration ALTER now succeeds on a table already holding invalid rows; they land as NULL while valid siblings compute correctly.
• Walked the full manual-fix path on a persistent datadir: original 0098 on 12.2 -> upgrade to 12.3.2 (breaks) -> roll back to 12.2 -> apply the ALTER above -> re-upgrade to 12.3.2 -> SELECT, INSERT and CHECK TABLE roms all OK with values preserved.

No unit test added: the failure only reproduces against a live MariaDB 12.3 server, which CI does not currently run.

AI assistance disclosure

This change was written with AI assistance (Claude Code). The AI diagnosed the root cause, identified the 12.3 version boundary and the table-open failure mode, wrote the replacement expression and the manual fix, and ran the MariaDB container testing described above. All of it was reviewed by me before submitting.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[greptile-apps]

Greptile Summary

This PR makes the MariaDB release-date generated column compatible with MariaDB 12.3. The main changes are:

• Replaces STR_TO_DATE with deterministic string reshaping.
• Builds a canonical datetime value with SUBSTRING and CONCAT.
• Keeps PostgreSQL and the other metadata branches unchanged.

Confidence Score: 4/5

The calendar-invalid input path needs a guard before merging.

• Fixed-width validation does not validate month, day, or time ranges.
• Strict MariaDB configurations can reject the resulting datetime during migration or row writes.

backend/alembic/versions/0098_generated_metadata_columns.py

Important Files Changed

Filename	Overview
backend/alembic/versions/0098_generated_metadata_columns.py	Replaces the MariaDB generated-column date parser, but calendar-invalid values can reach an implicit datetime conversion.

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
backend/alembic/versions/0098_generated_metadata_columns.py:137
**Invalid Dates Can Abort Writes**

The regex accepts calendar-invalid values such as `20201301T000000`, which this expression reshapes into an invalid datetime string. On MariaDB configurations that reject invalid implicit datetime conversions, evaluating the stored column can abort the migration for an existing row or reject a later insert instead of producing `NULL`; validate the calendar components before passing the value to `TIMESTAMPDIFF`.

_{Reviews (1): Last reviewed commit: "fix(db): don't use STR_TO_DATE in a gene..." | Re-trigger Greptile}

[Spinnich]

I know we basically already talked about this. But figured I'd throw my findings below for an automatic upgrade path for the few users who moved to alpha.1 and are currently on MariaDB v11 but may migrate to v12 in the future. Anyways, here's Claude lol.

Fix looks correct, and I like that the calendar_valid guard NULLs out impossible dates instead of aborting the INSERT under STRICT mode. Skipping the repair migration is a reasonable call for an alpha, especially since a DB already on 12.3 can't be repaired in place at all.

One thing worth considering though: the edit only helps installs that haven't run 0098 yet. A DB that already ran the original 0098 on MariaDB < 12.3 keeps the STR_TO_DATE definition baked into the table, so it stays a latent break until the day that server is bumped to 12.3+. If we want to proactively rescue that still-on-11.x cohort (fix them before they hit the landmine), it takes a small follow-up migration. Sketch:

def upgrade() -> None:
    connection = op.get_bind()
    if is_postgresql(connection):
        return  # PostgreSQL never used STR_TO_DATE

    # information_schema reads don't open the table, so this is safe even on a
    # 12.3 server where the table itself is unopenable.
    current = connection.execute(
        sa.text(
            "SELECT GENERATION_EXPRESSION FROM information_schema.COLUMNS "
            "WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'roms' "
            "AND COLUMN_NAME = 'generated_first_release_date'"
        )
    ).scalar()
    if not current or "str_to_date" not in current.lower():
        return  # already portable (fresh install / fixed 0098) -> no-op

    # Same expression 0098 now builds; MODIFY keeps the index in place.
    expr = _maria_first_release_date()
    op.execute(
        "ALTER TABLE roms MODIFY COLUMN generated_first_release_date "
        f"BIGINT GENERATED ALWAYS AS ({expr}) STORED"
    )

Key properties, all verified against real MariaDB containers:

• No-op where it should be. The information_schema guard skips fresh installs (which get the fixed 0098) and PostgreSQL, so it only rebuilds a genuinely stale column. On MariaDB 11.4 a single MODIFY COLUMN flips the baked definition, preserves idx_roms_generated_first_release_date, recomputes values (valid dates unchanged, invalid ones stay NULL), and CHECK TABLE reports OK.
• Reuse the expression, don't fork it. Building from the fixed _maria_first_release_date() keeps the two definitions from drifting.
• The hard limit. This can only fix a server still on < 12.3. On 12.3+ the table is already unopenable and even ALTER ... MODIFY fails with ERROR 1901 (the guard read still succeeds, but the ALTER can't run), so those DBs still need the manual rollback path from the description. A nice side effect: anyone who follows that "roll MariaDB back to < 12.3" step gets auto-fixed by this migration on the next startup instead of running the ALTER by hand.

Totally fine to leave it out for the alpha if the still-on-11.x population isn't worth the extra migration; just flagging the option and the exact shape it would take if we do want it.