Separate settings hierarchy for extensions + delete main settings that no longer exist from database

[stijn-uva]

The issue

The settings table is (name, value, tag) with a unique index on (name, tag). It carries no record of who declared a setting or when it was last seen.

Consequently:

• Settings left over from development and past refactors accumulate forever. They surface in the settings UI as text inputs in tabs under "Other", because manipulate_settings iterates {*all_settings.keys(), *definition.keys()} — anything in the DB gets rendered, declared or not. (webtool/views/views_admin.py:706)

• Why we can't just delete undeclared rows. A row can be undeclared for at least five different reasons, and right now they are indistinguishable:

  1. genuinely removed from 4CAT (safe to delete and what we're after)
  2. owned by an extension that is uninstalled (must keep — they may re-add it and we do not want behavior to change; plus would be annoying)
  3. owned by an extension that is installed but disabled (keep it)
  4. owned by a module that failed to import this boot — syntax error, missing dependency, a bug we ship and patch next week (must keep; module_loader.py:148-156 swallows this into missing_modules and only logs a warning)
  5. declared only on a first boot where module_config.bin did not yet exist (this is a little chicken/egg type thing that we just need to make sure doesn't cause problems i.e. accidental deletions)

• Two docstrings already promise this feature. ConfigManager.ensure_database claims "Deletes all stored settings not defined in 4CAT" — it does not; the deletion was dropped. ExtensionManipulator.uninstall_extension` claims "4CAT has general cleanup code for unreferenced settings" — but there is none.

Case 4 is the important one that worries me most and it applies to core processors just as much as to extensions. We do not want to delete a settings just because the module collector failed to import a module once.

[dale-wahl]

Updated issue text.

Here is what I am currently thinking: Make a new table (or could update the existing table, but I'll start with a new one to see how I like it). Something like:

CREATE TABLE settings_declarations (
  name             TEXT PRIMARY KEY,
  declared_by      TEXT NOT NULL,   -- worker type, or 'core:config_definition'
  owner_kind       TEXT NOT NULL,   -- 'core' | 'extension'  (binary to avoid deleting extension settings)
  extension_id     TEXT,            -- NULL for core
  category         TEXT NOT NULL,   -- display grouping; declared, else name.split('.')[0]
  category_label   TEXT,            -- optional display label ? would allow extensions to have sub categories
  sort_order       INTEGER,         -- sparse (10, 20, 30); NULL falls back to definition position - so we can reorder settings as we wish across workers
  is_managed       BOOLEAN DEFAULT FALSE,  -- we use both core and cache to have "indirect" settings; could be useful to have here
  first_seen       TIMESTAMP,
  last_seen        TIMESTAMP;
);

Also, it might be useful to archive settings instead of deleting...

CREATE TABLE settings_archive (
  name         TEXT,
  value        TEXT,
  tag          TEXT,
  declared_by  TEXT,
  archived_at  TIMESTAMP,
  archived_by  TEXT
);

I think that would give us everything we need (plus some useful category and ordering for UI improvements).

[dale-wahl]

I don't think we should do the extension.-prefix rename. I can still be a convention going forward.

Renaming e.g. selenium.browser → extensions.web_studies.selenium.browser means every config.get() call site, every Compatibility.required_settings entry, and every config block key has to change too. Any extension not updated at the same time falls back to defaults — so we lose the value, not just the row.

More fundamentally: a prefix tells you a setting is extension-owned. It can't tell you whether the owner is absent because it was uninstalled, disabled, or crashed on import like mentioned above. That's why I wanted to explore the database table solution which I now think is a lot better.

A settings_declarations table will it makes the delete decision decidable, and it lets the settings UI group by declared category instead of name.split(".")[0], which is what the prefix was going to buy us, without renaming anything. extensions.<id>. could still be used for collision-proofing between extensions but we'd have the actual table to really tell us and, if needed, resolve a collision. Which I am also going to ensure that our core settings win if there were a collision (which the convention could not do if it was ignored).