update docs

Author: dale-wahl

AGENTS.md

@@ -61,6 +61,8 @@ FourcatModule (common/lib/fourcat_module.py)
 
 ### Database Schema (key tables in `backend/database.sql`)
 - `settings` — key-value config store (name, value, tag).
+- `settings_declarations` — which module last declared each setting, and when it was last seen declared. A setting in `settings` with no row here is one nothing currently declares; that is not enough to remove it, since its extension may merely be uninstalled or disabled.
+- `settings_archive` — settings removed from `settings` are moved here rather than deleted, so removal is reversible.
 - `jobs` — job queue (jobtype, remote_id, timestamps, interval for recurring).
 - `datasets` — all datasets (key, type, key_parent for chaining, parameters as JSON, result_file, status, progress, etc.).
 - `datasets_owners` — many-to-many user-dataset ownership.
@@ -113,7 +115,9 @@ Invocation (the plain `docker compose` command uses `docker-compose.yml`):
 ## Configuration
 - **INI-based** (`config/config.ini`): Primary runtime config, read by `ConfigManager`. Docker's `docker_setup.py` syncs environment variables into this file.
 - **Database**: The `settings` table stores runtime-configurable settings (name/value/tag). `ConfigManager` reads from both INI and database, with memcached caching.
-- **Module config**: Module-defined `config` dicts (on worker classes) are collected by `ModuleCollector` at startup and cached to `config/module_config.bin`.
+- **Module config**: Module-defined `config` dicts (on worker classes) are collected by `ModuleCollector` at startup and cached to `config/module_config.bin`, with which module declared what in `config/module_config_provenance.bin` alongside it. The two files are deliberately separate: the first is merged straight into the config definition by every reader, including a front-end container that may still be running older code, so its shape is frozen — do not combine them. **Because these are caches written at start-up, a change to a module's `config` needs a back-end restart before it is visible.**
+- **Setting declarations**: a module may not redeclare a setting core already defines, or use a namespace reserved for core (`privileges.`, `flask.`, `4cat.`, `path.`, `datasources.`, `extensions.`, `logging.`). Such declarations are refused, not merged, because a definition controls a setting's `global` flag, `default` and `type`. See `ModuleCollector.collect_module_config()`.
+- **Start-up order**: `config.ensure_database()` runs *before* `ModuleCollector` in `backend/bootstrap.py`, because the collector reads `extensions.enabled` and `datasources.expiration` from the database. It therefore only ever sees the previous boot's module cache. Anything that needs the current module set — such as recording declarations — has to be a separate pass after the collector, not folded into `ensure_database()`.
 - **Legacy**: `config.py` in the repo root contains some legacy constants. Prefer `ConfigManager` / `config.ini` patterns.
 - **Extensions**: Installed under `config/extensions/`. Each extension can include its own `requirements.txt` (auto-installed by `setup.py`). Enabled/disabled via `extensions.enabled` setting.
 

config/extensions/README.md

@@ -37,4 +37,62 @@ folder. An extension containing both processors and a data source could look lik
 │          ├─ search_my_datasource.py
 ```
 
-In this scenario, `my_extension` would be a git repository within which all other files are contained. 
+In this scenario, `my_extension` would be a git repository within which all other files are contained.
+
+## Settings
+An extension can add its own settings to 4CAT's settings panel. Declare them in a `config` dictionary on the worker or
+processor class:
+
+```python
+class MyProcessor(BasicProcessor):
+    type = "my-processor"
+
+    config = {
+        "my_extension.api_key": {
+            "type": UserInput.OPTION_TEXT,
+            "default": "",
+            "help": "API key",
+            "tooltip": "Get one from the service's developer console"
+        }
+    }
+```
+
+Read them with `self.config.get("my_extension.api_key")`.
+
+Note that the `config` must be on the class, not on a data source's `__init__.py` - a data source's settings belong on
+its search or import worker, which 4CAT collects along with every other worker.
+
+### Naming
+Put your settings under a prefix of your own. The part before the first `.` decides which tab they appear under, so
+related settings sharing a prefix are grouped together. For new extensions, `extensions.<your extension id>.` is
+recommended, since it cannot collide with anything else.
+
+Some namespaces are reserved for 4CAT itself, and a setting declared in one of them will be **refused**: `privileges.`,
+`flask.`, `4cat.`, `path.`, `datasources.`, `extensions.` and `logging.`. This is so that an extension cannot claim a
+name 4CAT might use in a future version, and thereby take over its definition.
+
+A setting that a 4CAT release already declares is also refused - 4CAT's own definition always wins. Where two extensions
+declare the same setting, the first one wins, and which is 'first' does not depend on the machine. Refusals are written
+to the 4CAT log, so if a setting of yours does not show up, look there first.
+
+Several classes sharing a base class that declares `config` is fine: the setting is registered once, and inheriting it
+is not treated as a collision.
+
+### Settings that your code writes
+If a setting holds something your worker maintains rather than something an administrator sets - a cache timestamp, a
+list of options fetched from an API - mark it `"indirect": True`. It will then be kept out of the settings form, so it
+cannot be edited or overwritten by someone saving that page.
+
+### What happens to settings when an extension goes away
+4CAT does not delete an extension's settings. If your extension is switched off or uninstalled, its stored values are
+kept, and enabling or re-installing it restores the previous configuration instead of reverting to defaults. While the
+extension is not loaded, its settings are listed under 'unused settings' in the control panel, marked as kept on
+purpose.
+
+This means you do not need to worry about an administrator losing their configuration across an upgrade that briefly
+removes your extension. It also means that renaming one of your settings leaves the old value behind; 4CAT will not
+clean that up for you, because it cannot tell a rename from an uninstall.
+
+### Applying changes
+Declarations are read once, when the 4CAT back-end starts, and cached. After adding, renaming or removing a setting in
+your code, restart 4CAT before expecting the change in the interface.