fix(docker): migrate retained SearXNG settings

[RaresKeY]

Summary

Retained nonempty SearXNG settings can omit use_default_settings, so newer pinned images fail with missing settings instead of inheriting their current defaults. This adds an idempotent PyYAML-aware startup migration to every Compose variant; it inserts only the missing top-level inheritance key and atomically preserves existing custom settings, comments, secrets, ownership, mode, and the settings volume.

Target branch

• This PR targets dev, not main. All PRs land in dev; main is curated by the maintainer at each release. If your PR is on main by accident, click "Edit" on this PR and change the base.

Linked Issue

Fixes #6054

Type of Change

• Bug fix (non-breaking — fixes a confirmed issue)
• New feature (non-breaking — adds new behaviour)
• Breaking change (changes or removes existing behaviour)
• Refactor / cleanup (behaviour unchanged)
• Documentation only
• CI / tooling / configuration

Checklist

• I searched open issues and open PRs — this is not a duplicate.
• This PR targets dev
• My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
• I actually ran the app (docker compose up or uvicorn app:app) and verified the change works end-to-end. Type-checks and unit tests are not enough.

How to Test

1. Run python -m pytest -q tests/test_searxng_settings_migration.py tests/test_searxng_image_pinned.py tests/test_gpu_compose_standalone.py; the focused migration, image pin, and standalone Compose coverage should pass.
2. Start the SearXNG service with a fresh empty settings volume and confirm its health endpoint returns HTTP 200; the generated settings already contain use_default_settings: true and must remain byte-identical through the migration.
3. Replace only the test volume's settings.yml with a retained nonempty mapping that has a representative server.secret_key and custom search/UI values but no top-level use_default_settings, restart SearXNG, and confirm the key is inserted while all original bytes, ownership, mode, custom values, and secret remain intact. A second restart must make no further change, and the health endpoint must return HTTP 200.

Author validation: the focused command passed with 39 passed and one unrelated pre-existing SQLAlchemy deprecation warning. python -m py_compile scripts/migrate_searxng_settings.py tests/test_searxng_settings_migration.py passed. Isolated builds of the pinned SearXNG image returned HTTP 200 for both fresh and retained settings; the fresh file stayed byte-identical, and the retained file preserved its representative secret, custom values, and mode while the second migration was a no-op. Full Odysseus docker compose up was not run because Docker is unavailable in the controller and secretless test runner.

Visual / UI changes — REQUIRED if you touched anything that renders

Anything that changes what the UI looks like — buttons, icons, padding, colors, fonts, spacing, layout, CSS, HTML, SVG, or any static/js/ module that draws to the DOM — needs all of the following. PRs that change rendering without these WILL be closed.

• Screenshot or short clip of the change in the running app, attached below. Mobile screenshot too if the change affects mobile.
• Style match: the change uses Odysseus's existing visual language. Specifically:
  • Reuse existing CSS variables (--red, --fg, --bg, --card, --border, etc.) — do not introduce new color values, font sizes, or spacing units.
  • Reuse existing button/input/card/border classes. Don't invent parallel styling.
  • No Unicode emoji in UI or code. Use inline SVG (matching the monochrome icon style already in static/index.html) or plain text.
  • Monospaced font (Fira Code) for primary UI text. Don't override.
  • Dark theme is the default; any light-mode work must be wired through the existing theme system, not hard-coded.
• No new component patterns. If a similar widget already exists in the app, extend it instead of writing a parallel one.
• I am not an LLM agent submitting a bulk PR. If you are, please open an issue describing the problem first — bulk auto-generated PRs that don't match the project's visual style are closed on sight, even when the underlying fix is correct.

Screenshots / clips

N/A — no UI or rendering files changed.

[o3LL]

Reviewed 3cd6cdb6 against dev at 2c394704. Merge probe is clean.

The approach is right and the bug is real — I reproduced #6054 exactly against the pinned image with a retained settings volume. But the write path can't succeed in a real deployment as written, and the entrypoint turns that into a boot failure.

Findings

P0 · issue — the migration can never write on the upgrade path it targets

• Problem: fchown runs before fchmod, and since SearXNG's own entrypoint does chown -R searxng:searxng /etc/searxng, root no longer owns the temp file by the time the chmod lands — the Compose cap set (cap_drop: ALL plus CHOWN/SETGID/SETUID/DAC_OVERRIDE, no FOWNER) returns EPERM.
• Impact: the script exits 1, set -eu kills the wrapper before exec /usr/local/searxng/entrypoint.sh, and SearXNG never starts, so odysseus blocks on service_healthy — strictly worse than the 500 in #6054.
• Ask: swap the two calls so the chmod happens while the temp file is still root-owned.

os.fchown(fd, source_stat.st_uid, source_stat.st_gid)
os.fchmod(fd, stat.S_IMODE(source_stat.st_mode))

• Location: scripts/migrate_searxng_settings.py:123

I ran the pinned image with the Compose cap set against a searxng-owned retained settings.yml: as-is it prints SearXNG settings migration failed: [Errno 1] Operation not permitted and the container exits; with the two lines swapped the root endpoint returns 200, the key is inserted, and mode 640 and searxng:searxng ownership survive. It passes for a root-owned file, which is what an isolated hand-seeded volume looks like — a real upgrade is always the chowned case. Happy to paste the matrix if useful.

P1 · issue — any migration failure becomes a boot failure

• Problem: main() returns 1 for every OSError / UnicodeError / ValueError, and all three Compose variants invoke it unguarded under set -eu.
• Impact: a retained file that is non-UTF-8, a symlink, or has a non-mapping root now prevents SearXNG from starting at all, where today it boots and fails visibly on its own terms.
• Ask: make the migration advisory — append || true in the three entrypoints, or return 0 after printing the warning.
• Location: docker-compose.yml:113

P2 · issue — curated engine lists silently gain the full default set

• Problem: adding use_default_settings: true changes a standalone engines: list from "these are the engines" to "merge these into the defaults", which "preserves existing custom settings" in the summary doesn't cover.
• Impact: I booted the pinned image with a retained file pinning two engines; after migration /config reports 244 enabled engines, including ones the operator had removed.
• Ask: skip files carrying a top-level engines: key, or emit use_default_settings: {engines: {keep_only: [...]}} for them — at minimum say so in the log line and the PR body.
• Location: scripts/migrate_searxng_settings.py:90

P2 · issue scope — migrated volumes come up healthy with search still dead

• Problem: SearXNG's default search.formats is html only, so a retained file with no explicit formats list inherits defaults that exclude json.
• Impact: / returns 200 and the healthcheck goes green, but services/search/providers.py:156 requests format=json and gets a 403 — a quieter failure than the one #6052 reports today.
• Ask: also ensure json is present in search.formats, or note in the PR that those volumes still need a manual edit and keep Fixes pointed only at #6054.
• Location: scripts/migrate_searxng_settings.py:90

P3 · test — the new tests fail on Windows

• Problem: os.fchown and os.O_DIRECTORY are Unix-only, so every test that reaches the write path errors there.
• Impact: contributors running the suite on Windows get failures unrelated to their change.
• Ask: add the sys.platform == "win32" skip already used in tests/test_app_db_permissions.py:10.
• Location: tests/test_searxng_settings_migration.py:1

P3 · test — nothing covers the ordering that P0 breaks

• Problem: the ownership assertions run as the file's owner, so fchown is a no-op and fchmod can never fail.
• Impact: the one failure mode that matters inside the container is invisible to the suite.
• Ask: assert fchmod is called before fchown — a spy on both is enough.
• Location: tests/test_searxng_settings_migration.py:66

Open Questions

• #6052 is filed as a fresh git clone plus docker compose up -d --build, so the retained-volume diagnosis rests on the reporter reusing an existing searxng-data volume. Worth confirming with them before treating this PR as the fix for that report.

Validation

• Ran: reproduced #6054 on dev 2c394704 — pinned image, retained searxng-owned settings.yml, HTTP 500 with KeyError: 'default_doi_resolver'. Ran the migration on that volume under the Compose cap set (fails, P0) and with the calls swapped (200, bytes/secret/mode/owner preserved, second run a no-op), plus the engine-count and format=json checks above. Full suite on 3cd6cdb6: 5341 passed, 2 failed, 4 skipped — both failures are the known macOS ones. compileall clean on both new files.
• Not run: full docker compose up of the whole stack; the two GPU variants; a Linux host; SELinux relabelling of the new bind mount.
• Residual risk: I exercised the SearXNG service in isolation on arm64, so anything specific to a full Compose run or to x86 Linux is unverified.

PR Hygiene

• Targets dev, one focused change, title matches the Conventional Commits check, ready for review is on, no rendering changes so the screenshot requirement doesn't apply.
• Fixes #6054 is the right link and the right verb. Merge probe against dev 2c394704 is clean, and it also merges cleanly with #5894, which edits the adjacent ports: line.
• The "I actually ran the app" box is left unticked, which matches what you wrote — that gap is where the P0 sits, so the ask is one docker compose run against a retained volume once the ordering is swapped.

Only P0 blocks. If you'd rather not respin it yourself, say the word and I'll push the swap plus the ordering test to your branch or open a follow-up — I already have the reproduction set up.

Fixed in latest commits