ops: track mempalace-backup script + units in repo (#87) - #89
Merged
Conversation
The nightly pg_dump OOM fix was hand-deployed to
/usr/local/bin/mempalace-backup.sh on familiar and not tracked here, so a
host rebuild/reimage would silently lose it. Mirror the familiar-watchdog
pattern to close the drift:
- ops/familiar/mempalace-backup.sh — verbatim capture of the deployed
script (logic byte-identical; added a header comment only). Stops
llama-server-extractor for the dump window and restores its prior state
via an EXIT trap, freeing ~3.1G so earlyoom doesn't SIGTERM the dump's
postgres backend mid-COPY.
- ops/familiar/mempalace-backup.{service,timer} — the oneshot unit + 02:15
nightly timer, plus defense-in-depth resource hygiene (Nice=10, ionice
best-effort/7, OOMScoreAdjust=-500). The comment is explicit that these
protect the host-side dump pipeline only — earlyoom kills the in-container
postgres backend, so the llama-stop in the script remains the real guard.
- ops/scripts/install-mempalace-backup.sh — idempotent installer matching
install-familiar-watchdog.sh; copies script to /usr/local/bin, units to
/etc/systemd/system, arms the timer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the config drift flagged in #87 (Tail 2): the nightly mempalace
pg_dumpOOM fix was hand-deployed to/usr/local/bin/mempalace-backup.shon familiar and not tracked in this repo, so a host rebuild/reimage would silently lose it.Mirrors the established
familiar-watchdogpattern:ops/familiar/mempalace-backup.sh/usr/local/bin/mempalace-backup.sh; only a header comment added).ops/familiar/mempalace-backup.serviceoneshotunit + defense-in-depth resource hygiene.ops/familiar/mempalace-backup.timer02:15,Persistent=true.ops/scripts/install-mempalace-backup.shinstall-familiar-watchdog.sh).Why the fix works
familiar (15.5G) is RAM-oversubscribed. With
llama-server-extractor(~3.1G) running, thepg_dumpCOPYof the embedding-laden 410K-row drawers table tipped the host belowearlyoom's ~10%-free threshold, and earlyoomSIGTERM'd the postgres backend mid-COPY. The script stopsllama-server-extractorfor the dump window and restores its prior state via anEXITtrap (so it never resurrects a deliberately-stopped llama, e.g. during a benchmark). kg-extract retries against the dead endpoint (mempalace#307), so no triples are lost.Defense-in-depth nuance (honest framing)
The service unit adds
Nice=10,ionicebest-effort/7, andOOMScoreAdjust=-500. The unit comment is explicit that these protect the host-sidepg_dump/gzippipeline only — earlyoom kills the postgres backend running theCOPYinside themempalace-dbcontainer (separate process tree, separateoom_score), whichOOMScoreAdjuston this unit cannot reach. So the llama-stop in the script remains the real guard; the resource controls just reduce the dump's own footprint and keep it from starving interactive work.Verification
bash -nclean on both scripts;systemd-analyze verifyparsed the unit directives without complaint (only flagged the on-hostExecStartpath, expected off-host)./usr/local/bin/mempalace-backup.shviadiff.Not in scope / open tail
🤖 Generated with Claude Code