Reduce memory usage with puma_worker_killer and distro query optimization#919
Merged
Reduce memory usage with puma_worker_killer and distro query optimization#919
Conversation
…tion Add puma_worker_killer to restart workers that exceed memory limits, enable clustered mode with preload_app for copy-on-write savings, and refactor DistrosController#index to use find_each with selective columns instead of loading the entire distros table into memory twice.
Covers grouping by key, deduplication by pretty_name, sort order within and across groups, nil grouping_key exclusion, and scope filtering for the new model methods introduced in the memory optimization.
Move ungrouped distros filtering to a WHERE clause with selective columns instead of loading all records into memory. Default WEB_CONCURRENCY to 0 in development to avoid unnecessary clustered mode. Fix test that relied on generate_slug not running.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
puma_worker_killerto automatically restart Puma workers that exceed memory limits. Puma is now running in clustered mode withpreload_app!for copy-on-write memory savings.PumaWorkerKiller config:
PWK_RAM_MBenv var)Also refactors
DistrosController#indexwhich was loading the entire distros table into memory twice (via.to_a), then running three passes ofgroup_by/map/sort_byon each set. Now usesfind_eachwith selective columns (skippingraw_contenttext blob) and does deduplication in a single pass viaDistro.grouped_and_deduped.