[grug] Allow ten-second dirty-page decay in EP hero runs - #9182
Draft
mcwitt-agent wants to merge 2 commits into
Draft
mcwitt-agent wants to merge 2 commits into
mcwitt-agent wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Set the EP hero trainer's default jemalloc dirty-page decay to 10 seconds, allowing freed pages to be reused before reclamation. Keep jemalloc enabled with background purging, immediate muzzy-page decay, and two arenas. Explicit launch overrides still take precedence.
Immediate purging produced checkpoint-associated training stalls in the 64-GPU diagnostic: the total for six steps was 113.5–121.7 seconds, versus 97.7–99.4 seconds with purging disabled and 97.1–97.4 seconds without a save. After saves with purging disabled, allocator-reported resident memory was 64–69 GiB higher per node than in the preceding no-save condition with immediate purging. A finite decay interval is a minimal candidate for reducing the stalls while continuing to reclaim memory.
The 10-second setting has not been benchmarked. Keep this as a draft pending repeated-checkpoint measurements of total training time, worst-step latency, peak host memory, and memory recovery between saves. The diagnostic does not establish that this setting fixes production spikes or stays within production memory headroom.
Part of #9180.