[Refactor] T7: PreparedRequestQueue — parallel impl, consolidate with #903 (#661)#905
Open
XinhaoTheo wants to merge 1 commit into
Open
[Refactor] T7: PreparedRequestQueue — parallel impl, consolidate with #903 (#661)#905XinhaoTheo wants to merge 1 commit into
XinhaoTheo wants to merge 1 commit into
Conversation
…t#661 T7) Extract the tri-state preprocessing -> AR-engine handoff (prepared / inflight / aborted + lock + per-model preprocessing context) duplicated across moss_tts and moss_tts_local into one shared generic PreparedRequestQueue[CtxT, PrepT], and migrate both models onto it. - New scheduling/prepared_request_queue.py: generic tri-state registry; begin() reads the context and marks the request in-flight under one lock, preserving the original single-lock atomicity. publish() returns a non-observable bool; pop() returns the payload or None. - moss_tts + moss_tts_local delegate to the queue; the duplicated per-model tombstone + context code is deleted in the same change. stages.py wiring (abort_callback / set-context) unchanged. Behavior-preserving. - CPU contract test: the seven prepared/inflight/aborted scenarios plus a concurrency thread-storm.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
RFC #661 Template 7. The tri-state preprocessing→AR-engine handoff —
prepared/inflight/aborted+ a lock + the per-model preprocessing context — is duplicated line for line acrossmoss_ttsandmoss_tts_local(~80 LOC each). Per #661 (framework owns reusable mechanics; model dirs own model semantics), this extracts that shared machinery into one generic queue and migrates both models onto it.Note: #903 (@MelodyyyYin) implements the same template. happy to defer to #903 with a
Co-authored-bycredit, or fold my addition (below) into it.Modifications
sglang_omni/scheduling/prepared_request_queue.py: genericPreparedRequestQueue[CtxT, PrepT]owning the tri-state + lock + the opaque per-model context.begin()reads the context and marks the request in-flight under one lock (preserves the original single-lock atomicity so a concurrentclear_contextcan't strand a stale in-flight id);publish()returns a non-observablebool;abort/pop/fail_inflight/set_context/clear_context.moss_ttsandmoss_tts_localrequest_builders.pyto delegate to the queue; the duplicated per-model globals + lock are deleted in the same change.stages.pywiring (abort_callback / set-context) is unchanged.test_pipeline.pyassertions to read_QUEUE.*.Related Issues
Part of #661 (Template 7). Overlaps with #903 (same template) — see Motivation.
Accuracy Test
N/A — no model-side code (kernels / architecture / sampling) is touched. This is a behavior-preserving scheduling-layer refactor: the queue logic is line-for-line equivalent to the previous per-model code, and
publish()'sboolreturn is non-observable (callers ignore it). Per-model Seed-TTS accuracy CI is the backstop.Benchmark & Profiling
N/A — no performance-relevant change. Same lock, same operations per request; one
shared class replaces two copies.
Checklist