Skip to content

[Refactor] T7: PreparedRequestQueue — parallel impl, consolidate with #903 (#661)#905

Open
XinhaoTheo wants to merge 1 commit into
sgl-project:mainfrom
XinhaoTheo:t7-prepared-request-queue
Open

[Refactor] T7: PreparedRequestQueue — parallel impl, consolidate with #903 (#661)#905
XinhaoTheo wants to merge 1 commit into
sgl-project:mainfrom
XinhaoTheo:t7-prepared-request-queue

Conversation

@XinhaoTheo

Copy link
Copy Markdown

Motivation

RFC #661 Template 7. The tri-state preprocessing→AR-engine handoff — prepared / inflight / aborted + a lock + the per-model preprocessing context — is duplicated line for line across moss_tts and moss_tts_local (~80 LOC each). Per #661 (framework owns reusable mechanics; model dirs own model semantics), this extracts that shared machinery into one generic queue and migrates both models onto it.

Note: #903 (@MelodyyyYin) implements the same template. happy to defer to #903 with a Co-authored-by credit, or fold my addition (below) into it.

Modifications

  • New sglang_omni/scheduling/prepared_request_queue.py: generic PreparedRequestQueue[CtxT, PrepT] owning the tri-state + lock + the opaque per-model context. begin() reads the context and marks the request in-flight under one lock (preserves the original single-lock atomicity so a concurrent clear_context can't strand a stale in-flight id); publish() returns a non-observable bool; abort / pop / fail_inflight / set_context / clear_context.
  • Migrated moss_tts and moss_tts_local request_builders.py to delegate to the queue; the duplicated per-model globals + lock are deleted in the same change. stages.py wiring (abort_callback / set-context) is unchanged.
  • Updated both models' test_pipeline.py assertions to read _QUEUE.*.
  • New CPU contract test: the seven §13 prepared/inflight/aborted scenarios plus a concurrency thread-storm.

Related Issues

Part of #661 (Template 7). Overlaps with #903 (same template) — see Motivation.

Accuracy Test

N/A — no model-side code (kernels / architecture / sampling) is touched. This is a behavior-preserving scheduling-layer refactor: the queue logic is line-for-line equivalent to the previous per-model code, and publish()'s bool return is non-observable (callers ignore it). Per-model Seed-TTS accuracy CI is the backstop.

Benchmark & Profiling

N/A — no performance-relevant change. Same lock, same operations per request; one
shared class replaces two copies.

Checklist

  • Format your code according with pre-commit.
  • Add unit tests.
  • Update documentation / docstrings / example tutorials as needed.
  • Provide throughput / latency benchmark results and accuracy evaluation results as needed.

…t#661 T7)

Extract the tri-state preprocessing -> AR-engine handoff (prepared /
inflight / aborted + lock + per-model preprocessing context) duplicated
across moss_tts and moss_tts_local into one shared generic
PreparedRequestQueue[CtxT, PrepT], and migrate both models onto it.

- New scheduling/prepared_request_queue.py: generic tri-state registry;
  begin() reads the context and marks the request in-flight under one lock,
  preserving the original single-lock atomicity. publish() returns a
  non-observable bool; pop() returns the payload or None.
- moss_tts + moss_tts_local delegate to the queue; the duplicated per-model
  tombstone + context code is deleted in the same change. stages.py wiring
  (abort_callback / set-context) unchanged. Behavior-preserving.
- CPU contract test: the seven prepared/inflight/aborted scenarios plus a
  concurrency thread-storm.
@XinhaoTheo XinhaoTheo requested a review from shuaills as a code owner June 29, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant