Skip to content

Commit fee791d

Browse files
committed
fix: flush the stream pacer backlog when the wire goes quiet
The smoothing pacer reveals buffered content_block_delta bursts at a fixed 32 KB/s. When the model delivers a burst then pauses (between bursts, or finished a paragraph and is thinking), that burst sits in pending_stream and dribbles out while the wire is silent — the "it pauses, then slowly scrolls in the text it already had" stall. The pacer only earns its keep while bytes are actively flowing; once last_event_at goes stale (>90 ms, bumped on every SSE event incl. heartbeats) there is nothing left to smooth against, so flush the whole backlog in one tick. Pairs with maya 560edd4 (residue-drain gate fix) — together they stop already-received bytes from appearing later than they should. Bumps maya.
1 parent d7c0dcc commit fee791d

2 files changed

Lines changed: 25 additions & 3 deletions

File tree

maya

src/runtime/app/update/meta.cpp

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,25 @@ Step meta_update(Model m, msg::MetaMsg mm) {
120120
// worth preserving while the wire is keeping up; once it
121121
// gets ahead by a paragraph or more, the latency cost
122122
// dominates the aesthetic benefit.
123+
// Wire-quiescence flush. The smoothing pacer only earns its
124+
// keep while bytes are ACTIVELY flowing — it hides the visual
125+
// jump of a chunky multi-KB content_block_delta by revealing
126+
// it over a few frames. The moment the wire goes quiet (model
127+
// paused between bursts, finished a paragraph and is thinking,
128+
// or the render path was briefly backpressured and is now
129+
// catching up) there is nothing left to smooth AGAINST:
130+
// continuing to dribble the buffered tail at kBytesPerSec is
131+
// exactly the "it pauses, then slowly scrolls in the text it
132+
// already had" symptom. last_event_at is bumped on every SSE
133+
// event (text/json delta AND heartbeats), so a gap here means
134+
// the wire is genuinely idle — reveal the whole backlog now.
135+
bool wire_quiet = false;
136+
if (const auto* a = active_ctx(m.s.phase)) {
137+
if (a->last_event_at.time_since_epoch().count() != 0)
138+
wire_quiet = (now - a->last_event_at)
139+
> std::chrono::milliseconds(90);
140+
}
141+
123142
if (!m.d.current.messages.empty()
124143
&& m.d.current.messages.back().role == Role::Assistant)
125144
{
@@ -145,8 +164,11 @@ Step meta_update(Model m, msg::MetaMsg mm) {
145164
constexpr std::size_t kBurstFlushBytes = 8192;
146165

147166
std::size_t drip;
148-
if (msg.pending_stream.size() >= kBurstFlushBytes) {
149-
// Backlog dominates — flush it all this tick.
167+
if (msg.pending_stream.size() >= kBurstFlushBytes
168+
|| wire_quiet) {
169+
// Backlog dominates, OR the wire has gone quiet so
170+
// there's nothing left to smooth against — flush
171+
// it all this tick.
150172
drip = msg.pending_stream.size();
151173
} else {
152174
auto target = static_cast<std::size_t>(

0 commit comments

Comments
 (0)