Skip to content

fix: stamp response with sampler version atomically with the client - #2

Open
golldyck wants to merge 1 commit into
NousResearch:mainfrom
golldyck:fix/atomic-sampler-client-version
Open

fix: stamp response with sampler version atomically with the client#2
golldyck wants to merge 1 commit into
NousResearch:mainfrom
golldyck:fix/atomic-sampler-client-version

Conversation

@golldyck

@golldyck golldyck commented Jul 1, 2026

Copy link
Copy Markdown

What

_route_to_tinker in tinker_nemogym/tinker_responses_model.py now captures the active sampling client and its version stamp in a single lock acquisition. That guarantees the sampler_version written onto every response is the version of the client that actually produced the tokens.

Cause

The routing helper read the two values in two separate lock acquisitions:

  • client = get_sampling_client() at the top of the call (~line 172), then
  • sampler_version_at_call = get_sampler_version() just before sampling (~line 228).

Between those reads it renders the prompt and builds SamplingParams. The trainer hot-swaps the sampler between RL steps via set_sampling_client(new_client, save_count + 1). If that swap lands in the window between the two reads, the response is stamped with the new version while the old client (correctly captured by reference) serves the call.

The trainer then computes staleness = save_count - sampler_version (the HuggingFace BF16-mismatch drift diagnostic the version stamp exists for) and under-counts staleness. This is the in-flight hot-swap scenario the surrounding comments say they guard against.

Fix

  • Add get_sampling_client_and_version(), which returns (client, version) read under one acquisition of _sampling_client_lock, a consistent snapshot.
  • In _route_to_tinker, capture client, sampler_version_at_call = get_sampling_client_and_version() up front (before rendering) and drop the later second get_sampler_version() read.
  • The existing get_sampling_client() / get_sampler_version() getters are kept unchanged for compatibility. The helper is added to __all__.

Test

tests/unit/test_tinker_responses_model.py:

  • test_get_sampling_client_and_version_reads_atomically checks the combined getter returns a consistent snapshot.
  • test_route_stamps_version_of_client_that_served_across_hot_swap injects the trainer's set_sampling_client(B, v+1) swap inside the race window (from renderer.build_generation_prompt, which runs between the two reads; the window is synchronous, so no asyncio task can preempt it). It asserts client A still served the call and the stamped sampler_version is A's version (v), not B's (v+1).

The mid-route test fails on the two-lock code (assert 8 == 7) and passes with the snapshot helper. Verified under python3.11 with fastapi/pydantic/pytest/pytest-asyncio and a stubbed tinker.

_route_to_tinker read the active sampling client (get_sampling_client)
and its version stamp (get_sampler_version) in two separate lock
acquisitions, with prompt rendering and SamplingParams construction
between them. A concurrent set_sampling_client(new, version+1) — the
trainer's between-step hot-swap — landing in that window stamped the
response with a version that did not produce the sampled tokens, so the
trainer's staleness = save_count - sampler_version diagnostic (the HF
BF16-mismatch measurement the stamp exists for) under-counted drift.

Add get_sampling_client_and_version(), which reads both under one lock,
and capture (client, version) together at the top of _route_to_tinker
before rendering. The existing getters stay for compatibility.

Test: a hot-swap injected inside the render step (the exact race window,
which is synchronous so no asyncio task can preempt it) now leaves the
stamped sampler_version matching the client that served the call. The
test fails on the two-lock code and passes on the snapshot helper.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant