docs: warn that cuda-1.9 image + flash-attn silently produces NaN embeddings on Turing/T4#896
Open
moduvoice wants to merge 1 commit into
Open
Conversation
…eddings on Turing/T4 The README's headline Docker quickstart uses the cuda-1.9 multi-arch image, which correctly dispatches to the Turing binary on T4/RTX 2000-series GPUs but always defaults USE_FLASH_ATTENTION=True regardless of architecture. On real T4 hardware this reproduces as 100% NaN embedding vectors while the server returns HTTP 200 "Success" with no warnings, across two unrelated model architectures (Qwen3-Embedding-0.6B, BAAI/bge-small-en-v1.5), 3+ runs each. Document that Turing/T4 users must use the turing-1.9 tag or set USE_FLASH_ATTENTION=False, and that doing so can in turn hit a CUDA OOM at the default --max-batch-tokens=16384 for some models (recommend lowering to 2048). Also removes a stale --quantize CLI flag reference in tei_cloud_run.md that does not exist in cli_arguments.md or router/src/main.rs (likely copied from TGI docs). No code changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Motivation: The README's own headline "Docker" quickstart command uses the
cuda-1.9multi-architecture image. On a real Tesla T4 (Turing, 16GB), this image correctly auto-detects
the GPU and dispatches the Turing-compatible binary (
text-embeddings-router-75), but it stillstarts with
USE_FLASH_ATTENTION=True— that default is baked in at build time(
Dockerfile-cuda-all'sARG DEFAULT_USE_FLASH_ATTENTION=True) and is not adjusted perarchitecture the way the binary selection is (only the
turingbuild in.github/workflows/matrix.jsonoverrides it toFalse).This is more severe than the "precision issues" the README's own "Docker Images" table already
warns about for the Turing image: on a T4, the exact quickstart command from the README's
headline section produces 100% NaN embedding vectors, while the server logs
router/src/http/server.rs:737: Successand returns HTTP 200 for every request — no error orwarning anywhere. A production user following the README verbatim on a T4 would silently write
corrupted (all-
null) embeddings into their vector store.Repro (real Tesla T4, driver 550.163.01, Docker 24.0.9-1, TEI 1.9.3):
ghcr.io/huggingface/text-embeddings-inference:cuda-1.9(sha256:249a0bc8...)docker run --gpus all -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $modeldocker exec <container> env | grep FLASH→USE_FLASH_ATTENTION=Truecurl 127.0.0.1:8080/embed -d '{"inputs":"What is Deep Learning?"}'→[[null,null,null,...]](all dimensions NaN), HTTP 200Qwen/Qwen3-Embedding-0.6B(1024-dim) andBAAI/bge-small-en-v1.5(384-dim) — 3+ runs each, 100% reproduction rate-e USE_FLASH_ATTENTION=False, or switching to theturing-1.9image(sha256:dee4a149..., which defaults to
USE_FLASH_ATTENTION=False), immediately producesnormalized, valid embeddings (norm≈1.0) for both models — confirmed by A/B toggle on the same
hardware/model/input.
Qwen/Qwen3-Embedding-0.6Bfails to start atthe default
--max-batch-tokens=16384withDriverError(CUDA_ERROR_OUT_OF_MEMORY, "out of memory")during warmup; lowering to--max-batch-tokens 2048fixes it. (bge-small's defaultof 512 doesn't hit this.)
Changes:
telling Turing/T4 users to use
turing-1.9or set-e USE_FLASH_ATTENTION=False, and notingthe
--max-batch-tokensOOM trap that can follow.applies to the dedicated
turing-1.9image) to make clear it also applies to thecuda-1.9image used in the quickstart, which is not listed in that table at all.
AGENTS.mdsummarizing this and the other T4-relevant findings from this session(flash-attn default risk + fix, the
--max-batch-tokensOOM trap, and that TEI's defaultdtype — float16, no bf16 code path — is already safe on Turing).
docs/source/en/tei_cloud_run.md: remove a--quantizeCLI flag reference that does notexist in
docs/source/en/cli_arguments.mdorrouter/src/main.rs(0 matches in either) —likely copied over from TGI's docs, which does have this flag.
No code changes — this is a documentation-only PR. It does not change any default behavior,
Dockerfile build args, or entrypoint logic; it only documents what the current images already do.
I'm separately considering whether to open a follow-up issue proposing that
cuda-all-entrypoint.shpick a safeUSE_FLASH_ATTENTIONdefault based on thecompute_capitalready detects, since that would be a code change and is out of scope here.
AI disclosure: I used an AI coding assistant to help investigate this (including running the
repro on a real T4) and draft this PR description. I read and verified the diff and the repro
steps myself before submitting.
Before submitting
instasnapshots?Who can review?
@Narsil @alvarobartt