Skip to content

docs: warn that cuda-1.9 image + flash-attn silently produces NaN embeddings on Turing/T4#896

Open
moduvoice wants to merge 1 commit into
huggingface:mainfrom
moduvoice:docs/t4-turing-flash-attn-note
Open

docs: warn that cuda-1.9 image + flash-attn silently produces NaN embeddings on Turing/T4#896
moduvoice wants to merge 1 commit into
huggingface:mainfrom
moduvoice:docs/t4-turing-flash-attn-note

Conversation

@moduvoice

Copy link
Copy Markdown

What does this PR do?

Motivation: The README's own headline "Docker" quickstart command uses the cuda-1.9
multi-architecture image. On a real Tesla T4 (Turing, 16GB), this image correctly auto-detects
the GPU and dispatches the Turing-compatible binary (text-embeddings-router-75), but it still
starts with USE_FLASH_ATTENTION=True — that default is baked in at build time
(Dockerfile-cuda-all's ARG DEFAULT_USE_FLASH_ATTENTION=True) and is not adjusted per
architecture the way the binary selection is (only the turing build in
.github/workflows/matrix.json overrides it to False).

This is more severe than the "precision issues" the README's own "Docker Images" table already
warns about for the Turing image: on a T4, the exact quickstart command from the README's
headline section produces 100% NaN embedding vectors, while the server logs
router/src/http/server.rs:737: Success and returns HTTP 200 for every request — no error or
warning anywhere. A production user following the README verbatim on a T4 would silently write
corrupted (all-null) embeddings into their vector store.

Repro (real Tesla T4, driver 550.163.01, Docker 24.0.9-1, TEI 1.9.3):

  • Image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 (sha256:249a0bc8...)
  • Command: exactly the README headline command, unmodified:
    docker run --gpus all -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model
  • docker exec <container> env | grep FLASHUSE_FLASH_ATTENTION=True
  • curl 127.0.0.1:8080/embed -d '{"inputs":"What is Deep Learning?"}'
    [[null,null,null,...]] (all dimensions NaN), HTTP 200
  • Reproduced on two unrelated model architectures — Qwen/Qwen3-Embedding-0.6B (1024-dim) and
    BAAI/bge-small-en-v1.5 (384-dim) — 3+ runs each, 100% reproduction rate
  • Adding -e USE_FLASH_ATTENTION=False, or switching to the turing-1.9 image
    (sha256:dee4a149..., which defaults to USE_FLASH_ATTENTION=False), immediately produces
    normalized, valid embeddings (norm≈1.0) for both models — confirmed by A/B toggle on the same
    hardware/model/input.
  • Secondary trap: once Flash Attention is disabled, Qwen/Qwen3-Embedding-0.6B fails to start at
    the default --max-batch-tokens=16384 with DriverError(CUDA_ERROR_OUT_OF_MEMORY, "out of memory") during warmup; lowering to --max-batch-tokens 2048 fixes it. (bge-small's default
    of 512 doesn't hit this.)

Changes:

  1. README: add a prominent warning directly under the headline Docker quickstart command,
    telling Turing/T4 users to use turing-1.9 or set -e USE_FLASH_ATTENTION=False, and noting
    the --max-batch-tokens OOM trap that can follow.
  2. README: extend the existing "Docker Images" table warning (currently phrased as if it only
    applies to the dedicated turing-1.9 image) to make clear it also applies to the cuda-1.9
    image used in the quickstart, which is not listed in that table at all.
  3. New AGENTS.md summarizing this and the other T4-relevant findings from this session
    (flash-attn default risk + fix, the --max-batch-tokens OOM trap, and that TEI's default
    dtype — float16, no bf16 code path — is already safe on Turing).
  4. docs/source/en/tei_cloud_run.md: remove a --quantize CLI flag reference that does not
    exist in docs/source/en/cli_arguments.md or router/src/main.rs (0 matches in either) —
    likely copied over from TGI's docs, which does have this flag.

No code changes — this is a documentation-only PR. It does not change any default behavior,
Dockerfile build args, or entrypoint logic; it only documents what the current images already do.
I'm separately considering whether to open a follow-up issue proposing that
cuda-all-entrypoint.sh pick a safe USE_FLASH_ATTENTION default based on the compute_cap it
already detects, since that would be a code change and is out of scope here.

AI disclosure: I used an AI coding assistant to help investigate this (including running the
repro on a real T4) and draft this PR description. I read and verified the diff and the repro
steps myself before submitting.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline?
  • Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests? If applicable, did you include or update the insta snapshots?

Who can review?

@Narsil @alvarobartt

…eddings on Turing/T4

The README's headline Docker quickstart uses the cuda-1.9 multi-arch image, which
correctly dispatches to the Turing binary on T4/RTX 2000-series GPUs but always
defaults USE_FLASH_ATTENTION=True regardless of architecture. On real T4 hardware
this reproduces as 100% NaN embedding vectors while the server returns HTTP 200
"Success" with no warnings, across two unrelated model architectures (Qwen3-Embedding-0.6B,
BAAI/bge-small-en-v1.5), 3+ runs each. Document that Turing/T4 users must use the
turing-1.9 tag or set USE_FLASH_ATTENTION=False, and that doing so can in turn hit a
CUDA OOM at the default --max-batch-tokens=16384 for some models (recommend lowering
to 2048). Also removes a stale --quantize CLI flag reference in tei_cloud_run.md that
does not exist in cli_arguments.md or router/src/main.rs (likely copied from TGI docs).

No code changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant