VRAM and speed optimizations for GPU inference

[mingyi456]

Do you know how much VRAM this is supposed to take? When I set the n_gpu_layers argument to -1 in the code, I end up using more than 24GB of VRAM. I even tried changing the code to use different quantization levels, but the VRAM usage does not seem to change.

[mingyi456]

The VRAM usage seems to be due to not setting the kwarg flash-attn=True.

Edit: I still cannot get real time speed with my 4090, it seems to be due to the following log message:

load_tensors: tensor 'token_embd.weight' (q8_0) (and 0 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead

While I saw a discussion of the kobold.cpp repo that suggests this is harmless, I still see significant CPU utilization even after setting n_gpu_layers to -1, or 99.

By the way, setting use_mlock=True gives an additional small boost to speed, but the audio output is still not quite real time.

If I use the https://github.com/Lex-au/Orpheus-FastAPI frontend with LM Studio as the backend, it does not utilize the CPU significantly, and I guess this is why it manages to output in real time speed (although it does not feature audio streaming)

[mingyi456]

Further update: I finally managed to obtain real time inference speed on my 4090. It was not due to the embed tensor running on CPU, but because of enabling CUDAExecutionProvider for the onnx runtime for the snac model, which seems to be used for decoding from my understanding.

I first needed to install onnxruntime-gpu via pip, then install the cuDNN library from Nvidia, then add the directory containing the cudnn64_9.dll file to the PATH environmental variable. Finally my CPU is not utilized during inference, and I get real time speeds.

Now the only problem is that the maximum duration of the audio output is 25 seconds, even after setting the max_tokens to 32768.

[mingyi456]

I also managed to solve the issue of excessive VRAM/RAM consumption. It was due to the allocated KV cache, and this is because the n_ctx option was set to 0, which meant llama-cpp-python allocates the maximum context size allowed by the model. In this case, because Orpheus is based on Llama 3.2, the maximum context is 131072.

I noticed this because it appeared changing the max_tokens option (which affects the maximum number of tokens the model is allowed to generate, and this is totally separate from allocated context size, which is what actually affects VRAM/RAM consumption) does not affect the allocated VRAM in any way,

With flash-attn=True and n_ctx=65536, and a 8 bit quant of the Orpheus model weights, the total allocated VRAM is just over 12GB, and this is already extremely generous. There is plenty of room to shrink n_ctx, given that the default max_tokens is a measly 2048. Intuitively, the output tokens which correspond to audio should be much greater than the input tokens which correspond to the text prompt, so allocating 131072 tokens total context just to allow 2048 output tokens is plain silly.

There should also be absolutely no need to resort to using a 4 bit quant of the model weights, because the difference in VRAM consumption is very insignificant compared to the excessively gigantic KV cache if you blindly used the default settings in this repo.

Just make sure n_ctx is set to above max_tokens, so that the model does not end up forgetting its own context (which in this case is your text prompt), which would affect generation quality (which Ollama's default context limit is notorious for).