*Major T/s improvement* Use the Metal qmatmul MM kernels #2615
Conversation
* Add GGUF bf16 type support * Add non avx impl for vec_dot_bf16 * Fix from_u32 * Fix loading * Fix dequant of bf16
* Update kernels for metal bf16 * Fix typo * Check if have bfloat
* Test passes * All tests pass * Now all the tests really pass * Try out always using mm * Mirror llama.cpp metric * Mirror llama.cpp metric * Update test
|
@LaurentMazare if you could review, that would be great! More benchmarks with some smaller models can be found here: EricLBuehler/mistral.rs#903 (comment) |
|
Why is this still not close/? |
|
Without merging this MR, is candle still slower than llama.cpp/ggml right now? Or has this improvement already been implemented in other code submissions? |
|
@null-define without this, Candle Metal prompt performance is significantly reduced. This is because we aren't using the specialized Matrix-Matrix kernels, instead using Matrix-Vector kernels repeatedly which is slower. |
|
wondering why it isn't merged into main? Does candle is now not maintained well? |
|
This is an amazing improvement! After testing across 11 GGUF LLMs, the new code is 73% faster than the current version, exceeding llama.cpp speeds on my M3 Max.
Data
Computer Specs
@LaurentMazare What would it take to get this merged? |
|
@meg-huggingface Please consider merge it |
|
wow. this sounds amazing. we sure could use any speed boost we can get on metal. This original PR is from almost 5 months ago. Why is there no discussion as the reason its not merged yet? |
|
@LaurentMazare would this help with inference speed of Metal for Flux and SD3 image generation? |
|
The candle team seems abundant in this lib? |
|
just checking in again on this hanging PR. is there anyone out there that can review? do we need to do it different or fix something? thanks |
|
@LaurentMazare sorry to bug you but is there someone else we can ping to get this approved or at least some comment on why it's still sitting here for so many months? thanks |
|
I think HuggingFace abandoned the Candle project. |
|
is that real? |
|
I think it is now mainly community driven, and the core developers are lazy at merging new features, not even supporting new features, such as many low level ONNX ops. I couldn't see any response or support for it. |
|
maybe they should add some more admins that have merge authority. it seams like there are many people ready to work. |
|
LGTM |
…#2615) * Add GGUF BF16 support (huggingface#17) * Add GGUF bf16 type support * Add non avx impl for vec_dot_bf16 * Fix from_u32 * Fix loading * Fix dequant of bf16 * Update kernels for metal bf16 (huggingface#19) * Update kernels for metal bf16 * Fix typo * Check if have bfloat * Sync ggml metal kernels (huggingface#33) * Metal qmatmul mat-mat product (huggingface#39) * Test passes * All tests pass * Now all the tests really pass * Try out always using mm * Mirror llama.cpp metric * Mirror llama.cpp metric * Update test * Update test * fixed merge error --------- Co-authored-by: keighbee <kb@huggingface.co>
4 participants
This PR adds the automatic usage of Metal GGML quantized mat-mat kernels instead of always using the mat-vec kernels and upstreams a few related/necessary changes.
Before this change, Candle's Metal decoding performance was on-par with MLX and llama.cpp but the prompt performance was insufficient. After this change, the prompt performance (on the benchmark) was increased to a factor of about 2.5x faster than MLX and within 10% of llama.cpp - a performance boost by a factor of almost 6x.
This PR switches to only using the MV kernels when
D::Minus2of thexsinput tensor is equal to 1. This mirrors the logic in GGML.Besides utilizing the MM kernels, this PR also upstreams some required changes: