Hi, and thank you for this project. Amazing stuff!
I had a lot of trouble getting anything going, so here's my notes.
Hopefully it's useful to somebody else along the GPU-poor road.
Note that this is only a workaround, not a real fix to the underlying issue.
Using Linux Mint 21.2 and rustc 1.73.0.
So as I said, not using GPU at all, and my CPU is an old AMD FX (with AVX/FMA/FMA4/SSE4.2).
Not great I know, but I can run 7B stuff with llama.cpp, so I should be able to do something with candle too, right?
First of all, this CPU doesn't have AVX2, so I had to use RUSTFLAGS='-C target-feature=-avx2' to avoid the dreaded Illegal instruction.
(I knew this because I had sone a similar thing with llama.cpp.)
But even then, the simple example in the "get started" section wouldn't work for me. It just exited with the message
called Option::unwrap() on a None value
I noticed that in the backtrace there's a reference to gemm_f32::gemm::f32::fma::gemm_basic and the file gemm-common-0.16.14/src/gemm.rs.
So what I had to do was to comment out all the references to fma in that file, and that seems to have done the trick :D
I guess the gemm library is incomplete somehow?
Anyways, the end result is that I can now run madlad400-3b-mt, which was my goal.
Maybe not the speediest thing in the world, but I'm happy that it now works :D
Hi, and thank you for this project. Amazing stuff!
I had a lot of trouble getting anything going, so here's my notes.
Hopefully it's useful to somebody else along the GPU-poor road.
Note that this is only a workaround, not a real fix to the underlying issue.
Using Linux Mint 21.2 and rustc 1.73.0.
So as I said, not using GPU at all, and my CPU is an old AMD FX (with AVX/FMA/FMA4/SSE4.2).
Not great I know, but I can run 7B stuff with llama.cpp, so I should be able to do something with candle too, right?
First of all, this CPU doesn't have AVX2, so I had to use RUSTFLAGS='-C target-feature=-avx2' to avoid the dreaded Illegal instruction.
(I knew this because I had sone a similar thing with llama.cpp.)
But even then, the simple example in the "get started" section wouldn't work for me. It just exited with the message
I noticed that in the backtrace there's a reference to gemm_f32::gemm::f32::fma::gemm_basic and the file gemm-common-0.16.14/src/gemm.rs.
So what I had to do was to comment out all the references to fma in that file, and that seems to have done the trick :D
I guess the gemm library is incomplete somehow?
Anyways, the end result is that I can now run madlad400-3b-mt, which was my goal.
Maybe not the speediest thing in the world, but I'm happy that it now works :D