Add inference backend for Linux

[PrimalPimmy]

I'd like to add support for Linux. The idea is to use llamafile as the backend since it's portable and performs well on CPU/GPU without complex setups.

The plan is:

1. Update the Rust CLI to work on Linux (currently only the python server works on Linux, while the CLI runs on Linux but unimplemented).
2. Implement the Linux backend (server/backend/linux.py) to manage the llama cpp process using https://github.com/abetlen/llama-cpp-python.

I see that we have a cpu.rs file to implement that. However llamafile does support both CPU and GPU, might want to change the names to fit both later no? Or is there only CPU specific use case we are targetting?

Nonetheless, I have started to make a prototype for this.

[madclaws]

@PrimalPimmy Great !!

1. We use py server, because there is no support for mlx other than in python and swift (which is very early). But if we could run the models using rust/c libraries in linux, then we don't need to rely on python. But if that's not the case, lets use py server for linux too.

Regarding cpu.rs it's just a placeholder for linux. You can change it to whatever prefer.

[PrimalPimmy]

thanks @madclaws , I'll take a note of that. Currently I am just prototyping, so I will use the fastest way to get it working before I refactor, hence I'll use the py server for now.

[madclaws]

Also @PrimalPimmy Rust CLI works on linux too, it will show an unimplemented error, since no implementation is there.

[PrimalPimmy]

Indeed. Sorry I meant it was unimplemented. Will update the issue

[PrimalPimmy]

I have made a prototype:

I think we don't even need to go with llamafile, as I made the server run with llama cpp python API.

This prototype runs on the CPU on Linux nicely, we can implement GPU as well if it is available, or use flags.