eLLM can infer LLM on CPUs faster than on GPUs
-
Updated
Jun 18, 2026 - Rust
#
llm-infernece
Here are 5 public repositories matching this topic...
Efficient LLM inference on Slurm clusters.
inference speech-to-text vlm text-embedding multimodal audio-transcription llm vllm reward-model llm-infernece sglang llm-infrastructure
-
Updated
Jun 18, 2026 - Python
Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability.
-
Updated
Mar 9, 2026 - Python
⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.
macos rust metal self-hosted quantization mlx inference-engine kv-cache apple-silicon openai-api int4 llm local-llm local-ai ollama qwen llm-infernece on-device-inference ollama-alternative llama-cpp-alternative
-
Updated
Jun 19, 2026 - Python
Improve this page
Add a description, image, and links to the llm-infernece topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-infernece topic, visit your repo's landing page and select "manage topics."