World's fastest local inference engine: AlphaLlama

Local LLM inference benchmark: AlphaLlama vs Ollama, vLLM, SGLang, Strata, FreeToken and NInfer

InferArena races local inference engines on the same hardware, so you can pick the fastest way to run an LLM locally on your own GPU. AlphaLlama, powered by the AlphaInfer engine, is measured by exactly the same rules as every other engine, and stays on the board when it loses.

Current ranking (Overall, exact output)

1. AlphaLlama: 3.00× Ollama
2. ninfer: 1.44× Ollama
3. freetoken: 1.05× Ollama
4. vLLM: 1.05× Ollama
5. Ollama: 1.00× Ollama

Fastest single task per engine

• AlphaLlama: 1,175 tokens/s (rename across a C header, Qwen3.6-35B-A3B on RTX 3090)
• Strata: 425 tokens/s (rewrite a C++ file, Qwen3.8-Flash-Next on RTX 5090)
• Ollama: 350 tokens/s (work through a CSV table, Qwen3.6-35B-A3B on RTX 5090)
• vLLM: 278 tokens/s (chat: write a Python class, Qwen3.6-35B-A3B on RTX 5090)
• freetoken: 244 tokens/s (write unit tests, Qwen3.6-35B-A3B on RTX 5090)
• SGLang: 76 tokens/s (chat: write a Python class, Qwen3.8-27B on RTX 5090)

Questions

What is InferArena?

InferArena is an open benchmark for local LLM inference engines. Every engine runs the same models on the same GPUs, is driven by one OpenAI-compatible streaming client, and is scored on decode speed (tokens per second) over 25 public prompts: short chat, held-out code and document tasks, and copy-heavy code edits.

Which local inference engines are compared?

AlphaLlama, Ollama, vLLM, SGLang, Strata, FreeToken and NInfer, on GPUs from a laptop RTX 4070 to the RTX 5090, with Qwen 35B and 27B models and a large mixture-of-experts model that streams from system memory.

Which local LLM inference engine is fastest?

On the 2026-10 lab edition, AlphaLlama ranks first on the main board at 3.00× the speed of Ollama on the same GPU and model, followed by ninfer (1.44×), freetoken (1.05×), vLLM (1.05×), Ollama (1.00×). Rankings differ by category and by task: see the leaderboard for each GPU, model and kind of work.

How does AlphaLlama compare with Ollama, vLLM and SGLang?

AlphaLlama is measured with exactly the same client, prompts and settings as Ollama, vLLM and SGLang. The leaderboard shows its decode speed next to theirs on each GPU and model, the cases where it couldn't run or another engine leads (for example, NInfer on short chat on the RTX 5090), and whether each engine's output is reproducible.

What is AlphaInfer?

AlphaInfer is the engine backend inside AlphaLlama. Its speed-ups include verified multi-token-prediction (MTP) drafting, which keeps the output identical to plain greedy decoding, and expert offload that runs large mixture-of-experts models on 16 GB and 24 GB GPUs.

Is this a fair comparison?

Speeds are only compared on the same GPU and model. Exact decoding (output identical to the engine's own greedy decode) and fast modes that change the output are ranked separately, failures are listed with their reason, and every number comes from per-prompt token files. Anyone can run the open harness on their own engine and submit a result.