AlphaLlama v2: Qwen3.8-Flash-Next at 602 tokens a second on one RTX 5090
2 November 2026 · AlphaChat Team
Update, 3 November 2026: AlphaLlama 2.0.1 fixes a server crash on certain long Qwen3.8-Flash-Next prompts. Get it from alphachat.com/docs. The benchmark figures in this post were measured on 2.0.0.
AlphaLlama v2 is out, open source under the Apache 2.0 licence. It's a local inference engine for people who run large language models on their own NVIDIA GPU, and it's built for single-user decoding speed.
This post covers:
- the numbers;
- how we measured them;
- where it's faster and where it isn't.
Every figure below is on InferArena (2026-10 edition, results as of 2 November 2026), with its GPU, model, mode and date.
The headline: a ~125B model on one RTX 5090
Qwen3.8-Flash-Next is a mixture-of-experts model of about 125 billion parameters. We run it as an IQ3_XXS GGUF, the same file for both engines below. It doesn't fit in GPU memory, so the experts are streamed from system memory.
On a single RTX 5090, AlphaLlama v2 decodes it at 602.4 tokens a second on its best task, editing a C++ source file. Strata 0.2.8's best on the same GPU and the same model file is 425.1 (rewriting a C++ file, at its tuned setup). That's 1.42x, each engine's best against the other's.
| GPU | AlphaLlama v2, best task | Strata 0.2.8, best task | Best vs best |
|---|---|---|---|
| RTX 5090 | 602.4 tok/s (edit a C++ source file) | 425.1 (rewrite a C++ file; its tuned setup) | 1.42x |
| RTX 3090 | 288.5 (edit a C++ source file) | 181.0 (add Doxygen docs to a C++ file) | 1.59x |
| RTX 4080 16 GB | 231.8 (edit a C++ source file) | 180.7 (rename across a Python file) | 1.28x |
On the same prompt ("edit a C++ source file"), Strata runs at 378.6, 183.4 and 170.6 tok/s on the three cards. AlphaLlama leads by 1.59x, 1.57x and 1.36x.
These are fast-mode results. Fast mode is not token-identical: output may differ slightly from plain decoding. They're also peaks, measured on copy-heavy code edits where drafting helps most, not typical speed.
The full picture against Strata
- Overall: across all 25 InferArena prompts, AlphaLlama v2 is 1.09x Strata (1.07x to 1.13x per GPU).
- Edit & rewrite: AlphaLlama leads by 21–33%.
- Data & logs: AlphaLlama is ahead.
- Read & explain: Strata leads by about 6–11%.
The full per-task breakdown is on the board.
Qwen3.6-35B-A3B and Qwen3.8-27B on one RTX 3090
On a 24 GB RTX 3090, against Ollama 0.40.11 on the same card, the same model and in the same mode:
| Model | AlphaLlama v2 | Ollama 0.40.11 | Mode |
|---|---|---|---|
| Qwen3.6-35B-A3B | 1,175.2 tok/s (best task: rename across a C header) | fast | |
| Qwen3.6-35B-A3B | 913.2 (code-edit tasks, average) | 232.5 with MTP drafting: 3.9x | fast |
| Qwen3.6-35B-A3B | 409.4 (code edits) | 155.0: 2.6x | exact |
| Qwen3.8-27B | 658.5 (best task: rewrite a Python file) | fast | |
| Qwen3.8-27B | 482.8 (code-rewrite tasks, average) | 111.8 with MTP drafting: 4.3x | fast |
| Qwen3.8-27B | 159.2 (code-rewrite tasks, average) | 43.0: 3.7x | exact |
Ollama ran its own library builds (Q4_K_M, and MTP Q4_K_M for drafting) at its defaults. For everyday chat, the 27B runs at about 140–190 tok/s in fast mode.
How it's fast: MTP + n-gram drafting
Single-user decoding is limited by memory bandwidth: every token means reading the model's weights again. AlphaLlama gets several tokens out of each read:
- MTP drafting. The model's own multi-token-prediction head guesses the next 2–5 tokens.
- N-gram drafting. When the answer repeats text it has already seen, as code edits and rewrites do, AlphaLlama proposes whole spans of up to dozens of tokens.
- One verify pass. All drafts are checked in one wide pass. On mixture-of-experts models, a confidence gate verifies only the drafts likely to be accepted, since each extra token costs bandwidth.
- One captured GPU graph per decode round, with no per-step rebuilds or host stalls.
The details are in the AlphaInfer white paper.
Exact mode and fast mode
- Exact mode gives output identical to the engine's own plain greedy decoding, on the validated settings. On the 35B and 27B it still drafts. On Flash-Next it is plain decoding.
- Fast mode trades token identity for speed. Its output may differ slightly from plain decoding, and we label every fast-mode number "not token-identical".
- Defaults: on Qwen3.6-35B and Qwen3.8-27B, AlphaLlama runs in exact mode by default;
ALPHA_SPEED_MODE=1switches to fast mode. On Qwen3.8-Flash-Next it runs in fast mode by default;ALPHA_AI_MTP_K=0gives plain decoding, which repeats exactly within a build.
On Qwen3.8-Flash-Next, Strata 0.2.8 has no exact-output mode, so the Flash-Next comparison is fast mode against fast mode.
How we measured
InferArena is run by AlphaChat, and its protocol, prompts and harness are public:
- Each record is the median of two passes that agree within 10%.
- All runs use the same streaming client, on Linux.
- Comparisons are like for like: same GPU, same model file, same mode.
Other engines can submit results for review. If you get different numbers on your hardware, we want to hear about it.
Platforms
Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU, RTX 30-series or newer. The Windows (CUDA, Vulkan, CPU) and macOS packages run the standard engine. macOS packages are available for Apple Silicon (Metal, any M1 or later) and Intel Macs (CPU, AVX2), macOS 12 Monterey or later.
The Windows packages aren't code-signed yet, so some antivirus software may flag them; signing is in progress. Each package on the release page has a SHA-256 checksum.
Get it
- Latest packages: alphachat.com/docs (2.0.1)
- Source and the 2.0.0 release measured here: AlphaLlama v2.0.0 on GitHub (Apache 2.0)
- Downloads: alphachat.com/docs
- All results: alphachat.com/inferarena
- White paper: alphachat.com/docs/alphainfer
Questions, bug reports and benchmark runs are welcome on GitHub. Press: developer@alphachat.com.