How AlphaInfer makes local decoding fast

1. The problem: batch-1 decode is a byte problem

Generating one token for one user means reading every active weight of the model once, plus the attention cache, and doing very little arithmetic on each byte. On a consumer GPU this makes decode memory-bandwidth-bound, not compute-bound. The cost of a token is set by:

AlphaInfer attacks all four. Its working rule comes from repeated measurement: at batch 1, only removing bytes, removing launches from the critical path, or committing more tokens per weight pass pays. Fusing work into a kernel that is already bandwidth-saturated moves bytes around but saves no time.

2. One captured graph per decode round

A decode round (one forward pass over 1–8 token rows) is recorded once and replayed as a single CUDA graph. Everything that changes per request is ingress data written into fixed buffers, so graphs are never rebuilt during generation: token ids, positions, which expert lives where, and draft tokens. The engine owns its round capture, mixing ggml subgraphs and AlphaInfer's native kernels in one graph. It has no graph IR and no virtual dispatch inside the layer loop.

Measured on an RTX 3090, Qwen3.6-35B-A3B Q4_K_M, September 2026:

The 1.14× was measured on one 1,024-token prompt, 3 interleaved runs per arm, against a fork baseline of 150 tok/s with CUDA graphs on.

What did not pay. Saving a launch inside an already-captured graph is worth roughly nothing; replay hides it. What pays is deleting a latency-bound kernel from the critical path, which saves its full duration.

3. Speculative decoding, priced by the verify column

Speculative decoding drafts several tokens cheaply, then verifies them in one wider pass. Every accepted draft is a token that cost no extra weight pass. AlphaInfer supports three drafters:

• MTP head. The model's own multi-token-prediction layer drafts 2–5 tokens.
• Prompt lookup / suffix copy. For copy-heavy work (editing a file, renaming across a file, rewriting code), the draft is copied from the prompt or the output so far. It can be up to 16–64 tokens wide.
• DFlash2 block drafter for the dense 27B.

3.1 The insight: a draft's real price is its verify column

Profiling the 35B on the 3090 (2026-09-18) split the cost of one extra MTP draft into two parts:

The column is expensive because, in an MoE model, each extra verified token routes to experts the other tokens did not need, so it adds weight bytes. Tuning the drafter attacks the smaller half. AlphaInfer therefore optimises which drafts are worth verifying.

3.2 Confidence-gated verify width

The MTP head's max-softmax confidence is well calibrated: the probability that a draft is accepted tracks it closely. AlphaInfer verifies the leading drafts only while their cumulative confidence stays above a threshold τ. A draft is worth verifying only if its expected acceptance exceeds (column cost ÷ time per token).

The gate is off for the dense 27B in fast mode, where a column costs only ~0.9 ms against ~5 ms per token, so it cannot pay.

3.3 Script-adaptive draft vocabulary

To keep drafting cheap, the draft head scores a 131,072-row subset of the vocabulary. A fixed "first 131K ids" subset left 79% of Arabic, 55% of Russian and 38% of Korean output tokens impossible to draft. AlphaInfer picks the same number of rows per request from the Unicode scripts in the prompt, so drafting costs the same.

English was unchanged. A benchmark run on English prompts only had hidden this.

3.4 Copy-heavy work: many tokens per round

When the output largely repeats the input, lookup drafting commits whole spans per round. The 3090 maximum-speed record (2026-10-21) came from this: 1,153.5 tok/s single-stream on the 35B in fast mode, on a rewrite-the-file prompt. It averaged 24 tokens per round at 21 ms per round, with all 504 lookup drafts accepted.

3.5 Two drafters, two probability lines: MTP and the n-gram copy

Both drafters are judged by the same quantity: for the k-th drafted token, the probability that the verify pass accepts it, given that drafts 1 to k−1 were accepted. We call this the conditional hit rate. Multiplying the conditional rates for 1 to k gives the chain survival: the chance that a draft delivers at least k tokens. The two drafters draw very different lines, and how AlphaInfer combines them follows from that shape:

• MTP starts high and decays with depth.
• The n-gram copy starts lower on new text, but once it aligns it stays flat, because it is copying.

MTP: high at depth 1, falling with depth. Measured with MTP only, so the copy never takes the easy rounds:

• The falling line is the chain survival: it loses 10–13 points per depth on new text.
• The conditional rate stays flat, because min-p 0.5 stops drafting before a low-confidence token. A draft the head is unsure of is never verified.
• The head is well calibrated: its own top-1 probability is within 1–4 points of the measured hit rate at every depth.

Each extra depth also costs a verify column plus a draft step (§3.1), so MTP is capped at depth 3, which delivers 3.2–3.6 tokens per round.

The n-gram copy: low where text is new, high and flat where it repeats. Replayed at every output position, so this is what the copy would have done whether or not it was used:

• Short requests (A). The copy rarely applies (15.6% of positions), and its first copied token is right only 48% of the time, against MTP's 87%. Its line is the low one.
• Held-out tasks (V). The first copied token (0.844) is still below MTP's first draft (0.892), but the copy's conditional rate rises with offset, to 0.97 by token 15: a copy that has started to match keeps matching. Where it runs, it delivers 8.6 tokens per matched position, against MTP's ceiling of 4.
• Edits (R). The high line: 0.968 on the first token and still 0.828 chain survival at 15 tokens, so a matched position yields 13.3 tokens on average, which no MTP depth can approach.

The copies the server actually takes are better than this replay, because its policy filters out weak ones before they are verified: on V the first copied token is accepted 98.8% of the time, and chain survival at 15 tokens is 0.494 instead of 0.420.

How AlphaInfer combines them. Each round prefers MTP and switches to the copy only when two conditions hold:

• A suffix match exists and MTP's own first draft agrees with the copy's first token, which catches spurious repeats in new text.
• For short matches (under 6 tokens), the copy's expected tokens per millisecond beats the MTP draft's by 3%. The expectation is measured online for each match length.

On short requests the copy almost never fires and MTP carries the set. On held-out tasks the copy takes 12% of rounds but delivers 11.5 tokens in each. On edits it takes nearly half the rounds at 14 tokens each and halves the number of rounds. A verify round costs about the same however wide it is (§3.1), so rounds saved is time saved.

Result against Strata 0.2.8. The InferArena run of 2026-11-01/02 (AlphaLlama v2 release candidate in fast mode against Strata 0.2.8 at its own 32K launch configuration, one window per GPU, two passes with the second in reversed order, a fresh server per arm). Decode tok/s, per-prompt mean of the two passes, geometric mean per set:

The R lead is the copy line above. On A and V, where MTP carries most rounds, the two engines are level. Across the three GPUs the InferArena fast board's Speed Index is AlphaLlama 1.09, Strata 1.00.

4. Exact mode and fast mode

Speculative decoding can change the output text: a wider verify pass can round differently from a one-token pass, and in a near-tie that flips a token. AlphaInfer offers two modes.

Making speculation exact took specific engineering:

• Column-position invariance. The same token is verified in different column slots in different rounds, so every 2–8-column kernel must give bit-identical results for a column whatever its index. One tensor-core kernel broke this because the compiler contracted the even- and odd-column epilogues differently. It was fixed with explicitly rounded FMA/multiply intrinsics, and every verify kernel is now gated by a column-rotation test.
• Batch-invariant attention where the verify width would otherwise change the attention split.
• A teacher-forced comparator. It forces the drafts to the plain run's tokens and diffs logits bitwise at each position. Matching generated text over a short horizon is not evidence of exactness: our early "MTP == plain" results were argmax agreement over short outputs, and the comparator showed the first decoded position already differed at 2e-3.

Status as of 2026-10-29

The 35B at ctx 98,304 (staging off) is still open. The Flash-Next reset costs +0.3–0.5 s TTFT per request (the expert slots are recopied); decode is unchanged.

5. Running models larger than the GPU: expert offload v2

MoE models such as Qwen3.6-35B-A3B (22.7 GB) and Qwen3.8-Flash-Next (~125B parameters, ~71 GB at IQ3_XXS) do not fit a 16–24 GB card. AlphaInfer keeps the hottest experts in VRAM and serves the rest from system RAM. The design goal is no host round trip inside a decode round.

5.1 What the measurements taught us

• Shared slot banks. Per-layer fixed slots capped the warm hit rate at 98.5%, because early layers touch far more distinct experts. One shared bank across layers with the same expert types removed those warm misses entirely, using the same memory: 146.1 tok/s vs the fork's 137.1 on the 35B on a 4080, and 295 tok/s with MTP k = 3.
• "Async" copies are not free. Admission DMAs queued at the round boundary delayed the next round's own small uploads, because they share the link in issue order. Queuing them after the round launches made rounds +10–11% faster, with identical tokens (3090).
• Time the host before blaming the GPU. On Flash-Next, a 3.7 ms per-round gap was serial reads of hashed embedding rows from a 29 GB table on NVMe, not GPU kernels. Prefetching the round's rows first (fadvise WILLNEED) took one prompt from 97.9 to 111.7 tok/s, identical text.
• CPU thread placement is a first-order lever. On a 64-core Threadripper, a spinning miss pool on every core starved the GPU-launch thread through its SMT siblings. Capping and spreading the pool took plain decode from 66 to 112–116 tok/s with the same hit rate. AlphaInfer now isolates the launch thread's core by default.
• Prefill copies can block the host. Thousands of per-expert copies filled the copy queue and stalled kernel launches for 1.0 s at 16K tokens. Coalescing consecutive experts into one copy cut 16K prefill from 6.9 to 6.5 s with identical output (4080).
• Batched state folds. After each speculative round, the recurrent (GDN) state of every layer is folded with 3 launches instead of 72+, on a side stream that overlaps the next draft (3090: commit 0.70 → 0.34 ms per round).

5.2 The automatic path: AlphaServe detects the model and the GPU, then picks the offload layout

A user should not have to know whether a model fits. alphallama-serve -m model.gguf, with no other flags, reads the GGUF metadata and the GPU's free memory before loading anything, then sets every switch the operator left unset. It reads the architecture, MoE or dense, the size and bits per weight of the expert tensors, and whether a draft head is in the file or in a sidecar next to it. Any explicit setting wins; ALPHA_AUTO=0 turns the whole thing off. Because some engine switches are read before main(), the server re-executes itself once with the chosen settings, so they actually take effect.

The plain llama-server-derived binary in the same package routes itself the same way. When an MoE does not fit and the command line asks for nothing the server-only engine lacks, it hands the request to AlphaServe. This is roughly 8× faster on such a model than running it partially on the CPU.

Fused-infer: three memory tiers as one expert store

The GPU sees a model several times its VRAM as if it were resident:

Device-resolved experts (DRE). Decode reads each routed expert through that table. A non-resident expert is not copied over PCIe (~12 GB/s on these cards). Instead it is computed by CPU workers where it already sits in RAM (~50–70 GB/s), joined through a doorbell, with one CUDA graph per token instead of ~146 host-synchronised splits.

• The idea behind it: the CPU path is a second bandwidth pipe that runs in parallel with PCIe, so it routes around the link rather than waiting on it.
• It was first shown with the earlier expert cache's CPU-inline split (122B MoE, RTX 3090, 2026-07-19): CPU-only 12.4, GPU pool-direct 14.3, both pipes balanced 15.6 tok/s.
• Flash-Next on the RTX 4080, 2026-10-04, decode at 1K / 4K / 16K, the first DRE build vs the earlier expert cache: 45.6 / 42.9 / 33.2 → 48.3 / 50.4 / 41.6 tok/s.
• The auto profile now picks DRE for every MoE that does not fit, because it beat the expert cache at every pool size tried.

What the measurements taught us about the automatic path

• Pace the admission copies. Admission DMAs share the copy engine with each graph's small input uploads, which are served in submission order. Draft steps took 5.0 ms with admission on vs 1.6 ms off. A pacer that feeds admissions in 256 KB chunks brought drafts to 1.1–1.2 ms (Flash-Next, 4080).
• Size the CPU pool to the host, not to the core count. On a 64-core Threadripper, the default "every core but one" pool starved the decode and driver threads. 32 pool threads took Flash-Next from 66 to 112 tok/s (+70%, RTX 5090, 2026-10-22), with the same ~96% pool hit rate.
• Let the prompt borrow the expert pool. A long prompt streams nearly every expert per micro-batch chunk, about 43 GB over PCIe per chunk on Flash-Next. On 16 GB cards the auto profile prefills a 16–18K prompt in one 18,432-row micro-batch, which borrows pool memory and restores it before decode. That brought 16K time-to-first-token to parity with Strata on the 4080 (2026-10-20).
• Reuse the recurrent state across turns. Hybrid (attention + recurrent) models cannot drop a KV suffix, so chat turns used to re-prefill everything. Checkpointing the recurrent state at the prompt boundary makes the next turn continue from it. The output is byte-identical, and multi-turn TTFT fell from 3.1 to 0.32 s (−90%).
• Pinned memory has a host-side cost. The pinned RAM tier lets a 71 GB Flash-Next run on a 62 GB-RAM 3090 box with no settings. But any large pinned region feeds the kernel's memory-compaction daemon, which can make some starts run slow on a busy box (seen on both engines). Huge-page hints made this worse, so the tier is filled without them by default.

Measured on a bare -m (server decode counter, 2026-10-13)

Where a validated AlphaInfer row exists, it is faster than this generic path. On the 4080 Flash-Next protocol run (2026-10-28), AlphaInfer v2 gave 113.3 / 129.0 / 201.0 against the built-in path's 73.8 / 70.6 / 81.2. The automatic path is the floor: it makes any MoE GGUF run well on any card without a validated row.

6. Results

6.1 Dense and MoE models that fit: AlphaInfer vs NInfer (RTX 5090)

Same rented box, interleaved, NInfer at its own published invocation.

On a 12-prompt mix, d0–d3 plus 8 languages (2026-10-07, n = 36 paired):

NInfer leads on English MTP rounds; AlphaInfer leads on non-English prompts, through the script-adaptive draft rows.

6.2 A 22 GB model on a 16 GB card (RTX 4080, AlphaServe auto config, 2026-10-24)

6.3 Flash-Next (~125B MoE) with expert offload vs Strata 0.1.47 (2026-10-28, superseded)

Same IQ3_XXS file. AlphaInfer v2 in fast mode (not token-identical) (MTP k = 2, FAST verify, auto slots), selected with alphallama-serve --backend alphainfer: Flash-Next has no validated AlphaInfer row yet, so a bare -m still routes it to the built-in fused-infer engine (section 5.2; on the same protocol the built-in engine gave 73.8 / 70.6 / 81.2 on the 4080, 2026-10-28). Strata 0.1.47 (commit 011c9e60) at its defaults: spec 4, route tail skip 7, which skips low-ranked experts.

Benchmark protocol, through the AlphaServe server (2026-10-28). Set geometric means, one config for all prompts, both engines in the same window:

On the 4080, AlphaInfer leads Strata on all three sets. On the 3090, it leads on copy-heavy work and trails on A and V.

The engineering runs below break these down by prompt length. They are single instances, measured through the C API test program (the calls the server makes), not through the server.

RTX 4080 16 GB (decode tok/s, request 1 / request 2 with prefix reuse). AlphaInfer on 2026-10-27; R with the suffix-copy drafter on. Strata from the 2026-10-26 same-box run, a different day:

RTX 3090 24 GB (256 tokens, 2026-10-28, both engines in one window):

Reading these honestly

• On the 4080, AlphaInfer leads on fresh and copy-heavy text, and Strata leads on the 16K-token prompt.
• On the 3090, Strata still leads everywhere except copy-heavy work, in both the protocol and the engineering runs.
• The 3090 gap is round cost, not drafting: both engines accept ~2.4–2.5 tokens per round. Our decode cycle is ~25 ms against Strata's ~17.4 ms.
• Strata also keeps 90–97% of routed experts in VRAM by adapting to the context, against our ~74%.
• Closing this gap is active work. Single-instance numbers swing about ±10% run to run in both engines.

7. How we measure (and why it matters)

Most wrong performance conclusions we have made came from the measurement, not the engine. Our standing rules:

1. Use the competitor's protocol. Same prompts, context, chat template, token count and box. A speedup measured at a 7.5K-token context shrank from +20% to +0.2% at a competitor's ~200-token protocol.
2. Always run a same-session control arm. Throughput drifts ~4% across sessions on the same box.
3. Decompose before crediting. Split tok/s into ms per round (engine speed) and tokens per round (acceptance, which follows the generated text). A kernel change that alters the text can "win" by luck of the trajectory.
4. Time uninstrumented. Node-level GPU tracing inflates eager launches far more than captured graphs, so tracing makes a graph-based engine look relatively better than it is.
5. Check the hardware state. A PCIe link stuck at Gen1 and slow-host starts (memory-compaction stalls, about 1 in 3 starts on one box) each produced fake "regressions".
6. Re-sweep defaults on every new GPU architecture. One routing kernel was +0.85% on the 3090 and −3.0% on the 5090. Defaults are now keyed to the architecture they were measured on.

8. Limitations and open work (as of 2026-11-02)