262K on 32 GB: The Serving Stack That Changed the Desk

For the newer controlled context battery, including approximately 236K filled tokens, 60/60 needle recall, and 20/24 distant-fact synthesis, start with the standalone capability report.

I thought this investigation was finished.

The first round had already produced the conclusion I expected to publish: close the runaway thinking channel, turn empty answers into usable work, then route the right work locally. Own the substrate. Rent the frontier.

Then I changed the software around the model.

The GPU did not change. The base model did not change. But the delivered system did.

The original Q4_K_S stack measured 53.9 tok/s with a 32K window. The optimized daily profile allocated 262,144 tokens and measured roughly 138–145 wall tok/s on the HTML and SVG work I actually use to judge iteration speed.

That is the story of Part 4: same desk, different machine.

Same desk, different machine

The result was not one magical switch. It was a complete serving profile: llama.cpp b10488, one parallel slot, NVFP4-MTP-Q8attn weights, q4 KV cache, the native 262K allocation, and the model's built-in multi-token-prediction draft head at n-max 3.

Those rates are not directly interchangeable benchmarks: the Phase 1 number is a decode measurement, while the optimized range is end-to-end wall throughput on real artifacts. They describe the operating experience honestly; they do not isolate a single variable.

The more surprising question is why a 262K allocation fits on a 32 GB card at all.

Why 262K fits on 32 GB

Qwen3.8-27B is a dense 27B vision-language model. It is not a mixture of experts. Every token activates the model's parameters.

But dense does not mean conventional.

The model has 64 layers. Only 16 are full-attention layers, the kind that retain keys and values for prior tokens in a KV cache that grows with the sequence. The other 48 use Gated DeltaNet, a linear-attention design with a compact recurrent state.

For a newcomer, the practical distinction is simple:

• Full attention remembers the conversation by keeping a growing set of token records.
• DeltaNet carries forward a compressed state instead of retaining the same growing record at every layer.

So the KV cache grows meaningfully on one quarter of the network, not all 64 layers. That hybrid layout is the whole 32 GB story. A conventional 27B model with full attention in every layer would not have the same context economics.

The model's native context is 262,144 tokens. It advertises YaRN extension to one million; I did not test one million. The optional 0.86 GB vision projector was also not loaded. Both caveats matter.

The architecture made the window possible. The serving bakeoff determined whether it was useful.

Three rungs of evidence

Serving benchmarks become misleading when unlike measurements collapse into one number. I used three rungs instead: an easy synthetic probe to tune MTP, real artifacts to judge daily speed, and a narrow long-context recall test.

Each answers a different question.

Rung 1: the optimistic count probe

Qwen3.8 ships a built-in multi-token prediction (MTP) draft head. In plain language, it proposes several next tokens at once; the main model verifies the run and keeps what is correct.

On an easy prompt, acceptance can approach 100%. That makes MTP look extraordinary.

The prompt asked the model to count from 1 to 250, one integer per line. That regularity is almost ideal for speculative drafting. Q5_XL rose from 59.6 tok/s with MTP off to 174.9 at n-max 4; Q6_K rose from 54.6 to 165.2.

That is a real result. It is not the daily-work result.

Rung 2: artifacts people actually use

HTML and SVG had approximately 64–78% MTP acceptance, not nearly 100%. They landed around 120–145 wall tok/s rather than 175.

The decision rule was set before the grid: quality wins ties and modest speed gaps; a profile that cannot hold at least approximately 128K loses; and Windows needs roughly 1.5 GB of VRAM headroom to avoid a desktop-compositor surprise.

*Q6_K's generated self-test contradicted itself: it called get(4), which moves key 4 to most-recently-used, then expected put(5) to evict key 4. That disqualified the sample. It does not prove the cache implementation itself was wrong.

NVFP4 passed the LRU task and led the measured artifact-speed table. I selected this specific mixed-precision artifact because it combined that result with native-context headroom, not because “FP4 always wins.”

Rung 3: allocated context and tested recall

The context grid produced the most counterintuitive result. Reserving a larger window did not make a short prompt decode more slowly. The model still has to prefill the tokens actually sent, but unused allocation was not a short-chat tax in this test.

q8 KV at 262K on the selected NVFP4 weights did not fit. q4 KV at 262K did, at approximately 28.2 GB. The daily profile measured 168 tok/s on the optimistic count probe, compared with 170.9 at 32K q8 and 169.9 at 164K q8.

Then I tested recall. Every profile recovered an exact token buried in an approximately 48,151-token prompt. NVFP4 prefilling was about 3.1K tok/s at that length, versus roughly 2.2K on Q6.

That is the boundary of the evidence. 262K was allocated; approximately 48K was actually tested. One recovered token is useful evidence, not a 262K RAG evaluation or proof of general long-context reliability.

What the larger window is for

Context is the model's working surface. A larger allocation creates room for document packs, long code or research sessions, and agent traces that would otherwise need to be summarized or discarded. That matters operationally because unattended agents accumulate history far faster than a person typing into chat.

It does not make the model more intelligent, guarantee that it will use every token well, or remove retrieval and citation requirements. The window is capacity. The 48K needle is the tested evidence. Those are different claims.

The profile I kept

The daily command, stripped of local paths and private endpoint details, is short:

llama-server
  -m <Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf>
  -c 262144
  -ngl 999
  --parallel 1
  --cache-type-k q4_0
  --cache-type-v q4_0
  --host 0.0.0.0
  --port 1234
  --alias qwen3.8-27b
  --jinja
  -fa on
  --spec-type draft-mtp
  --spec-draft-n-max 3

The service remains private. No public endpoint is required, and none is published here.

What this changes

The first three parts of this series argued that local capability is a portfolio decision, not a hobby. This bakeoff sharpens that judgment.

Long context is no longer automatically a reason to rent. It still depends on architecture. A conventional full-attention model pays a different KV-cache bill. Qwen3.8's hybrid layout makes native context practical on consumer hardware.

Throughput is no longer the obvious agent bottleneck for this class of work. Real one-shot artifacts at roughly 140 tok/s make iterative and unattended loops materially different from the original 53.9 tok/s baseline. Whether that creates savings depends on whether the capacity is actually used; a faster machine does not create demand by itself.

The stack is part of the harness. Prompt posture, output contracts, engine, quantization, KV precision, context reservation, and speculative decoding all determine delivered capability. “Which model?” is still the wrong first question.

The better question is: which complete system produces the work you need, at the latency, privacy, and unit economics you can defend?

Continue with the controlled capability report

This article is the five-part investigation's serving-stack capstone. For the strategic conclusion and controlled evidence across bounded work, artifacts, and long context, read Useful AI Is Now Cheap: What One Consumer GPU Proved. That standalone controlled capability report is the strategic entry point; this series preserves the investigation and technical depth behind it.

What I did not test

• the advertised one-million-token YaRN extension
• a filled 262K prompt or comprehensive long-context RAG suite
• vision with the projector loaded
• NInfer, whose authors claim higher 5090 throughput on a Linux-only stack
• kekzl/imp, vLLM, or SGLang
• Q8 or BF16 weights
• repeated quality samples sufficient for statistical quality claims

One run can identify a promising operating point. It cannot settle every serving-stack question.

Method notes

• Measured August 19, 2026, on one consumer RTX 5090 system.
• Wall tok/s equals completion tokens divided by HTTP round-trip seconds.
• One sample per quality prompt unless noted.
• Count-probe speeds rank speculative-decoding settings; HTML/SVG wall speeds describe how the system felt.
• The architecture explanation and all tables in this post are transcribed into a dated, sanitized lab artifact. Private hostnames, IP addresses, machine names, and absolute paths are deliberately omitted.
• Series hub: Qwen3.8: Same Weights, Different Product
• Demo gallery: six verified one-shot apps