How Neural runs a 120-billion-parameter model on one gaming PC, and what was done to make it fast.

A 117B-parameter model on one gaming PC

Neural Server runs OpenAI's gpt-oss-120B on one RTX 3080 Ti with 12 GiB of VRAM and 64 GB of system RAM. It exposes an OpenAI-compatible API. It is a new design, built after the Neural research project closed in August 2026 at 0.98× llama.cpp: the CPU computes the missing experts where they sit instead of shipping them over PCIe. The August verdict on the original runtime still stands; this page is about what replaced it.

The core idea is to keep the dense core and about 430 expert slots in VRAM (250 static and 180 adaptive from the documented 5.5 GiB pool; about 500 with the launcher defaults now in use, --kv-ring 256 --scratch 6 --pool 6.05 and the packed store, which add about 45, 10 and 14 slots, CALCULATED). Every other expert is computed on the CPU directly from RAM, at the same time as the GPU computes the resident ones. Missing experts are therefore not copied over PCIe at 13.2 MB each.

The October 1 selected profile (v10) adds persistent CPU workers, bounded grouped prompt GEMMs and release of unused CUDA allocator cache. The new CPU change alone improved weighted decode from 23.8649 to 24.2807 tok/s, +1.74%, in the adaptive-placement study (MEASURED). A separate frozen-placement control improved from 19.8659 to 20.4391 tok/s, +2.885%; freezing is slower overall and is not the recommended configuration. The first-pass warm 3k prompt study reduced latency from 3.69655 to 3.43665 s, 7.03%, and freed about 1.11 GiB of allocator reservation after a long prompt (MEASURED). These are separate studies, not additive percentages.

A fresh October 1 Neural / llama.cpp / llama.cpp / Neural comparison replayed the same four saved API requests, beginning with a 13k coding context. Weighted all-answer decode was 24.6910 versus 14.3189 tok/s, 1.7244×; follow-ups were 25.0756 versus 14.7957 tok/s, 1.6948× (rates MEASURED, ratios CALCULATED). Initial prompt processing was 8.66645 versus 137.552015 s, 15.8718×, in fresh sessions. That is a fresh/session-start comparison against decode-tuned llama.cpp b10361, not the earlier warm-prompt metric or a 72% gain from the new CPU change.

The Sep 30 (4), v9 results remain historical: a warm 13k prompt took 6.8 versus 95.2 s, 13.9× (MEASURED, one run each; ratio CALCULATED). Follow-ups were 24.13 versus 15.33 tok/s, 1.57×, over nine turns per engine. Its 1.96× first-answer ratio used unusually slow llama.cpp first answers. No prompt-tuned llama.cpp configuration was measured in either study.

Correctness has several parts. The raw expert store is a byte-exact repack of the checkpoint's MXFP4 tensors. The optional packed store, used for the Sep 30 and October numbers, keeps the code bytes and re-encodes the scale rows so that outputs are bit-identical. The fuse, packed store, kernel scheduling, layer-order prefill, 64-row tiles and staging thread were checked as bit-identical at frozen residency (--refresh-every 0). The larger pool and --refresh-m 16 (admission rate) are device-class: they change which experts the GPU computes and can change answers. Adaptive residency depends on timing, so runs can differ byte for byte, as already observed on v4. The fused Triton core is not bit-exact against eager PyTorch, so its quality was judged against a fp32-order null envelope (4 sigma, 3 nulls; router agreement 861/864 layers) and accepted (ARCHITECTURE.md section 8).

The October changes preserve the same checkpoint, native MXFP4 precision, expert-selection rule and context capacity; no added quantization was introduced. The persistent kernel passed 432 native bit comparisons and 192 full-model decode-step comparisons at frozen residency; the selected repository suite passed 103 tests with 23 existing warnings (MEASURED). CPU-only adaptive and frozen serving repeats matched every corresponding generated token, as did the two optimized Neural repeats against llama.cpp. One earlier combined-profile adaptive session diverged on its last two answers. These checks do not establish universal adaptive determinism or numerical/quality equivalence across backends.

Use the opt-in start_neural_optimized.bat for the selected profile. Server CLI defaults and the original launcher/DLL are unchanged. The decode follow-up, first-pass review and their decode / prefill evidence retain commands, archives, hashes and rejected experiments.

Higher batch throughput; task quality is not cleared

A separate experimental backend runs a closed cohort of 16 independent coding requests. One clean split-graph timing screen completed 42,639 generated tokens in 1,454.0354679 seconds: 29.324594 aggregate tokens/s (MEASURED). Its cohort clock includes all prompt processing and K/V copying, followed by batch decode through natural final stops. The generated count includes 16 prefill-produced first tokens; the decode loop made 42,623 calls with zero fallback. The clock excludes startup, graph capture, diagnostic checking and HTTP. This is not HTTP client latency, a multi-turn tool workflow, or a single conversation's decode rate.

Full prompt and generated token IDs match the historical combined-graph batch B1/B2 workload, which measured 23.595025 and 23.622681 aggregate tokens/s. The clean split screen is 24.14–24.28% higher in throughput, or 19.44–19.54% lower in cohort wall time (CALCULATED). These are different percentage conventions. This is a later single screen against historical reports, not a contemporaneous ABBA comparison, a replicated result, or an isolated causal estimate. The earlier adaptive rates used different outputs and token counts and are excluded from this same-output chart; they do not establish a quality-equivalent gain.

The strict coding-task quality gate is NOT CLEARED. Frozen blinded static review checked the final implementations against predeclared task rubrics and found paired rubric regressions. All 64 responses were complete; rubric passes were 5/16 for adaptive A1, 3/16 for batch B1, 3/16 for batch B2 and 0/16 for adaptive A2. The generated code was not executed by that review, and these findings do not establish general model-quality degradation. Numerical equality to the batch reference preserves its answers; it does not supply a quality guarantee. The research does not establish a production or end-to-end gain, and no change is promoted to main. The October 1 production/llama.cpp comparison remains 24.6910 versus 14.3189 weighted decode tokens/s on its separate four-request replay workload.

The grouped multirow numerical audit passed 42,623 decode checks across all 16 natural finals. Its result verifies outputs only and has no timing measurement. The first timing screen was interrupted by a Windows reboot; the identical retry failed before generation at the host RAM-reserve guard. The split timing repeat was deferred and research is paused. No grouped rate is plotted; the chart shows only the two historical combined runs and the one completed split run. Receipt and report hashes, RAM readings, and terminal records are in the benchmark summary and the private final evidence manifest (repository access required).

Benchmark summary: metrics, method and hashes. Immutable private research-results evidence: split raw report, terminal admission, clocks and historical bindings, historical cohort summary, and locked static quality summary and review integrity record. Source links require private-repository access. Model/checkpoint and arithmetic precision are unchanged.

What it runs on and what it runs

The development machine has an RTX 3080 Ti with 12 GiB, an i7-11700K with AVX-512, 64 GB of DDR4 at 3001 MT/s (XMP; nominal DDR4-3000) in four DIMMs, and an NVMe drive. The server needs Windows 10 or 11, an NVIDIA card with at least 12 GiB, CUDA 13, an AVX-512 CPU, 64 GB of RAM and Python 3.12.

gpt-oss-120B is a 117B-parameter MoE in MXFP4 with 36 layers and 128 experts per layer. Four experts are active per token. One token therefore touches at most 144 of 4,608 experts, which is 3% (CALCULATED).

• Attention: hidden size 2880, 64 query heads, 8 K/V heads, head dimension 64. Layers alternate between sliding-window (128 tokens, 18 layers) and full attention, with attention sinks.
• One expert is 13,219,200 B raw and 12,839,040 B in the packed-scale store (MEASURED).
• The expert store is 57 GiB raw or 55.1 GiB packed (CALCULATED from the per-expert bytes). The checkpoint on disk is about 61 GB.
• Expert bytes touched per token are at most about 1.9 GB (CALCULATED as 144 × 13.2 MB). The budget section uses 88 experts, which is the per-token count of CPU-read experts, meaning the misses; the rest of the 144 are VRAM hits.
• Context length is 16,384 tokens.

VRAM holds the dense core in bf16 (about 2.9 GiB, CALCULATED), the K/V cache (1.1 GiB at 16,384 tokens, CALCULATED) and an expert slot pool. The original pool was configured at 5.5 GiB, giving 430 usable slots (250 static, 180 adaptive) plus 16 scratch. The launcher defaults later became --kv-ring 256 --scratch 6 --pool 6.05. --kv-ring 256 frees about 0.55 GiB, about 45 slots; --scratch 6 frees about 10 more, and the packed store adds about 14 at --pool 6.05 (CALCULATED).

Under the original 5.5 GiB pool and the raw store, the CPU-side store in RAM holds 4,178 experts in 51.4 GiB (CALCULATED from 4,178 × 13.2 MB). The packed store and larger pool reduce this. That is at the edge of what 64 GB has free next to other programs; the later runs cite 54–57 GiB free. Pinned buffers take about 0.3 GiB.

How one token and one prompt are processed

Memory topology: NVMe holds the checkpoint and the expert store. The store is memory-mapped, and the OS page cache in RAM serves the CPU kernels at 24–29 GB/s effective in the v4 server (MEASURED). That is an end-to-end rate for the mapped file, lower than the 38.6 GB/s ceiling of the kernel probe used in the budget section. PCIe carries admissions and prompt staging, with pinned host-to-device transfers at 21.4 GiB/s (MEASURED on this host in the earlier research project). Resident experts are read from VRAM slots.

1. Each layer is one CUDA graph of five fused Triton kernels: QKV, attention with split-K, output projection, router and top-k. These five kernels replace about 94 PyTorch kernels per layer. The whole core takes 3.4–4.0 ms per token instead of 10.0–10.7 ms (MEASURED).
2. The router kernels write the normalized hidden state, 4 expert ids, 4 weights and a hit mask straight into pinned host memory, with no copy.
3. The GPU runs MXFP4 GEMVs for the resident experts. A host thread waits on an event and calls the CPU kernel for 0–4 missed experts (8 threads, AVX-512). The original path uses OpenMP; v10's selected path uses a persistent native team.
4. The next layer's graph reads the CPU result zero-copy. After layer 36 a tail graph applies the final norm, lm_head and argmax or Gumbel sampling.

Per-layer messages bypass the copy engine. Before that change, 13.2 MB admission copies queued ahead of them, and the GPU wait fell from 9.5 to 4.8 ms per token when it was fixed (MEASURED). The 4.8 ms is the same figure as in the Sep 30 (2) row of the budget table.

VRAM residency and admission

The original v4 layout had 250 static slots and 180 adaptive slots. The current selected profile retains 250 static slots and has about 500 slots in total. Static slots hold coding-hot experts and are never evicted; adaptive slots follow the current request. Admission is copy-on-compute. When a miss is an admission candidate and one of 16 capture buffers is free, the CPU kernel writes the expert's bytes into a pinned buffer as it reads them, so there is no second store read. A copy stream then uploads the bytes into a victim slot, and the slot table flips when the copy lands.

Before v9, candidates were refreshed every 8 tokens, or every 4 in the first 32; v9 changed the candidates per refresh with --refresh-m 16. The admission rate is a device-class lever, so answers can differ. Prompt routing counts as at most 16 generated tokens (--prefill-weight 16). The v3 first answer ran at 0.83× llama.cpp (2 runs: 0.67× and 0.99×, MEASURED). RESULTS.md gives two causes: admission chasing the prompt's experts, and page faults from RAM pressure. The prompt weighting addressed the first; VRAM hit rate on the first answer went from 0.177 to 0.311 (MEASURED). The v3-to-v4 step also changed RAM residency, so this is not an isolated ablation.

RAM residency

The Sep 24 explanation was that a long prompt makes Windows trim expert pages, and the next answer faults them back 4 KB at a time. First answers showed 1,000–1,500 page faults per token against 10–350 for follow-ups (MEASURED, v4). This fits the trim explanation but does not isolate it. Three fixes addressed this: --release-resident 1, a warm-up ranking that fills free RAM, and --ws-min 1, a soft minimum working set. A later audit (Sep 30) found that the remaining first-answer faults are mostly compulsory first touches of released adaptive-slot experts, worth at most +1–2% (MEASURED/audit), plus stall windows under RAM pressure.

The CPU kernel

The decode kernel streams experts from RAM with 8 threads. Its threads share rows dynamically (schedule(dynamic)) and meet at one spinning barrier that yields to the OS. Each output row is reduced in a fixed order by a single thread, so scheduling does not change the result. This was verified bit-identical by content-hash comparison at frozen residency (MEASURED).

v10 removes the repeated OpenMP team entry/exit from supported fused decode calls. The caller remains a worker alongside seven persistent native threads, retaining the same 64/32-row chunks, scalar activation, scratch materialization, capture bytes and expert accumulation order. Jobs, settings and shutdown are serialized; capture stores are fenced before completion is published. Request counters report actual persistent jobs and reference-path fallbacks. The separate DLL defaults off and requires --kdll gptoss_cpu_persistent.dll --kernel-persistent 1 --kernel-fuse 1.

Floating-point controls are sampled from the actual OpenMP team, reproduced for the native workers and restored after work. Nonstandard rounding/DAZ/FTZ callers and unsupported configurations fall back to the reference path. Exactness is bounded by the tested environments, not every possible thread floating-point state. The DLL was built with GCC 16.1, -O3 -march=native -fopenmp -shared, for this Windows/AVX-512 host; it is not a portable binary.

Prompt processing

Prompts run layer-major in blocks of up to 4,096 tokens. Tokens are grouped by expert on the GPU. Resident experts use a GPU GEMM from the slot. A non-resident expert with 16 tokens or fewer goes to a CPU multi-token kernel on a worker thread. A non-resident expert with more tokens is copied through a pinned 4-deep ring to a scratch slot for a GPU GEMM.

--prefill-order layer runs each layer over all blocks, so each staged expert is copied once per layer instead of once per block. A 13k prompt staged 8,460 (layer, block, expert) triples; in block mode that meant one copy for each, and layer mode needed 2,717 staging copies for the same 8,460 pairs (MEASURED).

In the sixth pass the GEMM tile height BLOCK_M was raised from 16 to 64: the 16-row tensor-core tiles wasted rows, and the 13k prompt went from 10.78 / 12.22 s to 7.67 / 7.73 s in mirrored runs (MEASURED, step22), bit-identical by the in-process check (logits, every layer's K/V and the ring buffers equal). A staging producer thread that runs the copies of staged experts ahead of the main thread was neutral at BLOCK_M 16, when the GPU was the wall and the main thread waited 4.0 s for staged copies, and useful at 64 (stage wait 0.3 s). A cached Triton launch path shipped with it, exact. A grouped bias/SwiGLU epilogue per slot group was also exact but measured neutral: host launch time fell from 4.4 to 2.7-3.4 s while the wait for staging grew by the same amount, so the 13k prompt is balanced between host and GPU at about 7.2 s (MEASURED, step25), and the grouping stays opt-in. The instrumented split at BLOCK_M 16 (12.3 s) was: host GEMM-launch time 4.0 s over about 12,500 expert-GEMM calls of about 10 pointwise launches each, synchronous staging copies 2.45 s, copy-ring wait 1.3 s, routing sync 3.45 s, with the CPU multi-token kernel's 3.9 s on its worker thread never waited on (MEASURED, step19).

The October 1 grouped GEMM is a different lever: it combines independent expert/block GEMMs into fewer launches while preserving each pair's tiles and ordered dot accumulation. The selected profile uses groups of three, a soft 1,024-row workspace target and a guard of 1,024–4,096 new rendered tokens in a single block. Tiny and multi-block inputs retain the stock path. Address/pitch guards preserve vectorized loads where valid and retain an unaligned fallback. Synthetic and full-model hidden/logit/K/V checks passed; larger workspaces and long-prompt grouping regressed in warm serving, so they are excluded.

At the synchronized prompt/decode boundary, the profile releases unused CUDA allocator blocks without changing live weights, slots, K/V or graph addresses. Long-prompt trials reduced reservation from about 10.830 to 9.715 GiB; the first release cost about 6 ms, charged to prompt time (MEASURED). This is a memory saving, not an independently established decode-speed gain. The warm 3k improvement measures the first-pass profile, not a new timing experiment with persistent CPU workers added.

K/V cache and API

A rolling K/V ring serves the sliding-window layers, which frees about 0.55 GiB of VRAM (CALCULATED; half of the 1.1 GiB K/V). Ring checkpoints let a conversation resume without reprocessing. The API is OpenAI-compatible: /v1/chat/completions (streaming and not) and /v1/models. Tool calls use the gpt-oss harmony format, one per response. The server takes one request at a time, with no concurrent requests and no batching; all throughput figures are single-stream.

From the first server to October 1

The server grew out of the Neural research project on heterogeneous-memory MoE inference. That research closed in August 2026 with a runtime at 0.98× llama.cpp b10361 (MEASURED). A hybrid runtime followed, in which the CPU computes the misses. That hybrid went through three research steps, interleaved against llama.cpp each time: 0.82× with CPU-computed misses alone, 0.85–0.89× on fresh prompts and 1.26× warm with a fused graphed core and copy-on-compute admission, then 1.46–1.56× on short prompts with a static core, locked host memory and copy-engine-free small transfers (MEASURED, research builds). The server versions below start from that design. The Effect column mixes ratios against llama.cpp (v3 to v5) with A/B gains against Neural's own earlier or control build (the +% items from v5 to v10) and prompt times. Absolute values and protocols differ per row, so do not read the column as a trend.

Device-class means the lever changes which experts the GPU computes. A GPU-computed expert follows the pool path's bf16 rounding, while a CPU miss is fp32 throughout. Answers can then differ from before, in the same way that VRAM residency, which depends on timing, already makes runs differ.

The v7 figure of 22.07 tok/s and the v9 follow-up figure of 24.13 tok/s come from different benchmarks (an A/B on Neural alone, and conversation medians against llama.cpp) and must not be chained.

The prompt gain from v4 to v5 is not credited to the code. The unchanged Sep 24 build re-ran the 13k prompt at 15.6 s on Sep 29 (MEASURED), and why the PC got faster is not established. Later controlled and mirrored A/B studies support code gains at v8, v9 and v10; their small session counts and shared-host resource drift still limit precision.

Dead ends, kept on record

Failed levers stay in the record so nobody repeats them. Each entry below carries its evidence. The policy is that levers which change the arithmetic of the same experts are rejected, while levers that change only residency or timing are accepted and marked device-class.

• Cache policy: on a recorded code trace, LRU with admission of every miss reached 0.406 hit rate and Belady 0.632 (SIMULATED). All tested past-only rules moved more bytes than rate-limited copy-on-compute. Only an oracle beat it, by about 18% (SIMULATED).
• Speculation: verifying 8 drafts reads 4.72× the bytes of a single token (a trace-derived estimate, SIMULATED). Prompt-lookup drafting was estimated at 0.79–0.88× (SIMULATED), so it was not built. Both are estimates, not measurements.
• Larger or code-calibrated hot sets: +1–3 hit points out of sample (MEASURED). Prediction was falsified in the earlier research project, not in this evidence.
• GPU reading experts from page cache: bit-exact, but CPU plus GPU reached 28.5 GB/s against 29.2 GB/s for the CPU alone (MEASURED). This is a mapped-file measurement, different from the 38.6 GB/s kernel probe. The GPU reads over PCIe from the same DRAM, so the two paths share the wall: RAM bandwidth.
• Thread pinning (--kernel-affinity 2): −8% (MEASURED). The shipped pin had been a silent no-op and was fixed before the measurement.
• Doorbell (launching the next layer's graph before the CPU kernel finishes, with a GPU spin node waiting on a pinned flag): the first version's spin gave up during a >500 ms first-answer fault storm and the request failed; the robust port (whole-token re-run on a timeout, a mode switch for the first 32 tokens and after a >300-fault token) is exact including injected timeouts, but measured 22.32 vs 22.70 tok/s, −1.7% (MEASURED, 4 mirrored sessions), because the kernel barrier change had already removed the launch slack it targeted. Merged into main off by default as the record; with the doorbell off, the merged build reproduced the pre-merge build's answer hashes turn for turn at frozen residency (MEASURED, step28).
• Re-warming RAM in bulk after a long prompt: with re-warm, first answer 10.7 and follow-up 12.3 tok/s; without it, 13.3 and 12.6 tok/s (MEASURED).
• Larger prompt chunks (--prefill-chunk 8192 or 16384): faster on a 13k prompt but not bit-identical, and +0.3 s on a 3k prompt (MEASURED). Rejected.
• Third pass, also measured: --wait poll no gain (GPU wait 5.6 ms unchanged); admission by DMA from a 40 GiB pinned arena +3%, device-class and needing the pinned RAM; --zc-misses 1 (the GPU reading misses over PCIe) first answers −11% from bus sharing; a host-residual micro-patch cut admission bookkeeping 1.75 to 1.55 ms per token as predicted with no measurable tok/s gain (all MEASURED).
• Sixth pass: the multi-token CPU kernel's scheduling fix is bit-identical and +1–2.5% per call standalone but hidden behind the GPU in the server (its jobs are waited on for 0.0–0.2 s per prompt), not shipped; a cold-expert prefetch knob (a one-byte probe per expert and one PrefetchVirtualMemory call before each kernel call) was neutral in the cold regime it targets, 25.4 / 25.7 vs 25.7 / 25.1 tok/s off vs on with identical answer hashes (MEASURED, step26), and ships off; the grouped epilogue above, neutral.
• October grouped GEMMs with a large workspace passed equality but regressed warm 13k prompts from 7.484 / 7.211 to 10.499 / 10.339 s (MEASURED). A bounded workspace still slowed long prompts by about 2.3% in the completed mirrored study. Tiny prompts showed no useful gain. The selected profile groups only single-block inputs with 1,024–4,096 new rendered tokens.
• Masked GPU GEMVs passed 512 finite synthetic cases and frozen full-model checks but reduced weighted serving decode from 23.8298 to 23.0605 tok/s (MEASURED). The final adaptive control diverged on two answers; the output-matching subset also regressed. Masking stays off. Arithmetic correctness alone does not establish an end-to-end gain when GPU work overlaps CPU work.
• Lossless compression: a read-only 12-expert sample measured 4.6407% fewer bytes with raw DEFLATE level 1, but only 0.299 GB/s single-thread decompression. The sample's zero-order entropy was 3.868 bits per nibble (CALCULATED), not a universal compression bound. This codec cannot meet the runtime's bandwidth demand and was not integrated; model/store bytes are unchanged.
• Direct CUDA graph launch passed output checks but its best measured submission saving extrapolated to about 0.065 ms per token (CALCULATED). The extra handle ownership/RNG constraints were not justified, so ordinary replay remains in use.
• In the research project that preceded the server, seven cache-policy families were killed by oracles on an 80B model, LRU finishing within 2.8% of a cheating oracle, and a perfect next-token knowledge oracle returned no gain because 94% of misses were already pre-satisfied (MEASURED, research runtime); predictive expert paging is dead on this hardware twice over.

Self-review also caught errors in the work. A ring checkpoint bug had inflated a +7% claim, and the corrected figure is +2–3% (MEASURED). A DRAM probe timer bug had reported 52–55 GB/s against a CALCULATED 48 GB/s peak; a perf_counter cross-check gave about 40 GB/s (MEASURED).

Interleaved runs on one PC

The October 1 studies replay exact saved API request bodies: a 13k coding context and three follow-ups, with four answers capped at 512 tokens, temperature 0 and low reasoning effort. Fresh servers run serially in mirrored order: stock / persistent / combined / combined / persistent / stock for the adaptive study; stock / persistent / persistent / stock for the frozen control; optimized Neural / llama.cpp / llama.cpp / optimized Neural for the fresh comparison. Source/context and effective DLL fingerprints, request/response archives and full Neural prompt/generated-token hashes accompany the measurements.

• October decode rates are weighted timed tokens / timed seconds, not medians of per-turn speeds. Each engine's timing convention is retained: Neural times 511 tokens per 512-token answer; llama.cpp reports 512. Two sessions per setting support a modest workload-specific result, not a universal percentage.
• The CPU-only adaptive sessions matched all corresponding prompt and output tokens. One combined session diverged on its last two answers, so the all-session identity audit correctly failed. A matching first-two-answer subset was selected after observing that divergence and is not the headline CPU increment. All frozen-control sessions and both fresh optimized Neural sessions matched their corresponding full token sequences.
• The fresh initial-prompt comparison includes fresh/session-start behavior. The historical Sep 30 warm prompt rows and October warm Neural A/B studies are separate metrics. The warm 3k study discarded 13k and 3k warm-ups and checked zero cached-prefix tokens on measured requests.
• Matched API bodies across engines do not prove matching rendered tokens or numerical outputs. llama.cpp does not expose internal prompt-token hashes in this runner. No cross-backend quality-equivalence claim follows from the speed comparison.
• The older tools/bench_vs_llama.py --workload conversation results used follow-up medians over nine turns per engine (Sep 30 (4): ratios 1.63 / 1.56 / 1.61). Its visible-content hashes omit hidden reasoning/control-token identities. Historical rewrite comparisons used equal token spans and time-weighted 64-token windows.
• Frozen-residency equality checks compare every prompt-position logit, hidden state and complete layer K/V buffers for grouped prefill, and every step's full logits/hidden state plus final K/V for persistent decode. The CPU proof covers 128 / 3,000 / 13,000 prompt tokens and 64 steps per length, with actual persistent-job counts and zero fallbacks. Check timings include hashes and synchronization and are not serving speeds.

The selected suite passed 103 tests with 23 existing warnings; the 432 native cases and full-model checks ran separately. This is not a whole-directory pytest claim: existing standalone scripts parse CLI arguments during collection. Full reproduction details are in the decode evidence index, native proof/build record and serving archive audit.

Some rows are marked "not measured" on the public page. That means omitted or inconclusive, not zero. The three rewrite attempts for Sep 30 (2) were inconclusive, spoiled by stall windows or a llama.cpp outlier; the steady-state rewrite windows were +7–8% (MEASURED), the only measured signal. The rewrite leg was left out of Sep 30 (3) and (4) because a rewrite is not residency-frozen: it starts on experts that the preceding llama.cpp sessions had pushed out of the file cache, and its first ~300 tokens then measure disk reads (hard faults of about 5 ms per expert, a 64-token window at 8–10 tok/s) rather than the engine (MEASURED, step12, the same on three kernel builds).

Labels follow one convention: MEASURED, SIMULATED, CALCULATED or ASSUMED. Since 2026-09-30 11:31, when the PC hard-froze (Kernel-Power 41, no bugcheck; the cause is not established) while three agents ran heavy measurements at once, measurements are coordinated to run alone, one server or script at a time. Shared desktop RAM and GPU occupancy still vary and remain limitations.

The time budget per token

These are historical decode budgets through Sep 30 (4), retained with their original workloads. They are not a phase breakdown of v10.

The CPU phase is within a few ms of its floor. 88 CPU-read experts × 12.84 MB is 1.13 GB per token, which is 29.3 ms at the MEASURED 38.6 GB/s private-memory wall (CALCULATED); the measured 31–34 ms also carries the bus tax of the admission copies captured while the kernel reads (about 2.4 ms) and the barrier tails. Host DRAM reads top out near 40 GB/s (MEASURED), 83% of the CALCULATED 48 GB/s dual-channel peak.

DRAM traffic dominates, but the October follow-up found remaining software cost: entering/leaving an OpenMP team up to 36 times per token. The no-op probe saved about 35.3 microseconds per call with a 200-microsecond host gap (MEASURED, CPU-only), or about 1.27 ms over 36 calls (CALCULATED). That is a hypothesis bound, not an inference result; the serving study established the smaller workload-specific 1.74% adaptive decode gain. Doorbell and earlier host changes had failed to provide gains, but those trials did not exhaust software optimization.

The October stock follow-up phase medians were approximately 31.31 ms CPU calls, 4.84 ms GPU wait, 1.45 ms graph submissions, 0.90 ms admission and 3.04 ms other work (MEASURED). These marginal medians need not add to the median wall time. A direct graph-launch microbenchmark saved only about 0.065 ms per token when extrapolated over 36 launches (CALCULATED), so it was not integrated. Idle cache preparation, side-request K/V snapshots and streaming backpressure remain workload-specific latency hypotheses requiring their own correctness and client-wall tests.

One CALCULATED and never-run estimate is about 26 tok/s with the whole store pinned in RAM, requiring more RAM than this PC has. A DDR4-3200 Gear 1 setting was estimated at +4–6% on CPU reads (CALCULATED, never run). These hardware estimates are separate from the measured software improvements.

For prompts, the historical instrumented run (step25, g0) took 7.2 s: host launch 4.4 s in parallel with GPU 1.9 s (routing sync 1.2 s plus staging/H2D waits 0.7 s), with the 5.0 s CPU multi-token kernel hidden on its worker (MEASURED profile). The comparison run took 6.8 s. October's bounded grouping improved warm 3k latency; long-prompt grouping regressed. Grouping can reduce staging overlap, and the large-workspace trial increased reservation, but the counters do not isolate their contributions. The selected guard preserves the stock long-prompt path.

Where the weights sit and who computes them

The comparison target is llama.cpp build b10361 (14e78ddef, Clang 20.1.8 Windows x86_64) with -ngl 99 -ncmoe 31 -t 8 -c 16384 -fa on --jinja, the decode-tuned MoE offload. It maps the existing native MXFP4 GGUF, keeps the experts of 5 of the 36 layers in VRAM and the experts of the other 31 in system RAM. It computes those 31 layers' experts on the CPU where the weights already sit, so they never cross PCIe. Neural uses the same model in its existing native MXFP4 checkpoint/store without added quantization.

Neural's research runtime (August, not the server) took the opposite approach. It shipped experts over PCIe to the GPU, with multi-tier residency. It moved 1.84× fewer bytes per token, 850 MiB against about 1562 MiB (MEASURED). It still lost, because llama.cpp avoids the PCIe hop. In August the head-to-head on gpt-oss-120B was 13.65 against 13.90 tok/s, or 0.98× (MEASURED; against llama.cpp b10361 -ncmoe 31; it must not be chained with the other studies by protocol).

llama.cpp places experts per layer (5 layers on the GPU). Neural Server places individual experts adaptively per conversation. The CPU computes misses from RAM, and VRAM holds the experts that the current conversation uses most. Both sides work on each token at the same time. Neural then adds a fused graphed core, admission that copies while the CPU reads, and RAM tuning for prompts.

The fresh October 1 comparison used optimized Neural / llama.cpp / llama.cpp / optimized Neural and the same four saved API request bodies. Decode rows are weighted timed tokens / timed seconds; the initial-prompt row is a median across fresh sessions. Rates and times are MEASURED; ratios are CALCULATED. These are complete-runtime comparisons, not the increment from persistent workers alone.

The first-request client-wall medians were 30.38 versus 176.82 s; follow-up medians were 22.835 versus 39.405 s (MEASURED). The initial-prompt ratio includes session-start behavior and must not be spliced into the historical warm-prompt series. No prompt-tuned llama.cpp configuration was measured. Exact archived inputs, timings and the audit are in the fresh comparison and its verification record.

The following table retains the historical Sep 30 (4) medians against llama.cpp b10361. The 13k row is one warm run each; follow-up and first-answer rows are medians over nine turns per engine. Neural and llama.cpp columns are MEASURED; ratios are CALCULATED from unrounded medians. The rewrite figure below comes from Sep 30 (1).

Rewrite, a separate row from Sep 30 (1): 20.71 tok/s against 14.57 tok/s, 1.42× (MEASURED; 2 runs, 1.41–1.44×). It is a v6 result, not re-measured on v9 or v10.

Fairness notes matter here. A Sep 5 study of the research runtime (not the server) on the same PC, with llama.cpp at -ncmoe 33, a 2048-token context and background load, found llama.cpp 2.92× faster on fresh prompts (12.24 against 4.19 tok/s medians, MEASURED); the research runtime led by 9–11% only on a selected, repeatedly warmed 32-token trajectory. The protocols differ from the August 0.98× and from this page, so the studies must not be chained, and the server's fresh-prompt performance in that protocol was not re-measured. Compare warm against warm, use equal token spans, and keep free RAM about equal (54–57 GiB in the later runs).

The full comparison is on the results chart.

What these numbers do and do not show

• One PC and one model. The numbers come from a single machine running gpt-oss-120B, with a 16,384-token context. Throughput is single-stream.
• Full-model bit checks use frozen residency. CPU-only adaptive serving repeats and the fresh optimized Neural repeats also matched corresponding full tokens, but one combined adaptive session diverged on its last two answers. Timing changes placement between GPU bf16 rounding and CPU fp32. No universal adaptive determinism is claimed.
• The October CPU gain is modest and workload-specific: 1.74% adaptive, with a separate 2.885% frozen control. The roughly 1.72× fresh llama.cpp ratio is not a 72% gain from that CPU change. The historical Sep 30 1.96× first-answer ratio used unusually slow llama.cpp sessions.
• Equal API requests and token counts across backends do not prove equal rendered tokens, numerical outputs or quality. Native-kernel equality is scoped to tested FP environments; unsupported callers/configurations fall back. The fresh Neural repeats matching does not erase the earlier adaptive divergence.
• The prompt ratios are against a decode-tuned llama.cpp configuration; no prompt-tuned run was measured.
• The model only just fits in 64 GB of RAM next to other programs, so the first answer still pulls some experts from disk. The page-fault counts above are the measured indicator.
• The rewrite figure stands at 1.42× from the first Sep 30 measurement, because later attempts were not clean.
• The server handles one request at a time and needs Windows, NVIDIA and AVX-512.
• The persistent DLL targets this Windows/AVX-512 host. CLI defaults remain unchanged; the optimized wrapper is opt-in. No slow-client or multi-request throughput study was performed. This remains a research artifact; source links require access to the private repository.

The August research verdict on Neural's original runtime still stands for that runtime. The server is a new design, with the CPU computing misses, built after closure.