Measured, not promised.
Same PC and existing gpt-oss-120B model, with fresh server processes taking turns against the installed llama.cpp b10361, using its decode-tuned -ngl 99 -ncmoe 31 -t 8 -c 16384 -fa on --jinja settings. October 1 uses the exact same saved API requests for both engines: a 13,000-token code context and four answers capped at 512 tokens, in Neural / llama / llama / Neural order. Decode rates are total timed tokens divided by total decode time, with two sessions per engine.
The older dates keep their original methods and results. Their answer rates are medians, while October 1 uses weighted rates and fixed request replay. The prompt row is a warmed-up measurement through September 30 and a fresh-session measurement on October 1. The chart does not join those method changes as a continuous code improvement. The first two rows are historical short-context research results; the rewrite was not re-measured on October 1.
October 2 adds a separate research screen for 16 independent requests. It is aggregate backend throughput; the task-quality gate is not cleared, and there is no production promotion.
All answers, weighted decode
Oct 1, two sessions per engine
First answer after a long prompt
Reading a 13,000-token code prompt
fresh session; median of two runs
Neural, up to llama.cpp's speed (the line at 1×) faster than llama.cpp (faded: spread across runs and settings)
Each panel is Neural's speed divided by llama.cpp's, from the same set of alternating runs (the hollow point pairs the Sep 24 build's re-run with that night's llama.cpp run). The vertical whiskers show the run-to-run range. Above the solid line at 1× is faster than llama.cpp. The answer-writing panels share one scale; prompt reading has its own. October 1 uses weighted replay measurements instead of the older medians, and its prompt run starts a fresh session. These points are shown separately at the method change.
faster than llama.cpp up to llama.cpp's speed the unchanged Sep 24 build, measured again on Sep 29
October 2 · experimental backend throughput
One clean split-graph screen completed 16 independent requests at 29.3246 aggregate tokens/s: 42,639 generated tokens in 1,454.035 seconds. This includes prompt processing and K/V copying in the backend cohort clock. It is not one conversation's decode rate or an HTTP client measurement.
historical combined-graph batch clean split-graph screenAbsolute aggregate generated tokens/s; zero-based scale
The split screen reproduced all prompt and output token IDs from the historical combined batch B1/B2 runs. Its throughput was 24.14–24.28% higher (CALCULATED from measured clocks). This is one later screen against historical runs, not a fresh alternating comparison or a replicated causal gain. The 42,639 count includes 16 first tokens produced by prefill; there were 42,623 decode calls.
Task-quality gate: NOT CLEARED. Frozen static review found paired coding-rubric regressions; the generated code was not executed by that review. Matching the batch reference's token IDs preserves its answers, not a general quality guarantee. No production or end-to-end improvement is confirmed, and no research change is promoted to main.
The chart contains the two historical combined-graph timing runs and the single completed split-graph timing screen. No grouped timing bar is shown because no grouped run completed. October 1's separate NeuralServer production-runtime comparison, 24.6910 versus llama.cpp's 14.3189 tokens/s, uses a different workload and clock.
Research numbers and evidence
Research is paused. The grouped full natural numerical audit passed; no grouped timing completed, and the split timing repeat was deferred.
Benchmark summary: metrics, method and hashes. Private raw evidence: split raw timing report, terminal admission and historical comparison, and locked static task-quality summary, archived on the research-results branch. Repository access is required. Method and limits.
Production comparison: all numbers as a table
In the October 1 NeuralServer production-runtime comparison, the selected profile averaged 24.6910 tokens/s against llama.cpp's 14.3189 across all answers: 1.72× on this workload. First answers were 23.6047 versus 13.0566 tokens/s; follow-ups were 25.0756 versus 14.7957. Prompt processing in these fresh sessions took a median 8.666 versus 137.552 seconds. That prompt comparison is separate from September 30's warmed-up result. The installed llama configuration favours decode; this was not an exhaustive search for its best prompt settings.
The new software improvement is smaller than the engine-to-engine ratio. Keeping the eight CPU workers alive between expert calls raised Neural's decode rate by 1.74% in the normal adaptive comparison (23.8649 to 24.2807 tokens/s), and 2.885% in a fixed-placement control (19.8659 to 20.4391). Freezing placement was slower overall and is a diagnostic control, not the recommended setting. All corresponding output tokens matched in those CPU-only comparisons. A separate prefill study reduced warm 3k prompt latency by 7.03% (3.69655 to 3.43665 seconds); releasing unused CUDA cache saved about 1.11 GiB after a long prompt. These gains apply to different phases and workloads; they are not percentages to add together.
No additional quantization or checkpoint change. Native checks passed 432 bitwise cases, full-model checks compared 192 decode steps exactly, and 103 selected repository tests passed. One adaptive combined-profile run differed on its final two answers, so its strict output-identity gate failed and remains in the evidence. Both Neural repeats in the final llama comparison matched each other. API-request equality does not establish identical internal prompt tokens or output quality across engines. Neural times 511 decoded tokens per 512-token answer; llama reports 512, and each timing convention is retained.
Sources: decode study and reproduction, verified comparison, and prefill/cache review. All slow observations and failed experiments are retained. Source links require access to the private repository.
Two write-ups of this work: the technical documentation (architecture, every change with its number and verdict, method, the llama.cpp comparison) and a version for decision makers.
1.7–2.9%new CPU decode gain in adaptive and fixed-placement controls
7.03%lower warm 3k prompt latency in the separate prefill study
1.11 GiBless unused GPU allocator reservation after a long prompt
Development PC: NVIDIA RTX 3080 Ti 12 GB, Intel Core i7-11700K, 64 GB DDR4-3000, NVMe SSD. A token is about three quarters of an English word. Code is slower than prose: it calls on experts from all over the model, so more of the work lands on RAM, and RAM speed becomes the limit: expert reads remain the largest decode cost; the new worker scheduling reduces overhead without changing the weights or arithmetic precision.