C++20 inference engine for Qwen3.6-27B and Qwen3.8-27B (qwen3_5 hybrid: 48 Gated DeltaNet + 16 Gated Attention). Not a wrapper around llama.cpp, vLLM, SGLang, or MLC.
| License | MIT (LICENSE) |
| Weights | Apache-2.0 (Qwen) |
| Platforms | Windows / Linux x86-64 · CUDA sm_75/86/89 |
RapidLLM loads official HuggingFace block-FP8 directories and community GGUF (Q4_K / Q5_K / Q6_K / Q8_0) into one IR and runs them with the same scheduler:
Verified community GGUF: JonathanColetti/Qwen3.8-27B-Uncensored-GGUF. Its MTP Q4_K_M, Q5_K_M, Q6_K, and Q8_0 files use the supported qwen35 27B layout (65 blocks, including one NextN/MTP block). The noMTP-* files also load, but should be run with --spec off, ngram, or an external compatible drafter rather than MTP.
- 48 × Gated DeltaNet — O(1) recurrent state, no KV growth
- 16 × Gated Attention — GQA KV (FP16 or
--kv-type q8k_tq3v) - Speculative decode — built-in MTP, n-gram, or DFlash4 / DSpark block draft (
--spec dspark --draft PATH), verified with FP8, Q4, Q6, and Q8 targets
The CUDA path is a fused decode engine: packed Q4/Q6/Q8 GEMV, T=4 / T=12 CUDA graphs, occupancy-stable isolated kernels, and target-side spec verify. That is what beats vLLM / SGLang / BaoFan-tuning on a single Ada RTX 6000 48GB.
Same prompt 1,2,3, --max-new 16, --ctx 245760, greedy. Wall tok/s = n_new / (prefill + decode).
| Engine | Weights | Wall tok/s |
|---|---|---|
| RapidLLM | Q4_K_M GGUF + MTP | 89.56 |
| RapidLLM | Q8_0 GGUF + MTP T=12 | 86.46 |
| BaoFan-tuning (llama.cpp MTP-TurboQuant) | Q4 GGUF | 73.9 |
| RapidLLM | Q6_K GGUF + MTP | 60.41 |
| RapidLLM | official FP8 | 43.72 |
| SGLang | official FP8 | 21.59 |
| vLLM | official FP8 | 19.57 |
Q4 and Q8 RapidLLM beat BaoFan-tuning (73.9). Official FP8 is bandwidth-bound (~50–55 tok/s ceiling on this card); GGUF Q4 is the 90 tok/s path.
This separate, steady-state decode benchmark uses --ctx 32768 --max-new 64 --prompt "1,2,3" --fuse=on on one RTX 6000 Ada 48GB. It is not directly comparable to the 240k-context wall-clock table above. DFlash4 uses the 1.36B Qwen3.8-27B-DSpark drafter and target-side greedy verification. The current run keeps GGUF embeddings packed on device (rather than expanding the full vocabulary to F32), which restores Q6/Q8 32k-context headroom without changing generated IDs.
| Target weights | --spec off decode tok/s |
DFlash4 decode tok/s | Gain | Proposed / accepted |
|---|---|---|---|---|
| Qwen3.8 FP8 | 30.07 | 54.62 | +81.7% | 95 / 31 |
| Qwen3.8 MTP Q4_K_M | 38.75 | 70.01 | +80.7% | 32 / 31 |
| Qwen3.8 MTP Q6_K | 31.16 | 63.45 | +103.7% | 51 / 45 |
| Qwen3.8 MTP Q8_0 | 28.83 | 73.27 | +154.2% | 48 / 48 |
Every row matches target-only greedy decoding token-for-token. FP8, Q6, and Q8 use the verified three-token host-draft cap; Q4 uses cap one. Q4 deliberately stays on the exact T=2 verifier (anchor + one draft), avoiding its non-state-safe T=3 host verifier. The latest revalidation measured 54.62 / 70.01 / 63.45 / 73.27 decode tok/s (FP8 / Q4 / Q6 / Q8); the Q8 value is the mean of three independent runs (73.86, 73.24, 72.72). A three-run Q8 T=4 experiment remained exact (48/48 accepted) but averaged about 73.15 tok/s, so production remains on T=3.
For a direct algorithm comparison, SGLang's DFLASH block-size-7 path measured about 33 tok/s on this card, while its newer DSPARK path measured 50.83 tok/s. RapidLLM's DSpark verifier reached 54.62 tok/s on the same FP8 target and preserved target-only greedy IDs; SGLang DSPARK was faster than DFLASH but its final three IDs diverged, so that number is not an exactness-equivalent result.
The following is the like-for-like algorithm view from the raw-ID [1,2,3] / 64-token run. “Exact” means the complete generated ID sequence is identical to target-only greedy; a dash means that check was not part of the external engine's random-input throughput command.
| Implementation | Algorithm | Throughput (tok/s) | Exact | Measurement scope |
|---|---|---|---|---|
| RapidLLM | DSpark, cap 3 | 54.62 | yes | CUDA decode-only; 31/95 draft tokens accepted |
| SGLang 0.5.18 | DSPARK, block 7 | 50.83 | no | Three warmed /generate requests; final 3 IDs diverged |
| SGLang 0.5.18 | DFLASH, block 7 | ~33.0 | no | /generate; tail diverged from target-only |
| vLLM 0.28.0 | DSpark | 17.24–17.68 | — | bench throughput, random IDs; Triton FP8 fallback |
Thus RapidLLM DSpark is 7.5% faster than the SGLang DSPARK measurement and about 65% faster than its DFLASH measurement in this environment. The SGLang figures include HTTP serving overhead, while RapidLLM reports steady-state decode time, so the comparison is directional; only the RapidLLM row currently satisfies the exactness requirement.
Freshly re-measured on the same RTX 6000 Ada 48GB: one greedy request, raw input IDs [1,2,3], 64 generated tokens, 32k context, and one running request. SGLang numbers are the mean of three warmed direct /generate wall-clock requests, so they include serving/API overhead and must not be compared directly with RapidLLM's decode-only figures above.
| Engine / mode | Output tok/s | Notes |
|---|---|---|
| SGLang 0.5.18, target only | 22.81 | Mean of three warmed direct requests |
SGLang 0.5.18, DFLASH + Qwen3.8-27B-DSpark |
~33.0 | Block size 7; output diverged from target-only greedy in the 64-token tail |
SGLang 0.5.18, DSPARK + Qwen3.8-27B-DSpark |
50.83 | Mean of three warmed requests; +122.9% versus its target-only run; see exactness note below |
RapidLLM, DSpark cap 3 |
54.62 | Latest decode-only revalidation; greedy tokens exactly match its target-only result |
| vLLM 0.28.0, target only | 18.86 | vllm bench throughput, one random-ID request, 3 input / 64 output tokens |
vLLM 0.28.0, dspark + Qwen3.8-27B-DSpark |
17.24–17.68 | Same offline test; DSpark path loads with the Qwen DSpark config shim, but is slower here |
SGLang 0.5.18 DSPARK is functionally compatible, but its final three generated IDs differed from the target-only greedy output for this raw-ID test; its speed must therefore not be used where target-only greedy token identity is required. RapidLLM's published DSpark figures retain that exactness requirement. The isolated SGLang package reused the host's flashinfer_python 0.6.15.post1 although 0.5.18 requests 0.6.17, so the command explicitly bypasses that version gate; this is also an environment-specific, conservative baseline.
vLLM's throughput command uses random token IDs, so it is a length- and concurrency-matched throughput baseline rather than a raw-ID token-identity check. vLLM 0.28.0 registers Qwen3DSparkModel; this checkpoint labels itself DSparkDraftModel, so the DSpark test used a local config-only shim that changes that architecture name and symlinks the original weights. The isolated vLLM installation reused the host's Torch 2.11/CUDA dependencies; its optional deep_gemm extension had an ABI error and vLLM selected the Triton FP8 kernel. Treat these as conservative, reproducible environment-specific baselines.
# SGLang 0.5.18 DSPARK external baseline (direct /generate benchmark after warm-up).
# Measure a request using raw input_ids [1,2,3], temperature=0 and max_new_tokens=64.
SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK=1 sglang serve \
--trust-remote-code --model-path /path/Qwen3.8-27B-FP8 --context-length 32768 \
--mem-fraction-static 0.90 --max-total-tokens 32768 --max-running-requests 1 \
--max-mamba-cache-size 5 \
--chunked-prefill-size 256 --host 127.0.0.1 --port 30080 \
--speculative-algorithm DSPARK \
--speculative-draft-model-path /path/Qwen3.8-27B-DSpark \
--speculative-dspark-block-size 7 \
--speculative-draft-model-quantization unquant \
--mamba-radix-cache-strategy extra_buffer
# vLLM 0.28.0 target-only reference. --max-num-seqs 1 is required: the
# 32k model has only 229 available Mamba cache blocks, below the default 256.
vllm bench throughput --model /path/Qwen3.8-27B-FP8 --dataset-name random --num-prompts 1 \
--random-input-len 3 --random-output-len 64 --max-model-len 32768 \
--max-num-seqs 1 --gpu-memory-utilization 0.90 --trust-remote-code
# vLLM 0.28.0 DSpark experiment: build a config-only Qwen shim once, preserving
# the original weight files, then point --speculative-config at the shim.
mkdir -p /tmp/Qwen3.8-27B-DSpark-vllm
cp /path/Qwen3.8-27B-DSpark/config.json /tmp/Qwen3.8-27B-DSpark-vllm/
sed -i 's/DSparkDraftModel/Qwen3DSparkModel/' /tmp/Qwen3.8-27B-DSpark-vllm/config.json
ln -s /path/Qwen3.8-27B-DSpark/model.safetensors /tmp/Qwen3.8-27B-DSpark-vllm/model.safetensors
vllm bench throughput --model /path/Qwen3.8-27B-FP8 --dataset-name random --num-prompts 1 \
--random-input-len 3 --random-output-len 64 --max-model-len 32768 --max-num-seqs 1 \
--gpu-memory-utilization 0.90 --trust-remote-code \
--speculative-config '{"model":"/tmp/Qwen3.8-27B-DSpark-vllm","num_speculative_tokens":7,"method":"dspark"}'# Fastest (Q4 GGUF + MTP) — Ada 48GB, 240k ctx
rapidllm bench -m Qwen3.8-27B-MTP-Q4_K_M.gguf \
--device cuda --ctx 245760 --max-new 16 --prompt "1,2,3" \
--spec mtp --spec-n 3 --fuse=on
# DFlash4 reproducibility baseline: Ada RTX 6000 48GB, 32k ctx, greedy decode.
# The verified caps are FP8=3, Q4=1, Q6=3, and Q8=3.
# Set RAPIDLLM_DSPARK_CAP only to pin a benchmark to its verified optimum.
# Q4_K_M optimum: cap=1, 70.46 decode tok/s.
RAPIDLLM_DSPARK_CAP=1 rapidllm bench -m /path/Qwen3.8-27B-MTP-Q4_K_M.gguf \
--device cuda --ctx 32768 --max-new 64 --prompt "1,2,3" \
--spec dspark --draft /path/Qwen3.8-27B-DSpark --fuse=on
# Q6_K optimum: cap=3, 64.34 decode tok/s.
RAPIDLLM_DSPARK_CAP=3 rapidllm bench -m /path/Qwen3.8-27B-MTP-Q6_K.gguf \
--device cuda --ctx 32768 --max-new 64 --prompt "1,2,3" \
--spec dspark --draft /path/Qwen3.8-27B-DSpark --fuse=on
# Q8_0 optimum: cap=3, 73.27 decode tok/s (three-run mean).
RAPIDLLM_DSPARK_CAP=3 rapidllm bench -m /path/Qwen3.8-27B-MTP-Q8_0.gguf \
--device cuda --ctx 32768 --max-new 64 --prompt "1,2,3" \
--spec dspark --draft /path/Qwen3.8-27B-DSpark --fuse=on
# FP8 optimum: cap=3, 54.02 decode tok/s (re-measured with exact greedy output).
RAPIDLLM_DSPARK_CAP=3 rapidllm bench -m /path/Qwen3.8-27B-FP8 \
--device cuda --ctx 32768 --max-new 64 --prompt "1,2,3" \
--spec dspark --draft /path/Qwen3.8-27B-DSpark --fuse=on--spec dspark and --draft are a required pair. The listed RAPIDLLM_DSPARK_CAP values reproduce the verified maxima; normally omit it and the runtime will select the same cap from the target quantization. Always require a token-for-token comparison with --spec off before accepting a new quantization/cap pair; target verification alone does not make an implementation regression safe.
--ctx > 163840 enables compact KV (q8 K + TurboQuant-3 V) so 262k fits in 48GB. --kv-type f16 forces a full FP16 cache (OOM at 262k).
cmake -S . -B build -DRAPIDLLM_WITH_CUDA=ON
cmake --build build -j
rapidllm bench -m <hf-dir|file.gguf> --device cuda --ctx 32768
rapidllm serve -m <path> --host 0.0.0.0 --port 8080--spec off|ngram|mtp|auto|dspark. --spec dspark requires --draft pointing at a DSparkDraftModel directory.
Design: docs/architecture.md · Tech report: docs/TECH_REPORT.md · docs/技术报告.md