GPU Inference Benchmarks

Reproducible serving benchmarks for local GPU inference deployments. This repository starts with Qwen 3.6 27B NVFP4 on vLLM and is intended to grow with additional models, quantization formats, runtimes, and hardware.

Qwen 3.6 27B NVFP4 on vLLM

Configuration

• Model: Qwen 3.6 27B, NVIDIA ModelOpt NVFP4 checkpoint
• Runtime: vLLM 0.24.0, OpenAI-compatible chat endpoint
• GPU: NVIDIA RTX PRO 6000 Blackwell, 96 GB
• Serving features: FP8 KV cache, FlashInfer attention, chunked prefill, prefix caching, and three-token MTP speculative decoding
• Scheduler cap: 32,768 batched tokens

All standard-chat tests use deterministic sampling (temperature=0) with thinking disabled. Results are one-run serving characterizations using synthetic tokenized prompts, not confidence intervals or model-quality scores.

Executive Summary

• Interactive operating point: C4 reaches 311 output tok/s at 489 ms P99 TTFT for 512-input / 128-output chat requests.
• Highest point below a one-second P99 TTFT target: C8 at 447 output tok/s and 940 ms P99 TTFT.
• Saturation: C32 reaches 630 output tok/s. C64 only reaches 649 output tok/s while P99 TTFT grows to 7.34 s.
• Prefix caching: sharing an 8k-token prefix reduces mean TTFT from 10.38 s to 2.31 s (a 77.7% reduction).
• Long context: 32k-input requests at C8 reach 49.9 s P99 TTFT under the current 32k scheduler cap.
• MTP: three-token speculation improves C1 throughput by 67.8% and C8 by 29.0%. It does not materially increase the fully saturated ceiling.

TTFT is time to first token. TPOT is time per output token, excluding the first token. Cn denotes the maximum request concurrency.

Concurrency Curve

Workload: 512 input tokens, 128 generated tokens, thinking disabled, burst arrival.

Concurrency	Output tok/s	Mean TTFT	P99 TTFT	Mean TPOT
1	105.8	143 ms	148 ms	8.4 ms
2	188.6	182 ms	273 ms	9.1 ms
4	310.8	229 ms	489 ms	10.9 ms
8	447.3	367 ms	940 ms	14.6 ms
16	544.5	816 ms	1.85 s	22.2 ms
32	629.6	2.41 s	3.63 s	31.3 ms
64	649.4	5.48 s	7.34 s	46.8 ms

Prompt-Length Curve

All cases use C8, thinking disabled, and burst arrival.

Input / output tokens	Output tok/s	Mean TTFT	P99 TTFT	Mean TPOT
256 / 128	555.4	204 ms	511 ms	12.3 ms
2,048 / 256	327.0	1.16 s	3.52 s	19.3 ms
8,192 / 512	206.9	5.93 s	13.22 s	26.0 ms
32,768 / 512	72.1	40.54 s	49.89 s	30.0 ms

Prefix-Cache Effect

Workload: 8,192-token prefix, 512-token suffix, 128 generated tokens, C8.

Prefix distribution	Output tok/s	Mean TTFT	P99 TTFT
32 unique prefixes	61.8	10.38 s	14.00 s
One shared prefix	178.0	2.31 s	4.36 s

Reasoning and Arrival Rate

Thinking-enabled results use a 512-input / 1,024-output workload. Arrival tests use the standard 512-input / 128-output workload with C16.

Scenario	Output tok/s	P99 TTFT	Mean TPOT	MTP acceptance
Thinking, C1	140.7	162 ms	7.0 ms	75.1%
Thinking, C8	839.8	948 ms	8.8 ms	74.0%
Arrival rate: 2 req/s	243.1	586 ms	12.2 ms	57.3%
Arrival rate: 4 req/s	439.8	868 ms	20.0 ms	56.5%

Reasoning-mode throughput is not directly comparable to short chat output: each request has a much longer generation window and a higher speculative acceptance rate.

MTP Speculative-Decoding Comparison

Matched normal-chat workloads, with only the three-token MTP setting changed. Model, quantization, KV-cache type, attention backend, and scheduler settings were unchanged.

Concurrency	MTP on tok/s	MTP off tok/s	On vs. off	On P99 TTFT	Off P99 TTFT	On TPOT	Off TPOT
1	105.8	63.0	+67.8%	148 ms	136 ms	8.4 ms	15.0 ms
8	447.3	346.6	+29.0%	940 ms	893 ms	14.6 ms	16.5 ms
32	629.6	638.8	-1.4%	3.63 s	3.48 s	31.3 ms	23.4 ms

MTP improves decode efficiency at interactive and moderate concurrency. At C32, the server is capacity-limited and the difference falls within normal single-run variation.

Reproducing the Suite

The runner writes raw vLLM results, GPU telemetry, and local server snapshots to results/. Those host-local artifacts are intentionally excluded from Git.

export TOKENIZER=/path/to/local/model-or-tokenizer
export BASE_URL=http://127.0.0.1:8000
export MODEL=your-served-model-name

./scripts/run_suite.sh full
./scripts/build_report.py

Run a focused group with concurrency, length, prefix, reasoning, or arrival instead of full. Use the config group only for controlled server configuration A/B testing.

Scope and Limits

This report measures serving performance only. A model-quality suite such as IFEval, MMLU-Pro, GSM8K, coding evaluations, or function-calling evaluation should be run separately when comparing model weights or quantization formats.