Pro 6000 GPU vLLM Inference Benchmark Report: Pro6000 vs. H100 vs. A100-80GB

Test Environment and Methodology

Hardware and Software Stack

• GPU: Nvidia RTX Pro 6000 Blackwell Server Edition
• Inference Engine: vLLM
• Backend: CUDA + vLLM Continuous Batching
• Parallelism: --tensor-parallel-size = 1, Single-GPU inference, no model sharding

Pro 6000 GPU Details

• GPU: Nvidia RTX Pro 6000 Blackwell Server Edition
• Microarchitecture: Blackwell
• Compute capability: 12
• CUDA Cores: 24,064
• Tensor Cores: 852
• GPU Memory: 96GB GDDR7
• FP32 Performance: 126 TFLOPS

Benchmark Tool

Benchmark tests were executed using the official vLLM repository:

git clone https://github.com/vllm-project/vllm.git

The benchmark_serving.py script was used for all measurements.

Unified Benchmark Parameters

All models were tested with the same inference load configuration to ensure fair comparison:

--max-model-len 4096
--tensor-parallel-size 1
--num-prompts 50
--request-rate inf
--random-input-len 100
--random-output-len 600

This configuration simulates a realistic API / chat inference workload with moderate concurrency and long-form outputs.

Model Coverage

The benchmark includes three representative model tiers. Models download from Hugging Face:

Small Models (8B Class)

• meta-llama/Llama-3.1-8B (15 GB)
• deepseek-ai/DeepSeek-R1-Distill-Llama-8B (15GB)
• Qwen/Qwen3-8B (15GB)

Medium Models (12B–20B Class)

• google/gemma-3-12b-it (23GB)
• deepseek-ai/DeepSeek-R1-Distill-Qwen-14B (28 GB)
• Qwen/Qwen3-14B (28 GB)
• openai/gpt-oss-20b (4-bit, 13GB)

Large Models (32B–120B Class)

• deepseek-ai/DeepSeek-R1-Distill-Qwen-32B (62GB)
• Qwen/Qwen3-VL-32B-Instruct (63GB)
• openai/gpt-oss-120b (4-bit, 61 GB)

Total Throughput (tokens/sec)

Total Throughput reflects the overall decoding efficiency under sustained concurrency.

Observations:

• 8B models consistently deliver ~3,100–3,300 tokens/s
• GPT-OSS 20B (4-bit) achieves the highest throughput (~4,378 tokens/s) due to aggressive quantization and optimized architecture
• 32B models drop sharply to ~930–970 tokens/s
• GPT-OSS 120B further declines to ~1,780 tokens/s, despite quantization

✅ Key takeaway:

Pro 6000 performs best in the 8B–32B range, where compute utilization, memory bandwidth, and KV cache usage remain well balanced.

Request Throughput (req/s)

At 50 concurrent prompts:

• 8B models: ~4.4–4.6 req/s
• 14B models: ~2.6–2.9 req/s
• 32B models: ~1.3 req/s
• GPT-OSS 120B: ~2.5 req/s

Interestingly, GPT-OSS 120B maintains request throughput comparable to some 14B models.

Time to First Token (TTFT)

TTFT strongly affects perceived responsiveness.

• 8B models: 260–330 ms
• 14B models: 440–510 ms
• 32B models: ~1,000 ms
• GPT-OSS 120B: ~340 ms

Token Generation Latency (TPOT / ITL)

Per-token latency scales predictably with model size:

• 8B: ~13–18 ms/token
• 14B: ~28–31 ms/token
• 32B: ~58–61 ms/token
• GPT-OSS 120B: ~32 ms/token

Pro 6000 vs A100 vs H100: 300-Concurrent vLLM Stress Test

Under a 300-concurrent request stress test, the performance characteristics of Pro 6000, A100 80GB, and H100 diverge more clearly than in lower-concurrency benchmarks.

Pro 6000 shows the strongest scaling behavior under high concurrency, delivering the highest total throughput across all tested models. It leads significantly on smaller and medium models, reaching 8,990 tokens/s on Llama-8B and 5,160 tokens/s on Qwen-14B, and also outperforms both A100 and H100 on Qwen-32B. This indicates strong scheduler efficiency and stable memory behavior in heavy vLLM batching scenarios.

H100 remains consistently strong but does not dominate at high concurrency in this test. While it maintains high throughput on all model sizes, its advantage over Pro 6000 diminishes under extreme request pressure, especially for 8B and 14B models. H100 still performs well on 32B workloads, but the gap is narrower than expected given its higher hardware cost.

A100 80GB shows the weakest scaling under 300-concurrent load, particularly for larger models. Although it performs reasonably on 14B, throughput drops sharply on both 8B and 32B models, suggesting earlier saturation under heavy batching and less favorable cost-performance at high concurrency.

From a cost-performance perspective, the results suggest:

• Pro 6000 offers the best value for high-concurrency inference, especially for API-style workloads with many simultaneous requests.
• H100 is better suited for performance-critical or large-model workloads, but its price premium is harder to justify purely on throughput at extreme concurrency.
• A100 80GB is less competitive in sustained high-concurrency vLLM inference, particularly when compared against newer or better-balanced alternatives.

Special Analysis: Why GPT-OSS-120B Can Run on a Single Pro 6000 (96 GB)

Although GPT-OSS-120B is nominally a 120B-parameter model, the version published on Hugging Face is quantized to 4-bit precision, which significantly reduces its actual memory footprint compared to standard 16-bit (FP16/BF16) models.

In practice, a 120B model at 16-bit precision would require well over 200 GB of GPU memory, making single-GPU deployment impossible. However, with 4-bit quantization, the effective model size is reduced to roughly one quarter of its FP16 equivalent, bringing the total GPU memory requirement down to approximately 60–65 GB, including model weights and runtime overhead.

This reduced memory footprint allows GPT-OSS-120B to fit comfortably within the 96 GB VRAM of a single Pro 6000, without relying on multi-GPU tensor parallelism or offloading strategies. As a result, single-GPU inference becomes feasible, simplifying deployment and reducing operational complexity.

It is important to note that this deployment feasibility is a direct result of aggressive quantization, rather than raw GPU memory capacity alone. Inference performance and output quality may differ from full-precision variants, but for many inference-focused and cost-sensitive workloads, the 4-bit GPT-OSS-120B model provides a practical and efficient trade-off.

Final Conclusions

Under vLLM inference workloads, Pro 6000 demonstrates stable, predictable performance with clear scaling characteristics:

• Excellent efficiency in 8B–40B inference
• At 300 concurrent requests, throughput scaling and batching efficiency matter more than peak hardware specs, and in this scenario, Pro 6000 demonstrates the most favorable balance between performance and cost.
• priced lower than the A100 80GB and H100, and offers superior performance compared to the A100 80GB and H100.
• The Pro6000 can perform VLLM inference on the GTP-OSS-120b model from Hugging Face.

For production inference—especially API, agent, or SaaS scenarios—model selection matters more than maximum parameter count, and Pro 6000 is best utilized in the mid-to-high-end model segment.

Only $479/mo, Rent Pro 6000 Server for AI Inference >

Screenshot: Pro6000 vLLM Benchmark Results

Data Item Explanation in the Table

• Quantization: Indicates the numerical precision used for model weights, affecting memory usage and inference speed.
• Size (GB): Represents the approximate model memory footprint under the specified quantization, not necessarily peak GPU VRAM usage.
• Backend / Platform: Specifies the inference engine and runtime environment used for the benchmark.
• Tensor Parallel Size: Shows how many GPUs are used to run the model, with 1 meaning single-GPU inference.
• Max Model Length: Defines the maximum supported token length for input and output combined.
• Request Numbers: Indicates the total number of inference requests issued during the benchmark.
• Benchmark Duration (s): Measures the total time required to complete all benchmark requests.
• Total Input Tokens: The sum of all prompt tokens processed across requests.
• Total Generated Tokens: The total number of output tokens generated during the benchmark.
• Requests (req/s): Represents how many inference requests the system can handle per second.
• Input Throughput (tokens/s): Measures the rate at which input tokens are processed.
• Output Throughput (tokens/s): Measures the rate at which output tokens are generated.
• Total Throughput (tokens/s): Combines input and output token throughput to reflect overall inference efficiency.
• Median TTFT (ms): The median time from request submission to the first generated token.
• P99 TTFT (ms): The 99th-percentile time to first token, reflecting worst-case initial latency.
• Median TPOT (ms): The median time required to generate each output token after the first token.
• P99 TPOT (ms): The 99th-percentile per-token generation latency under load.
• Median ITL (ms): The median delay between consecutive output tokens during streaming generation.
• P99 ITL (ms): The 99th-percentile inter-token delay, indicating tail latency during output streaming.