NVIDIA RTX Pro 2000 Blackwell (16GB) Ollama Inference Benchmark Report

Test Overview

The test environment was standardized across all models to ensure fairness and reproducibility.

GPU: NVIDIA RTX Pro 2000 (Blackwell architecture)

• Microarchitecture: Blackwell
• Compute capability: 12
• CUDA Cores: 4352
• Tensor Cores: 5th Gen
• GPU Memory: 16GB GDDR7
• FP32 Performance: 17 TFLOPS

Backend Environment

• Inference framework: Ollama 0.15.4
• Quantization: 4-bit
• Execution mode: Single-model, single-GPU
• Prompt: Identical for all models

Although the prompt itself is simple, the length, structure, and detail of the generated answers vary significantly between models, which naturally affects token counts and generation time. This variation is considered part of the model’s real inference behavior rather than noise.

Input Settings

All models were tested with the same simple question:

“What is GPU?”

By keeping the prompt short and identical, the benchmark highlights pure decoding performance rather than prompt complexity or reasoning depth.

Models Evaluated

The benchmark covers a representative mix of DeepSeek, LLaMA, Gemma, and Qwen families:

• GPT-OSS: 20B
• DeepSeek-R1: 7B / 8B / 14B
• LLaMA 2: 7B / 13B
• LLaMA 3.1: 8B
• Gemma 3: 4B / 12B
• Qwen 3: 8B / 14B

All models fit comfortably within the 16 GB VRAM limit due to INT4 quantization.

Pro 2000 Ollama Benchmark Data Display

Token Generation Performance

Evaluation Token Rate (tokens/s)

With the addition of GPT-OSS 20B, the performance landscape becomes more interesting:

• Gemma 3 4B remains the fastest model at ~82 tokens/s, highlighting its strong decoding efficiency.
• LLaMA 2 7B achieves ~65 tokens/s, making it one of the most responsive mid-size models.
• GPT-OSS 20B delivers ~62 tokens/s, which is notable given its much larger parameter count.
• DeepSeek-R1 7B and LLaMA 3.1 8B operate in the 55–60 tokens/s range.
• 8B-class models generally stabilize around 48–55 tokens/s.
• 13B–14B models drop to ~28–35 tokens/s, showing a clear performance inflection point.

Key Observation:

GPT-OSS 20B performs closer to 7B–8B models than to 14B models, indicating that model architecture and quantization efficiency can outweigh raw parameter size on RTX Pro 2000.

GPU Utilization and System Load

Despite its entry-level positioning, RTX Pro 2000 shows strong hardware utilization:

• GPU utilization remains consistently high, between 87% and 96% across all models.
• GPT-OSS 20B sustains ~87% GPU utilization, comparable to smaller models.
• CPU usage stays below 10%, and RAM usage remains modest.

These results confirm that inference is GPU-bound, and that RTX Pro 2000 is being effectively saturated even by larger INT4 models.

Practical Implications

Based on the updated results:

• Best responsiveness: Gemma 3 4B, LLaMA 2 7B
• Best balance of size and speed: GPT-OSS 20B, LLaMA 3.1 8B
• Heavier models (13B–14B): Usable but better suited for non-interactive or batch workloads

For users operating within a 16GB GPU constraint, GPT-OSS 20B stands out as a surprisingly efficient large model option.

Conclusion

With the inclusion of GPT-OSS 20B, the NVIDIA RTX Pro 2000 Blackwell (16GB) demonstrates that it can handle not only small and mid-sized models, but also larger INT4-quantized architectures at reasonable token speeds.

For simple prompts such as “What is GPU?”, RTX Pro 2000 delivers:

• Smooth interactive performance with models up to 8B
• Competitive performance from GPT-OSS 20B
• Predictable scaling behavior as model size increases

Overall, RTX Pro 2000 Blackwell proves to be a capable entry-level inference GPU for Ollama-based deployments, especially when paired with well-optimized INT4 models.

Only $99/mo, Rent Pro2000 Blackwell Server >