Nvidia H100 vLLM Benchmark Results: Offline and Online Inference of LLMs on Hugging Face

• GPU: NVIDIA H100
• Microarchitecture: Hopper
• Compute capability: 9.0
• CUDA Cores: 14592
• Tensor Cores: 456
• Memory: 80GB HBM2e
• FP32 performance: 183 TFLOPS

• Impact of model size on performance: The smaller the model size, the higher the request throughput and total throughput. The performance of 7B and 8B models is significantly better than that of 14B and 32B models.
• GPU Utilization (GPU UTL): The larger the model size, the higher the GPU utilization. The GPU utilization of the 32B model is close to full load.
• GPU memory usage (GPU vRAM): In all tests, the GPU video memory usage rate is close to 90%, indicating that the GPU vRAM is close to full load.

• Impact of model size on performance: In online tests, the performance of 7B and 8B models is still significantly better than that of 14B and 32B models.
• GPU Utilization (GPU UTL): In online tests, GPU utilization is generally high, especially for larger models.
• GPU memory usage (GPU vRAM): In all tests, the GPU video memory usage rate is close to 90%, indicating that the GPU vRAM is close to full load.

• Use the benchmark_throughput.py script to simulate batch requests.
• Usually all requests are sent at once, and the system needs to process a large number of requests at the same time, which may cause resource competition and video memory pressure.
• More suitable for evaluating the maximum throughput and extreme performance of the system.

• Use the benchmark_serving.py script to simulate client requests.
• Requests are sent gradually, so the system can allocate resources more flexibly and reduce resource contention.
• More suitable for evaluating the performance of the system in an actual production environment.

• In online testing, requests are sent gradually, and the system can dynamically allocate resources (such as video memory and computing resources) based on the current load.
• In offline testing, all requests are sent at once, and the system needs to process a large number of requests simultaneously, which may cause resource competition and video memory pressure, thereby reducing performance.

• In online testing, video memory can be allocated more flexibly to each request, reducing video memory fragmentation.
• In offline testing, the video memory may be occupied by a large number of requests at the same time, resulting in insufficient video memory or the need for frequent recalculations (such as PreemptionMode.RECOMPUTE), which degrades performance.

• In online testing, requests are processed incrementally, and the system can better balance computing and I/O operations.
• In offline testing, a large number of requests arrive at the same time, which may lead to computing and I/O bottlenecks.

• In online tests, GPU utilization is generally more stable and the system can better utilize the computing power of the GPU.
• In offline testing, GPU utilization may fluctuate due to resource contention, resulting in performance degradation.

• Different input and output lengths have a significant impact on performance. Longer sequences increase video memory usage and computational complexity, which reduces throughput and increases latency. In actual projects, users should adjust the input and output lengths according to task requirements to optimize performance.
• Increasing the number of requests will improve system throughput, but may also lead to insufficient video memory or resource contention. Users should adjust the number of requests based on hardware configuration and task requirements to balance performance and resource usage.

• The larger the model size, the higher the video memory usage and computing requirements. It is recommended that users choose the appropriate model size based on the hardware configuration. For example, the H100 80GB GPU is suitable for running models from 7B to 32B, while models above 40B may require multi-GPU parallelism or hardware support with higher video memory.
• If the model size exceeds the GPU memory capacity (such as 80GB), it will lead to insufficient memory and significantly degraded performance. It is recommended that users avoid using models that exceed the hardware memory capacity in testing and production environments.

• GPU vRAM is one of the main performance bottlenecks. Users should monitor video memory usage to avoid performance degradation caused by insufficient video memory. Video memory usage can be optimized by adjusting the --gpu-memory-utilization parameter.
• A higher GPU utilization indicates that hardware resources are fully utilized, but if the utilization is close to 100%, it may cause performance bottlenecks. Users should adjust the model and parameters based on the test results to optimize GPU utilization.

• Offline testing is suitable for evaluating the extreme performance of the system, while online testing is more suitable for simulating the actual production environment. Users should choose the appropriate test mode according to project requirements.

• In actual projects, users should adjust test parameters (such as input and output lengths, number of requests, etc.) according to task requirements and hardware configuration to achieve better performance.
• When deploying a model, it is recommended to monitor system resource usage (such as video memory, GPU utilization, etc.) in real time to promptly identify and resolve performance bottlenecks.

H100 Server Rental，vllm h100 benchmark, h100 vllm performance testing, vllm online vs offline testing, h100 large language model inference, vllm hugging face models, h100 gpu server rental, vllm performance optimization, h100 deep learning inference