Evaluating Llama 3.3 70B inference on NVIDIA H100 and A100 GPUs
Throughput and time to first token for Llama 3.3 70B on four-GPU H100 and A100 systems, from 1 to 500 concurrent users.
NVIDIA NIM, TensorRT-LLM, genai-perf, 4× H100 SXM5, 4× A100 PCIe
Model Serving and Inference4 minute read