Application Benchmarking for HPC
Application benchmarking runs your actual scientific or engineering code on representative inputs to measure real performance on target hardware. While HPL and HPCG provide standardized comparisons, your application benchmark is the only reliable predictor of how the system will actually perform for your specific workload.
Strong Scaling
Fixed problem size, increasing node count. Measures how much faster the code runs with more resources. Plot efficiency = (1-node time) / (N × N-node time). Efficiency above 80% at your target scale indicates good parallelization. Below 50% signals a fundamental bottleneck limiting further scaling.
Weak Scaling
Problem size grows proportionally with node count — each node handles the same workload. Ideal efficiency = 1.0 (constant time). Measures communication overhead growth. More forgiving than strong scaling for most HPC codes — relevant for production workloads where you want to solve bigger problems, not just faster.
Amdahl's Law
Maximum speedup S = 1/(s + (1-s)/N) where s is serial fraction. A code with just 5% serial fraction is limited to 20× speedup regardless of how many nodes you add. Identify serial fractions early — they define your scaling ceiling and determine the optimal node count for cost-efficiency.
Roofline Model
Plot arithmetic intensity (FLOPS/byte) vs attainable performance to classify workload as memory-bound or compute-bound. Memory-bound codes benefit from faster memory (HBM) or better data reuse algorithms. Compute-bound codes benefit from GPU offload or vectorization. Determines which hardware investment delivers ROI.
Profiling Before Benchmarking
Profile your application with Intel VTune, NVIDIA Nsight, or Score-P before designing a benchmarking campaign. Identify the top 3 time-consuming routines and their bottleneck (compute, memory, or communication). Benchmarking without profiling wastes time optimizing the wrong component.
Representative Inputs
Benchmark inputs must represent production workloads — not toy problems. A 100-node benchmark with a small dataset may give misleading scaling results if production jobs use 10× more data. Use production-scale datasets wherever possible, and document clearly what the benchmark represents.
Vendor Benchmark Runs
For procurement, provide vendors with your application code and datasets and require them to demonstrate performance on their proposed hardware. Specify the exact metric (wall time, throughput, GPU utilization) and acceptable minimum. Vendor-run results on their own benchmarks are not comparable to your production workload.
Continuous Benchmarking
Include application benchmarks in monthly regression testing alongside HPL and STREAM. Software updates, MPI library changes, and driver updates can all impact application performance — sometimes significantly. A 10% slowdown from a routine OS update, undetected for months, wastes substantial allocation hours.