OSU MPI Benchmarks — Network Performance Testing
The OSU Micro-Benchmarks from Ohio State University are the standard suite for measuring MPI communication performance on HPC systems. They validate InfiniBand fabric quality, compare interconnect generations, and diagnose network bottlenecks that degrade parallel application performance.
Point-to-Point Latency
osu_latency measures the round-trip time for a small message between two processes on different nodes. InfiniBand HDR achieves ~1.2 µs round-trip; NDR ~1.0 µs; high-speed Ethernet ~8–15 µs. Latency is critical for tightly-coupled MPI with frequent small synchronization messages.
Point-to-Point Bandwidth
osu_bw measures unidirectional throughput between two nodes for increasing message sizes. InfiniBand NDR achieves ~48 GB/s; HDR ~24 GB/s; 100GbE ~11 GB/s. Bandwidth determines performance for large data transfers — checkpoint files, domain decomposition halos, parallel I/O.
Collective Benchmarks
osu_allreduce, osu_bcast, osu_alltoall — measure the performance of the MPI collective operations that dominate parallel application communication. Allreduce performance at scale directly determines distributed AI training throughput. Poor collective performance indicates fabric congestion or suboptimal routing.
GPU-Aware MPI Tests
OSU provides GPU variants testing GPUDirect RDMA performance — direct GPU-to-GPU transfers over InfiniBand bypassing CPU memory. osu_latency_cuda and osu_bw_cuda measure GPU memory transfers. GPUDirect RDMA should achieve near the same latency as CPU-based transfers (~5 µs vs ~50 µs without).
Fabric Acceptance Testing
Run OSU latency and bandwidth tests between every node pair in the cluster after installation. Any pair showing latency >2× median or bandwidth <80% expected indicates a fabric issue: wrong cable, bad port, incorrect QoS configuration, or firmware problem requiring investigation before production.
Diagnosing Bottlenecks
If OSU collective tests show poor scaling: check ibdiagnet for switch congestion, verify adaptive routing is enabled on the subnet manager, check for link speed degradation on any port in the path. Even a single 100 Gb/s link in an otherwise 200 Gb/s fabric can bottleneck all traffic routing through it.
Regression Testing
Run OSU benchmarks monthly as part of cluster health monitoring. Network performance degradation — from switch firmware bugs, cable aging, or configuration drift — appears gradually. A 10% latency increase over three months indicates a developing problem requiring proactive investigation.
Getting OSU Benchmarks
Free download from mvapich.cse.ohio-state.edu/benchmarks. Compile with your MPI implementation: ./configure CC=mpicc CXX=mpicxx && make. GPU variants require CUDA or ROCm. Run point-to-point tests with exactly 2 processes on different physical nodes for meaningful results.