Conducting a comparative performance analysis between HPC systems requires moving beyond "peak theoretical" numbers to empirical, workload-specific data. To identify the optimal configuration, you must evaluate how different architectures (CPU vs. GPU, Ethernet vs. InfiniBand, Shared vs. Parallel storage) handle your specific scientific kernels.

Here is a structured methodology for conducting a detailed comparative analysis.

1. Defining the Benchmarking Suite

A valid comparison requires a tiered approach, starting from synthetic hardware tests and moving to real-world application performance.

Tier 1: Micro-benchmarks (The "Pulse" Check):

STREAM: Measures sustainable memory bandwidth (GB/s).¹ Critical for memory-bound codes like CFD.
HPL (High Performance Linpack): Measures floating-point execution rate (²$R_{max}$).³ Useful for ranking raw compute power but often far from real-world utility.
OSU Micro-benchmarks: Measures point-to-point and collective communication latency and bandwidth.⁴ Essential for comparing InfiniBand (HDR/NDR) vs. Slingshot or RoCE.

Tier 2: Mini-Apps (The "Representative" Logic):

Use skeletonized versions of large codes (e.g., LULESH for hydrodynamics, HPCG for conjugate gradients) that mimic the communication and computation patterns of your actual production software.

Tier 3: Full Application Workloads:

Run a standard "Production Input" (e.g., a 1-million atom GROMACS simulation) across all systems using identical software versions (containers are highly recommended here for parity).

2. Analysis Framework: The Roofline Model

The most effective way to compare systems is the Roofline Model. It plots Arithmetic Intensity (Operations per Byte) against Attainable Performance (GFLOP/s).

Interpretation: If your code sits on the "slanted" part of the roof, it is Memory Bound; adding more cores won't help, but faster RAM or HBM (High Bandwidth Memory) will.⁵ If it sits on the "flat" part, it is Compute Bound; you need more GHz or vector units.
Comparative Use: Overlaying the Roofline of "System A" (e.g., AMD EPYC) and "System B" (e.g., NVIDIA H100) shows exactly where your application will gain the most benefit from a hardware upgrade.

3. Key Comparison Variables

When identifying the "optimal" configuration, compare these three pillars:

Variable	Metrics to Compare	Optimal Configuration Indicator
Compute Density	Time-to-Solution, Energy-per-Solution (Joules)	The system that completes the task with the lowest wall-clock time and power draw.
Interconnect	Injection Bandwidth, Bisection Bandwidth	High efficiency when scaling from 2 to 128 nodes; look for the "Scaling Efficiency" curve to remain above 80%.
Storage I/O	Metadata Ops/sec, Sustained Throughput	Look for minimal "I/O Wait" times during large-scale checkpointing.

4. Cost-Performance Analysis (TCO)

Performance isn't just about speed; it’s about Efficiency-per-Dollar. A system that is 10% faster but 50% more expensive is rarely the "optimal" choice for sustainable growth.

Normalized Throughput: Calculate $(Performance / Cost)$.
Energy Efficiency: In 2026, electricity costs and carbon footprints are primary constraints. Compare the Perf/Watt (Performance per Watt). A liquid-cooled system might have a higher upfront cost but a lower Total Cost of Ownership (TCO) over 5 years due to lower cooling overhead.⁶

5. Identifying the "Bottleneck" (Comparative Profiling)

Use Performance Co-Pilot (PCP) or Intel VTune to compare hardware counters across systems.⁷

Instruction per Cycle (IPC): If System A has higher IPC than System B on the same code, it suggests better branch prediction or instruction scheduling for your logic.
NUMA Locality: Compare performance degradation when tasks cross NUMA boundaries. This identifies if your code needs specific process-pinning optimizations to run optimally on high-core-count processors.

6. Comparative Analysis Checklist

[ ] Parity: Are you using the same compiler versions (e.g., GCC 14.1) and math libraries (MKL vs. AOCL) across both systems?
[ ] Topology: Are the nodes compared in the same network topology (e.g., both within a single leaf switch)?
[ ] Filesystem: Are the systems hitting the same storage backend, or is a slower storage tier skewing the "Time-to-Solution"?
[ ] Warm-up: Did you run "warm-up" iterations to ensure CPU/GPU frequency scaling (Turbo Boost) has stabilized?