Training Material Creation for HPC software development is about bridging the gap between "Code that works" and "Code that scales."

A standard developer writes code that runs on 1 core. An HPC developer writes code that must coordinate 100,000 cores without crashing. The training materials must focus on Parallel Paradigms, Memory Hierarchy, and the "Think in Vectors" mindset.

Here is the detailed breakdown of the curriculum structure, the "Code Clinic" methodology, and the hands-on lab modules, followed by the downloadable Word file.

1. The Core Curriculum: The "HPC Stack"

The training materials should be organized into four progressive modules.

Module 1: The Architecture (Know Your Hardware)

• Concept: You cannot write fast code if you don't understand the machine.
• Topics:
• NUMA (Non-Uniform Memory Access): Why accessing RAM on the "other" socket makes your code 2x slower.
• The Cache Line: Why reading data in "Strides" (skipping memory addresses) destroys performance.
• The Interconnect: Understanding Latency vs. Bandwidth.

Module 2: Parallel Programming Models

• Shared Memory (OpenMP):
• Analogy: One boss (Process), many workers (Threads) in the same room (Node).
• Key Lab: Taking a serial for loop and making it parallel with #pragma omp parallel for.
• Distributed Memory (MPI):
• Analogy: Many bosses in different buildings (Nodes) talking over the phone (Network).
• Key Lab: Implementing "Halo Exchange" (passing data boundaries to neighbors).
• Accelerator (CUDA / OpenACC):
• Analogy: Sending the heavy math to the army of graphical calculators (GPU).
• Key Lab: Offloading a matrix multiplication to the GPU using Directives (OpenACC) vs. Native Kernels (CUDA).

2. Material Formats: The "Code Clinic" Approach

Static slides put developers to sleep. Use these formats instead:

• The "Broken Code" Repo:
• Strategy: Do not give them a blank screen. Give them a GitHub repo with code that runs but is terribly slow.
• Task: "This code takes 10 minutes. Use a profiler to find the bug. Fix it. Goal: 1 minute."
• Visual Cheatsheets:
• Resource: One-page PDFs showing the syntax for common MPI calls (MPI_Send, MPI_Recv) and OpenMP Pragmas. Developers print these and tape them to their monitors.
• Performance Patterns:
• Resource: A library of "Good vs. Bad" patterns.
• Example: "Structure of Arrays (SoA)" vs. "Array of Structures (AoS)." (SoA is usually faster for vectorization).

3. The Debugging & Profiling Module

Teaching how to code is easy; teaching how to fix bugs in parallel is hard.

• Parallel Debugging: How to use TotalView or gdb to attach to Process #45 out of 100 processes when it crashes.
• Profiling: How to interpret a "Flame Graph" in Intel VTune or Nsight Systems. "Is my code CPU bound (math) or Memory bound (waiting for data)?"

4. Key Applications & Tools

Category	Tool	Usage
Compiler	Compiler Explorer (Godbolt)	A web tool to show how C++ code turns into Assembly. Great for teaching Vectorization.
Profiling	Intel VTune / Advisor	The gold standard for visualizing bottlenecks. "Advisor" specifically tells you what to change in your code.
Debugging	TotalView / DDT	Specialized debuggers that can visualize data arrays distributed across 100 nodes.
Repo	GitHub Classroom	Automates the distribution of the "Broken Code" assignments and auto-grades the performance improvements.