HPC Cluster Setup & Management is the holistic lifecycle of owning a supercomputer. It combines the initial "Build" phase with the ongoing "Run" phase.

While System Integration focuses on the physical wiring and initial burn-in, Cluster Management focuses on the software layer that keeps the system usable for scientists day after day. It answers the question: "How do I ensure Node 500 has the exact same Python version as Node 1, and how do I prevent User A from crashing the system for User B?"

Here is the detailed breakdown of the setup philosophy (Infrastructure as Code), the management pillars, and the toolset, followed by the downloadable Word file.

1. The Setup Philosophy: Infrastructure as Code (IaC)

In the past, admins manually installed software on servers. In modern HPC, this is forbidden. We treat servers like cattle, not pets.

• Golden Images: We create one perfect operating system image.
• Provisioning: When a node turns on, it downloads this image from the master server via the network (PXE Boot).
• Configuration Management: Tools like Ansible apply the final tweaks (creating user accounts, setting up printers, mounting storage).
• Result: If a node acts weird, we don't fix it. We reboot it, and it reloads a fresh, perfect copy of the OS in 5 minutes.

2. The Management Pillars

A. Workload Management (The Scheduler)

The heart of the cluster is the Scheduler (usually Slurm). Setting it up is an art form.

• Fairshare: Configured to ensure that if the "Chemistry Dept" used 90% of the cluster last month, their priority drops this month to give the "Physics Dept" a turn.
• Backfill: Configured to squeeze small jobs into the gaps left by large jobs, ensuring high utilization.
• Limits: Setting "Max Walltime" (e.g., 7 days) to prevent zombie jobs from running forever.

B. User Management

• Onboarding: Automating the creation of accounts. When a new user is added to the "HPC Group" in Active Directory, the cluster automatically creates their home directory /home/user and sets their storage quotas.
• Environment Modules: Managing the software complexity. We use Lmod so users can load specific versions of software (module load python/3.8) without conflicting with the system default.

C. High Availability (HA)

• Head Node Redundancy: The "Head Node" controls everything. If it dies, the cluster stops. We setup Pacemaker/Corosync to run two Head Nodes in active-passive mode. If Master A fails, Master B takes over the IP address instantly.

3. The "Day 2" Operations

Once the cluster is set up, the real work begins.

• Patching: You cannot simply run yum update on 1,000 nodes. We use a rolling update strategy: drain 50 nodes, update them, reboot, undrain, move to the next 50.
• Health Checks: Automated scripts (Node Health Check - NHC) run before every single job. If a node has low memory or a full disk, the script marks the node "OFFLINE" so the user's job doesn't crash immediately.

4. Key Applications & Tools

Category	Tool	Usage
Scheduler	Slurm	The industry standard for scheduling jobs and managing resources.
Provisioning	Warewulf / xCAT	Manages the OS images and PXE booting of compute nodes.
Config Mgmt	Ansible	Automates software installation and configuration updates across the cluster.
Environment	Lmod	Manages user software environments (Modules) dynamically.
Monitoring	Prometheus + Grafana	visualizes CPU temps, load, and queue status in real-time.
Health	NHC (Node Health Check)	Prevents jobs from starting on broken nodes.