Lustre Parallel Filesystem
What is Lustre?
Lustre is the world's most widely deployed parallel distributed filesystem for HPC. It powers the storage systems of the majority of TOP10 supercomputers — including Frontier, Perlmutter, and Summit — delivering aggregate I/O throughput in the terabytes per second range to hundreds of thousands of simultaneous clients.
Originally developed by Cluster File Systems and now maintained by the Lustre community (with major contributions from DDN, HPE, and Intel), Lustre is open-source (GPL) and POSIX-compliant, meaning existing applications work without modification.
Frontier at ORNL runs a Lustre filesystem providing 700 PB of storage and over 5 TB/s of aggregate I/O bandwidth — feeding 9,408 compute nodes simultaneously. This is what "parallel filesystem" means at scale.
Architecture: MDT, OST, and Clients
Lustre separates metadata from data — a fundamental design decision enabling independent scaling of both:
| Component | Full name | Stores | Typical count |
|---|---|---|---|
| MGS | Management Server | Filesystem configuration | 1 (often co-located with MDT) |
| MDS / MDT | Metadata Server / Target | Filenames, directories, permissions, layout info | 1–hundreds (DNE) |
| OSS / OST | Object Storage Server / Target | Actual file data in fixed-size objects | Tens to thousands |
| Client | Lustre client (kernel module) | Nothing — mounts the filesystem | Thousands of compute nodes |
When a client writes a file, Lustre stores the file's data stripes across multiple OSTs in parallel. The MDT only stores the layout (which OSTs hold which stripes) — the actual data transfer goes directly between client and OSTs over the high-speed interconnect.
Striping — The Key to Performance
Striping distributes a file's data across multiple OSTs, allowing parallel I/O from all nodes accessing that file simultaneously. The two critical parameters are:
- stripe_count — how many OSTs to spread the file across (default: 1, optimal for large files: 4–64+)
- stripe_size — chunk size per OST per stripe (default: 1 MB, adjust for access pattern)
For files smaller than 1 GB: stripe_count=1 (default). For files 1–100 GB: stripe_count=4–8. For files over 100 GB or frequently accessed by many nodes: stripe_count=16–64. Never stripe small files widely — metadata overhead dominates.
Distributed Namespace (DNE)
Early Lustre had a single MDS — a metadata bottleneck for workloads creating millions of files. DNE (Distributed Namespace Extension) introduced in Lustre 2.4 allows multiple MDTs, with directories spread across them. This dramatically increases metadata throughput for workloads like ML training that create massive numbers of small image files.
HSM — Hierarchical Storage Management
Lustre HSM enables automatic tiering between fast disk and slow tape. Frequently accessed data stays on the parallel filesystem; infrequently accessed data is transparently migrated to tape archives. When needed, it is recalled automatically on access — transparent to the application.
Parallel Filesystem Comparison
| Filesystem | Vendor | Strengths | Common deployment |
|---|---|---|---|
| Lustre | Open-source / DDN | Highest throughput, TOP500 dominance, open-source | National labs, universities, most HPC centers |
| GPFS / Spectrum Scale | IBM | POSIX, strong consistency, multi-protocol | IBM-heavy enterprises, some national labs |
| DAOS | Intel / open-source | Object store, NVMe-native, low latency | Frontier (alongside Lustre), next-gen systems |
| BeeGFS | ThinkParQ | Easy deployment, good for mid-scale | University clusters, AI labs |
| WekaFS | WEKA | Flash-optimized, cloud-native | AI/ML clusters, hybrid cloud HPC |
Best Practices for Users
- Never use home directories for compute I/O — home is typically NFS, not Lustre; always use
/scratch - Set stripe count before writing large files — you cannot re-stripe a file after it is written without copying it
- Avoid millions of small files in one directory — this hammers the MDT. Use subdirectory structures or HDF5/NetCDF containers
- Use collective I/O with MPI-IO or HDF5 — coordinate I/O across ranks to generate large sequential writes rather than millions of small random ones
- Respect scratch quotas and purge policies — scratch is not backed up and is regularly purged
Key Takeaways
- Lustre separates metadata (MDT) from data (OST) — enabling independent scaling of both
- Striping is the primary performance lever — wider stripes for large files accessed by many nodes
- DNE solves the metadata bottleneck for workloads with millions of files
- Use MPI-IO or HDF5 for collective I/O — never have thousands of ranks each open their own file independently
- Lustre is deployed at petabyte scale with TB/s throughput on the world's largest HPC systems