Slurm is a cluster management and job scheduling system. In simple terms, users reserve the computing power they need (CPU, RAM, GPU) for a specific duration and start their calculations (called jobs). Once the job is completed or the allotted time expires, the resources are returned to the pool so they can be assigned to other users' jobs.
All jobs are placed in a queue, where they are arranged based on job priority. The calculation of priority will be discussed later in this document.
Your entry point to the class is the login nodes. You can access it via ssh and start submitting jobs to be executed on the compute nodes.
Be aware that login nodes are resource-constrained and are limited to 40G RAM and 4 CPU cores per user. You should run your calculations on the compute node, not the login node.
If you want to learn more about Slurm, you can check out the quick start guide at https://slurm.schedmd.com/quickstart.html.
Compute nodes, categorized by hardware specifications, are grouped into partitions. You can list partitions with the command sinfo -s.
The command scontrol show partition provides more detailed information about partitions.
Depending on your job type, you can submit your job to a desired partition. Each partition has a separate job queue. You can view the queue of a specific partition by using the command squeue -p the_partition_name.
As of April 2024, there are four partitions in M3:
The Slurm cpu resource actually is a cpu-thread and not a cpu-core.
Independent of the resource request Slurm will always round up cpu to multiples of 2 for the allocation.
This guarantee that a job is not sharing cpu-cores with other jobs
and also means that the effective minimum per job is cpu=2.
And here is the list of other limitations that apply to each partition:
| cpu1 | cpu2-hm | cpu3-long | gpu-a30 | |
|---|---|---|---|---|
| Maximum time limit for jobs | 1day | 1day | 14days | 1day |
| Maximum number of jobs a user can run | - | - | - | - |
| Maximum resources a user can occupy | - | - | cpu=200, mem=2.25T | - |
| Maximum available resources of a node1 | cpu=64, mem=455G | cpu=128, mem=1.83T | ? 2 | cpu=96, mem=455G, gpu=2 |
| Maximum resources available to the partition | cpu=1280, mem=8.9T | cpu=256, mem=3.5T | cpu=768, mem=5.3T | cpu=576, mem=2.6T, gpu=12 |
| Price for a single core job | 1 | 1.3 | 2.6 | 1.7 (if gpu job, then 84) |
If you're interested in the specifications of the compute nodes, then:
| c001 - c020 | hm023 - hm024 | g025 - g030 | |
|---|---|---|---|
| CPU | Model: AMD EPYC 7343 16-Core Processor Threads per core: 2 Cores per socket: 16 Sockets: 2 Max MHz: 3940.6250 NUMA node0 CPUs: 0-15,32-47 NUMA node1 CPUs: 16-31,48-63 |
Model: AMD EPYC 7542 32-Core Processor Threads per core: 2 Cores per socket: 32 Sockets: 2 Max MHz: 2900.0000 NUMA node0 CPUs: 0-31,64-95 NUMA node1 CPUs: 32-63,96-127 |
Model: AMD EPYC 74F3 24-Core Processor Threads per core: 2 Cores per socket: 24 Sockets: 2 Max MHz:4037.5000 NUMA node0 CPUs: 0-23,48-71 NUMA node1 CPUs: 24-47,72-95 |
| RAM | Type: DIMM DDR4 Synchronous Registered Max MHz: 3200 MHz Module size: 32GiB Total: 512GiB |
Type: DIMM DDR4 Synchronous Registered Max MHz: 3200 MHz Module size: 64GiB Total: 2TiB |
Type: DIMM DDR4 Synchronous Registered Max MHz: 3200 MHz Module size: 32GiB Total: 512GiB |
| GPU | 0 | 0 | Product name: NVIDIA A30 Max clocks MHz: Graphics=1440 SM=1440 Memory=1215 Video=1305 FB memory total: 24576 MiB BAR1 memory total: 32768 MiB Per node: 2 GPU |