Slurm

Beginner’s Guide to SLURM

SLURM (Simple Linux Utility for Resource Management) is a job scheduler and resource management system commonly used in high-performance computing (HPC) environments. It allows users to submit and manage jobs on clusters or supercomputers. This guide provides a brief overview of SLURM and covers basic usage examples for sbatch and srun commands, along with common options for requesting resources such as memory, CPUs, and GPUs.

Introduction to SLURM

SLURM is a flexible and efficient framework for job submission and scheduling in HPC environments, enabling users to run parallel and distributed applications across multiple compute nodes in a coordinated manner. It serves as a job scheduler that queues, allocates, and launches both interactive and batch jobs.

Slurm provides two different types of jobs:

• Interactive Jobs: Access a compute node like you would via ssh
• Batch Jobs: Launch a job in the background

All jobs in the Slurm system are scheduled in a priority queue, meaning that heavy users may need to wait longer for their jobs to be launched. To manage jobs in the system, users can take advantage of several useful Slurm commands, including:

• • srun: Run a real time job, useful for launching interactive jobs
  • sbatch: Queue a batch job to be launched as available
  • scancel: Kill a running slurm job
  • sacct: View the history of your recently run jobs; Did they complete? fail?
  • sinfo: View the resources, nodes, gres on the slurm cluster
  • squeue: View the state of currently running jobs
  • sstat: Resource utilization by a particular job

These commands provide users with powerful tools to manage their jobs on the BABEL cluster and ensure that their work is completed efficiently and effectively. We will discuss these commands in more detail later.

Submitting Jobs with sbatch

To submit a batch job using sbatch, create a shell script (e.g., job_script.sh) that contains the necessary commands and configurations for your job. Then, use the following command to submit the job:

 $ sbatch job_script.sh

SLURM will assign a unique job ID to your job and enqueue it for execution. You can monitor the status of your job using various SLURM commands like squeue or sacct.

Running Jobs with srun

For interactive or non-batch jobs, you can use the srun command. It allows you to execute commands directly on compute nodes. Here’s an example:

 $ srun -n 4 ./my_program

SLURM will assign a unique job ID to your job and enqueue it for execution. You can monitor the status of your job using various SLURM commands like squeue or sacct.

Running Jobs with srun

For interactive or non-batch jobs, you can use the srun command. It allows you to execute commands directly on compute nodes. Here’s an example:

 $ srun -n 4 ./my_program

The -n option specifies the number of tasks you want to run. In the example above, we are running the my_program executable on four tasks.

Requesting Resources

SLURM provides options to request specific resources for your jobs, such as memory, CPUs, and GPUs.

Memory

To request a specific amount of memory for your job, use the --mem option with the desired value. For example, to request 8 GB of memory, use:

 $ #SBATCH --mem=8G

CPUs

SLURM allows you to request a specific number of CPUs for your job. Use the --cpus-per-task option to specify the number of CPUs needed. For example, to request 4 CPUs per task, use:

 $ #SBATCH --cpus-per-task=4

GPUs

If your job requires GPU resources, you can request them using the --gres option. For example, to request 2 GPUs, use:

 $ #SBATCH --gres=gpu:2

You can also specify the specific GPU type using the --gres option. For example, to request 2 NVIDIA V100 GPUs, use:

 $ #SBATCH --gres=gpu:v100:2

Here are some additional SLURM directives commonly which may make SLURM life more pleasant:

• #SBATCH --output=/home/<username>/path/output_report-%j.out: Specifies the path and filename pattern for the job’s standard output and error logs. %j is a placeholder that will be replaced with the job ID.
• #SBATCH --mail-type=END: Specifies the email notification types for job events. In this case, it is set to receive an email when the job ends.
• #SBATCH --mail-user=<username>@andrew.cmu.edu: Specifies the email address where the job-related emails will be sent.

Here are other useful arguments for srun/sbatch:

• -w, --nodelist: Nodes to run jobs (-x, --exclude is the opposite)
• -t, --time: Time limit (D-HH:MM:SS, 0=infinity) [explained mode below in interactive jobs]
• -o, --output: Output log files. It’s a good idea to flush output frequently to get timely output.

Submit your job within Python

Submitit is a lightweight tool for submitting Python functions for computation within a Slurm cluster. It basically wraps submission and provide access to results, logs and more

These are just a few common options for resource requests in SLURM. SLURM provides many more options for fine-grained resource management, job scheduling, and parallel computing. You can refer to the SLURM documentation for detailed information on all available options.

More Info

For more information on Slurm please consult the Slurm documentation, which provides detailed tutorials and resources for working with Slurm in an HPC environment.

Happy computing with SLURM!