How To Build a Slurm HPC Cluster on a Single Machine

In large-scale data analysis, using a queue job/distributed system is essential. Therefore, we need a mechanism that allows users to access cluster resources and request appropriate CPU, RAM, and GPU for their tasks. If they use more memory than they requested, the simplest solution is to kill the job. This helps avoid out-of-memory issues where the computer would freeze. Slurm is the most common cluster setup, but understanding how to create a Slurm cluster is not easy. Therefore, I created this blog series to guide you through setting it up on a single machine first, then scaling it later and using it effectively and efficiently.

This is Part 1 of a 3-part series where we'll build a complete Slurm cluster from scratch. In this first post, we'll cover the fundamentals by setting up a single-node Slurm cluster and understanding the core concepts.

Series Overview

• Part 1 (This Post): Introduction, Architecture, and Single Node Setup
• Part 2: Scaling to Production with Ansible
• Part 3: Administration and Best Practices

Why Slurm?

When it comes to job scheduling in HPC environments, several options exist including PBS, Grid Engine, and IBM's LSF. However, Slurm (Simple Linux Utility for Resource Management) stands out for several compelling reasons:

• Open Source: Free to use with a large, active community
• Scalability: Designed to scale from small clusters to the world's largest supercomputers
• Flexibility: Fine-grained control over job scheduling, resource allocation, and priority settings
• Integration: Works seamlessly with MPI, distributed computing frameworks (Spark, Ray, Dask), and monitoring tools
• Performance: Optimized for high throughput with minimal overhead

Understanding Slurm Architecture

Figure 1: The standard architecture of a multi-node Slurm cluster

Core Nodes

The main function of Slurm or any cluster is to connect computing machines with large numbers of CPUs, memory, and GPUs. It has a management system where users can request computing resources (16 cores, 32GB RAM, for example). It finds available computing machines and allocates resources to users. How can we design a system to do this at a high level?

We can divide it into 3 types of machines: login nodes, controller nodes, and compute nodes:

Controller nodes: They act as the controller, receive requests from users, allocate resources, and manage resources. Additionally, it's good practice to configure them with a database (SQL database) to store accounting information. This helps track who ran jobs, how they used computing resources, etc.

Login nodes: They act as the gateway, usually accessed via the public network. Users can SSH to login to the machine and request compute resources. The login node sends the request to the controller to decide whether there are available computing resources or not. Then it allocates resources or asks users to wait. Normally, without controller permission, users cannot "stand" on the compute nodes where the large resources actually reside.

Compute nodes: Simply have large resources and connect to the controller to wait for allocation.

Core Components

From the previous section and Figure 1, we can now identify the related services (software) that help the cluster connect together and work properly

slurmctld (Controller Daemon): The brain of the cluster, running on the controller node. It handles job scheduling, resource tracking, and communicates with compute nodes.

slurmd (Node Daemon): Runs on compute nodes to execute jobs and report status back to the controller.

slurmdbd (Database Daemon): Optional but recommended for storing job accounting data, resource usage tracking, and fair-share scheduling.

All In One - Single Node Setup

Figure 2: The Single Node Slurm Cluster Architecture

Starting with a single-node setup helps you understand how Slurm works before scaling up. This approach is perfect for learning and local development. For personal usage, you can configure it to use Slurm for resource allocation. According to Figure 2, we will now install everything on a single machine.

Virtual Machine and Docker

To manually set up the single-node Slurm cluster, instead of using your own computer, it is better to use a virtual machine or Docker:

Vagrant:

• To set up the virtual machine, install Vagrant: https://developer.hashicorp.com/vagrant
• Configure the new virtual machine: https://developer.hashicorp.com/vagrant/tutorials/networking-provisioning-operations/getting-started-provisioning
• To reproduce this tutorial, change the virtual machine box to "alvistack/ubuntu-20.04"

Docker:

• Install Docker: https://docs.docker.com/engine/install
• Create the container with base image: ubuntu:20.04
• For Docker, you are the root user, so you don't need to use sudo to run commands

Install Slurm dependencies

First, install the required Slurm components:

1
sudo apt-get update -y && sudo apt-get install -y slurmd slurmctld

Verify the installation:

1
# Locate slurmd and slurmctld
2
which slurmd
3
which slurmctld
4
# Output: /usr/sbin/slurmctld

Configuring slurm.conf

The slurm.conf file is the heart of your Slurm configuration. This file must be identical across all nodes in a cluster (but for now, we just have one node).

Create your slurm.conf:

1
cat <<EOF > slurm.conf
2
# slurm.conf for a single-node Slurm cluster
3
ClusterName=localcluster
4
SlurmctldHost=localhost
5
MpiDefault=none
6
ProctrackType=proctrack/linuxproc
7
ReturnToService=2
8
SlurmctldPidFile=/run/slurmctld.pid
9
SlurmctldPort=6817
10
SlurmdPidFile=/run/slurmd.pid
11
SlurmdPort=6818
12
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
13
SlurmUser=slurm
14
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
15
SwitchType=switch/none
16
TaskPlugin=task/none
17

18
# TIMERS
19
InactiveLimit=0
20
KillWait=30
21
MinJobAge=300
22
SlurmctldTimeout=120
23
SlurmdTimeout=300
24
Waittime=0
25

26
# SCHEDULING
27
SchedulerType=sched/backfill
28
SelectType=select/cons_tres
29
SelectTypeParameters=CR_Core
30

31
# ACCOUNTING (not enabled yet)
32
AccountingStorageType=accounting_storage/none
33
JobAcctGatherType=jobacct_gather/none
34
JobAcctGatherFrequency=30
35

36
# LOGGING
37
SlurmctldDebug=info
38
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
39
SlurmdDebug=info
40
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
41

42
# COMPUTE NODES (adjust CPUs and RealMemory to match your system)
43
NodeName=localhost CPUs=2 Sockets=1 CoresPerSocket=2 ThreadsPerCore=1 RealMemory=1024 State=UNKNOWN
44

45
# PARTITION CONFIGURATION
46
PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP
47
EOF
48

49
sudo mv slurm.conf /etc/slurm-llnl/slurm.conf

Starting Services

Start the Slurm daemons:

1
# Start slurmd (compute daemon)
2
sudo service slurmd start
3
sudo service slurmd status
4

5
# Start slurmctld (controller daemon)
6
sudo service slurmctld start
7
sudo service slurmctld status

Test your setup by submitting a simple interactive job:

1
srun --mem 500MB -c 1 --pty bash
2

3
# Check job details
4
squeue -o "%i %P %u %T %M %l %D %C %m %R %Z %N" | column -t

Enable Resource Limitation With cgroups

Let's test this problem first. Submit a job requesting 500MB and try to allocate much more:

1
srun --mem 500MB -c 1 --pty bash
2

3
# Try to allocate 1GB of memory (exceeding the 500MB limit)
4
declare -a mem
5
i=0
6
while :; do
7
    mem[$i]=$(head -c 100M </dev/zero | tr '\000' 'x')
8
    ((i++))
9
    echo "Allocated: $((i * 100)) MB"
10
done

Before submitting the job, memory usage is less than 200MB:

After allocating 1GB, the job is not killed due to missing control group (cgroup) configuration:

You'll notice the job continues running even after exceeding 500MB - that's the problem!

Now let's fix it with cgroups:

1
cat <<EOF > cgroup.conf
2
CgroupAutomount=yes
3
CgroupMountpoint=/sys/fs/cgroup
4
ConstrainCores=yes
5
ConstrainRAMSpace=yes
6
ConstrainDevices=yes
7
ConstrainSwapSpace=yes
8
MaxSwapPercent=5
9
MemorySwappiness=0
10
EOF
11

12
sudo mv cgroup.conf /etc/slurm-llnl/cgroup.conf

Update slurm.conf to use cgroup plugins:

1
sudo sed -i -e "s|ProctrackType=proctrack/linuxproc|ProctrackType=proctrack/cgroup|" \
2
            -e "s|TaskPlugin=task/none|TaskPlugin=task/cgroup|" /etc/slurm-llnl/slurm.conf

Enable cgroup in GRUB and reboot:

1
sudo sed -i 's/^GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1 /' /etc/default/grub
2
sudo update-grub
3
sudo reboot

After reboot, restart Slurm services:

1
sudo service slurmctld restart
2
sudo service slurmd restart

Now test again with the same memory allocation script - this time, the job will be killed when it exceeds the limit!

Enabling Accounting

Job accounting is essential for:

• Tracking who is using resources
• Monitoring job completion and failures
• Enforcing resource limits per user/group
• Fair-share scheduling

Install the required packages:

1
sudo apt-get install slurmdbd mariadb-server -y

Create the database and user:

1
sudo service mysql start
2

3
sudo mysql -e "CREATE DATABASE slurm_acct_db;"
4
sudo mysql -e "CREATE USER 'slurm'@'localhost' IDENTIFIED BY 'slurm';"
5
sudo mysql -e "GRANT ALL PRIVILEGES ON slurm_acct_db.* TO 'slurm'@'localhost';"
6
sudo mysql -e "FLUSH PRIVILEGES;"

Verify the database was created:

1
sudo mysql -e "SHOW DATABASES;"
2
sudo mysql -e "SELECT User, Host FROM mysql.user;"

Configure slurmdbd:

1
cat <<EOF > slurmdbd.conf
2
PidFile=/run/slurmdbd.pid
3
LogFile=/var/log/slurm/slurmdbd.log
4
DebugLevel=error
5
DbdHost=localhost
6
DbdPort=6819
7

8
# DB connection data
9
StorageType=accounting_storage/mysql
10
StorageHost=localhost
11
StoragePort=3306
12
StorageUser=slurm
13
StoragePass=slurm
14
StorageLoc=slurm_acct_db
15
SlurmUser=slurm
16
EOF
17

18
sudo mv slurmdbd.conf /etc/slurm-llnl/slurmdbd.conf
19
sudo service slurmdbd start

Update slurm.conf to enable accounting:

1
sudo sed -i -e "s|AccountingStorageType=accounting_storage/none|AccountingStorageType=accounting_storage/slurmdbd\nAccountingStorageEnforce=associations,limits,qos\nAccountingStorageHost=localhost\nAccountingStoragePort=6819|" /etc/slurm-llnl/slurm.conf
2

3
sudo sed -i -e "s|JobAcctGatherType=jobacct_gather/none|JobAcctGatherType=jobacct_gather/cgroup|" /etc/slurm-llnl/slurm.conf
4

5
sudo systemctl restart slurmctld slurmd

Add your cluster and user to accounting:

1
# Add cluster
2
sudo sacctmgr -i add cluster localcluster
3

4
# Add account for your user
5
sudo sacctmgr -i add account $USER Cluster=localcluster
6

7
# Add your user to the account
8
sudo sacctmgr -i add user $USER account=$USER DefaultAccount=$USER
9

10
sudo systemctl restart slurmctld slurmd

Now test accounting by submitting a job and viewing its details:

1
# Submit a test job
2
srun --mem 500MB -c 1 hostname
3

4
# View accounting information
5
sacct

Key Takeaways

In this first part of our series, we've covered:

1. Why Slurm: Understanding the advantages of Slurm over alternatives
2. Architecture: Core components (slurmctld, slurmd, slurmdbd) and their roles
3. Basic Setup: Installing and configuring a single-node cluster
4. Critical cgroups: Why resource limiting is essential (and how to enable it)
5. Accounting: Setting up job tracking and resource monitoring

References

1.Slurm Overview — Official documentation for Slurm workload manager

2.NVIDIA/deepops — Open-source cluster deployment toolkit (BSD-3-Clause License)

3.elasticluster — Elastic cluster provisioning tool (GPL-3.0 License)

This is Part 1 of the Omicslab series on building Slurm HPC clusters. Continue to Part 2 to learn about production deployment with Ansible.