14 Jan 2026

Slurm HPC Cluster Administration and Best Practices

Giang Nguyen

In Part 1 and Part 2, we built a complete Slurm HPC cluster from a single node to a production-ready multi-node system. Now let's learn how to manage, maintain, and secure it effectively.

This final post covers daily administration tasks, troubleshooting, security hardening, and integration with data processing frameworks.

Series Overview

Part 1: Introduction, Architecture, and Single Node Setup
Part 2: Scaling to Production with Ansible
Part 3 (This Post): Administration and Best Practices

Administration Overview

Managing a Slurm cluster involves several key areas:

Cluster Management: Build, maintain, and update the cluster via Ansible
User Management: Synchronize users across nodes with proper permissions
Login Security: Implement SSH hardening with 2FA or key pairs
Resource Management: Enforce limits and fair-share policies
Monitoring: Track performance and resource utilization
Troubleshooting: Diagnose and resolve issues

User and Resource Management

Adding Users and Groups

Slurm uses accounts (groups) to organize users and apply resource policies:

1
# Add a new account/group
2
sacctmgr add account research_team Description="Research Team"
3

4
# Add a user to an account
5
sacctmgr add user john account=research_team
6

7
# Add user with multiple accounts
8
sacctmgr add user alice account=research_team,dev_team DefaultAccount=research_team
9

10
# View accounts
11
sacctmgr show account
12

13
# View users
14
sacctmgr show user

Setting Resource Limits

Account-Level Limits

Control resources for entire groups:

1
# Limit CPU minutes (prevents monopolizing cluster)
2
sacctmgr modify account research_team set GrpCPUMins=100000
3

4
# Limit memory (in MB)
5
sacctmgr modify account research_team set GrpMem=500000
6

7
# Limit concurrent jobs
8
sacctmgr modify account research_team set GrpJobs=50
9

10
# Limit concurrent running jobs
11
sacctmgr modify account research_team set GrpJobsRun=20
12

13
# Limit number of nodes
14
sacctmgr modify account research_team set GrpNodes=10

User-Level Limits

Control individual user behavior:

1
# Limit jobs in queue
2
sacctmgr modify user john set MaxJobs=10
3

4
# Limit running jobs
5
sacctmgr modify user john set MaxJobsRun=5
6

7
# Limit wall time (in minutes)
8
sacctmgr modify user john set MaxWall=1440  # 24 hours
9

10
# Limit CPUs per job
11
sacctmgr modify user john set MaxCPUs=32
12

13
# View user limits
14
sacctmgr show user john withassoc format=user,account,maxjobs,maxsubmit,maxwall

Quality of Service (QoS)

QoS allows you to create service tiers with different priorities:

1
# Create QoS levels
2
sacctmgr add qos normal priority=100
3
sacctmgr add qos high priority=500 MaxWall=2-00:00:00 MaxJobs=5
4
sacctmgr add qos low priority=50
5

6
# Assign QoS to account
7
sacctmgr modify account research_team set qos=normal,high
8

9
# Users can specify QoS when submitting
10
sbatch --qos=high job_script.sh

Ensure equitable resource distribution:

1
# Set fair-share values (higher = more priority)
2
sacctmgr modify account research_team set fairshare=100
3
sacctmgr modify account dev_team set fairshare=50
4

5
# View fair-share tree
6
sshare -a
7

8
# View detailed fair-share info
9
sshare -A research_team --all

Node Management

Checking Node Status

1
# View all nodes
2
sinfo
3

4
# Detailed node information
5
sinfo -Nel
6

7
# Show node states
8
sinfo -N -o "%N %T %C %m %e %f"
9

10
# View specific node details
11
scontrol show node worker-01

Node states you'll encounter:

IDLE: Available for jobs
ALLOCATED: Running jobs
MIXED: Partially allocated
DRAIN: Won't accept new jobs (draining)
DRAINED: Fully drained
DOWN: Not responding

Node Maintenance

Draining a Node

When you need to perform maintenance:

1
# Drain node (won't accept new jobs, allows running jobs to finish)
2
scontrol update NodeName=worker-01 State=drain Reason="Hardware upgrade"
3

4
# Force drain (terminate running jobs)
5
scontrol update NodeName=worker-01 State=drain Reason="Emergency maintenance"
6

7
# Check drain reason
8
sinfo -R

Resuming a Node

After maintenance:

1
# Resume node
2
scontrol update NodeName=worker-01 State=resume
3

4
# Verify it's back
5
sinfo -n worker-01

Forcing Node Down

If a node is misbehaving:

1
# Mark node as down
2
scontrol update NodeName=worker-01 State=down Reason="Hardware failure"
3

4
# When fixed, resume
5
scontrol update NodeName=worker-01 State=resume

Adding New Compute Nodes

Update Ansible inventory (inventories/hosts):

1
[slurm_worker]
2
worker-01 ansible_host=192.168.58.11
3
worker-02 ansible_host=192.168.58.12
4
worker-03 ansible_host=192.168.58.13  # NEW

Run Ansible playbook:

1
ansible-playbook -i inventories/hosts cluster_slurm.yml --limit worker-03

Update slurm.conf on controller and all nodes (Ansible handles this)
Restart slurmctld:

1
sudo systemctl restart slurmctld

Verify the new node:

1
sinfo
2
scontrol show node worker-03

Monitoring and Troubleshooting

Checking Slurm Logs

Logs are essential for diagnosing issues:

1
# Controller logs
2
sudo tail -f /var/log/slurm/slurmctld.log
3

4
# Worker node logs (on compute nodes)
5
sudo tail -f /var/log/slurm/slurmd.log
6

7
# Database logs
8
sudo tail -f /var/log/slurm/slurmdbd.log
9

10
# Filter for errors
11
sudo grep "error" /var/log/slurm/*.log
12

13
# Filter for specific node
14
sudo grep "worker-01" /var/log/slurm/slurmctld.log
15

16
# Last 100 lines with context
17
sudo tail -100 /var/log/slurm/slurmctld.log

Common Issues and Solutions

Issue: Node Shows as DOWN

Diagnosis:

1
sinfo
2
# OUTPUT: worker-01    down   ...
3

4
scontrol show node worker-01
5
# Check "Reason" field

Solutions:

1
# 1. Check if slurmd is running
2
ssh worker-01 "sudo systemctl status slurmd"
3

4
# 2. Restart slurmd
5
ssh worker-01 "sudo systemctl restart slurmd"
6

7
# 3. Check network connectivity
8
ping worker-01
9

10
# 4. Check logs
11
ssh worker-01 "sudo tail -50 /var/log/slurm/slurmd.log"
12

13
# 5. Resume the node
14
scontrol update NodeName=worker-01 State=resume

Issue: Jobs Stuck in Pending

Diagnosis:

1
squeue
2
# See jobs in PD (pending) state
3

4
# Check why job is pending
5
squeue --start -j JOB_ID
6

7
# View detailed job info
8
scontrol show job JOB_ID

Common reasons:

Resources: Not enough resources available
Priority: Lower priority than other jobs
Dependency: Waiting for another job to complete
QOSMaxJobsPerUser: User has too many jobs running

Solutions:

1
# 1. Check available resources
2
sinfo -o "%P %a %l %D %N %C"
3

4
# 2. View job requirements
5
scontrol show job JOB_ID | grep -E "Partition|NumNodes|MinMemory"
6

7
# 3. Cancel job if needed
8
scancel JOB_ID
9

10
# 4. Modify pending job
11
scontrol update JobId=JOB_ID NumNodes=1

Issue: Jobs Failing Immediately

Diagnosis:

1
# Check job status
2
sacct -j JOB_ID
3

4
# View job output files
5
cat slurm-JOB_ID.out
6
cat slurm-JOB_ID.err

Common causes:

Script errors (check shebang line)
Missing executables
Resource limits exceeded
Permission issues

Issue: Accounting Database Not Working

Diagnosis:

1
# Check slurmdbd status
2
sudo systemctl status slurmdbd
3

4
# Test database connection
5
sudo mysql -u slurm -p slurm_acct_db -e "SHOW TABLES;"
6

7
# Check slurmdbd logs
8
sudo tail -50 /var/log/slurm/slurmdbd.log

Solutions:

1
# 1. Restart slurmdbd
2
sudo systemctl restart slurmdbd
3

4
# 2. Verify database credentials in slurmdbd.conf
5
sudo cat /etc/slurm-llnl/slurmdbd.conf
6

7
# 3. Check database permissions
8
sudo mysql -e "SHOW GRANTS FOR 'slurm'@'localhost';"
9

10
# 4. Restart slurmctld to reconnect
11
sudo systemctl restart slurmctld

System Logs with rsyslog

Our Ansible setup configures centralized logging:

1
# On controller (rsyslog server)
2
sudo tail -f /var/log/syslog
3

4
# Filter by hostname
5
sudo grep "worker-01" /var/log/syslog
6

7
# Filter by service
8
sudo grep "slurmd" /var/log/syslog
9

10
# Check authentication logs
11
sudo tail -f /var/log/auth.log

Security Best Practices

SSH Hardening

Secure your login nodes! HPC clusters are attractive targets for attackers.

For detailed SSH security setup, see our SSH Remote Server documentation.

Key recommendations:

Disable Password Authentication:

1
PasswordAuthentication no
2
PubkeyAuthentication yes

Implement 2FA with Google Authenticator or similar
Use SSH Key Pairs:

1
# Generate key on your machine
2
ssh-keygen -t ed25519 -C "your_email@example.com"
3

4
# Copy to cluster
5
ssh-copy-id user@controller-node

Limit SSH Access:

1
AllowUsers alice bob charlie
2
AllowGroups cluster_users
3

4
# Or deny specific users
5
DenyUsers baduser

Change Default Port (security through obscurity):

1
Port 2222

Munge Authentication

Munge provides authentication between Slurm components:

1
# Verify munge is running
2
sudo systemctl status munge
3

4
# Test munge
5
munge -n | unmunge
6

7
# Generate new key (do this on controller, then distribute)
8
sudo /usr/sbin/create-munge-key
9

10
# Copy key to all nodes (Ansible does this automatically)
11
sudo scp /etc/munge/munge.key worker-01:/etc/munge/
12

13
# Restart munge on all nodes
14
sudo systemctl restart munge

The munge key must be identical on all nodes and have proper permissions (0400, owned by munge).

Firewall Configuration

Restrict access to Slurm ports:

1
# Allow Slurm ports only from cluster network
2
sudo ufw allow from 192.168.58.0/24 to any port 6817  # slurmctld
3
sudo ufw allow from 192.168.58.0/24 to any port 6818  # slurmd
4
sudo ufw allow from 192.168.58.0/24 to any port 6819  # slurmdbd
5

6
# Allow SSH from anywhere
7
sudo ufw allow 22/tcp
8

9
# Enable firewall
10
sudo ufw enable

Shared Storage Best Practices

NFS Performance Tuning

Optimize NFS for your workload:

1
# /etc/fstab on compute nodes
2
controller-01:/home /home nfs4 rw,soft,rsize=262144,wsize=262144,timeo=14,intr 0 0

Parameters explained:

soft: Timeout after retry (vs hard which waits forever)
rsize/wsize: Read/write buffer size (larger = better performance)
timeo: Timeout value
intr: Allow interrupts

Storage Layout

Recommended directory structure:

1
/home/          # User home directories (SSD/NVMe)
2
  ├─ alice/
3
  ├─ bob/
4
  └─ charlie/
5

6
/mnt/data/      # Large datasets (HDD or object storage)
7
  ├─ shared/    # Common datasets
8
  ├─ projects/  # Project-specific data
9
  └─ scratch/   # Temporary data (auto-cleanup)
10

11
/opt/           # Shared software/modules
12
  ├─ anaconda/
13
  ├─ modules/
14
  └─ apps/

Quotas

Prevent users from filling up shared storage:

1
# Set user quotas
2
sudo setquota -u alice 50G 60G 0 0 /home
3
sudo setquota -u alice 500G 550G 0 0 /mnt/data
4

5
# Check quotas
6
quota -u alice
7

8
# View all quotas
9
sudo repquota -a

Integration with Data Processing Frameworks

One of Slurm's greatest strengths is integration with modern computing frameworks:

Apache Spark

You can install Spark/PySpark on the conda-base (pixi) environment. To run a Spark cluster on top of the Slurm cluster can be setup on this repo: https://github.com/vieomics/omicslab-kit/tree/main/spark-on-slurm

Submit Spark jobs to Slurm:

1
#!/bin/bash
2
#SBATCH --job-name=spark-job
3
#SBATCH --nodes=4
4
#SBATCH --ntasks-per-node=1
5
#SBATCH --cpus-per-task=8
6
#SBATCH --mem=32G
7

8
# Load Spark module
9
module load spark/3.5.0
10

11
# Run Spark application
12
spark-submit \
13
  --master yarn \
14
  --num-executors 4 \
15
  --executor-cores 8 \
16
  --executor-memory 28G \
17
  my_spark_app.py

Ray (Distributed ML)

1
#!/bin/bash
2
#SBATCH --job-name=ray-job
3
#SBATCH --nodes=2
4
#SBATCH --ntasks-per-node=1
5
#SBATCH --cpus-per-task=16
6
#SBATCH --gpus-per-node=2
7

8
# Start Ray cluster
9
ray start --head --port=6379
10
srun --nodes=1 --ntasks=1 ray start --address=$HEAD_NODE:6379
11

12
# Run Ray application
13
python ray_train.py

Dask

1
from dask_jobqueue import SLURMCluster
2
from dask.distributed import Client
3

4
cluster = SLURMCluster(
5
    cores=8,
6
    memory="16GB",
7
    processes=2,
8
    walltime="02:00:00",
9
    queue="compute"
10
)
11

12
cluster.scale(jobs=10)  # Request 10 jobs
13
client = Client(cluster)
14

15
# Your Dask code here

Nextflow (Bioinformatics)

1
process {
2
    executor = 'slurm'
3
    queue = 'compute'
4
    memory = '8 GB'
5
    time = '2h'
6
}

Run with:

1
nextflow run nf-core/rnaseq -profile test,singularity --outdir test_rnaseq_profile_outdir

Maintenance Tasks

Regular Updates

1
# Update cluster via Ansible
2
ansible-playbook -i inventories/hosts cluster_slurm.yml --tags update
3

4
# Update specific nodes
5
ansible-playbook -i inventories/hosts cluster_slurm.yml --limit worker-01,worker-02

Backup Critical Data

1
# Backup Slurm configuration
2
sudo cp /etc/slurm-llnl/slurm.conf /backup/slurm.conf.$(date +%Y%m%d)
3

4
# Backup accounting database
5
sudo mysqldump -u slurm -p slurm_acct_db > slurm_acct_backup_$(date +%Y%m%d).sql
6

7
# Backup user data (use rsync for efficiency)
8
sudo rsync -av /home/ /backup/home/

Monitoring Disk Space

1
# Check disk usage on all nodes
2
ansible all -i inventories/hosts -m shell -a "df -h"
3

4
# Check specific directory
5
ansible all -i inventories/hosts -m shell -a "du -sh /var/log/slurm"
6

7
# Find large files
8
find /home -type f -size +1G -exec ls -lh {} \;

Performance Optimization Tips

1. Tune Scheduler Parameters

1
# Increase scheduling frequency
2
SchedulerTimeSlice=30
3

4
# Prioritize recent submitters less
5
PriorityWeightAge=1000
6
PriorityWeightFairshare=10000
7

8
# Enable backfill scheduling with larger window
9
SchedulerType=sched/backfill
10
bf_window=1440  # 24 hours

2. Optimize Job Packing

1
# Use CR_CPU for CPU-bound jobs
2
SelectType=select/cons_tres
3
SelectTypeParameters=CR_CPU
4

5
# Or CR_Memory for memory-bound jobs
6
SelectTypeParameters=CR_Memory
7

8
# Or CR_Core for mixed workloads
9
SelectTypeParameters=CR_Core

3. Create Multiple Partitions

1
# Fast partition for short jobs
2
PartitionName=quick Nodes=worker-[01-02] Default=NO MaxTime=01:00:00 State=UP Priority=100
3

4
# Standard partition
5
PartitionName=standard Nodes=worker-[01-04] Default=YES MaxTime=2-00:00:00 State=UP Priority=50
6

7
# Long partition for extended jobs
8
PartitionName=long Nodes=worker-[03-04] Default=NO MaxTime=7-00:00:00 State=UP Priority=25
9

10
# GPU partition
11
PartitionName=gpu Nodes=gpu-[01-02] Default=NO MaxTime=1-00:00:00 State=UP Priority=75

4. Enable Job Arrays for Batch Processing

1
#!/bin/bash
2
#SBATCH --array=1-100%10  # 100 tasks, max 10 concurrent
3

4
# Process task based on array index
5
python process.py --input data_${SLURM_ARRAY_TASK_ID}.txt

Conclusion

Congratulations! You now have the knowledge to build, deploy, and manage a production Slurm HPC cluster. Let's recap the journey:

Part 1: Foundations

Understanding Slurm architecture
Single-node setup for learning
Critical cgroup configuration
Job accounting basics

Part 2: Production Deployment

Ansible automation
Multi-node cluster setup
Monitoring with Grafana
Slack alerting

Part 3: Administration (This Post)

User and resource management
Node maintenance and troubleshooting
Security hardening
Performance optimization
Framework integration

References

1.Slurm Overview — Official documentation for Slurm workload manager

2.NVIDIA/deepops — Open-source cluster deployment toolkit (BSD-3-Clause License)

3.elasticluster — Elastic cluster provisioning tool (GPL-3.0 License)

4.Ansible- Ansible for IT Automation DevOps

5.GitHub Repository-Omicslab HPC- Ansible scripts for setting up the Slurm HPC

This concludes the Omicslab series on building Slurm HPC clusters. Thank you for following along! We hope this guide helps you build and manage effective HPC infrastructure.

Series Overview

Administration Overview

User and Resource Management

Adding Users and Groups

Setting Resource Limits

Account-Level Limits

User-Level Limits

Quality of Service (QoS)

Fair-Share Scheduling

Node Management

Checking Node Status

Node Maintenance

Draining a Node

Resuming a Node

Forcing Node Down

Adding New Compute Nodes

Monitoring and Troubleshooting

Checking Slurm Logs

Common Issues and Solutions

Issue: Node Shows as DOWN

Issue: Jobs Stuck in Pending

Issue: Jobs Failing Immediately

Issue: Accounting Database Not Working

System Logs with rsyslog

Security Best Practices

SSH Hardening

Munge Authentication

Firewall Configuration

Shared Storage Best Practices

NFS Performance Tuning

Storage Layout

Quotas

Integration with Data Processing Frameworks

Apache Spark

Ray (Distributed ML)

Dask

Nextflow (Bioinformatics)

Maintenance Tasks

Regular Updates

Backup Critical Data

Monitoring Disk Space

Performance Optimization Tips

1. Tune Scheduler Parameters

2. Optimize Job Packing

3. Create Multiple Partitions

4. Enable Job Arrays for Batch Processing

Conclusion

Part 1: Foundations

Part 2: Production Deployment

Part 3: Administration (This Post)

References

Recent Articles

From sequencing data to an antimicrobial-resistance mechanism in Klebsiella pneumoniae and Pseudomonas aeruginosa

Building a Bacterial Genome Without Writing Code

How to Run nf-core Pipelines on the Omicslab Platform

Data Curation and Harmonization for Cancer Genomics Cohorts

Short-read Methylation Pipeline Using Nextflow

Short-read Somatic Variant Calling Pipeline Using Nextflow