Using Apptainer on AI.Panther

Apptainer (formerly Singularity) lets you run software inside containers on the cluster. Apptainer 1.4.1 is available on the login node and on every CPU, GPU and VDI node. Run containers inside a Slurm allocation: the login node has no GPU hardware at all, so --nv cannot work there, and it is shared by everyone.

If you want PyTorch, RAPIDS or Triton, the cluster already stages those images and you do not need to pull anything. See Using NGC Containers on AI Panther. This article covers Apptainer itself.

Loading the module

From login, or inside a job:

module load apptainer

That is the whole command on every node. MODULEPATH is configured per node so that the bare module load picks the build matching that node's CPU:

where	resolves to
login, CPU nodes (node01-17), VDI nodes	apptainer/1.4.1-gcc-13.3.0-nmn2ykt
GPU nodes (gpu01-12)	apptainer/1.4.1-gcc-13.3.0-qihxpzm

If an old script of yours carries module use /usr/local/spack/share/spack/modules/linux-ubuntu22.04-zen2, delete that line. That tree predates the cluster OS upgrade and the newest Apptainer in it is 1.3.4.

Any shell that loads a module must be a login shell

This is the most common container failure on the cluster, and it is not a container problem.

Slurm hands your job the environment of the shell you submitted from, including the login node's MODULEPATH. A shell that is not a login shell never re-reads /etc/profile.d on the compute node, so the login node's Intel-first module ordering stays in effect and module load gives you Intel binaries. gpu01-08 are AMD EPYC and have no AVX-512, so those binaries die immediately:

Illegal instruction (core dumped)

On a GPU node, the difference looks like this:

how the shell starts	MODULEPATH begins with	module load apptainer gives	apptainer --version
bash (plain #!/bin/bash, or srun --pty bash)	linux-ubuntu24.04-cascadelake	nmn2ykt, Intel build	Illegal instruction (core dumped)
bash -l	linux-ubuntu24.04-zen2	qihxpzm, AMD build	apptainer version 1.4.1

So:

• batch scripts start with #!/bin/bash -l
• interactive sessions use srun ... --pty bash -l

This applies to every module in the cluster software tree, not just Apptainer.

Running interactively

From the login node:

srun --partition=gpu1 --gres=gpu:1 --cpus-per-task=8 --time=1:00:00 --pty bash -l

Inside the allocation:

module load apptainer
apptainer exec --nv my_container.sif python my_script.py

Running via a Slurm batch script

#!/bin/bash -l
#SBATCH --partition=gpu1
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --time=4:00:00
#SBATCH --output=job_%j.out

module load apptainer

export APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"

apptainer exec --nv --cleanenv my_container.sif python my_script.py

A one-GPU job can take up to 24 CPUs on gpu1, 12 on gpu2 and 16 on h200 before it is asking for more than its share of the node.

Common options

option	what it does
--nv	binds the host NVIDIA driver libraries and binaries into the container. Without it, the container has no driver and no nvidia-smi: an NGC image reports "nvidia-smi": executable file not found in $PATH, and CUDA code fails to find a device.
--cleanenv	starts with a clean environment, so an active conda environment cannot shadow the packages inside the container. Read the next section before using it.
--bind /path	bind-mounts an additional host path into the container.

--cleanenv and CUDA_VISIBLE_DEVICES

--cleanenv strips the environment wholesale, and that includes the CUDA_VISIBLE_DEVICES that Slurm sets for your allocation. The container then sees every GPU on the node, so torch.cuda.device_count() reports 4 or 8 instead of the 1 or 2 you asked for. Nothing leaks, because the cgroup still blocks the devices you were not allocated and touching one raises an error on first use. What breaks is anything that trusts the count: torchrun auto-detection, --nproc_per_node, and any for i in range(device_count()) loop.

In a one-GPU job:

invocation	CUDA_VISIBLE_DEVICES inside
apptainer exec --nv	0
apptainer exec --nv --cleanenv	unset
apptainer exec --nv --cleanenv, with APPTAINERENV_CUDA_VISIBLE_DEVICES exported first	0

Add this line once per script, before the apptainer call:

export APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"

APPTAINERENV_ is the prefix Apptainer carries through --cleanenv, so this survives it.

What is bound into the container

Available inside the container with no flag:

• your home directory under /home1
• /shared, including /shared/containers and /shared/scratch
• /home2
• the current working directory

Everything else on the host is invisible inside the container until you bind it, including real mounts. Two that come up:

apptainer exec --nv --bind /localscratch my_container.sif python my_script.py
apptainer exec --nv --bind /aspendata my_container.sif python my_script.py

/localscratch is local NVMe and exists on gpu09-12 only. /aspendata is an NFS mount present on the compute nodes. Neither is bound by default. A path you have not bound fails with No such file or directory even though the same path works from a shell on the same node.

Caches

Your home directory is bind-mounted into the container, so anything the containerized application downloads into $HOME lands on /home1 exactly as a native run would. A single model checkpoint can be tens of gigabytes. The cluster therefore presets these for you:

variable	value
APPTAINER_CACHEDIR, APPTAINER_TMPDIR	/localscratch/$USER/apptainer/... on gpu09-12, /shared/scratch/$USER/apptainer/... everywhere else
HF_HOME, TORCH_HOME, TRITON_CACHE_DIR	/shared/scratch/$USER/cache/... in a job submitted from the login node

If you set any of these yourself, your value is kept. The Apptainer cache variables only matter when pulling or building an image; running an existing .sif never touches them.

Files under /shared/scratch and /localscratch are deleted after 60 days without modification, so a model cache there is re-downloaded after a long gap.

Pulling your own image

Supported, and the cache variables above already keep the layers off your home directory.

Inside an allocation:

module load apptainer
apptainer pull my_container.sif docker://ubuntu:24.04
apptainer cache clean

Images built on musl libc (Alpine, and anything derived from it) start and run normally, but the driver stack --nv binds in is glibc-linked and will not execute inside them:

FATAL:   exec /usr/bin/nvidia-smi failed: a shared library is likely missing in the image

Use a glibc-based image for GPU work. All NVIDIA NGC images are glibc-based.

GPU partitions

The NVIDIA driver on gpu01-12 is 580.173.02, a CUDA 13 driver, so images built against CUDA 12.x and 13.x both run. What changes between partitions is how much GPU memory you get:

partition	nodes	device	memory per device
gpu1	gpu01-04	A100-SXM4, 4 per node	40 GB
gpu2	gpu05-08	A100-PCIe, 4 per node	40 GB
h200	gpu09-10	H200, 8 per node	141 GB
h200_mig	gpu11-12	H200 2g.35gb MIG slices, 24 per node	35 GB
vdi-short, vdi-med, vdi-long	vgpu01-03	L40S-12Q vGPU, 8 slices per VM	12 GB

On h200_mig, nvidia-smi topo -m reports only PCIe and NUMA paths between devices, with no NVLink, so multi-GPU NCCL work belongs on gpu1, gpu2 or h200. Single-slice work runs fine. Note also that nvidia-smi -L there lists all eight physical H200s in the node while only your allocated slices are usable.

The VDI partitions cannot be allocated from the login node (srun returns Access/permission denied); they are reached through Open OnDemand.

The CPU nodes run containers fine, they simply have no GPU, so --nv is meaningless there and a CUDA-only image will not start.

If something fails

symptom	cause
Illegal instruction (core dumped)	shell is not a login shell: use #!/bin/bash -l or --pty bash -l
"nvidia-smi": executable file not found in $PATH, or CUDA finds no device	missing --nv
torch.cuda.device_count() too high	--cleanenv without APPTAINERENV_CUDA_VISIBLE_DEVICES
No such file or directory on a path that exists on the node	path is not bound, add --bind
a shared library is likely missing in the image	musl-based image, use a glibc-based one
home directory filling up	application cache inside the container, check HF_HOME and TORCH_HOME