NVIDIA publishes GPU-optimized container images through the NGC catalog. Research IT stages the ones with real demand and exposes them as modules, so you do not download or build anything.
This article covers those staged images and the things specific to them. It assumes you already know how to submit a job and request a GPU, which is covered in How to submit and run jobs on AI.Panther, and how Apptainer works generally, which is Using Apptainer on AI.Panther.
What is available
| Module |
Contains |
Sets |
pytorch-ngc/26.08 |
PyTorch 2.14, torchvision, Apex, TransformerEngine 2.18, Torch-TensorRT, torchtitan, CUDA 13 |
$PYTORCH_SIF |
rapids-ngc/26.08 |
cuDF 26.08, cuML, cuGraph, dask-cuDF, cuDF-polars, CuPy, CUDA 13 |
$RAPIDS_SIF |
tritonserver-ngc/26.09 |
Triton Inference Server 2.73, all backends |
$TRITONSERVER_SIF |
pip list inside an image is the real answer to what it contains:
module load pytorch-ngc/26.08
apptainer exec $PYTORCH_SIF pip list
Loading one of these modules pulls in Apptainer for you and sets the variable pointing at the image. You do not need module load apptainer first, and you do not need module use with a path.
The images are read-only and shared. Nothing is copied into your home directory when you run one, however many people use it at once.
Refreshes happen quarterly, between terms. Two versions of each are kept once a cycle has run, so you can pin the older one for the duration of a paper.
Running one
module load pytorch-ngc/26.08
apptainer exec --nv --cleanenv $PYTORCH_SIF python train.py
--nv and --cleanenv are both explained in Using Apptainer on AI.Panther. Two of their quirks bite harder with these images.
--cleanenv matters more here than usual. Your home directory is visible inside the container, so if you have a conda environment active, which you do if your prompt shows (base), its paths follow you in and Python can pick up your conda packages instead of the ones NVIDIA built. The symptom is a version mismatch or an import error that makes no sense for the image you loaded.
--cleanenv strips CUDA_VISIBLE_DEVICES, and you have to put it back. That variable is how Slurm tells your job which GPUs it was allocated. Without it, torch.cuda.device_count() reports every GPU on the node instead of the ones you were given. You cannot actually use the others, but anything that trusts the count will try: torchrun auto-detection, nproc_per_node, and any for i in range(torch.cuda.device_count()) loop. One line near the top of your script covers every apptainer call that follows:
export APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"
To pass one of your own variables into a clean environment, name it explicitly:
apptainer exec --nv --cleanenv --env MY_API_KEY="$MY_API_KEY" $PYTORCH_SIF python train.py
In a batch job
Use #!/bin/bash -l, not #!/bin/bash. The -l makes it a login shell. Without that, the compute node never sets up its own module paths.
Without it, Slurm hands your job the environment you submitted from, including the module search path your login node built. The login node and the GPU nodes are not all the same processor architecture, so a module load in a plain #!/bin/bash job can load a build compiled for the wrong one. On the A100 partitions that means Illegal instruction (core dumped) with nothing to explain it.
This is not specific to containers. It applies to anything you module load in a batch job, so add the -l to your existing scripts too.
Start with module purge. Slurm carries your submitting environment into the job, so whatever was loaded when you ran sbatch is still loaded inside it. If you had pytorch-ngc/26.05 loaded and your script asks for 26.08, the two conflict and the load fails. Purging first means the job does what the script says, not what your shell was doing when you hit submit.
#!/bin/bash -l
#SBATCH --job-name=torch
#SBATCH --partition=h200
#SBATCH --gres=gpu:H200:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=04:00:00
#SBATCH --output=%x-%j.out
module purge
module load pytorch-ngc/26.08
export APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"
apptainer exec --nv --cleanenv $PYTORCH_SIF python train.py
Partition names, walltime limits and the --gres syntax for each GPU type are in How to submit and run jobs on AI.Panther.
Interactively, request an allocation first and run the container inside it:
srun --partition=h200 --gres=gpu:H200:1 --cpus-per-task=8 --mem=64G --time=02:00:00 --pty bash -l
module purge
module load pytorch-ngc/26.08
export APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"
apptainer exec --nv --cleanenv $PYTORCH_SIF python
Multi-GPU
Multi-GPU training works on h200, gpu1 and gpu2.
It does not work on h200_mig or the vdi-* partitions. Those GPUs are partitioned slices with no peer-to-peer link between them, so NCCL cannot run collective operations across them and a job that tries will error or hang. Request a single GPU there. This is the hardware, not the container.
With the APPTAINERENV_CUDA_VISIBLE_DEVICES line above, torchrun sees exactly the GPUs you were allocated:
apptainer exec --nv --cleanenv $PYTORCH_SIF \
torchrun --standalone --nproc_per_node=4 train.py
What the container can see
A container does not see the whole filesystem. These are available with no extra flags:
| Path |
|
/home1, /home2 |
your home directory |
/shared |
everything under it, including /shared/scratch and /shared/projects/<name> |
| your current working directory |
bound automatically wherever it is |
Anything else needs an explicit --bind. /localscratch is the one you are most likely to want, and it is not bound by default because it only exists on the H200 nodes:
apptainer exec --nv --cleanenv --bind /localscratch $PYTORCH_SIF python train.py
See AI.Panther Storage Types for what each filesystem is for and which ones are purged.
Where model downloads go
Machine learning frameworks download a lot, and a single HuggingFace checkpoint can be tens of gigabytes. Because your home directory is visible inside the container, a download inside a container fills the file server exactly as one outside it would.
Your shell already points HF_HOME, TORCH_HOME and TRITON_CACHE_DIR at scratch space instead, and those carry through into the container. Which means:
- Cached models are purged like anything else in scratch. A model you have not touched in a couple of months is re-downloaded on next use. To keep one, put it in your project space and point
HF_HOME at it yourself.
- If you set
HF_HOME yourself, your choice wins and is carried into the container. Just do not point it at your home directory.
TRITON_CACHE_DIR here is for Triton the GPU kernel language, which PyTorch uses internally to compile kernels. It is unrelated to Triton Inference Server below. Two different projects share the name.
Serving a model with Triton
Triton Inference Server runs inside your own job, serves your client in that same job, and exits with it. There is no shared Triton endpoint on the cluster to connect to.
It earns its place when several clients fan into one GPU, several models share a GPU, you want dynamic batching, or you are combining models from different frameworks. For one model and one client, load the model directly in the PyTorch container instead; the network hop to a server in the same job is pure overhead.
Derive your ports from the job ID. Triton defaults to 8000, 8001 and 8002, so two jobs on one node collide and the second fails to bind.
BASE=$(( 10000 + (SLURM_JOB_ID % 20000) ))
Wait for readiness before starting your client. Loading a model takes seconds to minutes, and a client that starts immediately dies on connection refused.
until curl -sf "http://localhost:${BASE}/v2/health/ready" >/dev/null; do sleep 2; done
Lay the model repository out the way Triton expects, one directory per model with a config.pbtxt and at least one numbered version directory:
models/
my_model/
config.pbtxt
1/
model.pt
Keep the model repository somewhere the container can see, so under /shared or your home directory rather than /tmp on the host.
A complete job:
#!/bin/bash -l
#SBATCH --job-name=triton
#SBATCH --partition=h200_mig
#SBATCH --gres=gpu:H200-2g.35gb:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=02:00:00
#SBATCH --output=%x-%j.out
module purge
module load tritonserver-ngc/26.09
export APPTAINERENV_CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES"
BASE=$(( 10000 + (SLURM_JOB_ID % 20000) ))
HTTP=$BASE; GRPC=$((BASE+1)); METRICS=$((BASE+2))
REPO=/shared/projects/myproject/models
apptainer exec --nv --cleanenv $TRITONSERVER_SIF \
tritonserver --model-repository="$REPO" \
--http-port=$HTTP --grpc-port=$GRPC --metrics-port=$METRICS &
TRITON_PID=$!
until curl -sf "http://localhost:${HTTP}/v2/health/ready" >/dev/null; do sleep 2; done
echo "Triton ready on $HTTP"
module load pytorch-ngc/26.08
apptainer exec --cleanenv $PYTORCH_SIF python client.py --url "localhost:${GRPC}"
kill $TRITON_PID
Install the client library into your own environment with pip install tritonclient[all] rather than loading a second image.
Triton logs two things at startup that look like errors. Neither is:
CacheManager Init Failed. Error: -29. The response cache is off by default, shown as cache_enabled 0 further down the same log. Turn it on with --response-cache-byte-size if you want it.DCGM unable to start: DCGM initialization error. Triton's own GPU metrics need access the container does not have, so its /metrics endpoint serves CPU metrics only. GPU utilization for your job is collected cluster-wide regardless.
Troubleshooting
Illegal instruction (core dumped) from anything you loaded with module. Your script is #!/bin/bash instead of #!/bin/bash -l, so the job inherited the submit host's module paths and loaded a build for the wrong processor. Add the -l.
Module cannot be loaded, conflict in a batch job. A different version of the same module was loaded when you submitted and Slurm carried it in. Add module purge first.
torch.cuda.device_count() reports more GPUs than you asked for, and using them raises AcceleratorError. --cleanenv stripped CUDA_VISIBLE_DEVICES. Add the APPTAINERENV_CUDA_VISIBLE_DEVICES export.
Wrong package versions, or an import error that contradicts the image. Your conda or virtualenv environment is leaking in. Add --cleanenv.
No GPU inside the container. Missing --nv.
NCCL errors or hangs on multi-GPU work on h200_mig or a vdi-* partition. Those slices have no peer-to-peer link. Request one GPU, or move to h200, gpu1 or gpu2.
a shared library is likely missing in the image when using --nv. The image is Alpine or another musl-based distribution, and the host driver libraries --nv binds in are built against glibc. Every NGC image is glibc-based, so this only comes up with images you built yourself.
No such file or directory for a path you know exists. Check the table under "What the container can see" and add --bind.
Image file already exists when pulling your own image. Apptainer will not overwrite a SIF. Remove it or pull to a different filename.
Requesting another image
The catalog has far more than we stage, and each image is several gigabytes per retained version, so we add one when someone actually needs it. Open a ticket with the image name and a sentence on what you are running.
You can also pull NGC images into your own space with apptainer pull. The staged modules are the supported path and the ones we test, but nothing stops you running your own.