Symptom
A program loaded with module dies on a GPU node with no explanation:
Illegal instruction (core dumped)
With Apptainer it happens on the apptainer command itself, before any container starts:
module load apptainer
apptainer --version
Cause
Not the container, and not the default module path. In a login shell on a GPU node the module path is already correct.
Slurm hands your job the environment of the shell you submitted from, including the login node's MODULEPATH, which puts the Intel (cascadelake) builds first. A shell that is not a login shell never re-reads /etc/profile.d on the compute node, so that inherited ordering stays in effect and module load gives you Intel binaries. gpu01-08 are AMD EPYC and have no AVX-512, so those binaries die on the first instruction they cannot execute. gpu09-12 are Intel and run them.
Fix
Make the shell a login shell.
In batch scripts:
#!/bin/bash -l
Interactively:
srun --partition=gpu1 --gres=gpu:1 --time=1:00:00 --pty bash -l
Do not add a module use line. The linux-ubuntu22.04-* paths predate the cluster OS upgrade and the newest Apptainer in them is 1.3.4.
Verifying
On a GPU node, after module load apptainer, module list should show the Zen 2 build:
apptainer/1.4.1-gcc-13.3.0-qihxpzm
apptainer/1.4.1-gcc-13.3.0-nmn2ykt is the Intel build and is the one that crashes on gpu01-08. which apptainer shows the architecture in the path.
Notes
This applies to anything you module load in a job, not just Apptainer. Any Spack-provided software hits it the same way.