NOTE 1: For information on using installed packages on AI.Panther, please see this KB article for more information on using environment modules.
Introduction
To submit jobs on AI.Panther, the scheduler, Slurm Workload Manager, must be used to run the job. This differs from how you'd normally run a command, as you need to prepare a submission script and optionally make your code MPI capable.
All software must be run using the Slurm scheduler. The cluster is a shared resources; therefore, if a user runs code on the login server, it will cause login and performance issues for others. Users who run on the login node will have their code terminated, and continued disregard will result in the user being blocked from using these clusters.
Connecting to the AI.Panther
To access the AI.Panther HPC cluster, a SSH client must be installed on your device and used to connect to the HPC cluster.
The AI.Panther hostname for SSH is ai-panther.fit.edu
The cluster support using your TRACKS username and password. Please be aware that your TRACKS username is not your email address. It is only the part before the '@' symbol.
Resource Management
The cluster uses the Slurm Workload Manager to manage available resources and to distribute jobs to free compute nodes. Slurm also provides a queuing system; if not enough resources are available, it will hold your job until it can run it.
Slurm Example Submission Script
In order to submit a job to Slurm, a job submission script must be created. An example submission script is provided below:
#!/bin/bash
#SBATCH --job-name TestJob
#SBATCH --nodes 1
#SBATCH --ntasks 1
#SBATCH --mem=50MB
#SBATCH --time=00:15:00
#SBATCH --partition=short
#SBATCH --error=testjob.%J.err
#SBATCH --output=testjob.%J.out
module load mpich
echo "Starting at $(date)"
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is $(pwd)"
sleep 60
The only options you absolutely need are:
--job-name — a unique name for your job. This can be set to anything.--nodes — the number of nodes to request.--ntasks — the number of tasks in total across all nodes.--mem — amount of memory to request on each node. This is a hard limit and you will run into out-of-memory errors if you fail to provide the correct amount.--partition — the partition for your job. Valid partitions can be found by using sinfo.
If you need MPICH to run your jobs, set it to load at login using:
module initadd mpich
or by placing
module load mpich
in your submission script like above. Many more options are available for Slurm's submission scripts, and more information available by visiting: https://slurm.schedmd.com/sbatch.html.
Submitting a Job
Slurm has its own set of commands for job management. To submit your submission script, use
sbatch script.sh
Some other commands you may want to use are listed below.
squeue — lists the jobs that are currently running for everyone.sinfo — show node status.scancel — cancel a currently running job. sstat — show statistics for a job.
For a more in-depth look into Slurm and its respective commands, check out the Slurm quick start guide.
Optimizing your Submission Script
Slurm will attempt to run your job wherever it can place it, however, this is hugely dependent on how your submission script specifies its resources. Thus, if you can reduce your submission script requirements, your job has a much higher chance of being scheduled faster.
Memory Requirements
--mem is most often used to specify the amount of memory your job will take per node. However, this is largely dependent on how many tasks you can fit in a node or the number of nodes you'll require. If you don't specify the number of nodes you need, slurm won't balance out the tasks, often leading to out-of-memory errors on nodes where more jobs were placed than expected. Another issue arises when the cluster is under heavy use. Small pockets of resources are scattered through the cluster, and won't be easy to acquire when your job needs a fixed amount of memory per node.
To prevent this, we can use --mem-per-cpu instead. If each task only requires a certain amount of memory, you can specify this amount instead. This way, the scheduler can better allocate resources -- if tasks require more memory than what's available on a node, they'll be split, and if there are pockets of resources a single task can fit in, it will allocate that spot.
Partitions
Slurm's partitions can be considered job queues, each of which has an assortment of constraints such as job size limit, job time limit, users permitted to use it, etc. Jobs that only need a short amount of time to run, but a large number of processors, will have their jobs categorized differently than jobs that may need to run for days and require fewer processors. In addition, partitions can be used to group together nodes that have general hardware that others don't (ex, GPU partition has GPUs in its nodes).
To set a partition, use:
#sbatch --partition=[partition]
in your submission script, or specify it on the command line using --partition.
AI.Panther Partitions
| Partition Name |
Max Compute Time |
Max Nodes |
Allowed Groups |
short (default) |
45 minutes |
16 |
AI.Panther users |
med |
4 hours |
16 |
AI.Panther users |
long |
7 days |
16 |
AI.Panther users |
gpu1 |
7 days |
4 |
AI.Panther GPU users |
gpu2 |
7 days |
4 |
AI.Panther GPU users |
h200 |
7 days |
2 |
AI.Panther GPU users |
h200_mig |
7 days |
2 |
AI.Panther GPU users |
vdi-short |
1 hour |
3 |
AI.Panther users (Open OnDemand only) |
vdi-med |
8 hours |
3 |
AI.Panther users (Open OnDemand only) |
vdi-long |
7 days |
3 |
AI.Panther users (Open OnDemand only) |
The eternity partition has been removed (as of the July 20th 2026 maintenance window). Use long for extended CPU jobs, or the appropriate GPU partition for GPU work; both allow up to 7 days.
NOTE: gpu1 is 4 nodes (gpu01-gpu04), each with 4 NVIDIA A100 40 GB GPUs connected via SXM4 (NVLink). This partition is intended for jobs that need high inter-GPU bandwidth on a node, compared to the PCIe interconnect used by gpu2. For a comparison of the two topologies, see: https://infohub.delltechnologies.com/p/accelerating-hpc-workloads-with-nvidia-a100-nvlink-on-dell-poweredge-xe8545/
NOTE: gpu2 is 4 nodes (gpu05-gpu08), each with 4 NVIDIA A100 40 GB GPUs connected via PCIe.
NOTE: h200 is 2 nodes (gpu09-gpu10), each with 8 NVIDIA H200 GPUs connected via NVLink (SXM). Use this partition when your job needs whole H200 GPUs with high inter-GPU bandwidth.
NOTE: h200_mig is 2 nodes (gpu11-gpu12) where each H200 is partitioned into MIG slices of profile 2g.35gb (24 slices per node, 48 total across the partition). Use this partition when your job only needs a fraction of an H200; slices are shared across more jobs and typically start faster than whole-GPU requests on h200.
NOTE: vdi-short, vdi-med, and vdi-long run on 3 nodes (vgpu01-vgpu03). Each vgpu node exposes 8 NVIDIA L40S-12Q vGPU slices (12 GB VRAM each), backed by 4 physical L40S 48 GB cards on the underlying VDI hypervisor (2 slices per physical card, fully utilized). These partitions back the Open OnDemand interactive apps (VSCode, Jupyter, Desktop) and should be launched from the OOD portal, not from sbatch or srun on the login node. See "Interactive Jobs (Open OnDemand)" below.
Interactive Jobs (Open OnDemand)
Interactive sessions such as VSCode, Jupyter, and Desktop run on the vdi-* partitions and are launched from the Open OnDemand portal, not from sbatch or srun on the login node. When you request a session in OOD, the portal builds the Slurm submission for you and lands the job on one of the vgpu nodes.
Pick the vdi-* partition that matches how long you expect to keep the session open:
vdi-short: up to 1 hour, best for quick edits or troubleshooting.vdi-med: up to 8 hours, best for a full workday session.vdi-long: up to 7 days, best for long-running notebooks or persistent desktops.
Running GPU Jobs
Running GPU jobs is similar to running CPU jobs, with two additions: you must select a GPU partition and you must request GPUs with --gres.
Set the partition:
#SBATCH --partition=gpu1
Request N GPUs per node:
#SBATCH --gres=gpu:2
--gres counts GPUs per node, not for the whole job. If you request 4 nodes with --gres=gpu:2 you will get 8 GPUs total. The maximum per node depends on the partition: gpu1 and gpu2 have 4 GPUs, h200 has 8, h200_mig has 24 MIG slices, and each vdi-* node has 8 L40S-12Q vGPU slices.
Requesting a specific GPU model
To pin your job to a specific GPU model, use --gres=gpu:[type]:[N] with one of the types below (these are the canonical Slurm GRES names on this cluster; they are case-insensitive but shown here in the form they appear in gres.conf):
| GRES flag |
Where it runs |
--gres=gpu:A100-SXM4-40GB:N |
gpu1 (gpu01-gpu04), N up to 4 per node |
--gres=gpu:A100-PCI-40GB:N |
gpu2 (gpu05-gpu08), N up to 4 per node |
--gres=gpu:H200:N |
h200 (gpu09-gpu10), N up to 8 per node |
--gres=gpu:H200-2g.35gb:N |
h200_mig (gpu11-gpu12), N up to 24 per node |
--gres=gpu:L40S-12Q:N |
vdi-* (vgpu01-vgpu03, Open OnDemand only), N up to 8 per node |
Requesting a MIG slice on h200_mig
The h200_mig partition presents each H200 as a set of MIG slices using the 2g.35gb profile (about a quarter of an H200 with 35 GB of GPU memory). Each of the two nodes exposes 24 slices, so 48 total across the partition. Request slices with:
#SBATCH --partition=h200_mig
#SBATCH --gres=gpu:H200-2g.35gb:1
Because the slices are shared across more concurrent jobs than whole H200s, h200_mig will usually start faster than h200 when your workload does not need a full GPU.