What Staged Datasets are on AI.Panther?

Research IT stages common research datasets on shared storage so you do not download your own copy. They live under /shared/datasets, are read-only, and are visible from every node.

$AIP_DATASETS points at that directory in every shell without loading anything, so $AIP_DATASETS/culane is safe to hardcode in a job script.

What is staged

Using one

Loading a dataset module sets path variables and nothing else. Nothing goes on your PATH:

module load datasets/cifar-10
echo $CIFAR10_ROOT

import os, torchvision
ds = torchvision.datasets.CIFAR10(
    root=os.environ["CIFAR10_ROOT"], train=True, download=False)

To see the variables a dataset sets, along with its notes and citation:

module help datasets/cifar-10

Do not extract onto /shared

The large datasets are staged as archives and shards on purpose. Extracting them onto /shared turns every image read into a round trip to the Lustre metadata server, which slows the filesystem for everyone, including jobs that never touch your data.

Read from the archives directly, or unpack into node-local NVMe on the H200 nodes, which is /localscratch/$USER on gpu09-12 and is deleted after 60 days.

Terms and citation

Each dataset is staged under its upstream licence and those terms travel with it. CULane and CurveLanes are free for academic and non-academic use for non-commercial purposes, with no redistribution: the data must not leave the cluster. CIFAR-10 and CIFAR-100 carry no registration requirement.

Citation is a condition of use for all four. Each dataset directory has a CITATION.bib with the reference to use.

The licence text for each restricted dataset is readable at /shared/datasets/terms/.

ImageNet access

ImageNet is the one dataset you cannot read by default. Its terms grant use to the individual researcher who agreed to them, and extend that to colleagues only once each of them has also agreed. A notice in a README cannot deliver agreement, so /shared/datasets/imagenet-1k is readable only by members of the ad-idm-ai-panther-datasets-imagenet group.

To get access:

1. Read the terms at /shared/datasets/terms/imagenet-1k.txt, or at https://image-net.org/download.php.
2. Copy the agreement below, replacing [RESEARCHER_FULLNAME] with your own name and affiliation everywhere it says "Researcher".
3. Submit it as a ticket stating that you agree to be bound: https://help.fit.edu/TDClient/39/Portal/Requests/Service/8085/Submit-a-Ticket

The ticket is the record that you agreed. You will be added to the group from it, and /shared/datasets/imagenet-1k/README and the TERMS.txt beside it have the rest once you are in.

[RESEARCHER_FULLNAME] (the "Researcher") has requested permission to use the ImageNet database (the "Database") at Princeton University and Stanford University. In exchange for such permission, Researcher hereby agrees to the following terms and conditions:

1. Researcher shall use the Database only for non-commercial research and educational purposes.
2. Princeton University and Stanford University make no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
3. Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify the ImageNet team, Princeton University, and Stanford University, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database.
4. Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
5. Princeton University and Stanford University reserve the right to terminate Researcher's access to the Database at any time.
6. If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.
7. The law of the State of New Jersey shall apply to all disputes under this agreement.

Once you have access, train from the shard tars rather than extracting the 1.28 million training images onto Lustre.

Requesting a dataset

To ask for another dataset to be staged, submit a ticket with the dataset name, its source URL and roughly what you need it for.