Lesson 2: Data Management

Objectives¶

Move data on and off the cluster and place it across scratch, long-term, and node-local storage to optimize for runs.

• Transfer tools - Get data onto and off of the cluster with rsync, scp, and sftp.

• Storage types - Home directories, scratch vs. long-term storage, and node-local storage used during job execution.

• Best practices - Match storage to your data’s lifetime and keep code, environments, and data in separate places.

1. Data transfer tools¶

We’ll look at two categories of tools:

• rsync - For larger transfers, repeated syncs, and reliable transfers.

• scp / sftp - For small, quick, one-off transfers.

rsync¶

rsync syncs data over SSH and is the best choice when:

• you sync the same directories repeatedly

• you have large trees of files

• you want reliable, resumable transfers - helpful when a network connection drops mid-transfer

Sync local → RCC

rsync -avz --progress -e ssh local_folder/ <CNetID>@midway3.rcc.uchicago.edu:/scratch/midway3/<CNetID>/project_folder/

Sync RCC → local

rsync -avz --progress -e ssh <CNetID>@midway3.rcc.uchicago.edu:/scratch/midway3/<CNetID>/project_folder/ local_folder/

Common options:

• -a - Archive mode (preserves timestamps/permissions).

• -v - Verbose output.

• -z - Compress in transit.

• --progress - Show transfer progress.

• --delete - Make the destination match the source by deleting extra files (careful!).

References:

• RCC - rsync examples in the SSH docs: https://docs.rcc.uchicago.edu/ssh/main/

• DSI - rsync/scp guide: https://cluster-policy.ds.uchicago.edu/advanced-topics/rsync-scp/

🧰 - Remember the input data that we copied in Lesson 1.7 Step 4, you can also copy the input data from the local folder you downloaded it to the RCC cluster with this command:

rsync -avz --progress -e ssh /path/to/input/directory/on/your/laptop/ <CNetID>@midway3.rcc.uchicago.edu:/scratch/midway3/$USER/workshop-hpc-data

scp + sftp¶

Both tools transfer data to and from the cluster over SSH. Unlike rsync, they don’t sync two directories, so an interrupted copy (e.g. from a dropped connection) can’t be resumed - it has to start over.

scp(Secure Copy)

Use scp to copy files over SSH. Good for small-to-moderate transfers.

Copy a local file to RCC (Midway3 example):

scp path/to/local_file.txt <CNetID>@midway3.rcc.uchicago.edu:/home/<CNetID>/

Copy a directory recursively to RCC:

scp -r path/to/local_folder <CNetID>@midway3.rcc.uchicago.edu:/scratch/midway3/<CNetID>/

Copy a file from RCC to your laptop:

scp <CNetID>@midway3.rcc.uchicago.edu:/scratch/midway3/<CNetID>/results.csv .

sftp (Secure File Transfer Protocol)

sftp is like an interactive file browser over SSH.

Start an sftp session:

sftp <CNetID>@midway3.rcc.uchicago.edu

Useful commands inside sftp:

ls            # list remote files
lls           # list local files
cd DIR        # change remote directory
lcd DIR       # change local directory
get FILE      # download from remote
put FILE      # upload to remote
get -r DIR    # download directory
put -r DIR    # upload directory
bye           # exit

🧰 - prefixing commands with l makes them run on your local file system.

Examples:

Download a directory from remote to local:

sftp> cd /project/myproject  # Change to directory on the cluster
sftp> lcd ~/Downloads        # Change to the local Downloads folder on your laptop
sftp> get -r results         # Recursively download a directory from the cluster
# Downloads the 'results' directory to ~/Downloads/results

Upload a directory from local to remote:

sftp> cd /project/myproject   # Change to directory on the cluster
sftp> lcd ~/Documents/mycode  # Change to the local mycode directory on your laptop
sftp> put -r src              # Recursively upload a directory to the cluster
# Uploads the 'src' directory to /project/myproject/src

2. Storage types¶

Most clusters offer several storage types, each suited to a different purpose. We cover them generally here, then give the specifics for the RCC and DSI clusters.

General guidance:

• code in home

• data in project

• temporary job files in scratch

• heavy per-job I/O on node-local

Expanded into full best practices in §3.

Home Directories (/home/$USER)¶

This is your personal space on the cluster:

• Shell configuration - This space contains configuration files for the shell launched when you login and other cluster tools.

• Scripts and source code - You may want to use this space to store scripts and source code.

• Private to you - This is a space to store files owned by you and not shared with others, as your home directory is only accessible by you.

Research or Project Space¶

Project or research directories are the primary location for shared research data and code.

• Shared research assets - This space can often contain shared research code, input data, results data, analysis code and notebooks, and shared software environments.

• Group workspace - This space is the workspace for a research group.

• Built for sharing - This is a space for sharing, as most often everyone within a research group has access to these files.

Scratch Space¶

Scratch space is a high performance storage space for temporary data.

• Job I/O - This space is often used by running jobs to read data from and write data to.

• High performance - Scratch spaces often offer higher I/O performance.

• Not durable - Scratch spaces are often purged at regular intervals and are not backed up.

Node-local Storage¶

This is storage that is only available on and connected directly to a specific compute node.

• Fastest I/O - Because this is an NVMe or SSD storage device attached to the node, I/O performance is the best and fastest.

• Heavy file I/O - You may want to consider using the node-local storage if you have an algorithm that requires heavy file I/O and see a bottleneck in your code’s performance and runtime.

• Most ephemeral - This storage space is wiped as soon as a job releases the resource. In other words, once your job terminates on the cluster, the storage is purged.

Typical workflow (inside a batch script)

1. Stage inputs in - Copy your inputs from home or project space onto the node-local storage path.

2. Run against local disk - Run your algorithm pointed at node-local storage, often via a command-line argument like --inputdir that you thread through your code.

3. Copy results out - Copy your results back to home or project space before the job ends.

🧰 - Note you may have to figure out the size of the node-local storage so you know how much of your input data can be loaded to that space at a time. See specific cluster details below for info on how to do this.

RCC Cluster Specifics¶

The RCC cluster (Midway) offers all four storage types described above, plus long-term tiers. The specifics below are for Midway3, the cluster we use in this workshop; Midway2 and Beagle3 have slightly different paths and quotas. Quotas change over time, so treat the official RCC storage docs as the source of truth.

You can check your own usage at any time with:

rcchelp quota        # or:  quota -u $USER

Using node-local storage on the RCC cluster

Unlike the DSI cluster (below), RCC node-local scratch requires no special request, every job automatically gets a per-job directory, and its path is handed to you in two environment variables:

• Path - $TMPDIR and $SLURM_TMPDIR are both set to /tmp/jobs/${SLURM_JOB_ID}.

• Lifetime - It’s wiped automatically when the job completes, so copy results out before the job ends.

• Best for - High-throughput I/O of many small files (each < 4 MB).

Because the size varies by node and is not advertised as a Slurm resource (unlike DSI), you can’t read it from sinfo. You have to check it from on the compute node - $TMPDIR only exists inside a running job, so df -h "$TMPDIR" on a login node will just report “No such file or directory.”

One way to see the node-local storage size is via an interactive session (quick, one-off check).

Grab a short interactive job, then run df in the shell it drops you into:

srun --partition=schmidt-gpu --nodes=1 --ntasks=1 --cpus-per-task=2 --mem=4G --time=00:10:00 --pty bash -i
df -h "$TMPDIR"     # you are now on a compute node; $TMPDIR exists

Putting it together in a batch script - stage inputs onto the fast local disk, run, then copy results back to durable storage before the job exits:

#!/usr/bin/env bash
#SBATCH --job-name=local-io-example
#SBATCH --partition=schmidt-gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=32G
#SBATCH --time=02:00:00

# $TMPDIR / $SLURM_TMPDIR already point at this job's node-local scratch - no request needed
echo "Node-local scratch: $TMPDIR"

# 1. Copy inputs from durable storage (/project or /scratch) onto node-local disk
rsync -a /project/<group>/workshop-hpc-data/ "$TMPDIR"/data/

# 2. Run your code pointed at the fast local path
python train.py --inputdir "$TMPDIR"/data --outputdir "$TMPDIR"/results

# 3. Copy results back to durable storage BEFORE the job ends ($TMPDIR is wiped on exit!)
rsync -a "$TMPDIR"/results/ /project/<group>/results/

A worked example from the workshop.

The repo ships a concrete version of this pattern in slurm/06_nodelocal_predict.sbatch: it stages the SoS/SVS inputs and the trained model onto node-local disk, writes the many small per-shard prediction files there, aggregates them locally, and copies only the single discharge.csv back to durable storage.

The parts worth highlighting:

1. Pick the node-local directory (with a fallback). $SLURM_TMPDIR is the per-job disk physically on the compute node; fall back to a job-unique /tmp path if the site doesn’t set it:

LOCAL="${SLURM_TMPDIR:-/tmp/$USER/$SLURM_JOB_ID}"
mkdir -p "$LOCAL/input" "$LOCAL/predictions"

2. Always clean up on exit. A trap removes the local directory on success, error, or signal, so the job never strands data on the compute node:

cleanup() { rm -rf "$LOCAL"; }
trap cleanup EXIT

3. Stage inputs once with rsync -a. Copy the read-only inputs and the model onto local disk a single time, then point the code at the fast local paths. rsync -a preserves timestamps/permissions and skips anything already staged, so a requeue doesn’t re-copy the whole dataset:

rsync -a "$SOS_FILE" "$SVS_FILE" "$LOCAL/input/"
rsync -a "$MODEL" "$LOCAL/model.pt"

4. Do the many-small-files work locally. Each shard writes its per-reach CSV into $LOCAL/predictions/, and aggregation rolls them up locally - none of that churn touches the shared filesystem (exactly the many-small-files problem from §3).

5. Copy only the result back - before the job ends. Node-local disk is wiped on exit, so the single aggregated file is rsync’ed to durable storage while the job is still alive:

rsync -a "$LOCAL/discharge.csv" "$WORK_DIR/runs/discharge.csv"

🧰 - Notice this script needs no --gres=local request. On RCC, node-local scratch is automatic via $SLURM_TMPDIR (see the table above) - there is no local generic resource to ask for, and adding --gres=local:... would leave the job unschedulable.

That --gres=local:<SIZE> line is a DSI-only requirement, covered in the DSI specifics below.

DSI Cluster Specifics¶

The DSI cluster offers all four storage types described above. Quotas and node capacities change over time, so treat the official storage overview and node-local storage guide as the source of truth.

You can check your own usage at any time with:

dsiquota --home
dsiquota --project <NAME>
dsiquota --scratch
# You can just run `dsiquota` to display all of the command options

Using node-local storage on the DSI cluster

1. Request it in your job. Node-local disk is a Slurm generic resource (GRES). Add --gres=local:<SIZE> (size in GB) to your srun or sbatch request. The scheduler will only place your job on a node that advertises at least that much local disk.

# Interactive example: request a shell with 200 GB of node-local scratch
srun -p general --gres=local:200G --pty bash

2. Know where it lives. Slurm creates a per-job directory for you at /local/scratch/<CNetID>_<JOBID>/. It is removed automatically when the job ends, so you must copy results out before then.

The DSI docs don’t document an environment variable for this path, so build it from $USER (your CNetID) and $SLURM_JOB_ID - and confirm it inside the job:

LOCAL=/local/scratch/${USER}_${SLURM_JOB_ID}
ls -ld "$LOCAL"          # confirm Slurm created it
df -h "$LOCAL"           # check how much space is actually available

3. Request only what you need. Match the local: size to the smallest footprint you truly need so your job can land on more nodes (and you don’t block other users).

Figuring out how much a node has

Survey all nodes at once with sinfo, printing the node name (%N) and its GRES (%G) - local disk shows up in the local:disk: field alongside GPUs:

sinfo -N -o "%N %G"
# g002 gpu:...,local:disk:200G
# k001 gpu:h200:4,local:disk:1500G
# n001 gpu:...,local:disk:9500G   -> this node advertises 9500 GB of local disk

To inspect a single node in full detail, use scontrol and read its Gres= line:

scontrol show node <nodename>
# ... Gres=gpu:h200:4,local:disk:1500G   -> this node advertises 1500 GB of local disk

🧰 - See Lesson 3: Errors and Monitoring for more information on the sinfo and scontrol commands

Putting it together in a batch script

This follows the typical workflow above - stage inputs onto the fast local disk, run, then copy results back to durable storage before the job exits:

#!/usr/bin/env bash
#SBATCH --job-name=local-io-example
#SBATCH --partition=general
#SBATCH --gres=local:200G          # request 200 GB of node-local scratch
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=32G
#SBATCH --time=02:00:00

# Per-job directory Slurm created on the node's local NVMe/SSD
LOCAL=/local/scratch/${USER}_${SLURM_JOB_ID}

# 1. Copy inputs from network storage (/project or /net/scratch) onto node-local disk
rsync -a /project/<group>/workshop-hpc-data/ "$LOCAL"/data/

# 2. Run your code pointed at the fast local path
python train.py --inputdir "$LOCAL"/data --outputdir "$LOCAL"/results

# 3. Copy results back to durable storage BEFORE the job ends (local is wiped on exit!)
rsync -a "$LOCAL"/results/ /project/<group>/results/

3. Data management best practices¶

Each storage type above suits a different stage in the life of your data. Now that you have seen the specifics for the RCC and DSI clusters, this section covers how to use them well.

A little organization up front - deciding what is short-term vs. long-term, and where each piece lives - saves a lot of pain later.

A good test: six months from now, could you still answer which data was used, which version of the code ran, what parameters were set, and can I reproduce this result? The practices below are aimed at keeping that answer “yes.”

Match storage to the data’s lifetime¶

Think about how long you need each file and how it is accessed, then place it accordingly:

A few rules of thumb follow from this:

1. Store code in home - Keep source code, scripts, and configuration in your /home directory (or a Git repository).

2. Keep data in project - Store primary datasets, important results, and shared files in your /project directory.

3. Use scratch for jobs - Have running jobs read and write temporary files in a directory you create under /scratch, then copy final results back to /project when the job finishes.

4. Check your usage - Monitor your disk usage regularly so you don’t hit quotas mid-job (see the quota commands in §2).

5. Clean up regularly - Delete files you no longer need, especially from shared /scratch, to be a good cluster citizen.

Organize data for fast search and retrieval¶

When jobs need to find data in a large or growing collection (especially time series), a hierarchical directory layout beats one giant flat folder. Partition by the fields you filter on most - for time series, that is usually year/month/day:

/project/<group>/data/
  2026/
    07/
      15/
        station-A.parquet
        station-B.parquet
      16/
        ...

Why it helps:

• Targeted reads - A job can point straight at data/2026/07/ instead of scanning millions of files.

• Filesystem friendly - Parallel and scratch file systems slow down badly with huge flat directories; nesting keeps each directory small.

• Self-documenting - The path itself documents the data, and new data appends cleanly without reshuffling existing files.

Watch out for the many-small-files problem, too: thousands of tiny files are slow on shared storage.

Aggregate them into larger containers (e.g. .parquet, .tar, or .zarr), and remember that node-local storage is the right home for high-throughput small-file I/O.

🧰 - Keep raw inputs read-only and write derived/processed data to a separate directory. An accidental job then can’t corrupt your source data, and you can always re-derive results from the original.

Separate code, environments, and data¶

Code, environments, and data have different lifetimes and sharing needs, so keep them in separate locations rather than one mixed directory:

• Code - A Git repository (scripts, notebooks, job files). Version-controlled; never commit large data or secrets.

• Software environments - Build a conda/venv environment once in a known location (personal → /scratch, shared → /project) and activate it from every job; don’t rebuild it per run.

• Data - Inputs and results live in /project (durable) and /scratch (temporary), never inside the code repository.

The workshop repository is laid out this way:

• src/ for code

• config/ for configuration

• slurm/ for job scripts

• environment.yml for the environment

With the data kept on the cluster, outside the repo.

From notebooks to codebases¶

Notebooks are ideal for exploration and are a great first line of experiment tracking. But batch jobs run scripts, not notebooks, and copy-pasted cells are hard to reproduce. As soon as you are repeating an analysis or scaling it to batch jobs:

• Move logic into modules - Move reusable logic out of the notebook into small, importable scripts/modules in your codebase (src/).

• Keep notebooks for exploration - Keep the notebook for exploration and figures, importing from the codebase so the two stay in sync.

• Version the codebase - Put the codebase under version control so each job runs a known version of the code.

Track experiments, configs, and results¶

To make results reproducible, capture what produced them: the data, the code version, the parameters, and the outputs. Layer tools as your needs grow:

Two habits that pay off immediately:

• Keep the config with the results - Write each run’s exact parameters/config into that run’s output directory, so a result is self-describing.

  • The workshop does this: each experiment is a committed YAML in config/experiments/, and train_consensus.py writes the fully-resolved config to config_used.yaml in the run’s output directory.

• Give each run its own output directory - Key it by date or job ID (e.g. results/2026-07-15_<jobid>/), so runs never overwrite each other.

For a fuller walkthrough of these tools, see the Model, Data & Code Tracking guide.

✅ Verification Checkpoint¶

• You can move data on and off the cluster with rsync, scp, and sftp, and know when to reach for each.

• You can name the four storage types (home, project, scratch, node-local) and match each to the stage of data it suits.

• You know how to check your quotas and find the node-local path on both the RCC and DSI clusters.

• You can lay out a project so code, environments, and data live in separate, appropriately-durable locations.

Lesson Conclusion¶

You now know where data belongs on the cluster and how to get it there efficiently and safely.

Next we’ll submit jobs and watch them run - reading metrics and logs to diagnose failures and recover from interruptions.