Appendix

Configuring SSH authentication with GitHub (on the cluster)¶

To git clone, pull, and push the workshop repository (and your own repos) from the cluster, GitHub needs to recognize the machine you’re on. The cleanest way is an SSH key generated on the cluster and registered with your GitHub account. Once set up, Git operations to GitHub work without passwords or tokens.

🧰 - Don’t confuse the two SSH keys in play:

1. Laptop → cluster - lets you log in to the cluster (covered in §3; DSI requires this).

2. Cluster → GitHub - what this appendix sets up, so the cluster can talk to GitHub.

They are separate keys with separate purposes. The steps below are run while logged into the cluster.

1. Generate a key on the cluster¶

Log into the cluster, then create an ed25519 key (use the email tied to your GitHub account):

ssh-keygen -t ed25519 -C "your_github_email@example.com"

Press Enter to accept the default location (~/.ssh/id_ed25519). You may set a passphrase for extra security, or leave it empty for convenience.

2. Set correct permissions¶

SSH refuses to use keys with loose permissions:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519
chmod 644 ~/.ssh/id_ed25519.pub

3. Start the SSH agent and add the key¶

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

🧰 - The agent lives only for your current session. If a later session prompts for the key, just re-run these two lines.

4. Copy your public key¶

Print the public key and copy the entire line (it begins with ssh-ed25519):

cat ~/.ssh/id_ed25519.pub

🧰 - Only ever share the .pub (public) key. Never copy or send id_ed25519 (the private key).

5. Add the key to GitHub¶

1. Go to GitHub → Settings (top-right profile menu).

2. Select SSH and GPG keys.

3. Click New SSH key.

4. Give it a recognizable Title (e.g. UChicago cluster) so you can revoke it later.

5. Paste the public key into Key.

6. Click Add SSH key.

6. Test the connection¶

ssh -T git@github.com

The first time, accept the host fingerprint by typing yes. Success looks like:

Hi <your-username>! You've successfully authenticated, but GitHub does not provide shell access.

(That message is expected - GitHub doesn’t give a shell.)

7. Clone (or switch) using the SSH URL¶

Clone with the SSH form of the URL:

git clone git@github.com:chicago-aiscience/workshop-hpc-2026-sep.git

If you already cloned over HTTPS and want to switch that repo to SSH:

cd workshop-hpc-2026-sep
git remote set-url origin git@github.com:chicago-aiscience/workshop-hpc-2026-sep.git
git remote -v      # confirm it now shows git@github.com:...

Troubleshooting¶

• Permission denied (publickey) - The key isn’t loaded or isn’t on GitHub. Re-run ssh-add ~/.ssh/id_ed25519 (Step 3) and confirm the public key is listed under GitHub → Settings → SSH and GPG keys.

• Host key verification failed - You declined the fingerprint prompt in Step 6; run ssh -T git@github.com again and type yes.

• Still prompted for a username/password on git push - That repo’s remote is still HTTPS; switch it with the git remote set-url command in Step 7.

• Windows users - Generate the key inside WSL, not PowerShell; they keep separate SSH configurations. (This applies when setting up the laptop→cluster key, not on the cluster itself.)

References and further reading¶

The resources below informed the workshop and are good next reads if you want to go deeper. Each entry says what the resource covers and where in the series it applies. (The official RCC, DSI, and SLURM documentation is also linked inline throughout the lessons.)

SLURM and job management¶

• Slurm Fundamentals: Job Scheduling on HPC Clusters (Abhik Sarkar). An accessible primer on how SLURM schedules work on a cluster. Good background for the scheduler, partitions, and jobs material in Lesson 1.

• Job Management (NMSU HPC). A concise reference for submitting, monitoring, and controlling SLURM jobs. Complements the monitoring commands in Lesson 3.

• Basics of Running Jobs (NERSC). A large HPC center’s guide to batch and interactive jobs, giving broader context for Lessons 1 and 3.

Errors, exit codes, and troubleshooting¶

• Slurm Job Exit Codes (SchedMD). The official reference for the N:M exit-code format. Underpins the exit-code table in Lesson 3.

• Slurm Troubleshooting Guide (SchedMD). The official checklist for diagnosing failing and pending jobs (Lesson 3).

• Slurm Errors FAQ (University of Maryland). A plain-language catalog of common SLURM errors and their fixes. Cited directly in Lesson 3.

• Troubleshooting Failed HPCC Jobs (ICER, Michigan State). Worked examples of debugging failed jobs such as out-of-memory and timeouts. Reinforces the debugging workflow in Lesson 3.

Storage, scheduling policy, and checkpointing¶

• UChicago DSI Cluster Scheduling Policy (DSI). The DSI cluster’s own scheduling and preemption rules. The local reference behind the preemption and requeue discussion in Lesson 3.

• Checkpoint/Restart (ZIH HPC Compendium, TU Dresden). How to checkpoint and restart jobs, including DMTCP. Background for the checkpoint and requeue pattern in Lesson 3.

• Containerized Checkpoint/Restart (NERSC). Checkpointing for containerized workloads, an alternative to the in-code checkpointing shown in Lesson 3.

Parallelism and scientific workflows¶

• Parallel Computing (Sheffield HPC). An overview of parallel job types and of Amdahl’s and Gustafson’s laws. Cited in the “will parallelizing help?” discussion in Lesson 4.

• Evaluating Parameter Sweep Workflows in High Performance Computing. A study of parameter-sweep workflows, the classic job-array use case in Lesson 4.

• Scalable Workflows and Reproducible Data Analysis for Genomics (PMC). A real-world, multi-stage, reproducible workflow example that motivates the pipeline-and-dependency approach in Lesson 4.

• Many-Task Computing and Blue Waters (Katz et al.). A foundational discussion of many-task, embarrassingly parallel computing on HPC. Context for the job arrays in Lesson 4.

• Community Roadmap and Challenges in Scientific Workflows (arXiv:2410.14943). A community survey of open challenges in scientific workflows, mapping the wider landscape around Lesson 4.

• Workflows Community Initiative. A hub for workflow tools, standards, and community resources. Relevant to the workflow-manager section of Lesson 4.

• Snakemake. A widely used workflow manager for declaring stages and dependencies. The tool recommended in Lesson 4 as pipelines outgrow hand-wired dependency chains.

• Performance Analysis for Complex Scientific Workflows (Aliza Lisan, University of Oregon). A survey of performance-analysis and visualization tools for multi-task, AI-integrated workflows. Extends the monitoring ideas in Lesson 3 and the workflow ideas in Lesson 4.

Reproducibility, determinism, and provenance¶

• Reproducibility (PyTorch). The official notes on seeding, use_deterministic_algorithms, cuDNN flags, and DataLoader worker seeding. The basis for the determinism section of Lesson 5.

• Managing environments (conda). Exporting, locking, and rebuilding environments - the --no-builds and --explicit distinction used in Lesson 5.

• conda-lock. Generates true multi-platform lock files from an environment.yml. The next step past a hand-pinned spec in Lesson 5.

• Apptainer User Guide. Containers on HPC, including --nv for GPU access. Background for the “strongest pin” option in Lesson 5.

• Ten Simple Rules for Reproducible Computational Research (Sandve et al., PLOS Comput Biol). The short, widely cited statement of why recording provenance matters. Frames Lesson 5.

• A higher order estimate of the optimum checkpoint interval for restart dumps (Daly, Future Generation Computer Systems 22(3), 2006). The origin of the Young/Daly formula for choosing a checkpoint interval when interruptions are random. Further reading for the checkpointing in Lessons 3 and 5; see also Checkpointing à la Young/Daly: An Overview for a readable summary.

• Model, Data & Code Tracking guide (UChicago AI + Science). The companion guide on Git, DVC, pointer files, MLflow, and W&B. Referenced throughout Lessons 2 and 5.

AI + HPC and workload-aware computing¶

• AI-coupled HPC Workflow Applications, Middleware and Performance (arXiv:2406.14315). A survey of AI-in-HPC, AI-out-HPC, and AI-about-HPC integration patterns. The source for the “where AI fits in” section of Lesson 4.

• A Scientific Approach to Workload-Aware Computing on AWS (AWS HPC Blog). A framework for classifying workloads as tightly or loosely coupled, mixed, or variable. Cited in the workload-characteristics section of Lesson 4.

• Optimizing Compute-Intensive Tasks on AWS (AWS HPC Blog). Practical guidance on right-sizing compute-heavy jobs. Complements the resource and parallelism choices in Lesson 4.

• Optimal Price-Performance for HPC Workloads (AWS, 2019). Cost and performance trade-offs when sizing HPC workloads. Supplementary reading for Lesson 4.

Resources linked to in the workshop¶

Every external link that appears in the lesson content, grouped by where it is used. A handful of these also appear under References and further reading above - that section is for background reading, this one is a lookup for “what was that link in Lesson 3?”.

Used throughout the series¶

• Workshop repository - The repo every lesson works from; holds the environment file, configs, src/ code, and all SLURM batch scripts.

• Workshop input data (Google Drive). The two-basin subset of the SWOT Confluence products (~tens of MB) that attendees download and copy to the cluster.

• Model, Data & Code Tracking guide (UChicago AI + Science). The companion guide on Git, DVC, MLflow, and W&B, pointed to from Lessons 2 and 5.

• SLURM (SchedMD). The scheduler’s official documentation home, the source for every command reference in the series.

Lesson 0: HPC Introduction¶

• SWOT Confluence (GitHub organization). The upstream river-discharge project the workshop example is derived from.

• PO.DAAC (NASA JPL). The data center that distributes the SWOT Confluence products used as input.

• Windows WSL setup (this site). Prerequisite instructions for Windows attendees, carried over from the GitHub Actions workshop.

• Repository tour - The four directories walked through in the setup section: environment.yml, config/, src/, and slurm/.

Lesson 1: Getting Started¶

Accounts

• RCC general user accounts (RCC). How to request an RCC account and associate it with a PI.

• RCC user guide (this site). Our own summary of getting started on RCC.

• DSI cluster accounts (DSI). Eligibility rules and the request process for the DSI cluster.

• DSI user guide (this site). Our own summary of getting started on the DSI cluster.

Logging in

• RCC connection overview (RCC). The index of every supported way to reach the RCC cluster.

• RCC SSH (RCC). Terminal access, including the Duo two-factor step.

• RCC Open OnDemand (RCC). Browser-based access, useful for interactive sessions.

• RCC VS Code (RCC). Running a VS Code editor session against the cluster.

• DSI SSH and GitHub tutorial (DSI Clinic). The DSI equivalent of the SSH setup, and the basis for the SSH appendix above.

• DSI login nodes (DSI). What login nodes are for, and what you should not run on them.

• VS Code Remote - SSH (Microsoft). The extension that gives you an editor and terminal on a remote host.

Scheduler and partitions

• SLURM overview (SchedMD). What SLURM is and the pieces it is built from.

• SLURM architecture (SchedMD). The controller/daemon diagram behind the scheduler explanation.

• RCC partitions (RCC). The list of partitions, their hardware, and their limits - where to look before choosing --partition.

• Repository files - config/setup_env.sh, environment.yml, and slurm/02_train.sbatch - the environment setup and the first batch script submitted.

Lesson 2: Data Management¶

• RCC storage (RCC). Home, project, and scratch spaces with their quotas and purge policies.

• DSI storage overview (DSI). The equivalent storage map for the DSI cluster.

• DSI node-local storage (DSI). Why per-node scratch is fast and how to stage data into it.

• RCC rsync examples (RCC). Transfer commands for moving data to and from RCC.

• DSI rsync/scp guide (DSI). The same for the DSI cluster.

• Repository file - slurm/06_nodelocal_predict.sbatch - the worked example of staging inputs to node-local storage.

Lesson 3: Errors and Monitoring¶

SLURM command references (all SchedMD)

• sinfo - Node and partition state; see also the format specifiers used to build the custom views.

• sacctmgr - Accounting database, used here to list the associations that control partition access.

• squeue - The pending and running queue; also its format specifiers, job state codes, and job reason codes.

• scontrol - Full detail on a single job, and the place to read partition settings such as GraceTime.

• scancel - Cancelling jobs and sending arbitrary signals to them.

• sprio - The priority breakdown for pending jobs; the weighting is explained in the Multifactor Priority docs.

• srun - Launching job steps, including attaching to a running allocation.

• sacct - Post-hoc accounting metrics; the options reference covers the --format field widths used in the examples.

Monitoring and profiling

• nvidia-smi (NVIDIA). The GPU utilization and memory tool used to watch a job while it runs.

• PyTorch profiler API (PyTorch). The full reference for torch.profiler arguments.

• PyTorch profiler recipe (PyTorch). The recommended starting point; walks through the exact with profile(...) pattern in the lesson.

• TensorBoard profiler plugin (PyTorch). Visual trace viewing with a GPU utilization and kernel breakdown.

• Perfetto UI - Browser trace viewer for the trace.json that export_chrome_trace writes.

Errors and interruptions

• FAQ: Slurm Errors (University of Maryland). Plain-language explanations of the error messages in the common-issues table.

• DSI batch jobs (DSI). The local rules for preemption and requeue behind the checkpointing section.

• Repository files - slurm/03_checkpoint.sbatch and src/train_consensus.py - the full signal-handling wrapper and the training code it protects.

Lesson 4: Building Workflows¶

• Parallel Computing (Sheffield HPC). The overview of parallel job types behind the “will parallelizing help?” discussion.

• Amdahl’s law (Wikipedia). Why the serial fraction caps your speedup at a fixed problem size.

• Gustafson’s law (Wikipedia). The counterpoint: speedup scales when the problem grows with the core count.

• A Scientific Approach to Workload-Aware Computing on AWS (AWS HPC Blog). The tightly/loosely coupled classification used to characterize workloads.

• AI-coupled HPC Workflow Applications, Middleware and Performance (arXiv:2406.14315). The survey cited for where AI fits into HPC workflows.

• Repository files - slurm/02_train.sbatch, slurm/04_predict_array.sbatch, and slurm/05_aggregate_dep.sbatch - the train → array → aggregate chain built in this lesson, plus slurm/05_run_workflow.sh, the login-node launcher that submits all three with their dependencies wired up.

Lesson 5: Reproducibility and Checkpointing¶

• pip-tools - pip-compile --generate-hashes, the pip-side equivalent of a locked environment.

• PyTorch Lightning checkpointing (Lightning AI). Framework-managed checkpointing, an alternative to the hand-rolled version in the lesson.

• submitit (Meta). A Python interface for submitting and requeuing SLURM jobs, with signal handling built in.

• DMTCP (TU Dresden). Transparent, process-level checkpointing that needs no changes to your code.

• CRIU / containerized checkpointing (NERSC). Checkpoint and restore for containerized workloads.

• Repository files - environment.yml, src/config.py, config/experiments/baseline.yaml, config/paths.sh, src/build_manifest.py, and slurm/02_train.sbatch - the config, path, and manifest machinery that makes a run reproducible.