HPC Container Build Guide: Methodology & Best Practices

This document outlines the architecture and practical workflow for packaging custom Singularity (SIF) containers on High-Performance Computing infrastructure. It addresses the specific constraints of shared filesystems (Lustre/GPFS) and unprivileged user execution.

Rationale: Why Custom Containers?

Standard base images (e.g., from NVIDIA or Docker Hub) are often insufficient for scientific workflows on HPC systems. We build custom containers to address four critical gaps:

Missing System Dependencies: Minimalist base images lack tools like git, wget, and curl. Without these, Python package managers (pip) cannot install dependencies directly from version control systems.

Environment Isolation: To create robust virtual environments inside the container, we must ensure specific compilers and headers are present during the build phase.

HPC Integration: We must manually configure bind points (e.g., /scratch, /apps, /home) to ensure the container can interact with the host filesystem.

Reproducibility: Standardizing locales (UTF-8) and tzdata prevents encoding errors and timezone discrepancies during logging and data processing.

Filesystem Architecture & Constraints

Understanding the distinction between node-local and shared storage is critical for successful builds.

Filesystem Characteristics Use Case in Build Process /tmp (Node-Local)

• Fast: Local SSD or RAM-disk.

• Posix Compliant: Supports full metadata operations.

• Ephemeral: Data vanishes when the job/session ends.

Building the Sandbox. We use this to avoid lutimes / fakeroot errors common on shared filesystems. /scratch (Shared)

• Persistent: Visible across all nodes.

• Networked: Often Lustre or GPFS.

• Metadata Constraints: May fail on attribute modification under fakeroot.

Storage & Packaging. Used to store the final SIF image and datasets.

The Lifecycle Diagram

The build process is a four-stage pipeline designed to navigate privilege requirements and resource constraints.

┌──────────────────────────────┐
│ 0) Definition (.def) file    │
│    - Bootstrap: docker       │
│    - %post: install pkgs     │
│    - %env, %runscript        │
└───────────────┬──────────────┘
                │ (pull base + run %post)
                ▼
┌──────────────────────────────────────────┐
│ 1) Build SANDBOX on LOGIN node (/tmp)    │
│    - Needs --fakeroot                    │
│    - Writable rootfs                     │
│    - Avoids lutimes errors               │
└───────────────┬──────────────────────────┘
                │ (make visible cluster-wide)
                ▼
┌──────────────────────────────────────────┐
│ 2) RSYNC sandbox → /scratch/$USER        │
│    - Shared FS, accessible by all nodes  │
└───────────────┬──────────────────────────┘
                │ (compress with RAM/IO)
                ▼
┌──────────────────────────────────────────┐
│ 3) PACKAGE SIF on COMPUTE node           │
│    - No fakeroot required                │
│    - Uses mksquashfs (needs RAM)         │
│    - Use compute-node /tmp as TMPDIR     │
│    - Output to /scratch/$USER            │
└───────────────┬──────────────────────────┘
                │ (immutable, portable)
                ▼
┌──────────────────────────────────────────┐
│ 4) RUN the SIF anywhere                  │
│    - singularity exec --nv ...           │
│    - Bind /scratch, /home, /apps       │
└──────────────────────────────────────────┘

Step-by-Step Workflow

Step 0: Environment Preparation

Redirect Singularity's cache to local disk to prevent quota issues and metadata errors on the shared filesystem.

# Set paths to node-local storage
mkdir -p /tmp/$USER/singularity/{cache,tmp} /tmp/$USER/sifs /scratch/$USER/sifs

export SINGULARITY_CACHEDIR=/tmp/$USER/singularity/cache
export SINGULARITY_TMPDIR=/tmp/$USER/singularity/tmp
export REPO=/scratch/$USER/hpc-container-build

Step 1: Build Sandbox (Login Node)

Constraint: Must be run on a node allowing fakeroot. Action: We build into /tmp because it handles the file attribute changes required by package managers (apt/yum) gracefully.

singularity build --fakeroot \
    --tmpdir "$SINGULARITY_TMPDIR" \
    --sandbox /tmp/$USER/sifs/pt-sandbox \
    "$REPO/pt-2.8.0-cu129-devel.def"

Step 2: Migrate to Shared Storage

Constraint: /tmp is invisible to other nodes. Action: Move the uncompressed sandbox to global scratch space using rsync. The flags used ensure we do not attempt to preserve ownership attributes that we (as non-root users) cannot control.

rsync -a --delete --no-xattrs --no-owner --no-group \
    /tmp/$USER/sifs/pt-sandbox/ \
    /scratch/$USER/sifs/pt-sandbox

Step 3: Package Image (Compute Node)

Constraint: mksquashfs (the compression tool) is CPU and RAM intensive. Action: Submit a job or request an interactive session on a compute node. We do not need fakeroot here, as we are simply compressing an existing directory structure.

# 3.1 Request resources (Example: 16GB RAM)
salloc -N 1 -n 1 --mem=16G -p compute -t 02:00:00 --comment pytorch

# 3.2 Re-export variables on the compute node
mkdir -p /tmp/$USER/singularity/{cache,tmp}
export SINGULARITY_CACHEDIR=/tmp/$USER/singularity/cache
export SINGULARITY_TMPDIR=/tmp/$USER/singularity/tmp

# 3.3 Build the final immutable image
singularity build --notest \
    --tmpdir "$SINGULARITY_TMPDIR" \
    /scratch/$USER/sifs/pt-2.8.0-cu129-devel.sif \
    /scratch/$USER/sifs/pt-sandbox

Step 4: Validation

Verify the container has GPU access and necessary tools.

# Check Python/Torch
singularity exec --nv /scratch/$USER/sifs/pt-2.8.0-cu129-devel.sif \
    python -c "import torch; print(f'Torch: {torch.__version__}, CUDA: {torch.cuda.is_available()}')"

# Check Tools
singularity exec /scratch/$USER/sifs/pt-2.8.0-cu129-devel.sif git --version

Troubleshooting Common Failures

Symptom Diagnosis Resolution

• lutimes ... operation not permitted Fakeroot is attempting metadata changes on a shared filesystem (Lustre/GPFS) that doesn't support it for user namespaces. Build in /tmp (Step 1). Never build a sandbox directly on /scratch.

• signal: killed The build process ran out of memory, usually during the mksquashfs compression phase. Request a compute node with higher memory (16GB+) for Step 3.

• Sandbox Not Found You tried to access /tmp/$USER/sifs/pt-sandbox on a compute node, but that directory only exists on the login node. Ensure you rsync to /scratch (Step 2) before switching nodes.

Appendix: Minimal SLURM Template

This template demonstrates how to deploy the finished container in a production job.

#!/bin/bash
#SBATCH --comment pytorch
#SBATCH -J torch-training
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --gres=gpu:1
#SBATCH --mem=32G
#SBATCH -t 04:00:00
#SBATCH -p gpu

set -e

# 1. Localize Singularity caching
export SINGULARITY_CACHEDIR=/tmp/$USER/singularity/cache
export SINGULARITY_TMPDIR=/tmp/$USER/singularity/tmp
mkdir -p "$SINGULARITY_CACHEDIR" "$SINGULARITY_TMPDIR"

# 2. Define Image Path
SIF=/scratch/$USER/sifs/pt-2.8.0-cu129-devel.sif

# 3. Execution
# Note: -B /scratch binds the host scratch directory into the container
singularity exec --nv -B /scratch:/scratch "$SIF" bash -lc '
    echo "Container Active on $(hostname)"
    python -c "import torch; print(f\"GPU Detected: {torch.cuda.get_device_name(0)}\")"
    
    # Run your training script here
    # python train.py
'