smartspim-pipeline-v5.0.0

v5.0.0 — SLURM Deployment & Improved Cell Detection

Release of the aind-smartspim-pipeline v5.0.0, a Nextflow-based pipeline for end-to-end processing of SmartSPIM light-sheet microscopy datasets.

Highlights

New Deep Learning Cell Detection Model
This release includes a new cell detection model trained on a significantly expanded set of annotations, improving detection accuracy and generalization across diverse SmartSPIM datasets. The model integrates with the existing [cellfinder](https://github.com/brainglobe/cellfinder)-based segmentation step and is available alongside previous production models.

Full SLURM Cluster Compatibility
The pipeline now supports deployment on SLURM clusters using Singularity/Apptainer containers:

• Automated environment setup via environment/create_slurm_env.sh for bootstrapping cluster-ready environments
• Pre-built Singularity container support — containers can be pulled ahead of time to avoid resource-limited build failures on compute nodes
• Configurable SLURM partitions and queues through nextflow_slurm.config and nextflow_slurm_custom.config
• Dynamic resource allocation using SLURM environment variables (SLURM_JOBID, SLURM_JOB_CPUS_PER_NODE, SLURM_JOB_GPUS)
• Nextflow -resume support for restarting failed or interrupted runs without recomputing completed steps
• Optional AWS cloud output — results can be written directly to an S3 bucket by setting the --cloud flag

Pipeline Overview

The pipeline orchestrates eight modular processing steps: dataset validation, destriping (wavelet-based filtering + flat-field correction), stitching, fusion to OMEZarr format, Allen CCFv3 atlas registration, cell segmentation (via cellfinder), cell quantification, and result dispatching with Neuroglancer visualization links.

Required External Data

• [SmartSPIM template v1.0](https://open.quiltdata.com/b/aind-open-data/tree/SmartSPIM-template_2024-05-16_11-26-14/) — for CCF registration
• [SmartSPIM cell detection models](https://open.quiltdata.com/b/aind-benchmark-data/tree/mesoscale-anatomy-cell-detection/models/smartspim_production_models/) — for cell segmentation (includes the new expanded-annotation model)

Known Limitations

• The pipeline currently uses Nextflow DSL1, which is no longer supported by Nextflow. Migration to DSL2 is planned (see [#7](#7)).
• GPU processes require --nv in the container options to properly load GPU drivers.

License

MIT