|
1 | 1 | # How to build spack-stack at NAS |
2 | 2 |
|
3 | | -## Generic |
| 3 | +In the commands below some will be run on login nodes (with internet access) and some |
| 4 | +on compute nodes as, at NAS, you aren't allowed more than 2 processes on a login node. |
4 | 5 |
|
| 6 | +## Clone spack-stack |
| 7 | + |
| 8 | +``` |
| 9 | +git clone --recursive https://github.com/JCSDA/spack-stack.git -b release/1.9.0 spack-stack-1.9.3 |
| 10 | +``` |
| 11 | + |
| 12 | +## Grab interactive node |
| 13 | + |
| 14 | +Since NAS limits you to 2 processes on a login node, you'll need to grab an interactive node. For example: |
5 | 15 | ``` |
6 | | -git clone --recursive https://github.com/JCSDA/spack-stack.git -b release/1.9.0 spack-stack-1.9.1 |
| 16 | +qsub -I -V -X -l select=1:ncpus=128:mpiprocs=128:model=mil_ait -l walltime=12:00:00 -W group_list=s1873 -m b -N Interactive |
7 | 17 | ``` |
| 18 | +will get you a Milan node for 12 hours |
| 19 | + |
| 20 | +## Setup spack-stack on each node |
8 | 21 |
|
9 | | -## oneapi |
| 22 | +We will start on a login node with internet access. This is mainly needed for the |
| 23 | +`spack mirror create` command which downloads all the source code for the packages. |
10 | 24 |
|
11 | 25 | ``` |
12 | | -cd spack-stack-1.9.1 |
| 26 | +cd spack-stack-1.9.3 |
13 | 27 | . setup.sh |
14 | | -spack stack create env --name ue-oneapi-2024.2.0 --template unified-dev --site nas --compiler oneapi |
15 | | -cd envs/ue-oneapi-2024.2.0 |
16 | | -spack env activate . |
17 | | -spack concretize 2>&1 | tee log.concretize |
18 | | -spack install --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install |
19 | 28 | ``` |
20 | 29 |
|
21 | | -NOTE: You might need to run the `spack install` command multiple times because sometimes |
22 | | -it just fails. But then you run it more and more and it will eventually succeed. |
| 30 | +## Create environments |
| 31 | + |
| 32 | +We create two different environments, one for oneAPI and one for GCC. The commands below |
| 33 | +are used to create the environments. You only need to do this once. |
| 34 | + |
| 35 | +### oneAPI |
| 36 | + |
| 37 | +To create the oneAPI environment, do: |
23 | 38 |
|
24 | 39 | ``` |
25 | | -spack module tcl refresh -y |
26 | | -spack stack setup-meta-modules |
27 | | -spack env deactivate |
| 40 | +spack stack create env --name ue-oneapi-2024.2.0 --template unified-dev --site nas --compiler oneapi |
| 41 | +cd envs/ue-oneapi-2024.2.0 |
28 | 42 | ``` |
29 | 43 |
|
30 | | -## gcc |
| 44 | +### GCC |
| 45 | + |
| 46 | +To create the GCC environment, do: |
31 | 47 |
|
32 | 48 | ``` |
33 | | -cd spack-stack-1.9.1 |
34 | | -. setup.sh |
35 | 49 | spack stack create env --name ue-gcc-12.3.0 --template unified-dev --site nas --compiler gcc |
36 | 50 | cd envs/ue-gcc-12.3.0 |
| 51 | +``` |
| 52 | + |
| 53 | +## Activate environment |
| 54 | + |
| 55 | +Now enter the spack environment you just created: |
| 56 | + |
| 57 | +``` |
37 | 58 | spack env activate . |
| 59 | +``` |
| 60 | + |
| 61 | +NOTE: You need to make sure you do this in *any* terminal where you want to do any commmand |
| 62 | +below with this environment. |
| 63 | + |
| 64 | +## Concretize and create source cache |
| 65 | + |
| 66 | +``` |
38 | 67 | spack concretize 2>&1 | tee log.concretize |
39 | | -spack install --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install |
| 68 | +``` |
| 69 | + |
| 70 | +## Create source cache (LOGIN NODE ONLY) |
| 71 | + |
| 72 | +Because this step downloads all the source code for all packages and all versions, it |
| 73 | +should be done on a login node with internet access. |
| 74 | + |
| 75 | +``` |
| 76 | +spack mirror create -a -d /nobackup/gmao_SIteam/spack-stack/source-cache |
| 77 | +``` |
| 78 | + |
| 79 | +NOTE: Make sure you are in an environment when you run that `spack mirror create` command. Otherwise, |
| 80 | +you will download *EVERY* package and *EVERY* version in spack! |
| 81 | + |
| 82 | +## Install packages |
| 83 | + |
| 84 | +Our install process will actually have (at least) three steps. This is because of the `crtm` package |
| 85 | +which requires internet access at build time. |
| 86 | + |
| 87 | +### Install crtm dependencies (COMPUTE NODE) |
| 88 | + |
| 89 | +``` |
| 90 | +spack install -j 10 --verbose --fail-fast --show-log-on-error --no-check-signature --only dependencies crtm 2>&1 | tee log.install.crtm_dependencies |
| 91 | +``` |
| 92 | + |
| 93 | +### Install crtm (LOGIN NODE) |
| 94 | + |
| 95 | +``` |
| 96 | +spack install -j 2 --verbose --fail-fast --show-log-on-error --no-check-signature crtm 2>&1 | tee log.install.crtm |
| 97 | +``` |
| 98 | + |
| 99 | +Note we are only using 2 processes here because NAS limits you to 2 processes on a login node. |
| 100 | + |
| 101 | +### Install rest of packages (COMPUTE NODE) |
| 102 | + |
| 103 | +``` |
| 104 | +spack install -j 10 --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install.after_crtm |
40 | 105 | ``` |
41 | 106 |
|
42 | 107 | NOTE: You might need to run the `spack install` command multiple times because sometimes |
43 | 108 | it just fails. But then you run it more and more and it will eventually succeed. |
44 | 109 |
|
| 110 | +### Packages needing internet access to build |
| 111 | + |
| 112 | +If you encounter other packages that need internet access to build, you can install them with: |
| 113 | + |
| 114 | +``` |
| 115 | +spack install -j 2 --verbose --fail-fast --show-log-on-error --no-check-signature <package> |& tee log.install.<package> |
| 116 | +``` |
| 117 | + |
| 118 | +Then, once that package is built, you can go back to the compute node and run the `spack install` command again. |
| 119 | + |
| 120 | +## Update module files and setup meta-modules |
| 121 | + |
45 | 122 | ``` |
46 | 123 | spack module tcl refresh -y |
47 | 124 | spack stack setup-meta-modules |
| 125 | +``` |
| 126 | + |
| 127 | +## Deactivate environment |
| 128 | + |
| 129 | +``` |
48 | 130 | spack env deactivate |
49 | 131 | ``` |
0 commit comments