|
| 1 | +# Session 0 - Setup |
| 2 | + |
| 3 | +**What this session is for.** Later sessions assume a working software stack, the pipeline code on disk, and a data workspace with known path names. This session gets that ready, including what to run every time you open a new shell. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## 0.1 Environment (once) |
| 8 | + |
| 9 | +Use conda env `pecan-all-1.14` for all sessions. Activate it and confirm the packages later sessions need. If either check fails, stop and fix the environment before continuing. |
| 10 | + |
| 11 | +```bash |
| 12 | +conda activate <ENV_PATH_OR_NAME> # pecan-all-1.14 |
| 13 | + |
| 14 | +Rscript -e 'stopifnot( |
| 15 | + requireNamespace("arrow"), |
| 16 | + requireNamespace("dplyr"), |
| 17 | + requireNamespace("data.table"), |
| 18 | + requireNamespace("sf"), |
| 19 | + requireNamespace("terra"), |
| 20 | + requireNamespace("exactextractr"), |
| 21 | + requireNamespace("readr"), |
| 22 | + requireNamespace("stringr"), |
| 23 | + requireNamespace("lubridate"), |
| 24 | + requireNamespace("jsonlite") |
| 25 | +)' |
| 26 | + |
| 27 | +python - <<'PY' |
| 28 | +import dask |
| 29 | +import fiona |
| 30 | +import geopandas |
| 31 | +import numpy |
| 32 | +import pandas |
| 33 | +import pyarrow |
| 34 | +import shapely |
| 35 | +import tqdm |
| 36 | +PY |
| 37 | +``` |
| 38 | + |
| 39 | +--- |
| 40 | + |
| 41 | +## 0.2 Clone (once) |
| 42 | + |
| 43 | +Set `$CCMMF_BASE` to the directory that will hold your clones (`$CCMMF_BASE/src`) and data (`$CCMMF_BASE/ccmmf`). |
| 44 | + |
| 45 | +You also need two git repos: PEcAn and [cadwr-landuse](https://github.com/ccmmf/cadwr-landuse). Later sessions assume those repos live at `$CCMMF_BASE/src/pecan` and `$CCMMF_BASE/src/cadwr-landuse`. |
| 46 | + |
| 47 | +**New clones** |
| 48 | + |
| 49 | +```bash |
| 50 | +export CCMMF_BASE=/path/to/workdir |
| 51 | + |
| 52 | +mkdir -p "$CCMMF_BASE/src" |
| 53 | +cd "$CCMMF_BASE/src" |
| 54 | + |
| 55 | +# pecan -- develop |
| 56 | +git clone https://github.com/PecanProject/pecan.git |
| 57 | +cd pecan |
| 58 | +git checkout develop |
| 59 | + |
| 60 | +# cadwr-landuse -- main |
| 61 | +cd "$CCMMF_BASE/src" |
| 62 | +git clone https://github.com/ccmmf/cadwr-landuse.git |
| 63 | +cd cadwr-landuse |
| 64 | +git checkout main |
| 65 | +``` |
| 66 | + |
| 67 | +**Already cloned elsewhere.** |
| 68 | + |
| 69 | +```bash |
| 70 | +export CCMMF_BASE=/path/to/workdir |
| 71 | +mkdir -p "$CCMMF_BASE/src" |
| 72 | + |
| 73 | +# ln -s <existing clone> <tutorial path> |
| 74 | +ln -s /actual/path/to/pecan "$CCMMF_BASE/src/pecan" |
| 75 | +ln -s /actual/path/to/cadwr-landuse "$CCMMF_BASE/src/cadwr-landuse" |
| 76 | + |
| 77 | +ls -ld "$CCMMF_BASE/src/pecan" "$CCMMF_BASE/src/cadwr-landuse" |
| 78 | +``` |
| 79 | + |
| 80 | +--- |
| 81 | + |
| 82 | +## 0.3 Every new shell |
| 83 | + |
| 84 | +Activate the conda env from 0.1, set the same `$CCMMF_BASE` as in 0.2, pull the repo you need, and source `setup_env`. |
| 85 | + |
| 86 | +```bash |
| 87 | +conda activate <ENV_PATH_OR_NAME> # same env as 0.1 |
| 88 | + |
| 89 | +export CCMMF_BASE=/path/to/workdir # same as 0.2 |
| 90 | + |
| 91 | +# Pull only the repo you are using this session: |
| 92 | +git -C "$CCMMF_BASE/src/pecan" pull origin develop |
| 93 | +git -C "$CCMMF_BASE/src/cadwr-landuse" pull origin main |
| 94 | + |
| 95 | +# Optional overrides (only if you do not want the BASE defaults): |
| 96 | +# export CCMMF_ROOT=/path/to/data |
| 97 | +# export PRIOR_YEAR=2023 TARGET_YEAR=2024 |
| 98 | + |
| 99 | +source "$CCMMF_BASE/src/pecan/modules/data.remote/inst/ccmmf/documentation/setup_env.sh" |
| 100 | +``` |
| 101 | + |
| 102 | +--- |
| 103 | + |
| 104 | +## 0.4 Workspace (once) |
| 105 | + |
| 106 | +`setup_env` only stored the path strings. Create these folders once. Later sessions use the vars and assume the tree exists. |
| 107 | + |
| 108 | +```text |
| 109 | +$CCMMF_ROOT/ |
| 110 | + LandIQ/ |
| 111 | + raw/ # LANDIQ_RAW |
| 112 | + work/ # CADWR_WORK_DIR |
| 113 | + 03-final/ # LANDIQ_HARMONIZED |
| 114 | + gapfilled/ # LANDIQ_GAPFILLED |
| 115 | + HLS/ |
| 116 | + imagery/ # HLS_IMAGERY_ROOT |
| 117 | + MSLSP/ # MSLSP_NETCDF_ROOT |
| 118 | + CDL/ # CDL_DIR |
| 119 | + climate/ |
| 120 | + CHIRPS/ # CHIRPS_DIR |
| 121 | + CIMIS/ # CIMIS_DIR |
| 122 | + soils/ |
| 123 | + SSURGO/ # SSURGO_DIR |
| 124 | + lookups/ |
| 125 | + plant_traits/ # PLANT_TRAITS_DIR |
| 126 | + fertilization/ # FERTILIZATION_LOOKUPS |
| 127 | + products/ |
| 128 | + inventory/ # PRODUCTS_INVENTORY |
| 129 | + phenology/ |
| 130 | + tillage/ |
| 131 | + fertilization/ |
| 132 | + irrigation/ |
| 133 | + event_files/ |
| 134 | +``` |
| 135 | + |
| 136 | +Create the dirs: |
| 137 | + |
| 138 | +```bash |
| 139 | +mkdir -p "$LANDIQ_ROOT"/{raw,gapfilled} |
| 140 | +mkdir -p "$CADWR_WORK_DIR" "$LANDIQ_HARMONIZED" # 03-final; S3 skip or cadwr both land here |
| 141 | +mkdir -p "$HLS_ROOT"/{imagery,MSLSP} |
| 142 | +mkdir -p "$CDL_DIR" |
| 143 | +mkdir -p "$CLIMATE_ROOT"/{CHIRPS,CIMIS} |
| 144 | +mkdir -p "$SOILS_ROOT"/SSURGO |
| 145 | +mkdir -p "$LOOKUPS_ROOT"/{plant_traits,fertilization} |
| 146 | +mkdir -p "$PRODUCTS_INVENTORY"/{phenology,tillage,fertilization,irrigation,event_files} |
| 147 | +``` |
| 148 | + |
| 149 | +--- |
| 150 | + |
| 151 | +## 0.5 Confirm setup |
| 152 | + |
| 153 | +Confirm the code and data roots are real on disk. |
| 154 | + |
| 155 | +```bash |
| 156 | +ls "$CCMMF_CODE" # pipeline scripts and documentation |
| 157 | +# documentation events hls landiq-gapfill phenology tillage ... |
| 158 | + |
| 159 | +ls "$CCMMF_ROOT" # data workspace |
| 160 | +# CDL HLS LandIQ climate lookups products soils |
| 161 | +``` |
| 162 | + |
| 163 | +If you do not see the directories you made, go back and fix the section above before Session 1. |
| 164 | + |
| 165 | +--- |
| 166 | + |
| 167 | +## 0.6 NASA Earthdata |
| 168 | + |
| 169 | +Create an Earthdata Login account and store credentials in `~/.netrc` for HLS downloads (Session 2). |
| 170 | + |
| 171 | +1. Create a free account at [https://urs.earthdata.nasa.gov/](https://urs.earthdata.nasa.gov/) |
| 172 | +2. Store credentials in `~/.netrc`: |
| 173 | + |
| 174 | +```bash |
| 175 | +# Replace USERNAME and PASSWORD with your Earthdata Login values |
| 176 | +echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWORD" > ~/.netrc |
| 177 | +chmod 0600 ~/.netrc |
| 178 | +``` |
| 179 | + |
| 180 | +**Next:** [Session 1 - LandIQ crop identity](01-landiq.md). |
| 181 | + |
| 182 | +**Spine:** [tree README](../../README.md). |
0 commit comments