Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The AWS infrastructure reference, `INFRASTRUCTURE.md`, lives in the private `coe

**Handoff and operations**

- [`docs/TEAM_RUNBOOK.md`](docs/TEAM_RUNBOOK.md) - start here for active threads and next steps
- [`TEAM_RUNBOOK.md`](TEAM_RUNBOOK.md) - start here for active threads and next steps

**Data**

Expand Down Expand Up @@ -51,7 +51,7 @@ The AWS infrastructure reference, `INFRASTRUCTURE.md`, lives in the private `coe

## Roadmaps

**Resuming work after a handoff?** Start at [`docs/TEAM_RUNBOOK.md`](docs/TEAM_RUNBOOK.md). It is the dashboard of active threads (where each stands and the next step) and links into the detailed roadmaps below.
**Resuming work after a handoff?** Start at [`TEAM_RUNBOOK.md`](TEAM_RUNBOOK.md). It is the dashboard of active threads (where each stands and the next step) and links into the detailed roadmaps below.

**Roadmaps by area** - each section keeps its deferred and in-progress work at the bottom of its own README:

Expand Down Expand Up @@ -163,7 +163,7 @@ Two pieces of automation are designed and on the way to production-ready:
- A unified driver ([`etl/run_full_pipeline.py`](etl/run_full_pipeline.py)) that wraps every stage into one resumable command. Its current state and caveats are documented in the [ETL runbook](etl/README.md#experimental-orchestrator).
- An EventBridge `Batch -> run_all` trigger so statistics fire automatically on extraction completion.

Both are tracked in [`docs/TEAM_RUNBOOK.md`](docs/TEAM_RUNBOOK.md).
Both are tracked in [`TEAM_RUNBOOK.md`](TEAM_RUNBOOK.md).

### Tier data pipeline

Expand Down
2 changes: 1 addition & 1 deletion database/SCHEMA_BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -755,7 +755,7 @@ This is a per-lookup decision: enforce, retire, or accept-as-catalog. Numbers co
- [ ] **`network_subtype`:** Decide the `network` / `network_subtype` relationship (the lookup exists, `network` has no `subtype_id`). Effort: S.
- [ ] **`env_flow_season`:** Already FK'd from `env_flow_channel_seasonal.season_id`. No change unless other seasonal aggregations grow.
- [ ] **`watershed`:** Already a text FK (`channel_entity.watershed_short_code`). Decide whether to migrate to integer FK. Effort: S.
- [ ] **Normalize PWSID / drinking-water utility out of `du_urban_entity.community_agency`:** Utility names and the trailing 7-digit PWSID live as free text in `community_agency` (e.g. `Centerville and Redding Centerville CSD 4510011`). Decide whether to normalize the utility / PWSID into its own lookup. This overlaps the CWS schema build, since the draft `cws_entity` is one row per PWSID, so settle the PWSID representation during that design rather than in isolation (see [`TEAM_RUNBOOK.md` A1](../docs/TEAM_RUNBOOK.md) and [`water_user_categories.md`](topic_docs/cws/water_user_categories.md#roadmap)). Effort: S (decision), M (implementation).
- [ ] **Normalize PWSID / drinking-water utility out of `du_urban_entity.community_agency`:** Utility names and the trailing 7-digit PWSID live as free text in `community_agency` (e.g. `Centerville and Redding Centerville CSD 4510011`). Decide whether to normalize the utility / PWSID into its own lookup. This overlaps the CWS schema build, since the draft `cws_entity` is one row per PWSID, so settle the PWSID representation during that design rather than in isolation (see [`TEAM_RUNBOOK.md` A1](../TEAM_RUNBOOK.md) and [`water_user_categories.md`](topic_docs/cws/water_user_categories.md#roadmap)). Effort: S (decision), M (implementation).

Effort total: roughly two M-sized migrations (`unit`, `hydrologic_region`) plus a string of S-sized decisions and one-shot FK additions. None block current operation. The value is that the next person editing data cannot quietly drift from the controlled vocabulary.

Expand Down
2 changes: 1 addition & 1 deletion etl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -545,4 +545,4 @@ The table below is a **starting point**, and it covers mainly the **ingestion**
- **Community water system locations need attention and refinement** The list of community water system locations has been developing and changing. Speak with Kristin Dobbins and the research team about the goals for community water system locations, including export areas and systems served by demand unit.
- **Pipeline scenario-selection flags are inconsistent and could be rectified:** The same "which scenarios" is flagged differently across the pipeline's functions, which makes the runbook/pipeline harder to run. Recommendation: standardize on `--scenarios` (`nargs="*"`) plus `--all-scenarios` everywhere, document the set each `--all-scenarios` expands to, roll it through the per-module scripts and `run_all.py`.
- **Once scenario and tier metadata and descriptions settle, incorporate them as database data:** Descriptive text is intentionally left unfinished in the database, because it often isn't agreed upon when a scenario's identity row is created. Today the `scenario` row carries only identity, while the descriptive metadata lives in `scenario_hydroclimate_sibling` (name, short / long description, shared across a strategy's hydroclimate variants) and the `theme_scenario_link` / `scenario_tag_link` / `scenario_key_assumption_link` / `scenario_key_operation_link` tables, and some display extras (icons, short labels, per-scenario operations / assumptions) are still authored in the website rather than the database. Similarly, Tier labels and descriptions should eventually live in `tier_definition`. Once the final wording and classifications are agreed with the research and tier teams, load them as data into these tables and migrate the website-authored extras to the backend, so the API serves them from a single source of truth.
- Finish the pipeline orchestrator and add an EventBridge `Batch` to `run_all` trigger (so statistics fire automatically on extraction completion) is tracked as thread A8 in [`docs/TEAM_RUNBOOK.md`](../docs/TEAM_RUNBOOK.md#a8-automate-the-etl-pipeline-end-to-end-orchestrator--batch--run_all-trigger).
- Finish the pipeline orchestrator and add an EventBridge `Batch` to `run_all` trigger (so statistics fire automatically on extraction completion) is tracked as thread A8 in [`TEAM_RUNBOOK.md`](../TEAM_RUNBOOK.md#a8-automate-the-etl-pipeline-end-to-end-orchestrator--batch--run_all-trigger).
Loading