Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
1a90d8c
Add verified v2 metro parcel sources + generalized fetcher
SlowSpeedChase Jun 29, 2026
62b577c
Add value-per-acre 3D map notebook (downtown Austin slice)
SlowSpeedChase Jun 30, 2026
29eaf05
Add acreage cleaning + H3 metro layers to value-per-acre notebook
SlowSpeedChase Jun 30, 2026
1ac6940
Scale value-per-acre map to full metro with geometry-acreage fallback
SlowSpeedChase Jun 30, 2026
1fad17a
Improve map legibility: quantile turbo colors, legend, highway overlay
SlowSpeedChase Jun 30, 2026
8b1ac41
Add fiscal-productivity analysis: do the suburbs pay their way?
SlowSpeedChase Jun 30, 2026
9bff16a
Rebuild on full Travis 2025 certified roll; add reproducible roll pip…
SlowSpeedChase Jun 30, 2026
de46dea
Add road-infrastructure cost model to fiscal analysis
SlowSpeedChase Jun 30, 2026
0f73dc8
Add fire use-vs-pays-in analysis by AFD response area
SlowSpeedChase Jun 30, 2026
0c80c1e
Add apparatus-weighted coverage + distance-to-station to fire analysis
SlowSpeedChase Jun 30, 2026
0937bc9
Add synthesis report: value, infrastructure, and fire fiscal producti…
SlowSpeedChase Jun 30, 2026
107a03e
Fix requirements.txt newline (h3/markdown concatenation)
SlowSpeedChase Jun 30, 2026
321b369
Add parquet-free fire-fiscal master report, validation gate, palette,…
claude Jun 30, 2026
9303a7a
Execute fire-fiscal report locally: validate full roll, build map + c…
SlowSpeedChase Jun 30, 2026
c63b657
Add ranked colloquial inventory; fix palette-swatch spacing
SlowSpeedChase Jun 30, 2026
aef42ca
Add Kindle Scribe edition: e-ink page size + grayscale-safe map
SlowSpeedChase Jun 30, 2026
366a8da
Revise report voice: plainer, less AI; add city home values + area defs
SlowSpeedChase Jun 30, 2026
94e06c0
Plain-voice pass on deeper sections (§5, §13) + fix anchors
SlowSpeedChase Jul 1, 2026
bd665f4
Fix validated report errors + presentation polish
SlowSpeedChase Jul 1, 2026
382f9ab
Repo cleanup: reorganize pipelines, untrack heavy artifacts, refresh …
SlowSpeedChase Jul 1, 2026
1c0348e
Add design doc for report citation pass
SlowSpeedChase Jul 2, 2026
6b8a00d
Add Chicago-style citations + full bibliography to master report
SlowSpeedChase Jul 2, 2026
3129c5b
Add §15 'To be done' page + UNFINISHED watermark on every PDF page
SlowSpeedChase Jul 2, 2026
3c1378f
Clarify assessed value definitions and update citations for accuracy …
SlowSpeedChase Jul 7, 2026
53872e7
Address Eric review: apparatus link, citation [23] link, fix coverage…
SlowSpeedChase Jul 14, 2026
4b5b919
Add PR13 review discussion handout (markdown + print HTML) (#14)
SlowSpeedChase Jul 15, 2026
aa43274
Add data-provenance standard + retrofit deliverables (#14)
SlowSpeedChase Jul 15, 2026
7d04400
Move review/handoff asides into docs/working/ (#14)
SlowSpeedChase Jul 15, 2026
a8d6fe7
Resolve review items C & D: revenue/value convention + tighten §2.4 (…
SlowSpeedChase Jul 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,10 +1,35 @@
# Data files - too large for GitHub, can be regenerated
# ---------------------------------------------------------------- data files
# Raw downloads: regenerated by incident_pipeline/01_download_data.py
raw_data/
outputs/

# Outputs: keep the small tables/figures (csv/png/txt/md) so a fresh clone has
# results without re-running the pipeline; ignore the heavy generated artifacts
# (interactive map HTML, report PDF/HTML builds, folium maps).
outputs/*
!outputs/*.csv
!outputs/*.png
!outputs/*.txt
!outputs/*.md

# Large processed-data files: regenerated by the pipelines (see report §13 runbook)
processed_data/response_areas_final.geojson
processed_data/response_areas_with_demographics.geojson
processed_data/parcels_value_per_acre_metro.parquet
processed_data/metro_highways.geojson
processed_data/tx_places.zip
processed_data/tcad_2020_values.parquet
processed_data/tcad_2025_values.parquet
processed_data/tcad_gis_travis.parquet
processed_data/roads_48453.zip
processed_data/roads_48491.zip
processed_data/roads_48209.zip
processed_data/fire_stations.geojson
processed_data/fire_net_balance.geojson

# Notebook-generated maps
notebooks/*.html

# Python
# ---------------------------------------------------------------- environments
__pycache__/
*.py[cod]
*$py.class
Expand Down
6 changes: 6 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,9 @@ Before responding to any user message, check if any available skill applies (eve
Priority order:
1. Process skills first (brainstorming, debugging) - these determine HOW to approach the task
2. Implementation skills second - these guide execution

## Data provenance for every claim

**No number ships without its provenance.** Any quantitative claim in a deliverable — report, handout, README, issue/PR comment, chart caption — must be traceable to (1) its source data (file + columns), (2) the exact formula/transformation (including filters, weights, and null handling), and (3) a reproduction step (copy-paste snippet or `notebook.ipynb → cell N` / `script.py:line`). A figure a reader can't trace is a claim they can't check; don't publish it.

When you add or change a number in a deliverable, add or update its entry in `docs/METHODS.md` (the provenance ledger + standard) and leave a short pointer to it in the deliverable. Compute numbers in a notebook/script, never by hand. Prefer numbers that `notebooks/validation.ipynb` re-derives on every run.
81 changes: 55 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,25 +14,34 @@ python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Download data (~100MB from public APIs)
python 01_download_data.py
python incident_pipeline/01_download_data.py

# Run the pipeline
python 02_clean_incidents.py # Clean & classify incidents
python 03_create_crosswalk.py # Census tract → response area mapping + zoning
python 04_analysis.py # Calculate rates & statistical tests
python 05_visualize.py # Generate maps & charts
# Run the pipeline (from the repo root)
python incident_pipeline/02_clean_incidents.py # Clean & classify incidents
python incident_pipeline/03_create_crosswalk.py # Census tract → response area mapping + zoning
python incident_pipeline/04_analysis.py # Calculate rates & statistical tests
python incident_pipeline/05_visualize.py # Generate maps & charts

# Or run everything in one go
python incident_pipeline/run_all.py

# Optional: NFIRS cause analysis (requires manual data download)
python 06_nfirs_cause_analysis.py
python incident_pipeline/06_nfirs_cause_analysis.py
```

## Fire-Fiscal Report

A second analysis line — the fiscal-productivity / fire-fiscal report — asks whether each area of the Austin metro generates enough tax revenue to pay for the infrastructure and fire service it consumes. Its pipeline lives in `report_pipeline/` (TCAD roll parsing → metro parcel build → report figures → PDF), with the three models in `notebooks/` (`value_per_acre_metro.ipynb`, `fiscal_productivity.ipynb`, `fire_use_vs_pays.ipynb`) and a 24-check validation gate (`notebooks/validation.ipynb` → `outputs/validation_report.csv`).

The master report is [docs/FIRE_FISCAL_FULL_REPORT.md](docs/FIRE_FISCAL_FULL_REPORT.md) (rendered to `outputs/FIRE_FISCAL_FULL_REPORT.pdf` by `report_pipeline/14_build_report_pdf.py`); see its §13 runbook for the full rebuild order.

## Data Setup

**Data files are not included in this repository** due to size. Run `01_download_data.py` to fetch them from public APIs.
**Data files are not included in this repository** due to size. Run `incident_pipeline/01_download_data.py` to fetch them from public APIs.

### Automatic Download

`01_download_data.py` downloads:
`incident_pipeline/01_download_data.py` downloads:
- AFD fire incidents (2022-2024) from Austin Open Data Portal
- AFD response area boundaries from ArcGIS FeatureServer
- Fire station locations from ArcGIS FeatureServer
Expand All @@ -42,7 +51,7 @@ python 06_nfirs_cause_analysis.py

### NFIRS Data (optional)

The NFIRS cause analysis (`06_nfirs_cause_analysis.py`) requires a manual download:
The NFIRS cause analysis (`incident_pipeline/06_nfirs_cause_analysis.py`) requires a manual download:

1. Visit the [USFA NFIRS Data Center](https://www.usfa.fema.gov/nfirs/data/)
2. Download the annual Public Data Release files (2018-2021)
Expand Down Expand Up @@ -75,12 +84,12 @@ raw_data/

| Script | Input | Output | Description |
|--------|-------|--------|-------------|
| `01_download_data.py` | Public APIs | `raw_data/*` | Fetches all data; handles ArcGIS pagination for large datasets |
| `02_clean_incidents.py` | `raw_data/afd_incidents_2022_2024.csv` | `processed_data/incidents_clean.csv` | Parses dates, classifies incident types (structure, vehicle, outdoor, trash) |
| `03_create_crosswalk.py` | `raw_data/*` | `processed_data/tract_to_response_area_crosswalk.csv`, `response_area_demographics.csv`, `response_areas_with_demographics.geojson` | Area-weighted census allocation + zoning height mapping |
| `04_analysis.py` | `processed_data/*` | `outputs/summary_*.csv`, `outputs/statistical_tests.txt` | Per-capita rates, t-tests, ANOVA, correlations |
| `05_visualize.py` | `processed_data/*` | `outputs/map_*.html`, `outputs/chart_*.png` | Interactive Folium maps + matplotlib charts |
| `06_nfirs_cause_analysis.py` | `raw_data/nfirs/*` | `outputs/cause_*.csv`, `outputs/chart_cause_*.png` | Cause/heat source/origin analysis by housing type |
| `incident_pipeline/01_download_data.py` | Public APIs | `raw_data/*` | Fetches all data; handles ArcGIS pagination for large datasets |
| `incident_pipeline/02_clean_incidents.py` | `raw_data/afd_incidents_2022_2024.csv` | `processed_data/incidents_clean.csv` | Parses dates, classifies incident types (structure, vehicle, outdoor, trash) |
| `incident_pipeline/03_create_crosswalk.py` | `raw_data/*` | `processed_data/tract_to_response_area_crosswalk.csv`, `response_area_demographics.csv`, `response_areas_with_demographics.geojson` | Area-weighted census allocation + zoning height mapping |
| `incident_pipeline/04_analysis.py` | `processed_data/*` | `outputs/summary_*.csv`, `outputs/statistical_tests.txt` | Per-capita rates, t-tests, ANOVA, correlations |
| `incident_pipeline/05_visualize.py` | `processed_data/*` | `outputs/map_*.html`, `outputs/chart_*.png` | Interactive Folium maps + matplotlib charts |
| `incident_pipeline/06_nfirs_cause_analysis.py` | `raw_data/nfirs/*` | `outputs/cause_*.csv`, `outputs/chart_cause_*.png` | Cause/heat source/origin analysis by housing type |

## Data Sources

Expand Down Expand Up @@ -157,7 +166,7 @@ raw_data/

**Zoning coverage:** 508 of 711 response areas (71%). The 203 areas without zoning data are in Austin's ETJ where city zoning does not apply.

**Zoning-to-height mapping:** Each Austin `BASE_ZONE` code is mapped to a max height (ft) and estimated max stories based on the Land Development Code. The mapping is defined in `ZONING_HEIGHT_MAP` in `03_create_crosswalk.py`. Vertical mixed-use overlays (`-V` suffix) bump the minimum to 6 stories.
**Zoning-to-height mapping:** Each Austin `BASE_ZONE` code is mapped to a max height (ft) and estimated max stories based on the Land Development Code. The mapping is defined in `ZONING_HEIGHT_MAP` in `incident_pipeline/03_create_crosswalk.py`. Vertical mixed-use overlays (`-V` suffix) bump the minimum to 6 stories.

**Validation:** Higher multifamily % correlates with higher zoning-permitted stories (r=positive). CBD zones show highest avg stories (~21), SF zones lowest (~2-3).

Expand All @@ -174,18 +183,38 @@ raw_data/

```
fire-incident-analysis/
├── 01_download_data.py # Fetch data from APIs
├── 02_clean_incidents.py # Clean & classify incidents
├── 03_create_crosswalk.py # Census → response area mapping + zoning
├── 04_analysis.py # Calculate rates & run tests
├── 05_visualize.py # Generate maps & charts
├── 06_nfirs_cause_analysis.py # NFIRS cause code & building height analysis
├── quick_start.py # Convenience script
├── run_all.py # Run full pipeline
├── incident_pipeline/ # Original AFD incident/housing analysis (run from repo root)
│ ├── 01_download_data.py # Fetch data from APIs
│ ├── 02_clean_incidents.py # Clean & classify incidents
│ ├── 03_create_crosswalk.py # Census → response area mapping + zoning
│ ├── 04_analysis.py # Calculate rates & run tests
│ ├── 05_visualize.py # Generate maps & charts
│ ├── 06_nfirs_cause_analysis.py # NFIRS cause code & building height analysis
│ ├── census_variables.py # Shared Census variable definitions
│ ├── quick_start.py # Convenience script
│ └── run_all.py # Run full pipeline
├── report_pipeline/ # Fiscal-productivity / fire-fiscal report pipeline
│ ├── 12_parse_tcad_roll.py # Parse TCAD certified appraisal roll
│ ├── 13_build_metro_parcels.py # Build metro parcel set (Travis+Williamson+Hays)
│ ├── 14_build_report_pdf.py # Render the master report to PDF
│ ├── 15_build_report_figures.py # Report figures & diagrams
│ ├── v2_county_sources.py # County data source registry (PVS ratios, citations)
│ ├── v2_fetch_parcels.py # Fetch Williamson/Hays parcels from CAD servers
│ └── viz_palette.py # Shared Austin civic color palette
├── notebooks/ # Analysis notebooks
│ ├── fiscal_productivity.ipynb # Fiscal break-even models (A: land, B: road-miles)
│ ├── fire_use_vs_pays.ipynb # Fire service use vs. pays-in by response area
│ ├── validation.ipynb # 24-check validation gate → outputs/validation_report.csv
│ ├── interactive_map.ipynb # Interactive 3D fire-fiscal map
│ ├── value_per_acre_metro.ipynb # Metro value-per-acre surface
│ └── *.ipynb # Older EDA notebooks
├── legacy/ # Orphaned scripts kept for reference (parcel join,
│ # census-tract analysis, townhome cohort analysis)
├── raw_data/ # Downloaded data (gitignored)
├── processed_data/ # Cleaned/merged data (gitignored)
├── outputs/ # Results & visualizations (gitignored)
├── docs/ # Research context, design docs
├── docs/ # Research context, reports, design docs
│ ├── FIRE_FISCAL_FULL_REPORT.md # Fire-fiscal master report
│ ├── RESEARCH_CONTEXT.md # Policy background, glossary
│ └── plans/ # Analysis design documents
├── requirements.txt # Python dependencies
Expand Down
Loading