Skip to content

Commit 349fe41

Browse files
CopilotPapaBravo
Copilot
and
PapaBravo
authored
docs: add architecture.adoc covering context, building blocks, runtime view, and ADRs (#9)
* docs: add architecture.adoc and link from README Agent-Logs-Url: https://github.com/PapaBravo/zoomap/sessions/a441ff5d-d5a2-475b-a94c-6b3ac66843bc Co-authored-by: PapaBravo <2863907+PapaBravo@users.noreply.github.com> * docs: fix spelling and README link syntax per review Agent-Logs-Url: https://github.com/PapaBravo/zoomap/sessions/a441ff5d-d5a2-475b-a94c-6b3ac66843bc Co-authored-by: PapaBravo <2863907+PapaBravo@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: PapaBravo <2863907+PapaBravo@users.noreply.github.com>
1 parent ccc3947 commit 349fe41

2 files changed

Lines changed: 264 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ Features in descending priority are
99
* Full Text search for animals in list
1010
* Selecting animals in list to highlight on map
1111

12+
# Documentation
13+
14+
For a detailed description of the system design see [Architecture](docs/architecture.adoc).
15+
1216
# Plan
1317
* Simplest possible frontend (no react or similar)
1418
* Using osm poi data

‎docs/architecture.adoc‎

Lines changed: 260 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,260 @@
1+
= ZooMap Architecture
2+
:toc: left
3+
:toclevels: 3
4+
:sectnums:
5+
6+
== Context
7+
8+
ZooMap is a web application that lets zoo visitors browse and locate animals on an interactive map.
9+
A data pipeline periodically harvests animal POI data from open data sources, enriches it, and produces a static GeoJSON file that the frontend consumes directly.
10+
11+
=== System Context Diagram
12+
13+
[source,mermaid]
14+
----
15+
graph TD
16+
subgraph External Sources
17+
OSM_API["Overpass API\n(OpenStreetMap)"]
18+
WD_API["Wikidata API"]
19+
WP_API["Wikipedia REST API"]
20+
OSM_TILES["OSM Tile Server"]
21+
end
22+
23+
subgraph ZooMap
24+
PIPELINE["Pipeline\n(Python + dbt + DuckDB)"]
25+
APP["Frontend App\n(HTML / JS / Leaflet)"]
26+
GEOJSON["data/animals.geojson\n(static file)"]
27+
end
28+
29+
VISITOR["Zoo Visitor\n(Browser)"]
30+
31+
OSM_API -->|"POI nodes, ways & relations"| PIPELINE
32+
WD_API -->|"Species names & images"| PIPELINE
33+
WP_API -->|"Article summaries & thumbnails"| PIPELINE
34+
PIPELINE -->|"writes"| GEOJSON
35+
GEOJSON -->|"fetched at startup"| APP
36+
OSM_TILES -->|"background tiles"| APP
37+
APP -->|"interactive map & list"| VISITOR
38+
----
39+
40+
== Building Block View
41+
42+
=== Pipeline Components
43+
44+
The pipeline follows a *medallion architecture* (Bronze → Silver → Gold) implemented with Python scripts and dbt models backed by an embedded DuckDB database.
45+
46+
[cols="1,2,3", options="header"]
47+
|===
48+
| Component | Location | Responsibility
49+
50+
| `download_osm.py`
51+
| `pipeline/scripts/`
52+
| Queries the Overpass API for POI nodes, ways, and relations within the configured bounding box. Caches raw JSON responses, computes centroids for multi-point geometries, and loads results into the `raw_osm_elements` table in DuckDB (Bronze layer).
53+
54+
| `bronze_osm_pois.sql`
55+
| `pipeline/dbt/models/bronze/`
56+
| dbt model that casts raw JSON fields to typed columns, normalizing the flat OSM tag map into a structured table.
57+
58+
| `silver_pois.sql`
59+
| `pipeline/dbt/models/silver/`
60+
| dbt model that filters Bronze records (keeping only `attraction=animal` tags), extracts well-known OSM keys (`name`, `species`, `wikidata`, `wikipedia`), and produces a clean, deduplicated POI table.
61+
62+
| `enrich_wiki.py`
63+
| `pipeline/scripts/`
64+
| Reads QIDs and Wikipedia titles from the Silver layer, calls Wikidata and Wikipedia REST APIs, caches individual entity responses, and writes enrichment data to the `wiki_enrichment` table.
65+
66+
| `gold_animals.sql`
67+
| `pipeline/dbt/models/gold/`
68+
| dbt model that joins Silver POIs with enrichment data, coalesces display names and images, constructs Wikipedia URLs, and produces the final `gold_animals` table.
69+
70+
| `export_geojson.py`
71+
| `pipeline/scripts/`
72+
| Queries `gold_animals` from DuckDB, converts rows to GeoJSON Features (preserving both point and polygon geometries), validates the schema, and writes `data/animals.geojson`.
73+
74+
| `relevant_tags.csv`
75+
| `pipeline/dbt/seeds/`
76+
| Seed table that lists the OSM tag key/value pairs considered relevant for filtering.
77+
|===
78+
79+
=== Frontend Components
80+
81+
[cols="1,2,3", options="header"]
82+
|===
83+
| Component | Location | Responsibility
84+
85+
| `index.html`
86+
| project root
87+
| Shell page; defines the two-column layout (animal list + map container) and loads all scripts.
88+
89+
| `app.js`
90+
| project root
91+
| Fetches `data/animals.geojson` at startup, renders the Leaflet map with OSM tiles, populates the animal list, handles full-text search, and synchronises list selection with map highlights.
92+
93+
| `style.css`
94+
| project root
95+
| Flexbox-based layout and visual styling for the two-panel interface.
96+
97+
| `data/animals.geojson`
98+
| project root
99+
| Static GeoJSON FeatureCollection generated by the pipeline; serves as the application's database.
100+
|===
101+
102+
== Runtime View
103+
104+
=== Data Ingestion Flow
105+
106+
The pipeline is triggered by GitHub Actions (weekly schedule, push to `main`, or manual dispatch) and runs sequentially through four stages.
107+
108+
[source,mermaid]
109+
----
110+
sequenceDiagram
111+
participant GHA as GitHub Actions
112+
participant OSM as Overpass API
113+
participant PY1 as download_osm.py
114+
participant DBT as dbt (DuckDB)
115+
participant WD as Wikidata / Wikipedia APIs
116+
participant PY2 as enrich_wiki.py
117+
participant PY3 as export_geojson.py
118+
participant REPO as Git Repository
119+
120+
GHA ->> PY1 : run (bbox, osm_tags)
121+
PY1 ->> OSM : Overpass QL query
122+
OSM -->> PY1 : raw JSON (nodes/ways/relations)
123+
PY1 ->> DBT : INSERT INTO raw_osm_elements
124+
125+
GHA ->> DBT : dbt run (bronze → silver)
126+
DBT ->> DBT : bronze_osm_pois (cast & type)
127+
DBT ->> DBT : silver_pois (filter & normalise)
128+
129+
GHA ->> PY2 : run (wikidata_lang)
130+
PY2 ->> WD : Wikidata entity lookup (QIDs)
131+
WD -->> PY2 : taxon names, image filenames
132+
PY2 ->> WD : Wikipedia summary lookup (titles)
133+
WD -->> PY2 : article summaries & thumbnails
134+
PY2 ->> DBT : INSERT INTO wiki_enrichment
135+
136+
GHA ->> DBT : dbt run (gold)
137+
DBT ->> DBT : gold_animals (join & coalesce)
138+
139+
GHA ->> PY3 : run
140+
PY3 ->> DBT : SELECT * FROM gold_animals
141+
DBT -->> PY3 : result rows
142+
PY3 ->> REPO : write data/animals.geojson
143+
REPO ->> REPO : git commit & push
144+
----
145+
146+
=== Medallion Layers Summary
147+
148+
[cols="1,1,3", options="header"]
149+
|===
150+
| Layer | Storage | Content
151+
152+
| *Bronze*
153+
| DuckDB `raw_osm_elements`
154+
| Unmodified OSM data with typed columns; all original POIs retained.
155+
156+
| *Silver*
157+
| DuckDB `silver_pois`
158+
| Filtered and normalized POIs; only animal attractions with extracted tag columns.
159+
160+
| *Enrichment*
161+
| DuckDB `wiki_enrichment`
162+
| Species metadata fetched from Wikidata and Wikipedia.
163+
164+
| *Gold*
165+
| DuckDB `gold_animals`
166+
| Fully enriched, display-ready records with resolved names, descriptions, and image URLs.
167+
168+
| *Export*
169+
| `data/animals.geojson`
170+
| GeoJSON FeatureCollection consumed by the frontend; the single source of truth for the app.
171+
|===
172+
173+
== Architectural Decision Records
174+
175+
=== ADR-001: Static GeoJSON as Application Database
176+
177+
*Status*: Accepted
178+
179+
*Context*:
180+
The app needs animal POI data at runtime. Options range from a full backend API to a static file.
181+
182+
*Decision*:
183+
Serve a pre-generated `data/animals.geojson` file directly from the repository; the frontend fetches it once on startup.
184+
185+
*Alternatives Considered*:
186+
- *Backend REST API* – Requires a server, adds infrastructure complexity, overkill for a small, infrequently changing dataset.
187+
- *Hosted database (Postgres/Supabase)* – Same drawbacks; introduces network latency and costs.
188+
189+
*Consequences*:
190+
- (+) Zero server infrastructure; the app is fully static and can be hosted anywhere.
191+
- (+) Extremely fast startup; the browser caches the file.
192+
- (−) Data freshness is limited to pipeline run frequency (weekly by default).
193+
- (−) Entire dataset is downloaded even if the user views only a few animals.
194+
195+
---
196+
197+
=== ADR-002: DuckDB as Pipeline Database
198+
199+
*Status*: Accepted
200+
201+
*Context*:
202+
The pipeline needs an SQL engine to run dbt transformations without a running database server.
203+
204+
*Decision*:
205+
Use DuckDB as an embedded, file-based database throughout the pipeline.
206+
207+
*Alternatives Considered*:
208+
- *SQLite* – No native dbt adapter; limited analytical SQL capabilities (e.g., no UNNEST).
209+
- *PostgreSQL* – Requires a running server and credentials; heavy for a single-user pipeline.
210+
- *Pandas DataFrames* – Loses the dbt transformation layer and version-controlled SQL models.
211+
212+
*Consequences*:
213+
- (+) No external service to provision or maintain.
214+
- (+) Full SQL dialect including JSON functions, UNNEST, and window functions.
215+
- (+) dbt-duckdb provides a first-class adapter.
216+
- (−) Concurrent write access is limited; not suitable if multiple pipeline runs overlap.
217+
218+
---
219+
220+
=== ADR-003: dbt for Data Transformations
221+
222+
*Status*: Accepted
223+
224+
*Context*:
225+
The pipeline transforms raw OSM data through multiple stages. This logic could live in Python scripts or in SQL managed by a transformation framework.
226+
227+
*Decision*:
228+
Use dbt to define and run all data transformation models (Bronze → Silver → Gold).
229+
230+
*Alternatives Considered*:
231+
- *Pure Python (pandas/polars)* – Transformations would be harder to test, document, and version independently from orchestration code.
232+
- *Plain SQL scripts* – Possible, but dbt adds lineage tracking, seed management, and a standard project layout.
233+
234+
*Consequences*:
235+
- (+) Transformation logic is declarative SQL, easy to read and audit.
236+
- (+) dbt seeds manage the `relevant_tags` lookup table cleanly.
237+
- (+) Clear separation of concerns between ingestion (Python) and transformation (dbt).
238+
- (−) Adds a dependency and a learning curve for contributors unfamiliar with dbt.
239+
240+
---
241+
242+
=== ADR-004: Vanilla JavaScript Frontend
243+
244+
*Status*: Accepted
245+
246+
*Context*:
247+
The frontend needs to display an animal list and an interactive map. The project aims for the simplest possible implementation.
248+
249+
*Decision*:
250+
Use plain HTML, CSS, and JavaScript with Leaflet.js as the only runtime dependency.
251+
252+
*Alternatives Considered*:
253+
- *React / Vue / Svelte* – Would add a build step, larger bundle size, and framework complexity for a UI that has minimal interactivity.
254+
- *Server-side rendering* – Requires a server; contradicts the static-hosting goal.
255+
256+
*Consequences*:
257+
- (+) No build tooling required; the app can be opened directly in a browser.
258+
- (+) Minimal bundle size; fast load times.
259+
- (−) No component model; scaling to a richer UI would require manual DOM management.
260+
- (−) No type checking; refactoring is riskier as the codebase grows.

0 commit comments

Comments
 (0)