Skip to content

architecture.adoc

Latest commit

 

History

History
349 lines (263 loc) · 12.3 KB

architecture.adoc

File metadata and controls

349 lines (263 loc) · 12.3 KB
Raw

ZooMap Architecture

Table of Contents

1. Context

ZooMap is a web application that lets zoo visitors browse and locate animals on an interactive map. A data pipeline periodically harvests animal POI data from open data sources, enriches it, and produces a static GeoJSON file that the frontend consumes directly.

1.1. System Context Diagram

graph TD
    subgraph External Sources
        OSM_API["Overpass API\n(OpenStreetMap)"]
        WD_API["Wikidata API"]
        WP_API["Wikipedia REST API"]
        OSM_TILES["OSM Tile Server"]
    end

    subgraph ZooMap
        PIPELINE["Pipeline\n(Python + dbt + DuckDB)"]
        APP["Frontend App\n(HTML / JS / Leaflet)"]
        GEOJSON["data/animals.geojson\n(static file)"]
    end

    VISITOR["Zoo Visitor\n(Browser)"]

    OSM_API -->|"POI nodes, ways & relations"| PIPELINE
    WD_API  -->|"Species names & images"| PIPELINE
    WP_API  -->|"Article summaries & thumbnails"| PIPELINE
    PIPELINE -->|"writes"| GEOJSON
    GEOJSON  -->|"fetched at startup"| APP
    OSM_TILES -->|"background tiles"| APP
    APP -->|"interactive map & list"| VISITOR
Loading

2. Building Block View

2.1. Pipeline Components

The pipeline follows a medallion architecture (Bronze → Silver → Gold) implemented with Python scripts and dbt models backed by an embedded DuckDB database.

Component Location Responsibility

download_osm.py

pipeline/scripts/

Queries the Overpass API for POI nodes, ways, and relations within the configured bounding box. Caches raw JSON responses, reconstructs Polygon and MultiPolygon geometries from relation member ways, computes centroids for multi-point geometries, and loads results into the raw_osm_elements table in DuckDB (Bronze layer).

bronze_osm_pois.sql

pipeline/dbt/models/bronze/

dbt model that casts raw JSON fields to typed columns, normalizing the flat OSM tag map into a structured table.

silver_pois.sql

pipeline/dbt/models/silver/

dbt model that filters Bronze records (keeping only attraction=animal tags), extracts well-known OSM keys (name, species, wikidata, wikipedia), and produces a clean, deduplicated POI table.

enrich_wiki.py

pipeline/scripts/

Reads QIDs and Wikipedia titles from the Silver layer, calls Wikidata and Wikipedia REST APIs, caches individual entity responses, and writes enrichment data to the wiki_enrichment table. When the OSM species:wikidata tag contains multiple semicolon-separated QIDs, one enrichment record is produced per QID so that each species in a shared enclosure gets its own GeoJSON feature.

gold_animals.sql

pipeline/dbt/models/gold/

dbt model that joins Silver POIs with enrichment data, coalesces display names and images, constructs Wikipedia URLs, and produces the final gold_animals table. Because wiki_enrichment may contain multiple records per OSM element (one per species), the join naturally expands to one output row per species, each sharing the same geometry. The enclosure column always carries the raw OSM name tag to remain stable across multi-species expansions.

export_geojson.py

pipeline/scripts/

Queries gold_animals from DuckDB, converts rows to GeoJSON Features (preserving both point and polygon geometries), validates the schema, and writes data/animals.geojson.

relevant_tags.csv

pipeline/dbt/seeds/

Seed table that lists the OSM tag key/value pairs considered relevant for filtering.

2.2. Frontend Components

Component Location Responsibility

index.html

project root

Shell page; defines the two-column layout (animal list + map container) and loads all scripts.

app.js

project root

Fetches data/animals.geojson at startup, renders the Leaflet map with OSM tiles, populates the animal list, handles full-text search (including enclosure name), and synchronises list selection with map highlights. Multiple features sharing the same geometry (multi-species enclosures) each receive their own list entry; MultiPolygon geometries are handled natively by Leaflet.

style.css

project root

Flexbox-based layout and visual styling for the two-panel interface.

data/animals.geojson

project root

Static GeoJSON FeatureCollection generated by the pipeline; serves as the application’s database.

2.3. GeoJSON Data Model

data/animals.geojson is a RFC 7946 FeatureCollection. Each element in the features array represents one animal enclosure.

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": { ... },   // Point or Polygon
      "properties": { ... }  // see table below
    }
  ]
}

Geometry

The geometry field is one of three GeoJSON geometry types:

Type Usage

Point

Single coordinate [longitude, latitude]; used when the OSM source element is a node.

Polygon

Closed ring of coordinates; used when the OSM source element is a way or a relation with a single outer ring. The pipeline computes the centroid for display purposes but preserves the full polygon. Inner rings (holes) are included when present.

MultiPolygon

Multiple outer rings; used when an OSM relation contains more than one outer-role member way (e.g. a non-contiguous enclosure spread across two areas).

Properties

Field Type Required Description

id

string

✓

URL-safe identifier derived from the enclosure name (e.g. "african-penguin").

name

string

✓

Human-readable display name of the animal or enclosure (e.g. "African Penguin").

enclosure

string

✓

Raw OSM name tag of the enclosure element. For multi-species enclosures this value is the same across all species features sharing that enclosure, while name holds the individual species common name.

species

string

—

Scientific species name(s) sourced from the OSM species tag (e.g. "Spheniscus demersus"). Multiple species are separated by a semicolon. null when the tag is absent.

description

string

—

Short article extract fetched from the Wikipedia REST API. null when no Wikipedia link is available.

wikipedia

string (URL)

—

Full URL to the English Wikipedia article (e.g. "https://en.wikipedia.org/wiki/African_penguin"). null when no Wikidata link is found.

image

string (URL)

—

Wikimedia Commons thumbnail URL with ?width=320 appended. null when no image is available on Wikidata.

3. Runtime View

3.1. Data Ingestion Flow

The pipeline is triggered by GitHub Actions (weekly schedule, push to main, or manual dispatch) and runs sequentially through four stages.

flowchart TD
    GHA(["GitHub Actions\ntrigger"])

    subgraph Stage1["① OSM Ingestion"]
        PY1["download_osm.py"]
        OSM[/"Overpass API"/]
        BRONZE[("DuckDB\nraw_osm_elements\n[Bronze]")]
        PY1 -->|"Overpass QL query\n(bbox, osm_tags)"| OSM
        OSM -->|"raw JSON\n(nodes / ways / relations)"| PY1
        PY1 -->|"INSERT"| BRONZE
    end

    subgraph Stage2["② Bronze → Silver (dbt)"]
        DBT_BS["dbt run"]
        SILVER[("DuckDB\nsilver_pois\n[Silver]")]
        DBT_BS -->|"cast & type"| DBT_BS
        DBT_BS -->|"filter & normalize"| SILVER
    end

    subgraph Stage3["③ Wiki Enrichment"]
        PY2["enrich_wiki.py"]
        WD[/"Wikidata /\nWikipedia APIs"/]
        ENRICH[("DuckDB\nwiki_enrichment")]
        PY2 -->|"entity lookup (QIDs)"| WD
        WD  -->|"taxon names, image filenames"| PY2
        PY2 -->|"summary lookup (titles)"| WD
        WD  -->|"article summaries & thumbnails"| PY2
        PY2 -->|"INSERT"| ENRICH
    end

    subgraph Stage4["④ Gold + Export (dbt + Python)"]
        DBT_G["dbt run"]
        GOLD[("DuckDB\ngold_animals\n[Gold]")]
        PY3["export_geojson.py"]
        GEOJSON[/"data/animals.geojson"/]
        DBT_G -->|"join & coalesce"| GOLD
        GOLD  -->|"SELECT *"| PY3
        PY3   -->|"write + git commit & push"| GEOJSON
    end

    GHA --> Stage1
    BRONZE --> Stage2
    SILVER --> Stage3
    ENRICH --> Stage4
Loading

3.2. Medallion Layers Summary

Layer Storage Content

Bronze

DuckDB raw_osm_elements

Unmodified OSM data with typed columns; all original POIs retained.

Silver

DuckDB silver_pois

Filtered and normalized POIs; only animal attractions with extracted tag columns.

Enrichment

DuckDB wiki_enrichment

Species metadata fetched from Wikidata and Wikipedia.

Gold

DuckDB gold_animals

Fully enriched, display-ready records with resolved names, descriptions, and image URLs.

Export

data/animals.geojson

GeoJSON FeatureCollection consumed by the frontend; the single source of truth for the app.

4. Architectural Decision Records

4.1. ADR-001: Static GeoJSON as Application Database

Status: Accepted

Context: The app needs animal POI data at runtime. Options range from a full backend API to a static file.

Decision: Serve a pre-generated data/animals.geojson file directly from the repository; the frontend fetches it once on startup.

Alternatives Considered: - Backend REST API – Requires a server, adds infrastructure complexity, overkill for a small, infrequently changing dataset. - Hosted database (Postgres/Supabase) – Same drawbacks; introduces network latency and costs.

Consequences: - () Zero server infrastructure; the app is fully static and can be hosted anywhere. - () Extremely fast startup; the browser caches the file. - (−) Data freshness is limited to pipeline run frequency (weekly by default). - (−) Entire dataset is downloaded even if the user views only a few animals.

4.2. ADR-002: DuckDB as Pipeline Database

Status: Accepted

Context: The pipeline needs an SQL engine to run dbt transformations without a running database server.

Decision: Use DuckDB as an embedded, file-based database throughout the pipeline.

Alternatives Considered: - SQLite – No native dbt adapter; limited analytical SQL capabilities (e.g., no UNNEST). - PostgreSQL – Requires a running server and credentials; heavy for a single-user pipeline. - Pandas DataFrames – Loses the dbt transformation layer and version-controlled SQL models.

Consequences: - () No external service to provision or maintain. - () Full SQL dialect including JSON functions, UNNEST, and window functions. - (+) dbt-duckdb provides a first-class adapter. - (−) Concurrent write access is limited; not suitable if multiple pipeline runs overlap.

4.3. ADR-003: dbt for Data Transformations

Status: Accepted

Context: The pipeline transforms raw OSM data through multiple stages. This logic could live in Python scripts or in SQL managed by a transformation framework.

Decision: Use dbt to define and run all data transformation models (Bronze → Silver → Gold).

Alternatives Considered: - Pure Python (pandas/polars) – Transformations would be harder to test, document, and version independently from orchestration code. - Plain SQL scripts – Possible, but dbt adds lineage tracking, seed management, and a standard project layout.

Consequences: - () Transformation logic is declarative SQL, easy to read and audit. - () dbt seeds manage the relevant_tags lookup table cleanly. - (+) Clear separation of concerns between ingestion (Python) and transformation (dbt). - (−) Adds a dependency and a learning curve for contributors unfamiliar with dbt.

4.4. ADR-004: Vanilla JavaScript Frontend

Status: Accepted

Context: The frontend needs to display an animal list and an interactive map. The project aims for the simplest possible implementation.

Decision: Use plain HTML, CSS, and JavaScript with Leaflet.js as the only runtime dependency.

Alternatives Considered: - React / Vue / Svelte – Would add a build step, larger bundle size, and framework complexity for a UI that has minimal interactivity. - Server-side rendering – Requires a server; contradicts the static-hosting goal.

Consequences: - () No build tooling required; the app can be opened directly in a browser. - () Minimal bundle size; fast load times. - (−) No component model; scaling to a richer UI would require manual DOM management. - (−) No type checking; refactoring is riskier as the codebase grows.