Add Nepal Heritage Knowledge Graph and ETL pipeline

.gitignore

@@ -42,3 +42,12 @@ Eval.pdf
 # WIDOCO build artifacts (source of truth is ontology/HeritageGraph.ttl)
 ontology/HeritageGraph.merged.ttl
 ontology/HeritageGraph.build.ttl
+
+# KG build: binary triplestore + introspection cache (regenerable)
+data/fuseki/
+data/raw/shapes_spec.json
+# DANAM raw cache (non-commercial/academic source; regenerate with
+# `python scripts/kg/extract_danam.py`, optionally DANAM_MAX_PAGES=N).
+data/raw/danam/
+data/raw/danam.json
+/tmp/

.htaccess

@@ -10,7 +10,7 @@ AddType application/n-triples .n3
 AddType application/ld+json .jsonld
 # Rewrite engine setup
 RewriteEngine On
-# Served at the vocabulary namespace root (https://w3id.org/heritagegraph/).
+# Served at the vocabulary namespace root (https://cair-nepal.org/heritagegraph/).
 # If you deploy under a sub-path, change this to match (e.g. /heritagegraph).
 RewriteBase /
 

DISCOVERY_DANAM.md

@@ -0,0 +1,148 @@
+# DISCOVERY — DANAM as a HeritageGraph source (Phase 0)
+
+Status: **investigation only, no pipeline code written.** Awaiting approval + one
+licensing decision (see §4) before Phase 1.
+
+## 1. Repo conventions (summary of the pattern I must conform to)
+
+**Pipeline shape** (`scripts/kg/build_kg.sh`): `extract_*.py` → `transform.py` →
+`validate.py` → (`load_fuseki.sh` → `run_cq.py`). All paths/namespaces come from
+`scripts/kg/config.py`.
+
+- **Extractor contract** (`extract_wikidata.py`, `extract_osm.py`, `extract_unesco.py`):
+  fetch via `fetch.py` (curl + `config.USER_AGENT`), write a raw JSON cache to
+  `data/raw/<source>.json`, log counts to stderr. `extract_intangible.py` is the
+  exception — it both curates *and* emits TTL directly (hand-authored layer).
+- **Transform** (`transform.py`): per source, emits a self-contained named-graph
+  TTL to `data/kg/<source>.ttl` containing, for each entity: the typed entity,
+  an `crm:E53_Place` with `geo:asWKT`, a `crminf:I2_Belief` provenance node
+  (`prov:wasDerivedFrom`, `prov:generatedAtTime`, `hg:confidence_score`,
+  `hg:epistemic_stance`), and one `hg:DataSource` node (`add_source_node`). The
+  reverse link `asserts_about_entity` and all `owl:sameAs` / `skos:altLabel` /
+  `rdfs:seeAlso` go to a **separate** `crosswalk.ttl` so the closed shapes still
+  validate. `CONF` / `STANCE` dicts hold per-source confidence + stance.
+- **Closed-shape gating**: `allowed_for(cls)` reads `data/raw/shapes_spec.json`
+  (the set of property paths each class's `sh:closed` shape permits) and every
+  emitted property is gated through it. Where a required enum (e.g. architectural
+  style) is unknown, the entity is **typed to the nearest satisfiable super-class**
+  (e.g. `ArchitecturalStructure`) rather than inventing a value.
+  - ⚠️ `shapes_spec.json` is **git-ignored** and currently **absent**; it is
+    regenerated by `scripts/kg/shapes_introspect.py`, which `build_kg.sh` does
+    **not** currently call. Any DANAM integration must ensure it is generated
+    before `transform.py` runs (I'll add the step).
+- **Validate** (`validate.py`): unions Wikidata + UNESCO + intangible + a random
+  OSM sample, runs pyshacl with `inference="none"`, excludes crosswalk graphs.
+- **CQ answering**: `run_cq.py` / `run_all_cqs.py` already run *data-level* SELECT
+  queries against live Fuseki and return real bindings. The gap is in
+  `evaluation/run_abox_cq.py`, which loads only TBox + `kathmandu-mini-abox.ttl`
+  plus `examples/queries/abox-cq-samples.rq` — **and that `.rq` file contains only
+  prefix lines, zero queries.** That is literally why the report says
+  "0 passing SELECT queries."
+
+**Counts to flag:** `competency-questions.md` defines **CQ1–CQ32 (32 questions)**,
+and `QUALITY_REPORT.md` says "32/32 ASK". The task brief says "28 CQs / CQ01–CQ28".
+I will author **CQ01–CQ32** unless told otherwise.
+
+## 2. DANAM technical interface (probed 2026-06-28)
+
+DANAM runs **Arches v7** (CIDOC-CRM aligned). Priority-ordered findings:
+
+| Interface | Result |
+|---|---|
+| (a) Arches search API `GET /search/resources?format=json` | ✅ **Works.** Returns ES-style JSON. `results.hits.total.value = 4142` resources; `geo_aggs.doc_count = 4030` geolocated. Paginated **8 hits/page** (~518 pages). Each `_source` has: `resourceinstanceid`, `graph_id` (resource model), `root_ontology_class` (e.g. `crm:E71_Man-Made_Thing`), `displayname` (multilingual, incl. Devanagari), `displaydescription`, `points` (`{lat,lon}`), `geometries`, `map_popup`. |
+| (b) Per-resource JSON-LD `GET /resources/<uuid>` (and `?format=json-ld`) | ❌ **HTTP 500** for anonymous users — full CIDOC-CRM resource export is not publicly available. |
+| (c) sitemap / resource-id enumeration | ❌ `/sitemap.xml` 404. Enumeration is instead via the search API's paging (resourceinstanceids harvested page by page). |
+| (d) HTML scraping | Not needed — (a) is sufficient and stable. Public PDF reports exist at `/pdfs`. |
+
+**Conclusion:** extract via interface **(a)**, the search API — harvest all pages,
+caching each raw page. The available fields (label, description, coordinates,
+resource-model, root ontology class) are enough for typed, geolocated entities +
+provenance. Richer per-attribute data (precise dates, materials) is **not**
+publicly exportable, so those CQ dimensions stay DATA-GAP from DANAM.
+
+## 3. robots.txt & terms
+
+- `robots.txt`: `User-agent: * / Disallow:` — **nothing disallowed** (crawl-permitted).
+- **Citation (required):** *"Christiane Brosius and Axel Michaels (Eds.), Nepal
+  Heritage Documentation Project (https://danam.cats.uni-heidelberg.de), last
+  accessed: dd/mm/yyyy."*
+- **Copyright notice (verbatim):** *"All images in this database and in heidICON
+  are and may be only used for non-commercial documentary, academic or educational
+  purposes."*
+- No CC0/CC-BY grant is stated for the structured data; the project asserts an
+  academic, **non-commercial** framing and (EU) database context.
+
+## 4. ⚠️ Licensing wall — needs your decision before Phase 1
+
+The AWS Open Data release is **CC BY 4.0**, which *permits commercial use*. DANAM's
+stated terms are **non-commercial + citation-required**. Copying DANAM's factual
+records into a CC BY-licensed, commercial-use-OK open dataset is **not compatible**
+with those terms, and EU sui-generis database rights may attach to bulk extraction.
+This is the "licensing wall → stop and ask" case in the working agreement.
+
+Three workable resolutions (I recommend **A**):
+
+- **A — Link-only / crosswalk integration (safe, no relicensing).** Do not copy
+  DANAM facts into a redistributable graph. Mint `owl:sameAs` / `rdfs:seeAlso`
+  links from existing HeritageGraph entities to DANAM resource URIs in
+  `danam_crosswalk.ttl`, plus the DANAM `DataSource` node + citation. We redistribute
+  *links and citation*, not their database. **Limitation:** contributes little new
+  ABox detail, so it does *not* by itself close the ABox-CQ gap.
+- **B — Full DANAM graph, but quarantined under DANAM's own terms (not in the CC BY
+  bucket).** Build `data/kg/danam.ttl` for the *research* KG and CQ answering, in a
+  separate named graph labelled non-commercial/academic + required citation (the way
+  OSM is a separate ODbL file). Exclude it from the AWS public CC BY snapshot, or
+  publish it only in a clearly NC-licensed side-file. Closes the CQ gap locally;
+  keeps the public release clean.
+- **C — Seek written permission** from NHDP/Heidelberg to redistribute under stated
+  terms, then integrate fully. Highest fidelity, but blocks on an external reply.
+
+The mission's two goals interact with this: "close the ABox-CQ gap **thanks to
+DANAM**" is fully achievable only under **B or C**; under **A** the gap must be
+closed mainly from the existing sources, with DANAM adding links/citation.
+
+A second, smaller location question: in the previous task we consolidated
+`aws-open-data/heritagegraph/kg/` into `heritagegraph-core.ttl` (CC BY) +
+`heritagegraph-osm.ttl` (ODbL). The brief asks to emit `aws-open-data/.../kg/danam.ttl`.
+The canonical pipeline output is `data/kg/`; whether/where DANAM lands in the
+published snapshot depends on the §4 decision (B ⇒ its own NC side-file, not the
+CC BY core; A ⇒ only links).
+
+## 5. Proposed mapping, CONF/STANCE, and file list (pending §4)
+
+**DANAM → HeritageGraph (declared in `scripts/kg/mappings/danam.yaml`, read by the
+transform):**
+
+| DANAM resource model / signal | HeritageGraph target | Refinement |
+|---|---|---|
+| Monument (`E71_Man-Made_Thing`) | `hg:ArchitecturalStructure` | → `Temple` / `Stupa` / `Chaitya` / `DhungeDhara` / `BuddhistMonument` by DANAM monument-type term when present; else nearest super-class (`ArchitecturalStructure`) + count the demotion |
+| Art object / icon | `hg:Murti` / `hg:IconographicObject` | by object class term |
+| Inscription / text | CRM inscription/text class | — |
+| Anthropological / festival record | intangible layer (`Festival`/`RitualEvent`) | — |
+| `points {lat,lon}` | `crm:E53_Place` + `geo:asWKT` | — |
+| external ids / name+geo match | `owl:sameAs`/`seeAlso` → Wikidata/OSM/existing HG ids | → `danam_crosswalk.ttl` |
+
+**CONF / STANCE (justification):** DANAM is expert-curated by a Heidelberg academic
+project with scholarly documentation and peer review → `STANCE["danam"] = Scholarly`,
+`source_type = PublishedScholarship`, `CONF["danam"] = 0.9` (on par with Wikidata's
+scholarly 0.9; below UNESCO's authoritative 0.95; above OSM community 0.6).
+
+**Files I would add / touch in Phase 1 (under the chosen option):**
+- add `scripts/kg/extract_danam.py` (search-API harvester, paged + cached, ≤1 req/s,
+  backoff, descriptive UA)
+- add `scripts/kg/mappings/danam.yaml` (declarative mapping — the auditable artifact)
+- edit `scripts/kg/transform.py` (read the YAML; emit `danam.ttl` + `danam_crosswalk.ttl`
+  reusing `emit()`/`add_source_node()`/`allowed_for()` — no fork)
+- edit `scripts/kg/config.py` (`SOURCES["danam"]`, prefixes as needed)
+- edit `transform.py` `CONF`/`STANCE` (add `danam`)
+- edit `scripts/kg/build_kg.sh` (add the shapes_introspect step + extract_danam + note)
+- edit `scripts/kg/validate.py` (include danam.ttl in the validated union)
+- edit `scripts/kg/load_fuseki.sh` (load danam graphs)
+- add `release/evaluation/danam_mapping_audit.csv` + `.md` (generated from the YAML)
+- edit `.gitignore` (`data/raw/danam*` git-ignored per brief)
+
+**Phase 2 files:** add `scripts/kg/queries/abox/CQ01.rq … CQ32.rq`; extend
+`evaluation/run_abox_cq.py` to load the merged source graphs (or query Fuseki), run
+every SELECT, record row counts + up to 3 sample bindings, mark genuinely-unanswerable
+CQs `DATA-GAP`, and write `release/evaluation/abox_cq_report.{txt,csv}`; update
+`release/QUALITY_REPORT.md`.

aws-open-data/README.md

@@ -0,0 +1,191 @@
+# Using HeritageGraph — AWS Open Data Tutorial
+
+HeritageGraph is an RDF knowledge graph of Nepal's **tangible and intangible
+cultural heritage**: temples, stupas, monasteries, stone water spouts (*dhunge
+dhara*), UNESCO World Heritage sites, festivals, Guthi institutions, ritual
+roles, deities, and more. Every fact carries machine-readable **provenance**
+(source, confidence, epistemic stance) and links back to Wikidata or
+OpenStreetMap where available.
+
+Once published, the dataset will appear on the
+[Registry of Open Data on AWS](https://registry.opendata.aws/heritagegraph).
+Full ontology documentation lives at
+[https://cairnepal.github.io/heritagegraphontology/](https://cairnepal.github.io/heritagegraphontology/).
+Source code and releases are in the
+[CAIRNepal/heritagegraphontology](https://github.com/CAIRNepal/heritagegraphontology)
+repository on GitHub.
+
+---
+
+## Start here: guided notebook
+
+The fastest way to explore the data is the
+[**Get to Know the HeritageGraph KG**](get-to-know-the-heritagegraph-kg.ipynb)
+notebook (AWS Open Data template). It walks through bucket layout, loading Turtle
+files, entity statistics, a map of geolocated heritage, and provenance-aware
+SPARQL queries.
+
+**Run locally** (uses the bundled `heritagegraph/` data — no S3 bucket required):
+
+```bash
+git clone https://github.com/CAIRNepal/heritagegraphontology.git
+cd heritagegraphontology/aws-open-data
+pip install rdflib matplotlib boto3 jupyter
+jupyter lab get-to-know-the-heritagegraph-kg.ipynb
+```
+
+**Run against the public S3 bucket** (once provisioned): open the notebook,
+set `BUCKET` to the registry bucket name in the second code cell, and run all
+cells. Public buckets use unsigned S3 requests (no AWS account needed to read).
+
+---
+
+## Access the data
+
+### From Amazon S3
+
+After publication, files are under the `heritagegraph/` prefix, for example:
+
+```
+s3://<bucket>/heritagegraph/
+├── LICENSE.txt
+├── README.md
+├── ontology/          HeritageGraph OWL, SHACL shapes, LinkML source
+├── kg/                RDF Turtle — CC BY core + separate ODbL OSM layer
+└── examples/          SPARQL competency questions and sample results
+```
+
+Download a single file with the AWS CLI:
+
+```bash
+aws s3 cp s3://<bucket>/heritagegraph/kg/heritagegraph-core.ttl ./heritagegraph-core.ttl --no-sign-request
+```
+
+Sync the full dataset:
+
+```bash
+aws s3 sync s3://<bucket>/heritagegraph/ ./heritagegraph/ --no-sign-request
+```
+
+### From this repository
+
+Clone the repo and use the copy under `aws-open-data/heritagegraph/` — it matches
+the S3 layout exactly.
+
+---
+
+## What's in `heritagegraph/kg/`
+
+The KG ships as **two files**. The split is purely about licensing: the OSM layer
+carries OpenStreetMap's share-alike (ODbL) terms, so it is kept on its own and the
+rest is released as one graph under our CC BY 4.0.
+
+| File | License | Contents |
+|------|---------|----------|
+| `heritagegraph-core.ttl` | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | The HeritageGraph core: Wikidata-sourced facts (originally CC0), UNESCO World Heritage components, the curated intangible layer (festivals, Guthi, Kumari, deities, castes, rituals), and the crosswalk links between them. No OSM data. |
+| `heritagegraph-osm.ttl` | [ODbL](https://opendatacommons.org/licenses/odbl/1-0/) | ~7,589 geolocated physical features from OpenStreetMap (© OpenStreetMap contributors) plus the crosswalk links that reference them. |
+
+**Tip:** To avoid ODbL share-alike obligations, use only `heritagegraph-core.ttl`.
+Load `heritagegraph-osm.ttl` as well for full geographic coverage, but note any
+database that incorporates it must then be shared under ODbL. See
+`heritagegraph/LICENSE.txt` for full terms.
+
+**Scale:** ~7,850 tangible entities + a curated intangible layer (~138k triples
+in total). See `heritagegraph/examples/build_stats.json` for per-class counts.
+
+---
+
+## Quick start: Python + rdflib
+
+```python
+from rdflib import Graph
+
+g = Graph()
+# CC BY core only:
+g.parse("heritagegraph/kg/heritagegraph-core.ttl", format="turtle")
+# Add the OSM layer for full geographic coverage (adds ODbL share-alike terms):
+g.parse("heritagegraph/kg/heritagegraph-osm.ttl", format="turtle")
+
+print(f"{len(g):,} triples loaded")
+```
+
+### Example SPARQL — geolocated temples and stupas
+
+```python
+q = """
+PREFIX hg:   <https://cair-nepal.org/heritagegraph/>
+PREFIX crm:  <http://www.cidoc-crm.org/cidoc-crm/>
+PREFIX geo:  <http://www.opengis.net/ont/geosparql#>
+PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
+SELECT ?label ?wkt WHERE {
+  ?s a ?cls ; rdfs:label ?label ; crm:P55_has_current_location ?p .
+  ?p geo:asWKT ?wkt .
+  FILTER(?cls IN (hg:ArchitecturalStructure, hg:Stupa, hg:BuddhistMonument))
+} LIMIT 10
+"""
+for row in g.query(q):
+    print(row.label, "→", row.wkt)
+```
+
+More queries: `heritagegraph/examples/competency-questions.md` (32 competency
+questions with sample results in `competency-questions-data-results.txt`).
+
+---
+
+## Quick start: Amazon Neptune
+
+For production SPARQL at scale, load the Turtle files into
+[Amazon Neptune](https://aws.amazon.com/neptune/):
+
+1. Upload `heritagegraph/kg/*.ttl` to an S3 bucket Neptune can read.
+2. Use bulk load (`neptune-loader`) or the Neptune Workbench to import.
+3. Query with SPARQL 1.1 — the graph uses standard prefixes (`crm:`, `geo:`,
+   `prov:`, `hg:`).
+
+The ontology and SHACL shapes in `heritagegraph/ontology/` define the schema and
+validation rules for any data you add.
+
+Alternatives: [Apache Jena Fuseki](https://jena.apache.org/),
+[GraphDB](https://graphdb.ontotext.com/), or any RDF 1.1 triplestore.
+
+---
+
+## Namespaces
+
+| Purpose | IRI |
+|---------|-----|
+| Classes & properties | `https://cair-nepal.org/heritagegraph/` |
+| Instance identifiers | `https://data.cair-nepal.org/heritagegraph/` |
+| Source vocabulary | `https://data.cair-nepal.org/heritagegraph/source/` |
+
+---
+
+## Licensing
+
+HeritageGraph (ontology, SHACL shapes, and the `heritagegraph-core.ttl` graph)
+is released by CAIR-Nepal under **CC BY 4.0**.
+
+> Attribution: *"HeritageGraph © CAIR-Nepal, CC BY 4.0
+> (https://cair-nepal.org/heritagegraph/)"*
+
+Some underlying facts are **sourced from** third parties — Wikidata (originally
+CC0), the UNESCO World Heritage Centre, and OpenStreetMap. The OpenStreetMap layer
+is the only one carrying onward obligations: it is governed by the share-alike
+**ODbL** license and is therefore published as a separate file,
+`heritagegraph-osm.ttl` (© OpenStreetMap contributors). Any combined database that
+includes the OSM layer must comply with ODbL; `heritagegraph-core.ttl` can be used
+under CC BY 4.0 on its own. Full terms:
+[`heritagegraph/LICENSE.txt`](heritagegraph/LICENSE.txt).
+
+---
+
+## Learn more
+
+| Resource | Link |
+|----------|------|
+| Registry entry | [registry.opendata.aws/heritagegraph](https://registry.opendata.aws/heritagegraph) |
+| Ontology docs | [cairnepal.github.io/heritagegraphontology](https://cairnepal.github.io/heritagegraphontology/) |
+| GitHub repository | [github.com/CAIRNepal/heritagegraphontology](https://github.com/CAIRNepal/heritagegraphontology) |
+| Guided notebook | [get-to-know-the-heritagegraph-kg.ipynb](get-to-know-the-heritagegraph-kg.ipynb) |
+| Bucket README | [heritagegraph/README.md](heritagegraph/README.md) |
+| Contact | info@cair-nepal.org · [CAIR-Nepal](https://cair-nepal.org) |