feat(structural): genuine PMVR3 disclosure series unlocks TOKI period cohorts (measured)

Two operator-audited consecutive Park Mavera III cumulative-realized-sales disclosures (2019-10-31, 2019-11-30) added; the 2019-12-31 Level-2 case stays frozen. No new architecture; nothing fabricated.

- The existing revision/reconciliation guards derived 4 VALID period cohorts at the room-type stratum level: Oct->Nov 2+1 (7 @ 1016244.2957) and 3+1 (2 @ 1435225.00); Nov->Dec 2+1 (5 @ 1003690.00) and 4+1 (1 @ 1684074.65). 3 strata are reconciliation-blocked (delta_count=0, delta_total!=0): 5+1 Oct->Nov; 3+1 & 5+1 Nov->Dec -- no division by zero, never reinterpreted as sales. 4+1 Oct->Nov (delta=0) yields no cohort. The overall TOTAL-row difference is never used as a clean cohort.
- toki_cohort_moments moved unavailable -> AVAILABLE (toki_cohort_avg_price/count + room-type shares). dataset_status now reports reconciliation_blocked_strata.
- HONEST MEASURED RESULT: these TOKI moments are now observed but have NO simulated counterpart in the current structural model (identification_report.observed_without_simulated_counterpart) -- a model-mapping gap, not a data gap, and no TOKI/primary-market mechanism was added. So rank(J_TOKI) stays 0, rank(J_combined) stays 2, and the status is unchanged NOT_IDENTIFIED (dim 6). Snapshot updated; the before->after comparison shows the 5 newly-unlocked TOKI moments with rank 2->2.
- Level-2 real_records/toki.json unchanged; consumer path frozen; theta/optimizer/mechanism/thresholds/terminology untouched.
- 177 tests passing (genuine PMVR3 cohort values asserted exactly)

Author: onatozmenn

README.md

@@ -4,7 +4,7 @@
 [![Data refresh](https://github.com/onatozmenn/sold/actions/workflows/kfe-refresh.yml/badge.svg)](https://github.com/onatozmenn/sold/actions/workflows/kfe-refresh.yml)
 [![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-175%20passing-brightgreen.svg)](tests/)
+[![Tests](https://img.shields.io/badge/tests-177%20passing-brightgreen.svg)](tests/)
 [![Data](https://img.shields.io/badge/data-TCMB%20%C2%B7%20T%C3%9C%C4%B0K-informational.svg)](#data-sources)
 
 > Infer the **realized transaction price** of a Turkish home from its **asking** price — a provenance-aware valuation engine.
@@ -301,7 +301,7 @@ tests/               # offline unit / end-to-end tests
 ## Testing
 
 ```bash
-pytest -q             # 175 tests, fully offline (no network or API key required)
+pytest -q             # 177 tests, fully offline (no network or API key required)
 ```
 
 ## Methodology & References
@@ -327,7 +327,8 @@ Negotiation-margin figures from Turkish market reporting: İstanbul ≈ 10%, Ank
 - [x] **Structural econometric core** — mechanism-aware generalized **Nash bargaining** (`P = ηB+(1−η)S`, `η` estimated, not hard-coded) fit by **Simulated Method of Moments**; TCMB-anchored hedonic fair value (relative premiums only, no listing intercept as level); structural UYAP auctions with the **statutory legal floor** (`muhammen_bedel` preserved as appraised value `Q`, never the reserve; partially-observed floors not fabricated); KAP `η`-calibration moments with a corporate mechanism shift; TOKİ cumulative-disclosure differencing into room-type cohort moments. Replaces weak-label aggregation as the core; the provenance registry and validated KAP/TOKİ/UYAP Level-2 records are kept; the consumer path is frozen as an optional future *validation* channel. No SaleProbability model yet
 - [x] **Statutory-floor fix, TCMB double-count audit & identification diagnostics** — corrected the İİK acceptance floor to `max(0.5·Q, priority_claims) + realization_costs`; audited the fair-value anchor so a contemporaneous TL/m² level is not trend-adjusted twice (KFE ratio only when rolling an older anchor); expanded the UYAP/KAP/TOKİ structural dataset schemas + observed-moment constructors (area semantics preserved, `log(sale/appraisal)` KAP moments, revision-guarded TOKİ cohorts); added `sold structural identify` (Jacobian rank / singular values / condition number / weak directions / profile diagnostics → `NOT_IDENTIFIED` → sensitivity mode)
 - [x] **Genuine structural dataset (measured, not synthetic)** — wired the three validated Level-2 records as the genuine audited seed under [`validation/structural/`](validation/structural/) (`source_audited=true`, kept distinct from fixtures): **1 UYAP auction** (sold; appraised `Q`=4.5M → winning 4.545M; parcel/net areas distinct, gross null; partial legal floor), **1 KAP negotiated disposal** (prior-appraisal 5.2M → sale 5.508M; `value_method=negotiation`), **1 TOKİ disclosure** (PMVR3 → **0** derivable period cohorts: differencing needs ≥2 consecutive disclosures). `m_obs` is rebuilt from this genuine data — **3 single-observation means** (`uyap_sale_prob`, `uyap_win/appraisal`, `kap log(sale/appraisal)`); variances/quantiles reported **unavailable** at n=1 (never guessed). Measured result: `sold structural dataset` + `sold structural identify` → **`NOT_IDENTIFIED`** (rank 2 / dim 6, condition number →∞), sensitivity mode; a three-way `WEAKLY_IDENTIFIED` status and moment-provenance/unavailability reporting were added
-- [x] **Identification-contribution diagnostics (source-specific Jacobian + snapshots)** — `sold structural identify` reports per-source Jacobian ranks (`J_UYAP`/`J_KAP`/`J_TOKİ`/`J_combined`) restricted to each family's genuinely observed moments — answering *which source adds an independent parameter direction* (a local moment-sensitivity diagnostic, **not** causal or standalone identification) — plus a `--save-snapshot` before→after comparison (newly-unlocked moments, sample-size increases, rank / smallest-non-zero-singular-value / condition-number / status deltas). **Measured on the current genuine data: `J_UYAP` rank 1 (`uyap_sale_prob` degenerate at 1.0), `J_KAP` rank 1, `J_TOKİ` rank 0 (single disclosure, no cohort), `J_combined` rank 2 → still `NOT_IDENTIFIED` (dim 6).** Unlocking `toki_cohort_moments` / `uyap_win-over-appraisal_sd` / `kap_log_ratio_sd` requires additional **genuine audited** records (a second consecutive PMVR3 disclosure; a second sold + one unsold UYAP auction; a second KAP negotiated disposal) — operator-supplied official records; none were fabricated, so those moments stay unavailable and the status is unchanged. The moment-unlock machinery is verified with clearly-labelled fixtures (not counted in the genuine report). **Next: operator-audited genuine records → rerun identify (data, not architecture)**
+- [x] **Identification-contribution diagnostics (source-specific Jacobian + snapshots)** — `sold structural identify` reports per-source Jacobian ranks (`J_UYAP`/`J_KAP`/`J_TOKİ`/`J_combined`) restricted to each family's genuinely observed moments — answering *which source adds an independent parameter direction* (a local moment-sensitivity diagnostic, **not** causal or standalone identification) — plus a `--save-snapshot` before→after comparison (newly-unlocked moments, sample-size increases, rank / smallest-non-zero-singular-value / condition-number / status deltas). **Measured on the current genuine data: `J_UYAP` rank 1 (`uyap_sale_prob` degenerate at 1.0), `J_KAP` rank 1, `J_TOKİ` rank 0 (single disclosure, no cohort), `J_combined` rank 2 → still `NOT_IDENTIFIED` (dim 6).** Unlocking `toki_cohort_moments` / `uyap_win-over-appraisal_sd` / `kap_log_ratio_sd` requires additional **genuine audited** records (a second consecutive PMVR3 disclosure; a second sold + one unsold UYAP auction; a second KAP negotiated disposal) — operator-supplied official records; none were fabricated, so those moments stay unavailable and the status is unchanged. The moment-unlock machinery is verified with clearly-labelled fixtures (not counted in the genuine report)
+- [x] **Genuine PMVR3 disclosure series → TOKİ period cohorts unlocked** — added two operator-audited consecutive Park Mavera III cumulative-realized-sales disclosures (`2019-10-31`, `2019-11-30`; the `2019-12-31` Level-2 case frozen). The existing revision/reconciliation guards derived **4 valid period cohorts** (Oct→Nov 2+1 & 3+1, Nov→Dec 2+1 & 4+1) and **reconciliation-blocked 3 strata** with `Δcount=0, Δtotal≠0` (5+1 Oct→Nov; 3+1 & 5+1 Nov→Dec — no division by zero, never reinterpreted as sales); the overall TOTAL-row difference is never used as a clean cohort. `toki_cohort_moments` moved **unavailable → available** (5 cohort/composition moments). **Honest measured result: these TOKİ moments are now observed but have _no simulated counterpart_ in the current structural model (a model-mapping gap, not a data gap — no new mechanism was added), so `rank(J_TOKİ)` stays 0, `rank(J_combined)` stays 2, and the status is unchanged `NOT_IDENTIFIED`.** Mapping TOKİ primary-market moments to `θ` is a deliberately-deferred future structural-model extension
 - [ ] **SaleProbability** model (`P(sold ≤ N days)`) trained on collected outcomes
 - [ ] Live, ToS-reviewed fetchers for the public label sources
 - [ ] Broker-vs-benchmark analytics over an aggregate anonymized dataset

src/sold/cli.py

@@ -1513,7 +1513,8 @@ def structural_dataset_cmd() -> None:
     )
     typer.echo(
         f"        geçerli dönem kohortu {t['valid_derived_period_cohorts']} "
-        f"· revizyonla-bloklanan {t['revision_blocked_cohorts']}"
+        f"· revizyonla-bloklanan {t['revision_blocked_cohorts']} "
+        f"· mutabakatla-bloklanan strata {t.get('reconciliation_blocked_strata', 0)}"
     )
     na = st["non_audited_records"]
     typer.secho(
@@ -1618,6 +1619,12 @@ def structural_identify_cmd(
         typer.echo(f"  Moment provenance: {rep['moment_provenance']}")
     for un in rep.get("unavailable_moments", []):
         typer.echo(f"  Eksik moment: {un['moment']} [{un['source']}] — {un['reason']}")
+    owc = rep.get("observed_without_simulated_counterpart", [])
+    if owc:
+        typer.secho(
+            f"  Gözlenen ama sim KARŞILIĞI YOK (model-eşleme boşluğu; Jacobian'a girmez): {owc}",
+            fg=typer.colors.YELLOW,
+        )
     color = {
         "IDENTIFIED": typer.colors.GREEN,
         "WEAKLY_IDENTIFIED": typer.colors.YELLOW,

src/sold/structural/datasets.py

@@ -103,6 +103,7 @@ def _toki_status(disclosures: list[dict], toki_result: dict) -> dict:
         "room_type_cumulative_strata": int(strata),
         "valid_derived_period_cohorts": int(len(toki_result.get("cohorts", []))),
         "revision_blocked_cohorts": int(len(toki_result.get("revisions", []))),
+        "reconciliation_blocked_strata": int(len(toki_result.get("reconciliation", []))),
     }
 
 

src/sold/structural/identify.py

@@ -229,6 +229,10 @@ def identification_report(
     )
     dim = len(free_names)
     n_moments = len(keys)
+    # GÖZLENEN ama mevcut yapısal modelde SİMÜLE KARŞILIĞI OLMAYAN momentler (ör. TOKİ
+    # kohort momentleri): m_obs'ta VAR ama Jacobian'a GİRMEZ — çünkü simülatör bu
+    # mekanizmayı (henüz) üretmez. Bu bir MODEL-eşleme boşluğudur (veri değil), dürüstçe raporlanır.
+    observed_unmatched = sorted(k for k in (m_obs or {}) if k not in set(keys))
 
     report: dict = {
         "dataset": summary,
@@ -238,6 +242,7 @@ def identification_report(
         "moment_keys": keys,
         "moment_provenance": provenance or {},
         "unavailable_moments": unavailable or [],
+        "observed_without_simulated_counterpart": observed_unmatched,
         "sample_sizes": sample_sizes or {},
         "jacobian_step": step,
     }

tests/test_structural.py

@@ -462,9 +462,10 @@ def test_genuine_datasets_load_and_reconcile_with_level2():
     assert k["sale_price"] == 5_508_474.60 and k["appraisal_value"] == 5_200_000
     assert k["reference_price_type"] == "prior_appraisal"
     assert bool(k["negotiated"]) is True and bool(k["related_party"]) is False
-    # TOKİ: 1 disclosure → 0 dönem kohortu (tek disclosure farklanamaz)
-    assert len(g["toki_disclosures"]) == 1
-    assert g["toki_result"]["cohorts"] == []
+    # TOKİ: 3 ardışık denetlenmiş disclosure → 4 geçerli dönem kohortu (mutabakat-bloklu strata hariç)
+    assert len(g["toki_disclosures"]) == 3
+    assert len(g["toki_result"]["cohorts"]) == 4
+    assert g["toki_result"]["revision_detected"] is False
 
 
 def test_dataset_status_genuine_counts():
@@ -475,7 +476,9 @@ def test_dataset_status_genuine_counts():
     assert g["kap"]["audited_eligible_disposals"] == 1
     assert g["kap"]["negotiated_calibration_observations"] == 1
     assert g["kap"]["prior_appraisal_observations"] == 1 and g["kap"]["appraisal_observations"] == 0
-    assert g["toki"]["audited_disclosures"] == 1 and g["toki"]["valid_derived_period_cohorts"] == 0
+    assert g["toki"]["audited_disclosures"] == 3
+    assert g["toki"]["valid_derived_period_cohorts"] == 4
+    assert g["toki"]["reconciliation_blocked_strata"] == 3
     assert st["non_audited_records"]["uyap"] == 0
 
 
@@ -500,7 +503,8 @@ def test_build_observed_moments_reports_provenance_and_unavailable():
     un_moments = {u["moment"] for u in un}
     assert "uyap_win_over_appraisal_sd" in un_moments  # n=1 sold
     assert "kap_log_ratio_sd" in un_moments            # n=1 negotiated
-    assert "toki_cohort_moments" in un_moments         # 0 cohorts
+    # TOKİ kohort momenti artık AVAILABLE (3 gerçek disclosure → 4 kohort), unavailable DEĞİL
+    assert "toki_cohort_avg_price" in m and "toki_cohort_moments" not in un_moments
     assert m["kap_log_ratio_mean"] == pytest.approx(np.log(5_508_474.60 / 5_200_000), abs=1e-6)
 
 
@@ -644,3 +648,47 @@ def test_second_kap_unlocks_log_ratio_sd_fixture():
     )
     built = build_observed_moments(None, df, None)
     assert "kap_log_ratio_sd" in built["moments"]  # ≥2 müzakereli → sd AÇILDI
+
+
+# --- GERÇEK PMVR3 serisi (3 denetlenmiş disclosure) dönem kohortları ---------- #
+def test_genuine_pmvr3_series_derives_expected_cohorts():
+    g = load_genuine_datasets()
+    res = g["toki_result"]
+    assert res["revision_detected"] is False
+    cohorts = {(c["period_start"], c["period_end"], c["room_type"]): c for c in res["cohorts"]}
+    # Oct→Nov geçerli
+    c = cohorts[("2019-10-31", "2019-11-30", "2+1")]
+    assert c["cohort_count"] == 7 and c["cohort_total"] == pytest.approx(7113710.07)
+    assert c["cohort_avg"] == pytest.approx(1016244.2957142857)
+    c = cohorts[("2019-10-31", "2019-11-30", "3+1")]
+    assert c["cohort_count"] == 2 and c["cohort_avg"] == pytest.approx(1435225.00)
+    # Nov→Dec geçerli
+    c = cohorts[("2019-11-30", "2019-12-31", "2+1")]
+    assert c["cohort_count"] == 5 and c["cohort_avg"] == pytest.approx(1003690.00)
+    c = cohorts[("2019-11-30", "2019-12-31", "4+1")]
+    assert c["cohort_count"] == 1 and c["cohort_avg"] == pytest.approx(1684074.65)
+    assert len(res["cohorts"]) == 4  # 4+1 Oct→Nov (delta=0) kohort DEĞİL
+    # mutabakat-bloklu (delta_count=0 & delta_total≠0): 5+1 Oct→Nov, 3+1 & 5+1 Nov→Dec
+    recon = {(r["room_type"], r["period_end"]) for r in res["reconciliation"]}
+    assert ("5+1", "2019-11-30") in recon
+    assert ("3+1", "2019-12-31") in recon
+    assert ("5+1", "2019-12-31") in recon
+    assert len(res["reconciliation"]) == 3
+
+
+def test_genuine_toki_moment_available_but_no_sim_counterpart():
+    # TOKİ kohort momenti artık AÇIK, ama mevcut modelde sim KARŞILIĞI YOK → J_TOKI rank 0
+    g = load_genuine_datasets()
+    built = build_observed_moments(g["uyap"], g["kap"], g["toki_result"])
+    assert "toki_cohort_avg_price" in built["moments"]
+    assert built["provenance"]["toki_cohort_avg_price"] == "toki"
+    ctx = context_from_datasets(g["uyap"], g["kap"])
+    rep = identification_report(
+        ctx, StructuralParams(), DEFAULT_FREE, m_obs=built["moments"],
+        auctions=g["uyap"], kap=g["kap"], toki_result=g["toki_result"],
+        provenance=built["provenance"],
+    )
+    assert "toki_cohort_avg_price" in rep["observed_without_simulated_counterpart"]
+    assert rep["source_jacobians"]["J_TOKI"]["rank"] == 0  # sim karşılığı yok
+    assert rep["source_jacobians"]["J_combined"]["rank"] == 2  # değişmedi
+    assert rep["status"] == "NOT_IDENTIFIED"

validation/structural/identify_snapshot.json

@@ -3,6 +3,11 @@
   "n_observed_moments": 3,
   "available_moments": [
     "kap_log_ratio_mean",
+    "toki_cohort_avg_price",
+    "toki_cohort_count",
+    "toki_share_2+1",
+    "toki_share_3+1",
+    "toki_share_4+1",
     "uyap_sale_prob",
     "uyap_win_over_appraisal_mean"
   ],
@@ -12,7 +17,12 @@
     "uyap_win_over_appraisal_sd": 1,
     "kap_log_ratio_mean": 1,
     "kap_log_ratio_sd": 1,
-    "toki_cohort_moments": 0
+    "toki_cohort_moments": 4,
+    "toki_cohort_avg_price": 4,
+    "toki_cohort_count": 4,
+    "toki_share_2+1": 4,
+    "toki_share_3+1": 4,
+    "toki_share_4+1": 4
   },
   "rank": 2,
   "smallest_nonzero_singular_value": 0.5793385803552313,

validation/structural/toki.json

@@ -1,12 +1,42 @@
 [
+  {
+    "project_id": "PMVR3",
+    "as_of_date": "2019-10-31",
+    "observation_role": "cumulative_realized_sales",
+    "table_semantics": "cumulative_realized_sales_by_room_type",
+    "source_audited": true,
+    "_validation_ref": "operator-audited PMVR3 official disclosure series (2019-10-31)",
+    "_note": "GENUINE manually-audited official Park Mavera III (PMVR3) cumulative realized-sales disclosure, as-of 2019-10-31, same published disclosure series and table semantics as the frozen 2019-12-31 Level-2 record. Operator-audited official table values supplied as structured fields; the aggregate adapter normalizes them — the system does NOT claim raw-PDF extraction. Official TOTAL row (audit only, NOT used for differencing): count=372 total_price=359650908.98 total_gross_m2=67112.34 average_price=966803.52 average_m2_price=5358.94.",
+    "strata": [
+      {"room_type": "2+1", "count": 153, "total_price": 106285859.62, "total_gross_m2": 20461.86, "average_price": 694678.82, "average_m2_price": 5194.34, "cum_count": 153, "cum_total": 106285859.62},
+      {"room_type": "3+1", "count": 76, "total_price": 74899258.35, "total_gross_m2": 14202.85, "average_price": 982516.56, "average_m2_price": 5273.54, "cum_count": 76, "cum_total": 74899258.35},
+      {"room_type": "4+1", "count": 23, "total_price": 27248070.90, "total_gross_m2": 4935.96, "average_price": 1184698.73, "average_m2_price": 5520.77, "cum_count": 23, "cum_total": 27248070.90},
+      {"room_type": "5+1", "count": 120, "total_price": 151217720.11, "total_gross_m2": 27511.67, "average_price": 1260147.67, "average_m2_price": 5496.49, "cum_count": 120, "cum_total": 151217720.11}
+    ]
+  },
+  {
+    "project_id": "PMVR3",
+    "as_of_date": "2019-11-30",
+    "observation_role": "cumulative_realized_sales",
+    "table_semantics": "cumulative_realized_sales_by_room_type",
+    "source_audited": true,
+    "_validation_ref": "operator-audited PMVR3 official disclosure series (2019-11-30)",
+    "_note": "GENUINE manually-audited official Park Mavera III (PMVR3) cumulative realized-sales disclosure, as-of 2019-11-30, same published disclosure series and table semantics as the frozen 2019-12-31 Level-2 record. Operator-audited official table values supplied as structured fields; the aggregate adapter normalizes them — the system does NOT claim raw-PDF extraction. Official TOTAL row (audit only, NOT used for differencing): count=381 total_price=375610239.03 total_gross_m2=68424.69 average_price=985853.65 average_m2_price=5489.40.",
+    "strata": [
+      {"room_type": "2+1", "count": 160, "total_price": 113399569.69, "total_gross_m2": 21408.07, "average_price": 708747.31, "average_m2_price": 5297.05, "cum_count": 160, "cum_total": 113399569.69},
+      {"room_type": "3+1", "count": 78, "total_price": 77769708.35, "total_gross_m2": 14568.99, "average_price": 997047.54, "average_m2_price": 5338.03, "cum_count": 78, "cum_total": 77769708.35},
+      {"room_type": "4+1", "count": 23, "total_price": 27248070.90, "total_gross_m2": 4935.96, "average_price": 1184698.73, "average_m2_price": 5520.77, "cum_count": 23, "cum_total": 27248070.90},
+      {"room_type": "5+1", "count": 120, "total_price": 157192890.09, "total_gross_m2": 27511.67, "average_price": 1309940.75, "average_m2_price": 5713.68, "cum_count": 120, "cum_total": 157192890.09}
+    ]
+  },
   {
     "project_id": "PMVR3",
     "as_of_date": "2019-12-31",
     "observation_role": "cumulative_realized_sales",
     "table_semantics": "cumulative_realized_sales_by_room_type",
     "source_audited": true,
     "_validation_ref": "validation/real_records/toki.json",
-    "_note": "GENUINE audited TOKİ disclosure derived from validated Level-2 Park Mavera III (PMVR3) aggregate record. This is a SINGLE disclosure (one as-of date); repeated consecutive disclosures for the SAME project are required to difference cumulative strata into period cohorts, so ZERO period cohorts can be derived from this alone. The offered_inventory population is NOT paired with realized sales (different populations); only the cumulative_realized_sales strata are carried for future period differencing.",
+    "_note": "GENUINE audited TOKİ disclosure derived from validated Level-2 Park Mavera III (PMVR3) aggregate record (FROZEN Level-2 case). Now the third point in a consecutive PMVR3 cumulative-realized-sales series (with 2019-10-31 and 2019-11-30); the audited cumulative strata values remain unchanged. The offered_inventory population is NOT paired with realized sales; only cumulative_realized_sales strata are carried for period differencing.",
     "strata": [
       {"room_type": "2+1", "cum_count": 165, "cum_total": 118418019.69},
       {"room_type": "3+1", "cum_count": 78, "cum_total": 78003128.35},