feat(structural): genuine audited dataset + m_obs from real data + measured identification

Structural dataset expansion sprint over the already-validated UYAP/KAP/TOKI families (no new source/model/quality abstraction). Make the dataset REAL and MEASURE what it identifies.

- Genuine audited seed under validation/structural/ (source_audited=true), derived from the frozen Level-2 records, kept distinct from fixtures: 1 UYAP auction (sold; appraised Q=4.5M -> winning 4.545M; parcel 509 / unit-net 32.5 distinct, gross null; partial legal floor), 1 KAP negotiated disposal (prior-appraisal 5.2M -> sale 5.508M; value_method=negotiation), 1 TOKI disclosure (PMVR3 -> 0 derivable period cohorts: differencing needs >=2 consecutive disclosures).
- datasets.py: load_genuine_datasets + dataset_status; CLI 'sold structural dataset' reports genuine audited observations separately from fixtures (UYAP sold/unsold/winning/offer/bidder/exact-floor; KAP eligible/negotiated/appraisal/prior; TOKI disclosures/projects/strata/cohorts/revision-blocked).
- moments.build_observed_moments: m_obs rebuilt from genuine data with moment provenance + unavailable (reasons). Single-observation variance is UNAVAILABLE (NaN), never guessed as 0.
- KAP negotiated-calibration subset (value_method must explicitly support negotiation; non-related alone does NOT infer negotiation).
- identify: moment Jacobian restricted to genuinely OBSERVED moments (simulated-but-unobserved sd moments do not inflate rank); three-way status NOT_IDENTIFIED / WEAKLY_IDENTIFIED (full rank but ill-conditioned or flat FREE-parameter profiles) / IDENTIFIED; no observation-count threshold.
- MEASURED result on genuine data: m_obs = 3 single-observation means (uyap_sale_prob, uyap_win/appraisal, kap log(sale/appraisal)); sold structural identify -> NOT_IDENTIFIED (rank 2 / dim 6, condition number ->inf), sensitivity mode. Reported honestly.
- Level-2 (KAP 963554, TOKI PMVR3, UYAP 16766356960) unchanged; consumer path frozen; no SaleProbability, no weak supervision, no fourth source, no fabricated fields/floors/counts/cohorts/identification.
- 169 tests passing

Author: onatozmenn

README.md

@@ -4,7 +4,7 @@
 [![Data refresh](https://github.com/onatozmenn/sold/actions/workflows/kfe-refresh.yml/badge.svg)](https://github.com/onatozmenn/sold/actions/workflows/kfe-refresh.yml)
 [![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-161%20passing-brightgreen.svg)](tests/)
+[![Tests](https://img.shields.io/badge/tests-169%20passing-brightgreen.svg)](tests/)
 [![Data](https://img.shields.io/badge/data-TCMB%20%C2%B7%20T%C3%9C%C4%B0K-informational.svg)](#data-sources)
 
 > Infer the **realized transaction price** of a Turkish home from its **asking** price — a provenance-aware valuation engine.
@@ -131,7 +131,7 @@ Each public source enters as **structural moments under its own mechanism**, nev
 
 For an ordinary listing, **asking price is a noisy strategic signal of the seller reservation** (not ground truth, not a ceiling): `S` is conditioned on asking, fair value and tightness, `B`/`S` are drawn, trades (`B ≥ S`) retained, and the **conditional-on-trade** closing distribution returned — median, mean, an **80% structural interval**, trade probability, and **mechanism-transfer sensitivity**. This is a **structural inference, never an observed closing price or a measured ordinary-resale accuracy**.
 
-**Identification before estimation.** Optimizer convergence is *not* identification. `sold structural identify` inspects the actual dataset and computes a numerical moment Jacobian `J(θ)=∂m_sim/∂θ'` (central differences, common random numbers) reporting rank, singular values, condition number and weakly-identified directions, plus per-parameter profile diagnostics for `θ` and the mechanism shifts. If `rank(J) < dim(θ)` the status is **`NOT_IDENTIFIED`** and prediction runs in **sensitivity mode**. Until an identified fit exists, `sold structural value` is labelled a **structural-method prototype using provisional parameters** — not a measured ordinary-resale model. See [`src/sold/structural/`](src/sold/structural/) and `sold structural value` / `identify` / `estimate`.
+**Identification before estimation.** Optimizer convergence is *not* identification. `sold structural identify` inspects the **actual audited dataset** (`sold structural dataset` reports genuine audited observations separately from fixtures) and computes a numerical moment Jacobian `J(θ)=∂m_sim/∂θ'` (central differences, common random numbers) **restricted to the moments that are genuinely observed** — reporting available vs unavailable moments (with reasons), moment provenance by source, rank, singular values, condition number, weakly-identified directions, and per-parameter profile diagnostics. The status is three-way: `NOT_IDENTIFIED` (`rank(J) < dim(θ)`), `WEAKLY_IDENTIFIED` (full rank but severely ill-conditioned or flat profiles), or `IDENTIFIED`; the first two run prediction in **sensitivity mode**. No observation-count threshold is used as the criterion. Until an identified fit exists, `sold structural value` is labelled a **structural-method prototype using provisional parameters** — not a measured ordinary-resale model. See [`src/sold/structural/`](src/sold/structural/) and `sold structural dataset` / `identify` / `value` / `estimate`.
 
 ## Broker Data Flywheel
 
@@ -301,7 +301,7 @@ tests/               # offline unit / end-to-end tests
 ## Testing
 
 ```bash
-pytest -q             # 161 tests, fully offline (no network or API key required)
+pytest -q             # 169 tests, fully offline (no network or API key required)
 ```
 
 ## Methodology & References
@@ -325,7 +325,8 @@ Negotiation-margin figures from Turkish market reporting: İstanbul ≈ 10%, Ank
 - [x] **Direct-label quality gate (pre-ML)** — mandatory `origin` (`consumer_submission` / `test_fixture` / `demo_seed` / `manual_import`) so `asking_to_closing_labels()` **excludes test/demo by default** (opt-in `include_non_production=True`) and fixtures never inflate the genuine count; `quality_status` (`accepted`/`flagged`/`rejected`) that **hard-rejects only structurally impossible values** (non-positive price, closing-before-listing) and merely **flags** unusual ratios (extreme close-to-ask, final-above-initial, suspicious duration, duplicate) while preserving the original self-reported values; a privacy-preserving duplicate-candidate **fingerprint** (one-way SHA-256 over bucketed canonical non-personal fields that flags submissions collapsing to the **same canonical transaction fingerprint** — a canonical-fingerprint collision, **not** general near-duplicate similarity detection, and it does **not** identify a property or seller); genuine vs test/demo reported as **separate counts**
 - [ ] **First genuine real-world label** — exactly one *actual* seller-submitted completed residential sale passing through the product path + quality gate. **Current genuine direct-label count: 0** — the end-to-end test proves the acquisition *path* works, not that a real-world label has been acquired
 - [x] **Structural econometric core** — mechanism-aware generalized **Nash bargaining** (`P = ηB+(1−η)S`, `η` estimated, not hard-coded) fit by **Simulated Method of Moments**; TCMB-anchored hedonic fair value (relative premiums only, no listing intercept as level); structural UYAP auctions with the **statutory legal floor** (`muhammen_bedel` preserved as appraised value `Q`, never the reserve; partially-observed floors not fabricated); KAP `η`-calibration moments with a corporate mechanism shift; TOKİ cumulative-disclosure differencing into room-type cohort moments. Replaces weak-label aggregation as the core; the provenance registry and validated KAP/TOKİ/UYAP Level-2 records are kept; the consumer path is frozen as an optional future *validation* channel. No SaleProbability model yet
-- [x] **Statutory-floor fix, TCMB double-count audit & identification diagnostics** — corrected the İİK acceptance floor to `max(0.5·Q, priority_claims) + realization_costs`; audited the fair-value anchor so a contemporaneous TL/m² level is not trend-adjusted twice (KFE ratio only when rolling an older anchor); expanded the UYAP/KAP/TOKİ structural dataset schemas + observed-moment constructors (area semantics preserved, `log(sale/appraisal)` KAP moments, revision-guarded TOKİ cohorts); added `sold structural identify` (Jacobian rank / singular values / condition number / weak directions / profile diagnostics → `NOT_IDENTIFIED` → sensitivity mode). **Next: public structural dataset expansion → observed moments → identification → only then real SMM estimation**
+- [x] **Statutory-floor fix, TCMB double-count audit & identification diagnostics** — corrected the İİK acceptance floor to `max(0.5·Q, priority_claims) + realization_costs`; audited the fair-value anchor so a contemporaneous TL/m² level is not trend-adjusted twice (KFE ratio only when rolling an older anchor); expanded the UYAP/KAP/TOKİ structural dataset schemas + observed-moment constructors (area semantics preserved, `log(sale/appraisal)` KAP moments, revision-guarded TOKİ cohorts); added `sold structural identify` (Jacobian rank / singular values / condition number / weak directions / profile diagnostics → `NOT_IDENTIFIED` → sensitivity mode)
+- [x] **Genuine structural dataset (measured, not synthetic)** — wired the three validated Level-2 records as the genuine audited seed under [`validation/structural/`](validation/structural/) (`source_audited=true`, kept distinct from fixtures): **1 UYAP auction** (sold; appraised `Q`=4.5M → winning 4.545M; parcel/net areas distinct, gross null; partial legal floor), **1 KAP negotiated disposal** (prior-appraisal 5.2M → sale 5.508M; `value_method=negotiation`), **1 TOKİ disclosure** (PMVR3 → **0** derivable period cohorts: differencing needs ≥2 consecutive disclosures). `m_obs` is rebuilt from this genuine data — **3 single-observation means** (`uyap_sale_prob`, `uyap_win/appraisal`, `kap log(sale/appraisal)`); variances/quantiles reported **unavailable** at n=1 (never guessed). Measured result: `sold structural dataset` + `sold structural identify` → **`NOT_IDENTIFIED`** (rank 2 / dim 6, condition number →∞), sensitivity mode; a three-way `WEAKLY_IDENTIFIED` status and moment-provenance/unavailability reporting were added. **Next: genuine repeated TOKİ disclosures + more audited UYAP/KAP observations → rerun identify (data, not architecture)**
 - [ ] **SaleProbability** model (`P(sold ≤ N days)`) trained on collected outcomes
 - [ ] Live, ToS-reviewed fetchers for the public label sources
 - [ ] Broker-vs-benchmark analytics over an aggregate anonymized dataset

src/sold/cli.py

@@ -1483,17 +1483,59 @@ def structural_estimate_cmd(
     )
 
 
+@structural_app.command("dataset")
+def structural_dataset_cmd() -> None:
+    """GERÇEK denetlenmiş yapısal gözlem durumu — fixture/illustratif kayıtlardan AYRI."""
+    from .structural import dataset_status
+
+    st = dataset_status()
+    g = st["genuine"]
+    typer.secho("Yapısal veri kümesi durumu (GERÇEK denetlenmiş)", fg=typer.colors.CYAN, bold=True)
+    u = g["uyap"]
+    typer.echo(
+        f"  UYAP: denetlenmiş açık artırma {u['total_audited_auctions']} "
+        f"(satılan {u['sold']} · satılmayan {u['unsold']})"
+    )
+    typer.echo(
+        f"        kazanan teklif {u['winning_bids_observed']} · teklif sayısı {u['offer_counts_observed']} "
+        f"· artıran {u['bidder_counts_observed']} · tam yasal-taban {u['exact_legal_floors_observed']}"
+    )
+    k = g["kap"]
+    typer.echo(
+        f"  KAP: uygun elden çıkarma {k['audited_eligible_disposals']} "
+        f"(müzakere-kalibrasyon {k['negotiated_calibration_observations']}) · "
+        f"appraisal {k['appraisal_observations']} · prior-appraisal {k['prior_appraisal_observations']}"
+    )
+    t = g["toki"]
+    typer.echo(
+        f"  TOKİ: denetlenmiş açıklama {t['audited_disclosures']} · proje {t['projects_represented']} "
+        f"· oda-tipi kümülatif strata {t['room_type_cumulative_strata']}"
+    )
+    typer.echo(
+        f"        geçerli dönem kohortu {t['valid_derived_period_cohorts']} "
+        f"· revizyonla-bloklanan {t['revision_blocked_cohorts']}"
+    )
+    na = st["non_audited_records"]
+    typer.secho(
+        f"  Denetlenmemiş (fixture/illustratif, GERÇEK sayıma KATILMAZ): "
+        f"UYAP {na['uyap']} · KAP {na['kap']} · TOKİ {na['toki']}",
+        fg=typer.colors.YELLOW,
+    )
+
+
 @structural_app.command("identify")
 def structural_identify_cmd(
-    auctions_file: Optional[Path] = typer.Option(None, "--auctions", help="UYAP açık artırma kayıtları (JSON)"),
-    kap_file: Optional[Path] = typer.Option(None, "--kap", help="KAP elden çıkarma kayıtları (JSON)"),
+    auctions_file: Optional[Path] = typer.Option(None, "--auctions", help="UYAP kayıtları (JSON) — verilirse gerçek seti geçersiz kılar"),
+    kap_file: Optional[Path] = typer.Option(None, "--kap", help="KAP kayıtları (JSON)"),
     toki_file: Optional[Path] = typer.Option(None, "--toki", help="TOKİ açıklamaları (JSON)"),
-    demo: bool = typer.Option(False, "--demo", help="Sentetik veriyle diagnostiği göster"),
+    demo: bool = typer.Option(False, "--demo", help="Sentetik veriyle diagnostiği göster (gerçek değil)"),
 ) -> None:
-    """Yapısal KİMLİKLENDİRME raporu: dataset sayıları + Jacobian rank/SVD/koşul + profiller.
+    """Yapısal KİMLİKLENDİRME raporu — VARSAYILAN olarak GERÇEK denetlenmiş veri kümesinden.
 
-    Optimizer yakınsaması KİMLİKLENDİRME DEĞİLDİR. rank(J) < dim(θ) ise NOT_IDENTIFIED ve
-    tahmin sensitivity moduna geçer. Epistemik katı: veri yoksa 0 raporlanır (uydurma yok).
+    Dataset sayıları + kullanılabilir/eksik momentler (+ neden) + moment provenance +
+    Jacobian rank/SVD/koşul + zayıf yönler + eta/kayma profilleri + 3'lü durum. Optimizer
+    yakınsaması KİMLİKLENDİRME DEĞİLDİR. rank(J)<dim → NOT_IDENTIFIED; tam rank ama ağır
+    kondisyon bozukluğu/düz profil → WEAKLY_IDENTIFIED; ikisi de değilse IDENTIFIED.
     """
     import json
 
@@ -1503,20 +1545,22 @@ def structural_identify_cmd(
         DEFAULT_FREE,
         MomentContext,
         StructuralParams,
+        build_observed_moments,
         context_from_datasets,
         difference_disclosures,
         identification_report,
-        kap_observed_moments,
         load_auctions,
+        load_genuine_datasets,
         load_kap_disposals,
         observed_moments,
         simulate_negotiations,
-        uyap_observed_moments,
     )
 
     auctions_df = kap_df = None
     toki_res = None
     m_obs: dict = {}
+    provenance: dict = {}
+    unavailable: list = []
 
     if demo:
         theta0 = StructuralParams(eta=0.6)
@@ -1525,48 +1569,52 @@ def structural_identify_cmd(
         neg = simulate_negotiations(rng, V, theta0, 20000, mechanism="kap")
         tr = neg["traded"]
         m_obs = observed_moments(kap_realized=neg["price"][tr], kap_appraisal=V[tr])
+        provenance = {k: "synthetic" for k in m_obs}
         ctx = MomentContext(
             auction_appraised=np.array([]), auction_floors=np.array([]),
             kap_appraisal=np.ones(60), reps=300,
         )
     else:
-        if auctions_file:
-            auctions_df = load_auctions(
-                json.loads(Path(auctions_file).read_text(encoding="utf-8"))
-            )
-            m_obs.update(uyap_observed_moments(auctions_df))
-        if kap_file:
-            kap_df = load_kap_disposals(
-                json.loads(Path(kap_file).read_text(encoding="utf-8"))
-            )
-            m_obs.update(kap_observed_moments(kap_df))
-        if toki_file:
-            toki_res = difference_disclosures(
-                json.loads(Path(toki_file).read_text(encoding="utf-8"))
-            )
+        if auctions_file or kap_file or toki_file:
+            if auctions_file:
+                auctions_df = load_auctions(json.loads(Path(auctions_file).read_text(encoding="utf-8")))
+            if kap_file:
+                kap_df = load_kap_disposals(json.loads(Path(kap_file).read_text(encoding="utf-8")))
+            if toki_file:
+                toki_res = difference_disclosures(json.loads(Path(toki_file).read_text(encoding="utf-8")))
+        else:
+            genuine = load_genuine_datasets()  # VARSAYILAN: gerçek denetlenmiş veri
+            auctions_df, kap_df, toki_res = genuine["uyap"], genuine["kap"], genuine["toki_result"]
+        built = build_observed_moments(auctions_df, kap_df, toki_res)
+        m_obs, provenance, unavailable = built["moments"], built["provenance"], built["unavailable"]
         ctx = context_from_datasets(auctions_df, kap_df)
 
     rep = identification_report(
         ctx, StructuralParams(), DEFAULT_FREE, m_obs=m_obs,
         auctions=auctions_df, kap=kap_df, toki_result=toki_res,
+        provenance=provenance, unavailable=unavailable,
     )
     ds = rep["dataset"]
     typer.secho("Yapısal kimliklendirme raporu", fg=typer.colors.CYAN, bold=True)
     typer.echo(
-        f"  UYAP: toplam {ds['uyap_total']} · satılan {ds['uyap_sold']} · "
-        f"satılmayan {ds['uyap_unsold']}"
-    )
-    typer.echo(
-        f"        teklif gözlemli {ds['uyap_offer_count_observed']} · artıran gözlemli "
-        f"{ds['uyap_bidder_count_observed']} · tam yasal-taban {ds['uyap_exact_legal_floor_observed']}"
+        f"  UYAP: toplam {ds['uyap_total']} · satılan {ds['uyap_sold']} · satılmayan {ds['uyap_unsold']} "
+        f"· teklif göz. {ds['uyap_offer_count_observed']} · artıran {ds['uyap_bidder_count_observed']} "
+        f"· tam-taban {ds['uyap_exact_legal_floor_observed']}"
     )
     typer.echo(f"  KAP müzakereli elden çıkarma: {ds['kap_negotiated_disposals']}")
-    typer.echo(f"  TOKİ geçerli proje-dönem strata: {ds['toki_valid_project_period_strata']}")
+    typer.echo(f"  TOKİ geçerli proje-dönem kohortu: {ds['toki_valid_project_period_strata']}")
     typer.echo(
-        f"  Yapısal parametre (dim θ): {rep['n_structural_parameters']} · "
-        f"gözlenen moment: {rep['n_observed_moments']}"
+        f"  dim(θ): {rep['n_structural_parameters']} · kullanılabilir moment: {rep['n_observed_moments']}"
     )
-    color = typer.colors.GREEN if rep["status"] == "IDENTIFIED" else typer.colors.RED
+    if rep.get("moment_provenance"):
+        typer.echo(f"  Moment provenance: {rep['moment_provenance']}")
+    for un in rep.get("unavailable_moments", []):
+        typer.echo(f"  Eksik moment: {un['moment']} [{un['source']}] — {un['reason']}")
+    color = {
+        "IDENTIFIED": typer.colors.GREEN,
+        "WEAKLY_IDENTIFIED": typer.colors.YELLOW,
+        "NOT_IDENTIFIED": typer.colors.RED,
+    }.get(rep["status"], typer.colors.RED)
     typer.secho(
         f"  DURUM: {rep['status']}  (rank {rep['rank']} / dim {rep['n_structural_parameters']}) "
         f"· mod: {rep['prediction_mode']}",
@@ -1577,16 +1625,17 @@ def structural_identify_cmd(
         sv = ", ".join(f"{x:.2e}" for x in rep["singular_values"])
         typer.echo(f"  Tekil değerler: [{sv}]  ·  koşul sayısı: {rep['condition_number']:.2e}")
     for wd in rep.get("weakly_identified_directions", []):
-        typer.echo(f"  Zayıf-kimliklendirilmiş yön (s={wd['singular_value']:.2e}): {wd['direction']}")
+        typer.echo(f"  Zayıf yön (s={wd['singular_value']:.2e}): {wd['direction']}")
     for pname, pr in (rep.get("profiles") or {}).items():
         rr = pr.get("relative_range")
         flag = "ZAYIF" if pr.get("weakly_identified") else "belirgin"
         rr_txt = f"{rr:.2e}" if isinstance(rr, (int, float)) else "—"
         typer.echo(f"  Profil {pname}: göreli hedef aralığı {rr_txt} → {flag}")
     if rep["status"] != "IDENTIFIED":
         typer.secho(
-            "  Not: rank < dim → optimizer sonucu NOKTA TAHMİNİ olarak SUNULMAZ; sensitivity "
-            "mode. (Optimizer yakınsaması ≠ kimliklendirme.) Sonraki iş: veri kümesi genişletmesi.",
+            "  Not: rank < dim veya zayıf kimliklendirme → optimizer sonucu NOKTA TAHMİNİ olarak "
+            "SUNULMAZ; sensitivity/prototip mode. (Optimizer yakınsaması ≠ kimliklendirme.) "
+            "Sonraki iş: gerçek UYAP/KAP/tekrarlı TOKİ gözlem genişletmesi.",
             fg=typer.colors.YELLOW,
         )
 

src/sold/structural/__init__.py

@@ -29,6 +29,14 @@
     simulate_negotiations,
     trade_mask,
 )
+from .datasets import (
+    GENUINE_DIR,
+    dataset_status,
+    load_genuine_datasets,
+    load_kap_records,
+    load_toki_records,
+    load_uyap_records,
+)
 from .hedonic import HedonicPremium, roll_unit_price, tcmb_fair_value
 from .identify import (
     dataset_summary,
@@ -45,6 +53,7 @@
 from .moments import (
     MomentContext,
     align,
+    build_observed_moments,
     context_from_datasets,
     observed_moments,
     simulated_moments,
@@ -88,12 +97,20 @@
     "MomentContext",
     "observed_moments",
     "simulated_moments",
+    "build_observed_moments",
     "context_from_datasets",
     "align",
     "estimate_smm",
     "SMMResult",
     "smm_objective",
     "nelder_mead",
+    # gerçek veri kümesi (denetlenmiş)
+    "load_genuine_datasets",
+    "dataset_status",
+    "load_uyap_records",
+    "load_kap_records",
+    "load_toki_records",
+    "GENUINE_DIR",
     # kimliklendirme (identification)
     "moment_jacobian",
     "identification_report",

src/sold/structural/auction.py

@@ -180,7 +180,7 @@ def auction_moments(sold, win_over_appraisal) -> dict:
     return {
         "uyap_sale_prob": sale_prob,
         "uyap_win_over_appraisal_mean": float(ratios.mean()) if ratios.size else float("nan"),
-        "uyap_win_over_appraisal_sd": float(ratios.std()) if ratios.size > 1 else 0.0,
+        "uyap_win_over_appraisal_sd": float(ratios.std()) if ratios.size > 1 else float("nan"),
     }