feat(structural): source-specific Jacobian + snapshot identification-contribution report

Identification-targeted diagnostics over the genuine audited dataset (no new source/model/threshold/terminology; nothing fabricated).

- source_jacobian_ranks: per-family Jacobians J_UYAP/J_KAP/J_TOKI restricted to each family's genuinely OBSERVED moments + J_combined; reports numerical rank of each. Answers 'which source adds an independent parameter direction' (a LOCAL moment-sensitivity diagnostic, NOT causal or standalone identification).
- build_observed_moments now returns per-moment sample_sizes; identification_report adds source_jacobians, sample_sizes and smallest-non-zero singular value.
- snapshot before->after comparison (snapshot_metrics/save_snapshot/load_snapshot/compare_snapshots): newly-unlocked moments, sample-size increases, rank / smallest-nonzero-sv / condition-number / status deltas. CLI: sold structural identify --save-snapshot.
- MEASURED on the current genuine data (validation/structural/): J_UYAP rank 1 (uyap_sale_prob degenerate at 1.0), J_KAP rank 1, J_TOKI rank 0 (single disclosure, no cohort), J_combined rank 2 -> still NOT_IDENTIFIED (dim 6). Baseline saved to validation/structural/identify_snapshot.json.
- EPISTEMIC: no additional genuine audited records were available or fabricated, so toki_cohort_moments / uyap_win-over-appraisal_sd / kap_log_ratio_sd stay UNAVAILABLE and status is unchanged. The moment-unlock machinery is verified with clearly-labelled FIXTURES (not counted in the genuine report): a 2nd sold auction unlocks the win/appraisal sd, an unsold auction makes sale_prob non-degenerate, two consecutive PMVR3 disclosures unlock a TOKI period cohort, a 2nd KAP negotiated disposal unlocks the log-ratio sd.
- Estimation gate unchanged; theta/optimizer/mechanism/thresholds/terminology untouched; Level-2 records + consumer path frozen.
- 175 tests passing

Author: onatozmenn

README.md

@@ -4,7 +4,7 @@
 [![Data refresh](https://github.com/onatozmenn/sold/actions/workflows/kfe-refresh.yml/badge.svg)](https://github.com/onatozmenn/sold/actions/workflows/kfe-refresh.yml)
 [![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-169%20passing-brightgreen.svg)](tests/)
+[![Tests](https://img.shields.io/badge/tests-175%20passing-brightgreen.svg)](tests/)
 [![Data](https://img.shields.io/badge/data-TCMB%20%C2%B7%20T%C3%9C%C4%B0K-informational.svg)](#data-sources)
 
 > Infer the **realized transaction price** of a Turkish home from its **asking** price — a provenance-aware valuation engine.
@@ -301,7 +301,7 @@ tests/               # offline unit / end-to-end tests
 ## Testing
 
 ```bash
-pytest -q             # 169 tests, fully offline (no network or API key required)
+pytest -q             # 175 tests, fully offline (no network or API key required)
 ```
 
 ## Methodology & References
@@ -326,7 +326,8 @@ Negotiation-margin figures from Turkish market reporting: İstanbul ≈ 10%, Ank
 - [ ] **First genuine real-world label** — exactly one *actual* seller-submitted completed residential sale passing through the product path + quality gate. **Current genuine direct-label count: 0** — the end-to-end test proves the acquisition *path* works, not that a real-world label has been acquired
 - [x] **Structural econometric core** — mechanism-aware generalized **Nash bargaining** (`P = ηB+(1−η)S`, `η` estimated, not hard-coded) fit by **Simulated Method of Moments**; TCMB-anchored hedonic fair value (relative premiums only, no listing intercept as level); structural UYAP auctions with the **statutory legal floor** (`muhammen_bedel` preserved as appraised value `Q`, never the reserve; partially-observed floors not fabricated); KAP `η`-calibration moments with a corporate mechanism shift; TOKİ cumulative-disclosure differencing into room-type cohort moments. Replaces weak-label aggregation as the core; the provenance registry and validated KAP/TOKİ/UYAP Level-2 records are kept; the consumer path is frozen as an optional future *validation* channel. No SaleProbability model yet
 - [x] **Statutory-floor fix, TCMB double-count audit & identification diagnostics** — corrected the İİK acceptance floor to `max(0.5·Q, priority_claims) + realization_costs`; audited the fair-value anchor so a contemporaneous TL/m² level is not trend-adjusted twice (KFE ratio only when rolling an older anchor); expanded the UYAP/KAP/TOKİ structural dataset schemas + observed-moment constructors (area semantics preserved, `log(sale/appraisal)` KAP moments, revision-guarded TOKİ cohorts); added `sold structural identify` (Jacobian rank / singular values / condition number / weak directions / profile diagnostics → `NOT_IDENTIFIED` → sensitivity mode)
-- [x] **Genuine structural dataset (measured, not synthetic)** — wired the three validated Level-2 records as the genuine audited seed under [`validation/structural/`](validation/structural/) (`source_audited=true`, kept distinct from fixtures): **1 UYAP auction** (sold; appraised `Q`=4.5M → winning 4.545M; parcel/net areas distinct, gross null; partial legal floor), **1 KAP negotiated disposal** (prior-appraisal 5.2M → sale 5.508M; `value_method=negotiation`), **1 TOKİ disclosure** (PMVR3 → **0** derivable period cohorts: differencing needs ≥2 consecutive disclosures). `m_obs` is rebuilt from this genuine data — **3 single-observation means** (`uyap_sale_prob`, `uyap_win/appraisal`, `kap log(sale/appraisal)`); variances/quantiles reported **unavailable** at n=1 (never guessed). Measured result: `sold structural dataset` + `sold structural identify` → **`NOT_IDENTIFIED`** (rank 2 / dim 6, condition number →∞), sensitivity mode; a three-way `WEAKLY_IDENTIFIED` status and moment-provenance/unavailability reporting were added. **Next: genuine repeated TOKİ disclosures + more audited UYAP/KAP observations → rerun identify (data, not architecture)**
+- [x] **Genuine structural dataset (measured, not synthetic)** — wired the three validated Level-2 records as the genuine audited seed under [`validation/structural/`](validation/structural/) (`source_audited=true`, kept distinct from fixtures): **1 UYAP auction** (sold; appraised `Q`=4.5M → winning 4.545M; parcel/net areas distinct, gross null; partial legal floor), **1 KAP negotiated disposal** (prior-appraisal 5.2M → sale 5.508M; `value_method=negotiation`), **1 TOKİ disclosure** (PMVR3 → **0** derivable period cohorts: differencing needs ≥2 consecutive disclosures). `m_obs` is rebuilt from this genuine data — **3 single-observation means** (`uyap_sale_prob`, `uyap_win/appraisal`, `kap log(sale/appraisal)`); variances/quantiles reported **unavailable** at n=1 (never guessed). Measured result: `sold structural dataset` + `sold structural identify` → **`NOT_IDENTIFIED`** (rank 2 / dim 6, condition number →∞), sensitivity mode; a three-way `WEAKLY_IDENTIFIED` status and moment-provenance/unavailability reporting were added
+- [x] **Identification-contribution diagnostics (source-specific Jacobian + snapshots)** — `sold structural identify` reports per-source Jacobian ranks (`J_UYAP`/`J_KAP`/`J_TOKİ`/`J_combined`) restricted to each family's genuinely observed moments — answering *which source adds an independent parameter direction* (a local moment-sensitivity diagnostic, **not** causal or standalone identification) — plus a `--save-snapshot` before→after comparison (newly-unlocked moments, sample-size increases, rank / smallest-non-zero-singular-value / condition-number / status deltas). **Measured on the current genuine data: `J_UYAP` rank 1 (`uyap_sale_prob` degenerate at 1.0), `J_KAP` rank 1, `J_TOKİ` rank 0 (single disclosure, no cohort), `J_combined` rank 2 → still `NOT_IDENTIFIED` (dim 6).** Unlocking `toki_cohort_moments` / `uyap_win-over-appraisal_sd` / `kap_log_ratio_sd` requires additional **genuine audited** records (a second consecutive PMVR3 disclosure; a second sold + one unsold UYAP auction; a second KAP negotiated disposal) — operator-supplied official records; none were fabricated, so those moments stay unavailable and the status is unchanged. The moment-unlock machinery is verified with clearly-labelled fixtures (not counted in the genuine report). **Next: operator-audited genuine records → rerun identify (data, not architecture)**
 - [ ] **SaleProbability** model (`P(sold ≤ N days)`) trained on collected outcomes
 - [ ] Live, ToS-reviewed fetchers for the public label sources
 - [ ] Broker-vs-benchmark analytics over an aggregate anonymized dataset

src/sold/cli.py

@@ -1529,6 +1529,7 @@ def structural_identify_cmd(
     kap_file: Optional[Path] = typer.Option(None, "--kap", help="KAP kayıtları (JSON)"),
     toki_file: Optional[Path] = typer.Option(None, "--toki", help="TOKİ açıklamaları (JSON)"),
     demo: bool = typer.Option(False, "--demo", help="Sentetik veriyle diagnostiği göster (gerçek değil)"),
+    save_snapshot_flag: bool = typer.Option(False, "--save-snapshot", help="Mevcut sonucu genuine snapshot olarak kaydet (karşılaştırma tabanı)"),
 ) -> None:
     """Yapısal KİMLİKLENDİRME raporu — VARSAYILAN olarak GERÇEK denetlenmiş veri kümesinden.
 
@@ -1543,24 +1544,30 @@ def structural_identify_cmd(
 
     from .structural import (
         DEFAULT_FREE,
+        GENUINE_DIR,
         MomentContext,
         StructuralParams,
         build_observed_moments,
+        compare_snapshots,
         context_from_datasets,
         difference_disclosures,
         identification_report,
         load_auctions,
         load_genuine_datasets,
         load_kap_disposals,
+        load_snapshot,
         observed_moments,
+        save_snapshot,
         simulate_negotiations,
+        snapshot_metrics,
     )
 
     auctions_df = kap_df = None
     toki_res = None
     m_obs: dict = {}
     provenance: dict = {}
     unavailable: list = []
+    sample_sizes: dict = {}
 
     if demo:
         theta0 = StructuralParams(eta=0.6)
@@ -1587,12 +1594,13 @@ def structural_identify_cmd(
             auctions_df, kap_df, toki_res = genuine["uyap"], genuine["kap"], genuine["toki_result"]
         built = build_observed_moments(auctions_df, kap_df, toki_res)
         m_obs, provenance, unavailable = built["moments"], built["provenance"], built["unavailable"]
+        sample_sizes = built["sample_sizes"]
         ctx = context_from_datasets(auctions_df, kap_df)
 
     rep = identification_report(
         ctx, StructuralParams(), DEFAULT_FREE, m_obs=m_obs,
         auctions=auctions_df, kap=kap_df, toki_result=toki_res,
-        provenance=provenance, unavailable=unavailable,
+        provenance=provenance, unavailable=unavailable, sample_sizes=sample_sizes,
     )
     ds = rep["dataset"]
     typer.secho("Yapısal kimliklendirme raporu", fg=typer.colors.CYAN, bold=True)
@@ -1631,6 +1639,35 @@ def structural_identify_cmd(
         flag = "ZAYIF" if pr.get("weakly_identified") else "belirgin"
         rr_txt = f"{rr:.2e}" if isinstance(rr, (int, float)) else "—"
         typer.echo(f"  Profil {pname}: göreli hedef aralığı {rr_txt} → {flag}")
+    # Kaynağa özgü Jacobian rank (hangi kaynak BAĞIMSIZ bir parametre yönü ekliyor?)
+    sj = rep.get("source_jacobians") or {}
+    if sj:
+        typer.secho(
+            "  Kaynak-özgü Jacobian rank (yerel moment-duyarlılığı; NEDENSEL/tek-başına değil):",
+            fg=typer.colors.CYAN,
+        )
+        for fam in ("J_UYAP", "J_KAP", "J_TOKI", "J_combined"):
+            blk = sj.get(fam, {})
+            typer.echo(f"    {fam}: rank {blk.get('rank', 0)} / moment {blk.get('n_moments', 0)}")
+    # Snapshot: kaydet ya da önceki genuine snapshot ile karşılaştır
+    snap_path = Path(GENUINE_DIR) / "identify_snapshot.json"
+    if save_snapshot_flag:
+        save_snapshot(rep, snap_path)
+        typer.secho(f"  Snapshot kaydedildi: {snap_path}", fg=typer.colors.GREEN)
+    elif not demo:
+        prev = load_snapshot(snap_path)
+        if prev:
+            cmp = compare_snapshots(prev, snapshot_metrics(rep))
+            typer.secho("  Önceki genuine snapshot → mevcut (identification-katkı):", fg=typer.colors.CYAN)
+            typer.echo(f"    Yeni açılan moment: {cmp['moments_newly_unlocked'] or '—'}")
+            inc = [c['moment'] for c in cmp['moments_sample_increased']]
+            typer.echo(f"    Örneklem artan: {inc or '—'}")
+            typer.echo(f"    rank: {cmp['rank']['before']} → {cmp['rank']['after']}")
+            typer.echo(
+                f"    en küçük ≠0 tekil değer: {cmp['smallest_nonzero_singular_value']['before']} → "
+                f"{cmp['smallest_nonzero_singular_value']['after']}"
+            )
+            typer.echo(f"    durum: {cmp['status']['before']} → {cmp['status']['after']}")
     if rep["status"] != "IDENTIFIED":
         typer.secho(
             "  Not: rank < dim veya zayıf kimliklendirme → optimizer sonucu NOKTA TAHMİNİ olarak "

src/sold/structural/__init__.py

@@ -39,10 +39,15 @@
 )
 from .hedonic import HedonicPremium, roll_unit_price, tcmb_fair_value
 from .identify import (
+    compare_snapshots,
     dataset_summary,
     identification_report,
+    load_snapshot,
     moment_jacobian,
     profile_objective,
+    save_snapshot,
+    snapshot_metrics,
+    source_jacobian_ranks,
 )
 from .kap import (
     KAP_FIELDS,
@@ -114,8 +119,13 @@
     # kimliklendirme (identification)
     "moment_jacobian",
     "identification_report",
+    "source_jacobian_ranks",
     "dataset_summary",
     "profile_objective",
+    "snapshot_metrics",
+    "save_snapshot",
+    "load_snapshot",
+    "compare_snapshots",
     # tahmin
     "StructuralClosingPredictor",
 ]

src/sold/structural/identify.py

@@ -22,6 +22,9 @@
 
 from dataclasses import replace
 
+import json
+from pathlib import Path
+
 import numpy as np
 import pandas as pd
 
@@ -78,6 +81,50 @@ def msim(x: np.ndarray) -> dict:
     return J, keys
 
 
+def _rank_and_svd(J: np.ndarray) -> dict:
+    """Bir Jacobian bloğunun sayısal rank'ı + tekil değerleri + en küçük sıfır-olmayan sv."""
+    if J.size == 0 or J.shape[0] == 0:
+        return {"rank": 0, "n_moments": int(J.shape[0]), "singular_values": [], "smallest_nonzero_sv": None}
+    s = np.linalg.svd(J, compute_uv=False)
+    smax = float(s.max()) if s.size else 0.0
+    tol = max(J.shape) * np.finfo(float).eps * smax
+    nz = s[s > tol]
+    return {
+        "rank": int((s > tol).sum()),
+        "n_moments": int(J.shape[0]),
+        "singular_values": [float(x) for x in s],
+        "smallest_nonzero_sv": float(nz.min()) if nz.size else None,
+    }
+
+
+def source_jacobian_ranks(
+    params: StructuralParams,
+    ctx: MomentContext,
+    free_names: tuple[str, ...],
+    m_obs: dict,
+    provenance: dict | None = None,
+    seed: int = 12345,
+    step: float = JACOBIAN_STEP,
+) -> dict:
+    """Kaynağa özgü (J_UYAP / J_KAP / J_TOKİ) ve BİRLEŞİK Jacobian sayısal rank'ları.
+
+    Her aile YALNIZCA o ailenin katkı verdiği (gözlenen) momentlerin satırlarını kullanır.
+    Amaç: "hangi kaynak BAĞIMSIZ bir parametre yönü ekliyor?" (sadece "hangi kaynak daha
+    çok satır ekledi?" değil). NOT: kaynağa özgü rank NEDENSEL ya da TEK-BAŞINA parametre
+    kimliklendirmesi DEĞİLDİR; mevcut yapısal model altında yerel moment-duyarlılığı tanısıdır.
+    """
+    obs_keys = list(m_obs.keys()) if m_obs else None
+    J, keys = moment_jacobian(params, ctx, free_names, seed=seed, step=step, moment_keys=obs_keys)
+    prov = provenance or {}
+    out: dict = {}
+    for fam in ("uyap", "kap", "toki"):
+        rows = [i for i, k in enumerate(keys) if prov.get(k) == fam]
+        Jf = J[rows, :] if rows else np.zeros((0, len(free_names)))
+        out[f"J_{fam.upper()}"] = _rank_and_svd(Jf)
+    out["J_combined"] = _rank_and_svd(J)
+    return out
+
+
 def _weight(obs: np.ndarray) -> np.ndarray:
     return np.diag(1.0 / (np.abs(obs) + 1e-3) ** 2)
 
@@ -162,6 +209,7 @@ def identification_report(
     toki_result: dict | None = None,
     provenance: dict | None = None,
     unavailable: list | None = None,
+    sample_sizes: dict | None = None,
     seed: int = 12345,
     step: float = JACOBIAN_STEP,
 ) -> dict:
@@ -190,6 +238,7 @@ def identification_report(
         "moment_keys": keys,
         "moment_provenance": provenance or {},
         "unavailable_moments": unavailable or [],
+        "sample_sizes": sample_sizes or {},
         "jacobian_step": step,
     }
 
@@ -199,8 +248,10 @@ def identification_report(
                 "status": "NOT_IDENTIFIED",
                 "rank": 0,
                 "singular_values": [],
+                "smallest_nonzero_singular_value": None,
                 "condition_number": float("inf"),
                 "weakly_identified_directions": [],
+                "source_jacobians": {},
                 "prediction_mode": "sensitivity_mode",
                 "reason": "no_observed_moments" if n_moments == 0 else "no_free_parameters",
             }
@@ -212,6 +263,8 @@ def identification_report(
     tol = max(J.shape) * np.finfo(float).eps * smax
     rank = int((s > tol).sum())
     cond = float(smax / s.min()) if s.size and s.min() > 0 else float("inf")
+    nz = s[s > tol]
+    smallest_nonzero_sv = float(nz.min()) if nz.size else None
 
     # Zayıf yönler: küçük tekil değerlere karşılık gelen sağ-tekil vektörler
     _, s_full, vt = np.linalg.svd(J, full_matrices=False)
@@ -256,9 +309,83 @@ def identification_report(
             "status": status,
             "rank": rank,
             "singular_values": [float(x) for x in s],
+            "smallest_nonzero_singular_value": smallest_nonzero_sv,
             "condition_number": cond,
             "weakly_identified_directions": weak_dirs,
+            "source_jacobians": source_jacobian_ranks(
+                params, ctx, free_names, m_obs or {}, provenance, seed=seed, step=step
+            ),
             "prediction_mode": mode,
         }
     )
     return report
+
+
+# --------------------------------------------------------------------------- #
+# Snapshot karşılaştırması — identification-KATKI raporu (yeni batch etkisi)
+# --------------------------------------------------------------------------- #
+def snapshot_metrics(report: dict) -> dict:
+    """Bir identification raporundan snapshot metriklerini çıkarır (önceki/sonraki kıyas)."""
+    return {
+        "n_structural_parameters": report.get("n_structural_parameters"),
+        "n_observed_moments": report.get("n_observed_moments"),
+        "available_moments": sorted((report.get("moment_provenance") or {}).keys()),
+        "sample_sizes": report.get("sample_sizes", {}),
+        "rank": report.get("rank"),
+        "smallest_nonzero_singular_value": report.get("smallest_nonzero_singular_value"),
+        "condition_number": report.get("condition_number"),
+        "weak_directions": [
+            wd.get("direction") for wd in report.get("weakly_identified_directions", [])
+        ],
+        "status": report.get("status"),
+    }
+
+
+def save_snapshot(report: dict, path) -> Path:
+    path = Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(
+        json.dumps(snapshot_metrics(report), ensure_ascii=False, indent=2), encoding="utf-8"
+    )
+    return path
+
+
+def load_snapshot(path) -> dict | None:
+    path = Path(path)
+    if not path.exists():
+        return None
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def compare_snapshots(before: dict, after: dict) -> dict:
+    """Önceki genuine snapshot ile mevcut arasındaki identification-KATKI değişimi.
+
+    'Kaç satır eklendi?' değil, 'hangi moment açıldı / hangi yön güçlendi?' sorusuna yanıt.
+    """
+    b_moms = set(before.get("available_moments", []))
+    a_moms = set(after.get("available_moments", []))
+    ss_b = before.get("sample_sizes", {}) or {}
+    ss_a = after.get("sample_sizes", {}) or {}
+    increased = [
+        {"moment": k, "before": int(ss_b.get(k, 0)), "after": int(ss_a.get(k, 0))}
+        for k in sorted(set(ss_a) | set(ss_b))
+        if int(ss_a.get(k, 0)) > int(ss_b.get(k, 0))
+    ]
+    return {
+        "moments_newly_unlocked": sorted(a_moms - b_moms),
+        "moments_sample_increased": increased,
+        "rank": {"before": before.get("rank"), "after": after.get("rank")},
+        "smallest_nonzero_singular_value": {
+            "before": before.get("smallest_nonzero_singular_value"),
+            "after": after.get("smallest_nonzero_singular_value"),
+        },
+        "condition_number": {
+            "before": before.get("condition_number"),
+            "after": after.get("condition_number"),
+        },
+        "weak_directions": {
+            "before": before.get("weak_directions"),
+            "after": after.get("weak_directions"),
+        },
+        "status": {"before": before.get("status"), "after": after.get("status")},
+    }