Skip to content

Commit 3c45509

Browse files
committed
feat(uyap): inspect full June candidate set
1 parent 857cf87 commit 3c45509

9 files changed

Lines changed: 2300 additions & 19 deletions

File tree

README.md

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
[![Data refresh](https://github.com/onatozmenn/sold/actions/workflows/kfe-refresh.yml/badge.svg)](https://github.com/onatozmenn/sold/actions/workflows/kfe-refresh.yml)
55
[![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
66
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
7-
[![Tests](https://img.shields.io/badge/tests-827%20passing-brightgreen.svg)](tests/)
7+
[![Tests](https://img.shields.io/badge/tests-844%20passing-brightgreen.svg)](tests/)
88
[![Data](https://img.shields.io/badge/evidence-UYAP%20%C2%B7%20KAP%20%C2%B7%20TCMB%20%C2%B7%20TOK%C4%B0-informational.svg)](#provenance-audited-public-structural-evidence)
99

1010
> A **mechanism-aware structural econometric prototype** that infers a **structural transaction-price distribution** for a Turkish home from an **asking-price signal** and public economic evidence.
@@ -261,6 +261,35 @@ sold uyap campaign --all-provinces --phase acquire `
261261
--cdp-endpoint http://127.0.0.1:9222
262262
```
263263

264+
For large result sets, screen first and reserve UI work for genuine exceptions:
265+
266+
```powershell
267+
# Refresh privacy-safe property hints on completed discovery windows (no raw description retained)
268+
sold uyap campaign --all-provinces --phase discover `
269+
--date-from 2026-06-01 --date-to 2026-06-30 `
270+
--max-windows-per-province 5 --rescreen --concurrency 8 `
271+
--cdp-endpoint http://127.0.0.1:9222
272+
273+
# Inspect exact document manifests for residential candidates; downloads no document bytes
274+
sold uyap campaign --all-provinces --phase acquire `
275+
--date-from 2026-06-01 --date-to 2026-06-30 `
276+
--max-windows-per-province 10 --residential-only --manifest-only --concurrency 8 `
277+
--cdp-endpoint http://127.0.0.1:9222
278+
279+
# Download the unique candidate-bound sale notice and result UDFs, then extract/reconcile/audit
280+
sold uyap campaign --all-provinces --phase acquire `
281+
--date-from 2026-06-01 --date-to 2026-06-30 `
282+
--max-windows-per-province 10 --residential-only --direct --concurrency 8 `
283+
--cdp-endpoint http://127.0.0.1:9222
284+
```
285+
286+
`--direct` revalidates the candidate manifest immediately before download, validates ZIP/XML bounds,
287+
corroborates each document type semantically, writes content-addressed artifacts atomically, and runs the
288+
existing deterministic extraction, same-asset reconciliation and audit pipeline. Exact manifest-URI ODF
289+
fallback is serialized because UYAP keeps that viewer state session-current. Ambiguous/missing manifests,
290+
source error envelopes and semantic group/content mismatches remain explicit UI/manual work; no arbitrary
291+
first/last document is selected.
292+
264293
For bounded historical backfill, increase the per-province budget gradually; completed discovery and
265294
acquisition checkpoints are independent, so rerunning the command advances rather than starting over:
266295

@@ -303,6 +332,17 @@ work is never reported as complete and returns a nonzero status for automation.
303332
> adapters bind search results to exact AJAX response/cardinality and document actions to exact KAYIT,
304333
> response URI, modal row and action fingerprint.
305334
335+
> **Full June candidate inspection (2026-07-12).** The fast path screened all **6,305 newly discovered**
336+
> terminal-sale candidates across `2026-06-01..2026-06-30` without retaining raw listing descriptions.
337+
> Including 18 previously known candidates rediscovered in those windows, the inspection ledger covers
338+
> **6,323/6,323 operational candidates**: **6,066 audit-complete** and **257 `MANUAL_REQUIRED`**, with zero
339+
> incomplete inspections. Audit decisions are **3,990 `ADMISSIBLE_COMPLETED_SALE`**, 1,774 `PENDING_REVIEW`,
340+
> 136 `RECONCILIATION_FAILED`, 142 `MISSING_APPRAISAL`, 22 `DUPLICATE`, and 2
341+
> `MISSING_AUCTION_PRICE`. Manual reasons remain fail closed: 164 multi-result sets did not agree, 61 had no
342+
> result document, 24 returned source error envelopes, and 8 had manifest-group/content semantic mismatches.
343+
> No record was admitted automatically. Local gitignored evidence contains 12,173 native UDFs and 164
344+
> exact-URI ODF transformations (about 90 MiB total).
345+
306346
> **Live status.** UYAP Live Browser Pilot 1 reached **`PASS`** on the real **2026/263 Esas** record. A later bounded Ankara batch (`2026-06-18..2026-06-24`) collected five terminal-sale cards through native UDF artifacts: two new records were explicitly admitted, two were identified as existing observations (one primary ID and one alias), and one remains blocked for human review. The history endpoint is an observed internal same-origin interface, **not** an official public API. The workflow does **not** automate authentication, is **not** unattended continuous ingestion, and does **not** prove universal layout coverage. The automated test suite remains fully offline.
307347
308348
### UYAP Live Browser Pilot 1 (live verification)
@@ -517,7 +557,7 @@ tests/ # offline unit / end-to-end tests
517557
## Testing
518558

519559
```bash
520-
pytest -q # 827 tests, fully offline (no network or API key required)
560+
pytest -q # 844 tests, fully offline (no network or API key required)
521561
```
522562

523563
The automated suite is fully offline. The **UYAP live browser pilot is a separate, operator-run verification** (a real user-controlled session against the live site) and is **not** part of the offline CI suite.

docs/DEVELOPMENT_HISTORY.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -976,3 +976,34 @@ terminal `Satıldı` candidates as the prior run. This demonstrates that the ear
976976
cards under-counted source coverage; it does not alter the already completed 3/3 candidate-bound acquisitions
977977
or admit any evidence. Focused validation after the change: **120 bulk/campaign tests passed**; the live
978978
two-province benchmark completed in 0.86 seconds and the nationwide benchmark completed in 6.34 seconds.
979+
980+
## 2026-07-12 — Full June screening and direct evidence acquisition
981+
982+
The nationwide fast discovery path was applied to all five UYAP-bounded windows covering
983+
`2026-06-01..2026-06-30`. It reconciled 14,357 result cards and discovered 6,305 unique terminal `Satıldı`
984+
candidates in 128.69 seconds. A second endpoint pass classified every candidate from `malAciklama` using
985+
token-bounded, privacy-safe property hints; raw descriptions were never persisted. The resulting scope was
986+
2,049 residential candidates and 4,256 non-residential/unknown candidates.
987+
988+
Two additional bounded same-origin stages avoided thousands of serial modal interactions. Candidate-bound
989+
document manifests were scanned in 62.98 seconds: 1,939 candidates had exactly one sale notice and one result
990+
document, 107 required UI/manual fallback because result documents were multiple or missing, and 3 returned
991+
explicit source error envelopes. Direct acquisition revalidated each manifest, downloaded content-addressed
992+
native UDFs, validated ZIP/XML safety and semantic document type, then reused the existing deterministic
993+
extract/reconcile/audit pipeline. OneDrive atomic-replace contention is retried with a short bounded backoff.
994+
For portal cases where the candidate download returned the wrong type, the exact manifest-owned viewer ODF
995+
was parsed under equivalent ZIP/XML limits; opaque document URIs are stored only as SHA-256 fingerprints.
996+
Because viewer state is session-current, this fallback is serialized per candidate.
997+
998+
The same pipeline was then run across the non-residential and unknown property hints, so the final ledger is
999+
not a residential-only sample. The 6,305 newly discovered candidates plus 18 previously known candidates
1000+
rediscovered in the June windows produce **6,323/6,323 terminal inspection outcomes**: 6,066 audit-complete
1001+
and 257 `MANUAL_REQUIRED`, with zero incomplete inspections. Audit decisions are 3,990
1002+
`ADMISSIBLE_COMPLETED_SALE`, 1,774 `PENDING_REVIEW`, 136 `RECONCILIATION_FAILED`, 142
1003+
`MISSING_APPRAISAL`, 22 `DUPLICATE`, and 2 `MISSING_AUCTION_PRICE`. Manual reasons are 164 non-agreeing
1004+
multi-result sets, 61 missing result documents, 24 source error envelopes, and 8 manifest-group/content
1005+
semantic mismatches. Multi-result candidates were automated only when every result UDF reported the same
1006+
explicit `İhale Bedeli` and no non-personal asset descriptor conflicted; all result documents were retained
1007+
rather than selecting an arbitrary first or last row. Admission remained a separate human action, so genuine
1008+
structural UYAP evidence remained unchanged. Final gitignored operational evidence: 12,173 UDFs, 164 exact-URI
1009+
ODFs and approximately 90.16 MiB of artifacts.

src/sold/cli.py

Lines changed: 75 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2086,6 +2086,23 @@ def uyap_status_cmd(store_dir: Optional[str] = typer.Option(None)) -> None:
20862086
typer.echo(f" inceleme blokerleri: {s['review_blockers']} · admissible: {s['admissible']} · admitte: {s['admitted']} · dışlanan: {s['excluded_non_terminal']}")
20872087

20882088

2089+
@uyap_app.command("inspection")
2090+
def uyap_inspection_cmd(
2091+
date_from: Optional[str] = typer.Option(None, "--date-from"),
2092+
date_to: Optional[str] = typer.Option(None, "--date-to"),
2093+
store_dir: Optional[str] = typer.Option(None),
2094+
) -> None:
2095+
"""Audit ve fail-closed manuel sonuçları ayrı tutan aday inceleme defterini günceller."""
2096+
from .ingestion.uyap.bulk import finalize_inspection_statuses
2097+
2098+
result = finalize_inspection_statuses(
2099+
store_dir, date_from=date_from, date_to=date_to
2100+
)
2101+
typer.secho(f"UYAP aday inceleme sonucu: {result}", fg=typer.colors.CYAN, bold=True)
2102+
if not result["all_inspected"]:
2103+
raise typer.Exit(code=2)
2104+
2105+
20892106
@uyap_app.command("show")
20902107
def uyap_show_cmd(
20912108
file_id: Optional[str] = typer.Option(None, "--file-id", help="Dosya/Esas No ile eşleştir (ör. 2026/23)"),
@@ -2392,6 +2409,10 @@ def uyap_campaign_cmd(
23922409
max_records: Optional[int] = typer.Option(None, "--max-records", min=1, help="Acquire fazında toplam işlenecek en fazla kart"),
23932410
fast_discovery: bool = typer.Option(True, "--fast/--ui", help="Keşifte same-origin endpoint havuzu veya seri UI"),
23942411
concurrency: int = typer.Option(8, "--concurrency", min=1, max=16, help="Hızlı keşifte eşzamanlı istek sınırı"),
2412+
rescreen: bool = typer.Option(False, "--rescreen", help="Keşif metadatasını COMPLETE pencerelerde de yeniden sınıflandır"),
2413+
residential_only: bool = typer.Option(False, "--residential-only", help="Acquire kuyruğuna yalnız açık konut işaretli adayları al"),
2414+
manifest_only: bool = typer.Option(False, "--manifest-only", help="Acquire adaylarının belge manifestini hızlı endpoint ile tara; belge indirmez"),
2415+
direct_acquisition: bool = typer.Option(False, "--direct", help="Manifest-uygun adayların iki native UDF belgesini hızlı ve KAYIT-bound edin"),
23952416
dry_run: bool = typer.Option(False, "--dry-run", help="Planı göster; tarayıcıya bağlanma"),
23962417
store_dir: Optional[str] = typer.Option(None, help="Çalışma deposu / checkpoint dizini"),
23972418
genuine_path: Optional[str] = typer.Option(None, help="Genuine uyap.json yolu (yalnız duplicate denetimi)"),
@@ -2427,6 +2448,24 @@ def uyap_campaign_cmd(
24272448
typer.secho("--phase discover veya acquire olmalı", fg=typer.colors.RED)
24282449
raise typer.Exit(code=1)
24292450
campaign_phase = aliases[normalized_phase]
2451+
if rescreen and (campaign_phase != PHASE_DISCOVERY or not fast_discovery):
2452+
typer.secho("--rescreen yalnız --phase discover --fast ile kullanılabilir", fg=typer.colors.RED)
2453+
raise typer.Exit(code=1)
2454+
if residential_only and campaign_phase != PHASE_ACQUISITION:
2455+
typer.secho("--residential-only yalnız --phase acquire ile kullanılabilir", fg=typer.colors.RED)
2456+
raise typer.Exit(code=1)
2457+
if manifest_only and (campaign_phase != PHASE_ACQUISITION or not fast_discovery):
2458+
typer.secho("--manifest-only yalnız --phase acquire --fast ile kullanılabilir", fg=typer.colors.RED)
2459+
raise typer.Exit(code=1)
2460+
if direct_acquisition and (campaign_phase != PHASE_ACQUISITION or not fast_discovery):
2461+
typer.secho("--direct yalnız --phase acquire --fast ile kullanılabilir", fg=typer.colors.RED)
2462+
raise typer.Exit(code=1)
2463+
if direct_acquisition and manifest_only:
2464+
typer.secho("--direct ve --manifest-only birlikte kullanılamaz", fg=typer.colors.RED)
2465+
raise typer.Exit(code=1)
2466+
if direct_acquisition and concurrency > 8:
2467+
typer.secho("--direct için --concurrency en fazla 8 olabilir", fg=typer.colors.RED)
2468+
raise typer.Exit(code=1)
24302469
resolved_to = date_to or dt.date.today().isoformat()
24312470
resolved_from = date_from
24322471
if campaign_phase == PHASE_DISCOVERY and not resolved_from:
@@ -2450,17 +2489,20 @@ def uyap_campaign_cmd(
24502489
acquisition_scope = selected
24512490
if campaign_phase == PHASE_DISCOVERY:
24522491
full_plan = build_discovery_campaign_plan(
2453-
selected, resolved_from, resolved_to, state=campaign_state,
2492+
selected, resolved_from, resolved_to,
2493+
state={"windows": []} if rescreen else campaign_state,
24542494
newest_first=True, max_windows_per_province=max_windows_per_province,
24552495
)
24562496
plan = limit_campaign_tasks(full_plan, max_provinces)
24572497
deferred_discovery_tasks = len(full_plan) - len(plan)
24582498
else:
24592499
full_plan = build_acquisition_queue(
2460-
store_dir, selected, date_from=resolved_from, date_to=date_to
2500+
store_dir, selected, date_from=resolved_from, date_to=date_to,
2501+
residential_only=residential_only,
24612502
)
24622503
preliminary_blockers = acquisition_queue_blockers(
2463-
store_dir, selected, date_from=resolved_from, date_to=date_to
2504+
store_dir, selected, date_from=resolved_from, date_to=date_to,
2505+
residential_only=residential_only,
24642506
)
24652507
if max_provinces is not None:
24662508
work_provinces = {task["province"] for task in full_plan}
@@ -2494,14 +2536,16 @@ def uyap_campaign_cmd(
24942536
blocker_provinces,
24952537
date_from=resolved_from,
24962538
date_to=date_to,
2539+
residential_only=residential_only,
24972540
)
24982541
if campaign_phase == PHASE_ACQUISITION else []
24992542
)
25002543

25012544
discovery_mode = "fast" if fast_discovery else "ui"
2545+
property_scope = "konut" if residential_only else "tüm-taşınmaz"
25022546
typer.secho(
25032547
f"UYAP campaign · faz={campaign_phase} · aktif il={len(active_provinces)} · "
2504-
f"görev={len(plan)} · keşif={discovery_mode}",
2548+
f"görev={len(plan)} · keşif={discovery_mode} · kapsam={property_scope}",
25052549
fg=typer.colors.CYAN, bold=True,
25062550
)
25072551
by_province: dict[str, int] = {}
@@ -2540,6 +2584,30 @@ def uyap_campaign_cmd(
25402584
cdp_endpoint, store_dir=store_dir, genuine_path=genuine_path,
25412585
request_delay_ms=delay_ms, stall_seconds=stall_timeout,
25422586
)
2587+
if campaign_phase == PHASE_ACQUISITION and manifest_only:
2588+
result = collector.run_fast_manifest_scan(
2589+
plan, concurrency=concurrency, max_records=max_records
2590+
)
2591+
typer.secho(f"Belge manifest taraması: {result}", fg=typer.colors.CYAN, bold=True)
2592+
if (
2593+
result.get("stopped_reason")
2594+
or result.get("manifest_failures")
2595+
or deferred_acquisition_tasks
2596+
):
2597+
raise typer.Exit(code=2)
2598+
return
2599+
if campaign_phase == PHASE_ACQUISITION and direct_acquisition:
2600+
result = collector.run_fast_direct_acquisition(
2601+
plan, concurrency=concurrency, max_records=max_records
2602+
)
2603+
typer.secho(f"Doğrudan native edinim: {result}", fg=typer.colors.CYAN, bold=True)
2604+
if (
2605+
result.get("stopped_reason")
2606+
or result.get("retryable_failures")
2607+
or deferred_acquisition_tasks
2608+
):
2609+
raise typer.Exit(code=2)
2610+
return
25432611
totals = {
25442612
"windows": 0, "cards": 0, "acquired": 0, "failed": 0,
25452613
"incomplete": deferred_acquisition_tasks,
@@ -2564,7 +2632,9 @@ def uyap_campaign_cmd(
25642632
remaining = max_records
25652633
if campaign_phase == PHASE_DISCOVERY:
25662634
if fast_discovery:
2567-
result = collector.run_fast_discovery(plan, concurrency=concurrency)
2635+
result = collector.run_fast_discovery(
2636+
plan, concurrency=concurrency, force=rescreen
2637+
)
25682638
totals["windows"] += result["windows_processed"]
25692639
totals["cards"] += result["result_cards_inspected"]
25702640
totals["saturated_unresolved"] += result["saturated_windows_unresolved"]

0 commit comments

Comments
 (0)