blocksense-v3.6.1-engineering.md

Urban Dossier NYC v3.6.1 - Complete Backend Engineering Document (Pensar-Hardened)

Target reader: backend engineer / Codex / Claude Code / team manager

Date: 2026-04-11 Hackathon: NVIDIA Spark Hack NYC 2026 Hardware target: Acer Veriton GN100 / DGX Spark / NVIDIA GB10 Grace Blackwell / 128GB unified memory Runtime target: 100% local, offline-capable Team scope owned by us: backend analysis engine, overview precompute service, detail realtime analysis, data-provider abstraction, evidence, brief generation, batch/watchlist, backend-only smoke tests Not our scope: final frontend implementation, final dataset selection, final Skill implementation details owned by teammates

---

Project Overview & Scoring Alignment

Urban Dossier NYC v3.6.1 is a fully local civic analysis backend designed for two user-visible product states and one batch mode:

1. a whole-map overview state
2. a clicked-point detail state
3. a batch watchlist state

The backend is not just a data display layer. It is designed to:

• expose city-scale overview layers
• recompute local priorities after a click
• explain why those priorities rose to the top
• attach traceable evidence
• produce a concise local brief
• pass a real Pensar/Apex security review without hiding or disabling the real product surface

What we already have

• local data assets in the workspace
• a legacy direct-query backend path that already works on parquet/demo assets
• real remote machine access and validation over SSH
• RAPIDS benchmark evidence on the official machine
• a frontend prototype already oriented around overview + click + detail

What v3.6 merges

v3.6 is the merged version of:

• v3
  • deterministic trend / priority / watchlist code skeletons
  • action-first reporting
  • explicit evidence verification
• v3.5.2
  • correct overview/detail product split
  • detail-only user ranking
• v3.5.3
  • Skill interface preserved
  • direct-query fallback made first-class
  • remote SSH reality incorporated
• v3.6.1
  • Pensar/Apex hardening requirements
  • no-cheating security gate
  • security-focused API constraints for the prize path

How this maps to the judging rubric

Technical Execution & Completeness

• overview precompute + detail realtime + batch reuse
• provider abstraction for Skill vs direct mode
• deterministic ranking and evidence
• frontend-ready API contract

NVIDIA Ecosystem & Spark Utility

• accommodates NemoClaw / OpenClaw / Nemotron
• RAPIDS/cuDF for overview precompute and heavy aggregations
• DuckDB/parquet for local realtime filtering
• remote benchmark path already proven

Value & Impact

• overview helps users orient at map scale
• detail gives local prioritized issues after click
• user preference changes actual local ranking
• watchlist supports civic monitoring workflows

Frontier Factor

• same city data consumed in overview, detail, and watchlist modes
• overview is fixed and fast; detail is personalized and realtime
• provider abstraction keeps the system runnable before Skill is ready

Least Likely to get Hacked

• the same real backend must be testable by Pensar/Apex
• no fake-safe test mode is allowed
• every future backend change must pass a standing security gate

---

0. Constraints And Assumptions

0.1 Confirmed constraints

1. All AI inference and core data processing must run locally.
2. Primary data source must be NYC Open Data.
3. The demo target environment is the Acer GN100 / DGX Spark class machine.
4. We own backend only.
5. Frontend and final dataset selection are handled by teammates.
6. The official rules include a bounty for "Least Likely to get Hacked" that explicitly requires running Pensar and fixing the issues it surfaces.

0.2 Working assumptions

1. NemoClaw / OpenClaw / a local model runtime will likely exist on the official machine, but backend must degrade cleanly if not.
2. Skill outputs may exist later, but must not be a blocker for backend testing now.
3. The first frontend release wants:
   • one overall overview layer
   • one single-category layer per button
   • click-to-detail
   • detail-only drag ranking
4. The prize-evaluation path should prefer direct mode until SkillDataProvider is hardened and tested under the same security gate.

0.3 Explicit non-goals

This document does not finalize:

• the UI presentation details
• the final list of all datasets
• the final Skill implementation internals
• the final deck phrasing
• any "special test-only" behavior that differs from the real backend exposed to the frontend

This document is about the backend system we should actually build.

---

1. Ground Truth We Must Design Around

1.1 Team split

Current split:

• backend: us
• frontend interaction: teammates
• final dataset curation / Skill: teammates
• benchmark work: teammates

Therefore, the backend must:

• expose stable contracts
• be runnable without waiting on other workstreams
• preserve future integration paths

1.2 Local workspace reality

The local workspace already contains:

• engineering docs from previous iterations
• a legacy urban_dossier backend prototype
• local processed/parquet/demo assets
• older score-first report logic
• 311, EMS, and Fire raw/parquet assets already present locally

That means we already have enough material to test backend logic now.

1.3 Remote SSH reality

The official-machine remote environment already has:

• ~/nyc_open_data with real NYC data
• parquet conversion scripts
• a minimal DuckDB pipeline
• RAPIDS benchmark scripts
• a lightweight ~/urban_dossier experimental directory

But it does not yet have a complete service implementing:

• overview API
• detail API
• trend engine
• priority engine
• watchlist API
• frontend-ready contract

1.4 Remote smoke-test reality

We already confirmed over SSH that local-detail-style real data queries are feasible against real remote assets.

Examples already observed in rough smoke tests near a Manhattan point:

• non-empty collision counts
• non-empty rodent counts
• non-empty 311 counts
• non-empty public toilet nearby results

That proves we are not blocked on backend feasibility.

---

2. Product Definition

2.1 What Urban Dossier v3.6.1 is

Urban Dossier is a fully local civic analysis backend with three consumption modes:

1. Overview mode
   • whole-map, pre-rendered, fixed layers
2. Detail mode
   • clicked-point, radius-based, realtime, user-priority-aware
3. Watchlist mode
   • batch reuse of detail logic with fixed platform ordering

2.2 What it is not

It is not:

• a simple score dashboard
• a single-address due diligence tool
• a frontend-owned ranking system
• a Skill-dependent prototype that cannot run without teammates finishing their part

2.3 Primary user experience

1. User opens the map
2. User sees an overall map or one category map
3. User clicks a point
4. User enters detail mode
5. User may reorder categories
6. Backend recomputes local priorities in realtime
7. UI shows local actions, why-now signals, evidence, and a brief

---

3. Core Product Interpretation

3.1 Overview state

Overview should only support:

• one overall map
• one single-category map per visible category button

Overview should not:

• accept user drag order
• recompute custom weighted citywide maps for each user ranking
• emit fully personalized local recommendations

Overview is a navigation layer.

3.2 Detail state

Detail is where user preference matters.

After a click:

• frontend sends priority_order
• backend converts order to weights
• backend recomputes local priority actions in realtime

This is where the product earns most of its value.

3.3 Watchlist state

Watchlist is a batch consumer of the same analysis core.

It should:

• not use per-user ordering
• use a fixed platform/default priority configuration
• reuse the same detail logic on many seeds / areas / cells

---

4. Backend Principles

4.1 Deterministic first

All judgment should be deterministic wherever possible:

• retrieval
• trends
• ranking
• pattern detection
• evidence checks

The LLM should only organize the final expression.

4.2 Same API regardless of source mode

Frontend should not care whether backend is currently using:

• Skill-curated inputs
• direct hardcoded retrieval

4.3 Shared core, multiple consumers

Overview, detail, and watchlist should reuse one shared analysis core rather than each inventing separate business logic.

4.4 Honest degradation

If a dataset or provider is missing:

• return coverage gaps
• return empty cells or empty sections honestly
• do not fabricate results
• do not replace missing data with reassuring scores

4.5 Security is a first-class contract

The backend must not treat security as a post-hoc polish task.

At minimum:

• heavy endpoints must be bounded
• overview category selection must be whitelisted
• local-only model endpoints must be allowlisted
• cross-origin access must be restricted to known frontend origins
• the same core backend used in the demo must also be the backend tested by Pensar

4.6 No cheating rule

For the Pensar/Apex prize path, the backend is not allowed to:

• disable risky endpoints only during the security scan while keeping them enabled in the demo
• swap in fake static responses only for pentest runs
• detect Pensar/Apex and return safer behavior only to that tool
• hide watchlist/detail functionality from the scanner if those features exist in the product
• claim Skill mode is secure if the award path actually runs only in direct mode

---

5. High-Level Architecture

                     +----------------------------------+
                     |             API Layer            |
                     | overview / detail / coverage /   |
                     | categories / watchlist / health  |
                     +----------------+-----------------+
                                      |
                                      v
                     +----------------------------------+
                     |       Shared Analysis Core       |
                     | current state / trends /         |
                     | patterns / priority / evidence / |
                     | scoring / report                 |
                     +----------------+-----------------+
                                      |
                     +----------------+-----------------+
                     |       Data Provider Layer        |
                     +-----------+----------------------+
                                 |
                  +--------------+--------------+
                  |                             |
                  v                             v
     +---------------------------+  +---------------------------+
     | SkillDataProvider         |  | DirectQueryDataProvider   |
     | future preferred path     |  | current test default      |
     +---------------------------+  +---------------------------+

Key consequence:

• product behavior lives above the provider layer
• source-specific logic lives below it

---

6. Runtime Modes

6.1 Data mode env var

Add:

URBAN_DOSSIER_DATA_MODE = auto | skill | direct

6.2 Meaning

• direct
  • use hardcoded direct data retrieval only
  • current default for testing
• skill
  • require Skill outputs
  • fail clearly if unavailable
• auto
  • use Skill if available
  • otherwise fall back to direct query

6.3 Current default

For current backend testing:

URBAN_DOSSIER_DATA_MODE=direct

6.4 Future recommended default

When Skill is stable:

URBAN_DOSSIER_DATA_MODE=auto

---

7. System Modes

7.1 Overview mode

Purpose:

• initial map load
• button-based switching between overall and single-category layers

Properties:

• pre-rendered
• broad
• fast
• not user-ranked
• no long-form local recommendation logic

7.2 Detail mode

Purpose:

• clicked-point analysis
• realtime local ranking

Properties:

• point + radius input
• drag-order aware
• action-first output
• evidence-backed
• optional brief generation

7.3 Watchlist mode

Purpose:

• overnight / scheduled / manual batch analysis

Properties:

• fixed default platform ordering
• batch seeds or areas
• reuses detail logic

---

8. Category Configuration

The backend must be config-driven.

Do not hardcode category names inside ranking logic.

Recommended first-release config shape:

categories:
  safety:
    label: "Safety"
    map_driving: true
    detail_rankable: true
    signals:
      - collisions
      - rodent
      - ems_response
      - fire_dispatch
      - sanitation_311

  traffic:
    label: "Traffic"
    map_driving: true
    detail_rankable: true
    signals:
      - collisions
      - transit_access
      - bike_routes
      - open_streets

  facilities:
    label: "Facilities"
    map_driving: true
    detail_rankable: true
    signals:
      - parks_access
      - public_toilets
      - linknyc
      - civic_facilities
      - restaurant_context

8.1 Why config-driven matters

This lets the frontend later:

• rename categories
• merge or split categories
• choose which are visible as overview buttons

without forcing us to rewrite the ranking engine.

---

9. Frontend-Facing Interpretation

9.1 Overview request model

Overview requests should accept:

• view_mode = overall | category
• category_id
• viewport
• zoom
• optional render_mode

Overview should not accept priority_order.

9.2 Detail request model

Detail requests should accept:

• latitude
• longitude
• radius_m
• priority_order
• time_window_days

This is the only place where drag order should matter.

9.3 Why this split is correct

This is the only split that simultaneously gives:

• frontend clarity
• manageable overview precompute cost
• meaningful local personalization

---

10. Data Provider Contract

Everything above the provider layer should consume the same interface.

10.1 Required provider methods

class DataProvider:
    def get_overview_layer(self, view_mode: str, category_id: str | None, viewport: dict | None, zoom: int | None) -> dict: ...
    def get_point_signals(self, latitude: float, longitude: float, radius_m: int, time_window_days: int) -> dict: ...
    def get_local_timeseries(self, latitude: float, longitude: float, radius_m: int, time_window_days: int) -> dict: ...
    def get_baselines(self) -> dict: ...
    def get_context_items(self, latitude: float, longitude: float, radius_m: int) -> dict: ...
    def get_coverage(self) -> dict: ...

10.2 Why this is the right abstraction

If this contract stays stable:

• frontend API stays stable
• detail logic stays stable
• Skill arrival later only changes the provider implementation

---

11. Provider Implementations

11.1 SkillDataProvider

This is the preferred future path.

It should read from:

• standardized signal tables
• precomputed overview layers
• area timeseries
• baselines
• manifest files

It assumes teammate-produced Skill outputs are ready.

11.2 DirectQueryDataProvider

This is the currently enabled test path.

It should directly read from:

• processed parquet
• cached precomputed JSON
• cached reports
• local CSVs when unavoidable
• demo fixtures only as honest fallback

This provider should be built by refactoring the older backend retrieval logic, not by inventing a second ad hoc app.

11.3 Provider get_coverage() contract

Every provider's get_coverage() must include provider_ready: bool. The auto-mode selector uses this field to decide whether to use the provider or fall back:

# SkillDataProvider
def get_coverage(self):
    return {"provider_ready": False, ...}  # until Skill outputs are ready

# DirectQueryDataProvider
def get_coverage(self):
    return {"provider_ready": True, ...}  # always operational

11.4 Important rule

Direct mode is not temporary trash code.

It is the operational fallback path for:

• current testing
• teammate delays
• on-site failures

---

12. Standardized Data Contract

The future Skill/data side should still target standardized outputs.

12.1 Required tables

1. point_events
2. area_timeseries
3. building_signals
4. context_places
5. baselines
6. manifest

12.2 Minimum schemas

point_events

• source_dataset
• event_type
• subtype
• latitude
• longitude
• h3_r8
• h3_r9
• event_date
• borough
• zip
• optional bbl
• optional bin

h3_r10 is optional, not required in first release.

area_timeseries

• source_dataset
• metric_name
• area_type
• area_id
• period_start
• period_end
• period_label
• metric_value

building_signals

• source_dataset
• bbl
• bin
• latitude
• longitude
• status_date
• signal_type
• signal_value

context_places

• source_dataset
• place_type
• latitude
• longitude
• h3_r8
• borough
• zip

baselines

• signal_name
• area_type
• area_scope
• p25
• p50
• p75
• updated_at

manifest

• dataset freshness
• coverage notes
• missing-field warnings
• source-to-standard mapping

12.3 Direct-mode compatibility rule

Direct mode is allowed to assemble these shapes from whatever local assets exist, as long as the shapes exposed upward are stable.

---

13. Overview Layer Design

13.1 Supported overview outputs

Overview should only expose:

• overall
• one layer per single category button

For a first release with three categories:

• overall
• safety
• traffic
• facilities

13.2 What overview should not do

Overview should not:

• use user drag order
• produce priority_actions
• generate local narrative conclusions
• call the LLM on map-switch interactions

13.3 Preferred output unit

Preferred first implementation:

• H3 cells at resolution r8

13.4 Overview artifacts

Recommended precompute outputs:

• overview_overall_h3_r8.parquet
• overview_safety_h3_r8.parquet
• overview_traffic_h3_r8.parquet
• overview_facilities_h3_r8.parquet
• overview_manifest.json

13.4.1 How overview_overall should be computed

overview_overall should not be guessed ad hoc at implementation time.

For the first release, compute it as a fixed platform-weighted combination of category layers:

OVERVIEW_DEFAULT_WEIGHTS = {
    "safety": 0.40,
    "traffic": 0.30,
    "facilities": 0.30,
}


def build_overall_overview(cell_scores_by_category):
    """
    cell_scores_by_category shape:
    {
        "892a1072d6fffff": {"safety": 0.82, "traffic": 0.55, "facilities": 0.61},
        ...
    }
    """
    overall = {}
    for h3_cell, categories in cell_scores_by_category.items():
        weighted_sum = 0.0
        total_weight = 0.0
        for category_id, weight in OVERVIEW_DEFAULT_WEIGHTS.items():
            if category_id in categories and categories[category_id] is not None:
                weighted_sum += categories[category_id] * weight
                total_weight += weight
        overall[h3_cell] = None if total_weight == 0 else round(weighted_sum / total_weight, 4)
    return overall

Important rule:

• overview overall uses fixed platform weights
• detail ranking uses user weights
• these two systems should stay separate

13.5 Overview fallback

If overview layers are missing, /api/overview must still return a valid response:

{
  "schema_version": "v3.6",
  "mode": "overview",
  "view_mode": "overall",
  "category_id": null,
  "cells": [],
  "coverage": {
    "overview_ready": false,
    "missing_categories": ["overall", "safety", "traffic", "facilities"]
  },
  "ui_message": "Overview not yet available. Click a point for realtime detail analysis.",
  "data_mode": "direct"
}

---

14. Detail Layer Design

14.1 Detail is the core product value

Detail mode should:

1. resolve clicked point
2. query local signals in radius
3. query local or area-linked timeseries
4. compute trends
5. detect patterns
6. apply ranking weights
7. generate local action priorities
8. build evidence and gaps
9. optionally generate a brief

14.2 Detail input

Required:

• latitude
• longitude
• radius_m
• priority_order

Optional:

• time_window_days
• category_subset

14.3 Detail output

{
  "schema_version": "v3.6",
  "mode": "detail",
  "data_mode": "direct",
  "target": {},
  "priority_profile": {
    "order": ["safety", "traffic", "facilities"],
    "weights": {
      "safety": 1.0,
      "traffic": 0.72,
      "facilities": 0.52
    }
  },
  "priority_actions": [],
  "why_now": [],
  "current_state": {},
  "detail_items": {
    "map_points": [],
    "nearby_facilities": [],
    "building_flags": [],
    "recent_incidents": []
  },
  "trends": {},
  "patterns": [],
  "evidence_table": [],
  "data_gaps": [],
  "scores": {},
  "report_summary": "",
  "report_markdown": ""
}

14.4 Meaning of detail_items

detail_items is for concrete UI objects, not for aggregate reasoning.

Suggested subgroups:

• map_points
• nearby_facilities
• building_flags
• recent_incidents

Difference from current_state:

• current_state = aggregate metrics
• detail_items = concrete rows / objects displayed in UI

---

15. User Preference Model

15.1 Where preference matters

Only detail mode should use user drag order.

15.2 Weight conversion

Use a simple exponential decay:

weight(rank) = decay ^ (rank - 1)

Recommended default:

decay = 0.72

Approximate weights:

• rank 1 -> 1.00
• rank 2 -> 0.72
• rank 3 -> 0.52
• rank 4 -> 0.37
• rank 5 -> 0.27

15.3 What weights affect

Weights should affect:

• local priority ranking
• tie-breaking
• report emphasis

Weights should not affect:

• raw retrieval counts
• evidence truth
• overview layer selection

---

16. Data Priorities

16.1 Core detail-driving signals

These should drive the first ranking version:

• 311
• collisions
• rodent
• EMS
• Fire
• housing violations
• AEP
• PLUTO / HPD building anchors

16.2 Context signals

These enrich current_state and detail_items, but should not dominate ranking:

• parks
• public toilets
• LinkNYC
• civic facilities
• restaurant context
• bike routes / open streets if available

16.3 Overview-driving signals

Overview should prefer signals that are:

• broad in coverage
• aggregatable
• stable enough for map rendering

---

17. Trend Engine

Detail ranking needs trend signals, even if overview is simpler.

Each major signal should compute:

• recent_delta
• seasonal_delta
• baseline_gap
• persistence

17.1 Reference implementation

def compute_recent_delta(values_by_period):
    recent = values_by_period.get("last_30d", 0)
    previous = values_by_period.get("prev_30d", 0)
    if previous == 0:
        return {"change_pct": None, "label": "no_baseline"}
    pct = (recent - previous) / previous * 100
    return {
        "change_pct": round(pct, 1),
        "label": "rising" if pct > 10 else "falling" if pct < -10 else "stable",
    }


def compute_seasonal_delta(values_by_period):
    current = values_by_period.get("last_90d", 0)
    same_last_year = values_by_period.get("same_90d_last_year", 0)
    if same_last_year == 0:
        return {"change_pct": None, "label": "no_baseline"}
    pct = (current - same_last_year) / same_last_year * 100
    return {
        "change_pct": round(pct, 1),
        "label": "rising" if pct > 15 else "falling" if pct < -15 else "stable",
    }


def compute_baseline_gap(current_value, baseline_dist):
    if current_value is None:
        return {"gap_pct": None, "percentile_label": "unknown"}
    p50 = baseline_dist.get("p50", current_value)
    if p50 == 0:
        return {"gap_pct": None, "percentile_label": "unknown"}
    gap = (current_value - p50) / p50 * 100
    if current_value <= baseline_dist.get("p25", 0):
        label = "below_average"
    elif current_value <= baseline_dist.get("p50", 0):
        label = "average"
    elif current_value <= baseline_dist.get("p75", 0):
        label = "above_average"
    else:
        label = "high"
    return {"gap_pct": round(gap, 1), "percentile_label": label}


def compute_persistence(values_by_period, threshold=None):
    periods = values_by_period.get("quarterly_values", [])
    if not periods or threshold is None:
        return {"consecutive_above": 0}
    count = 0
    for v in reversed(periods):
        if v > threshold:
            count += 1
        else:
            break
    return {"consecutive_above": count}

17.2 Direct-mode note

In direct mode, timeseries may be assembled more roughly from available local tables.

That is acceptable for testing if:

• schema is stable
• evidence is honest
• caveats are explicit

---

18. Pattern Detection

Pattern detection should be rule-based and conservative.

Recommended first rules:

• sanitation complaints rising + rodent rising
• collision pressure rising + EMS response worsening
• housing violations unresolved + AEP/building risk present

18.1 Reference implementation

def detect_patterns(trends):
    patterns = []

    sanitation = trends.get("sanitation_311", {})
    rodent = trends.get("rodent", {})
    collision = trends.get("collision", {})
    ems = trends.get("ems_response", {})
    housing = trends.get("housing_violations", {})
    aep = trends.get("aep", {})

    if sanitation.get("direction") == "worsening" and rodent.get("direction") == "worsening":
        patterns.append({
            "pattern_id": "sanitation_rodent_pressure",
            "label": "Sanitation and rodent pressure are worsening together",
            "severity": "high",
            "evidence_ids": ["trend_311_sanitation", "trend_rodent"],
        })

    if collision.get("direction") == "worsening" and ems.get("direction") == "worsening":
        patterns.append({
            "pattern_id": "safety_emergency_pressure",
            "label": "Street safety pressure and emergency response pressure are worsening together",
            "severity": "high",
            "evidence_ids": ["trend_collision", "trend_ems_response"],
        })

    if housing.get("direction") == "worsening" and aep.get("direction") in {"worsening", "high"}:
        patterns.append({
            "pattern_id": "building_stress_pattern",
            "label": "Building-level housing stress remains elevated",
            "severity": "medium",
            "evidence_ids": ["trend_housing_violations", "aep_building_signal"],
        })

    return patterns

18.2 Language rule

Use:

• "worsening together"
• "worth prioritizing"
• "may indicate"
• "public records suggest"

Do not overstate causality.

---

19. Priority Engine

This remains the heart of detail mode.

19.1 Core formula

priority_score =
    category_weight
  * severity
  * momentum
  * confidence
  * actionability

19.2 Reference implementation

def compute_priority_actions(current_state, trends, baselines, priority_weights, signal_to_category):
    candidates = []

    for signal_name, trend in trends.items():
        config = PRIORITY_SIGNALS.get(signal_name)
        if not config:
            continue

        category_id = signal_to_category.get(signal_name, "other")
        category_weight = priority_weights.get(category_id, 0.30)

        severity = compute_severity(current_state, signal_name, baselines)
        momentum = compute_momentum(trend)
        confidence = compute_confidence(trend, current_state, signal_name)
        actionability = config.get("actionability", 0.5)

        priority_score = category_weight * severity * momentum * confidence * actionability

        if priority_score > 0.05:
            candidates.append({
                "signal": signal_name,
                "category_id": category_id,
                "action": config["action_template"],
                "severity": round(severity, 2),
                "momentum": round(momentum, 2),
                "confidence": round(confidence, 2),
                "actionability": round(actionability, 2),
                "category_weight": round(category_weight, 2),
                "priority_score": round(priority_score, 3),
                "evidence_ids": config["evidence_ids"],
            })

    candidates.sort(key=lambda x: x["priority_score"], reverse=True)
    for i, c in enumerate(candidates, start=1):
        c["rank"] = i
    return candidates[:5]


def compute_severity(current_state, signal_name, baselines):
    """
    0-1 scale based on current value vs baseline p75.
    """
    mapping = {
        "rodent": ("safety", "rodent_positive_500m"),
        "collision": ("safety", "collision_count_500m"),
        "ems_response": ("safety", "ems_median_response_seconds"),
        "311_sanitation": ("safety", "safety_311_recent_count"),
        "housing_violations": ("housing", "open_class_c"),
    }
    if signal_name not in mapping:
        return 0.5
    module, key = mapping[signal_name]
    value = current_state.get(module, {}).get(key, 0) or 0
    baseline = baselines.get(signal_name, {})
    p75 = baseline.get("p75", max(value * 1.5, 1))
    return min(value / p75, 1.0)


def compute_momentum(trend):
    """
    0-1 scale based on rate of change and persistence.
    """
    score = 0.0
    rd = trend.get("recent_delta", {}).get("pct") or trend.get("recent_delta", {}).get("change_pct")
    sd = trend.get("seasonal_delta", {}).get("pct") or trend.get("seasonal_delta", {}).get("change_pct")
    persistence = trend.get("persistence", {}).get("consecutive") or trend.get("persistence", {}).get("consecutive_above", 0)
    if rd is not None and rd > 0:
        score += min(rd / 50, 0.4)
    if sd is not None and sd > 0:
        score += min(sd / 50, 0.3)
    score += min(persistence * 0.1, 0.3)
    return min(score, 1.0)


def compute_confidence(trend, current_state, signal_name):
    """
    0-1 scale based on data completeness.
    """
    mapping = {
        "rodent": ("safety", "rodent_positive_500m"),
        "collision": ("safety", "collision_count_500m"),
        "ems_response": ("safety", "ems_median_response_seconds"),
        "311_sanitation": ("safety", "safety_311_recent_count"),
        "housing_violations": ("housing", "open_class_c"),
    }
    module, key = mapping.get(signal_name, ("safety", signal_name))
    has_current = current_state.get(module, {}).get(key) is not None
    has_trend = trend.get("direction") not in (None, "insufficient_data")
    has_baseline = trend.get("baseline_gap", {}).get("gap_pct") is not None
    return (0.4 if has_current else 0) + (0.35 if has_trend else 0) + (0.25 if has_baseline else 0)


PRIORITY_SIGNALS = {
    "rodent": {
        "action_template": "Allocate additional pest control resources to this area",
        "evidence_ids": ["rodent_500m", "trend_rodent"],
        "actionability": 0.80,
    },
    "collision": {
        "action_template": "Review traffic safety measures in high-incident corridors",
        "evidence_ids": ["collisions_500m", "trend_collision"],
        "actionability": 0.75,
    },
    "ems_response": {
        "action_template": "Review EMS coverage and response capacity",
        "evidence_ids": ["ems_response", "trend_ems"],
        "actionability": 0.70,
    },
    "311_sanitation": {
        "action_template": "Increase sanitation enforcement and collection frequency",
        "evidence_ids": ["311_sanitation", "trend_311"],
        "actionability": 0.85,
    },
    "housing_violations": {
        "action_template": "Prioritize HPD inspection for buildings with open violations",
        "evidence_ids": ["hv_open_c", "trend_housing"],
        "actionability": 0.65,
    },
}

19.3 Why user ranking belongs here

This is the correct place for personalization because it changes interpretation and ranking, not raw facts.

---

20. Secondary Scores

Scores still have value, but only as secondary output.

They can support:

• normalization
• compatibility with older UI expectations
• secondary display blocks

20.1 No-data rule

If a dimension has no usable data, return None, not a reassuring number.

This fixes the earlier risk where absent data looked like a good score.

20.2 Implementation requirement

Each scoring module must check whether its required inputs are present before computing a score. If all key inputs are missing or None, the function must return None instead of a number.

def _has_any_data(module: dict, keys: list[str]) -> bool:
    """Return True if at least one key has a non-None value."""
    return any(module.get(k) is not None for k in keys)

Example for safety:

if not _has_any_data(safety, ["rodent_positive_500m", "sanitation_311_recent_count",
                               "ems_avg_response_seconds", "fire_avg_response_seconds"]):
    safety_score = None  # do NOT return 100

The overall score must use weighted average of available (non-None) scores only, re-normalizing the weights.

---

21. Evidence And Verification

Every surfaced issue should remain traceable.

21.1 Evidence responsibilities

Evidence must support:

• priority_actions
• why_now
• patterns
• report_markdown

21.2 Reference flow

def build_evidence(query_evidence, trends, patterns):
    """
    `query_evidence` should be produced by each provider/query function.
    This function aggregates and enriches those entries with trend and
    pattern evidence so downstream verification has concrete evidence_ids.
    """
    evidence = list(query_evidence or [])
    seen_ids = {e["evidence_id"] for e in evidence if e.get("evidence_id")}

    for signal_name, trend in trends.items():
        direction = trend.get("direction")
        recent = trend.get("recent_delta", {}).get("pct") or trend.get("recent_delta", {}).get("change_pct")
        seasonal = trend.get("seasonal_delta", {}).get("pct") or trend.get("seasonal_delta", {}).get("change_pct")
        persistence = trend.get("persistence", {}).get("consecutive") or trend.get("persistence", {}).get("consecutive_above", 0)
        gap = trend.get("baseline_gap", {}).get("gap_pct")

        trend_id = f"trend_{signal_name}"
        if trend_id not in seen_ids and direction not in (None, "insufficient_data"):
            summary_bits = [f"direction={direction}"]
            if recent is not None:
                summary_bits.append(f"recent_delta={recent}%")
            if seasonal is not None:
                summary_bits.append(f"seasonal_delta={seasonal}%")
            if gap is not None:
                summary_bits.append(f"baseline_gap={gap}%")
            if persistence:
                summary_bits.append(f"persistence={persistence}")
            evidence.append({
                "evidence_id": trend_id,
                "source": "computed_trend",
                "date": "computed",
                "summary": f"{signal_name} trend: " + ", ".join(summary_bits),
            })
            seen_ids.add(trend_id)

    for pattern in patterns:
        pattern_id = pattern.get("pattern_id")
        evidence_id = f"pattern_{pattern_id}"
        if pattern_id and evidence_id not in seen_ids:
            evidence.append({
                "evidence_id": evidence_id,
                "source": "computed_pattern",
                "date": "computed",
                "summary": pattern.get("label") or pattern.get("description") or pattern_id,
            })
            seen_ids.add(evidence_id)

    return evidence


def verify_priority_actions(priority_actions, evidence_table):
    valid_ids = {e["evidence_id"] for e in evidence_table}
    verified = []
    for item in priority_actions:
        if all(eid in valid_ids for eid in item.get("evidence_ids", [])):
            verified.append(item)
    return verified

21.2.1 Evidence production rule

The provider/query layer must already emit raw evidence entries during retrieval, for example:

{
    "evidence_id": "collisions_500m",
    "source": "h9gi-nx95",
    "date": "2024-2026 local extract",
    "summary": "244 collision records within 500m in the selected window",
}

build_evidence() is responsible for:

• aggregating provider-emitted evidence
• adding computed trend evidence
• adding computed pattern evidence
• returning the final evidence_table used by verification and reporting

21.3 Direct-mode rule

Because direct retrieval may be rougher than future Skill outputs, evidence and caveat honesty are especially important in direct mode.

---

22. Report Generation

The model remains a writer, not the judge.

22.1 Model role

The model does not:

• discover issues
• override rankings
• invent causes

The model does:

• write a concise brief from verified structured data

22.2 Required report order

1. priority actions
2. why these issues now
3. current local state
4. evidence-backed patterns
5. data gaps

22.3 Fallback requirement

If the model endpoint is unavailable, detail mode should still return:

• priority_actions
• why_now
• current_state
• evidence_table
• data_gaps

and a deterministic fallback brief string.

22.4 Pensar-safe model rules

To reduce avoidable prompt/data injection risk:

• the prompt should prefer structured values, evidence ids, and bounded summaries
• raw external free-text fields should be minimized, truncated, or sanitized before entering the prompt
• local model endpoints should be allowlisted to local addresses only for the prize path
• remote hosted model endpoints should not be used for the "Least Likely to get Hacked" evaluation path
• if model configuration is invalid or unavailable, the backend must fall back rather than retrying indefinitely or leaking configuration detail

---

23. API Surface

23.0 CORS

The FastAPI app must include CORSMiddleware, but must not use allow_origins=["*"] in the prize-ready build.

For the Pensar-hardened path:

• allow only the known frontend origins used by the real demo
• load them from env, for example URBAN_DOSSIER_ALLOWED_ORIGINS
• keep health readable, but do not make heavy endpoints universally cross-origin callable

from fastapi.middleware.cors import CORSMiddleware
app.add_middleware(
    CORSMiddleware,
    allow_origins=[
        "http://localhost:3000",
        "http://127.0.0.1:3000",
        "http://localhost:5173",
        "http://127.0.0.1:5173",
    ],
    allow_methods=["GET", "POST"],
    allow_headers=["Content-Type", "X-Urban-Dossier-Token"],
)

23.0.1 Endpoint protection baseline

For the prize-ready build:

• GET /api/health may remain open
• GET /api/categories and GET /api/coverage may remain open if they expose no sensitive internal paths
• POST /api/overview, POST /api/analyze-point, and POST /api/watchlist/run should require a simple local demo token such as X-Urban-Dossier-Token
• watchlist must be treated as a protected heavy endpoint, not a casual public endpoint

23.0.2 Request-budget rules

These are required so Pensar does not surface obvious resource-exhaustion findings:

• category_id must be whitelisted to overall or known map-driving categories
• radius_m should be capped to a demo-safe maximum, recommended 1000 or 1500
• time_window_days should be capped to a demo-safe maximum, recommended 730
• watchlist.seeds must have a hard upper bound, recommended 10
• duplicate watchlist seeds should be deduplicated before analysis
• heavy requests should have a timeout budget and fail closed with a structured error

23.1 POST /api/overview

Requires:

• X-Urban-Dossier-Token
• category_id omitted for overall
• category_id restricted to known map-driving categories for view_mode="category"

Request:

{
  "view_mode": "overall",
  "category_id": null,
  "viewport": {
    "north": 40.92,
    "south": 40.49,
    "east": -73.68,
    "west": -74.27
  },
  "zoom": 11,
  "render_mode": "h3_cells"
}

Response:

{
  "schema_version": "v3.6",
  "mode": "overview",
  "view_mode": "overall",
  "category_id": null,
  "layer_mode": "h3_r8",
  "cells": [],
  "coverage": {
    "overview_ready": true,
    "available_categories": ["overall", "safety", "traffic", "facilities"],
    "missing_categories": []
  },
  "data_mode": "direct",
  "freshness": {
    "generated_at": "2026-04-11T07:30:00Z"
  }
}

23.2 POST /api/analyze-point

Requires:

• X-Urban-Dossier-Token
• bounded radius_m
• bounded time_window_days
• priority_order filtered to known detail-rankable categories

Request:

{
  "latitude": 40.7579,
  "longitude": -73.9999,
  "radius_m": 500,
  "priority_order": ["safety", "traffic", "facilities"],
  "time_window_days": 365
}

Response:

{
  "schema_version": "v3.6",
  "mode": "detail",
  "data_mode": "direct",
  "target": {},
  "priority_profile": {
    "order": ["safety", "traffic", "facilities"],
    "weights": {
      "safety": 1.0,
      "traffic": 0.72,
      "facilities": 0.52
    }
  },
  "priority_actions": [],
  "why_now": [],
  "current_state": {},
  "detail_items": {
    "map_points": [],
    "nearby_facilities": [],
    "building_flags": [],
    "recent_incidents": []
  },
  "trends": {},
  "patterns": [],
  "evidence_table": [],
  "data_gaps": [],
  "scores": {},
  "report_summary": "",
  "report_markdown": ""
}

23.3 GET /api/categories

Returns:

• category labels
• default order
• map-driving flags
• detail-rankable flags

23.4 GET /api/coverage

Returns:

• current data_mode
• provider availability
• overview readiness
• signal availability by category

23.5 POST /api/watchlist/run

Runs batch analysis with fixed platform ordering.

Security rules:

• require X-Urban-Dossier-Token
• reject requests above the seed cap
• apply timeout / cancellation budget
• reuse detail logic, but do not allow unbounded synchronous fan-out

23.6 GET /api/health

Recommended to return:

• data provider status
• model endpoint status
• overview cache readiness
• report fallback readiness

---

24. Watchlist Mode

Watchlist should not be its own logic tree.

It should reuse the detail engine on multiple seeds.

24.1 Recommended flow

1. choose seed cells / areas / ZIPs
2. apply fixed default ordering
3. run detail analysis in batch
4. collect top issues and top areas
5. emit watchlist JSON and optional markdown summaries

24.2 Reference implementation

def run_watchlist(area_seeds, default_priority_order):
    results = []
    for seed in area_seeds:
        result = run_detail_analysis(
            latitude=seed["latitude"],
            longitude=seed["longitude"],
            radius_m=seed.get("radius_m", 500),
            priority_order=default_priority_order,
            time_window_days=365,
        )
        results.append(result)
    return results

---

25. Current Legacy Backend Assets We Should Reuse

The older local backend already contains directly useful ideas:

• address resolution
• parquet filtering
• radius-based direct metric retrieval
• evidence creation
• fallback report generation

v3.6.1 should not discard those.

Instead:

• refactor them into DirectQueryDataProvider
• preserve deterministic pieces where still valid
• replace only what is score-first or due-diligence-specific

---

26. Direct Query Strategy

This is crucial because v3.6.1 must be runnable before Skill is ready.

26.1 Direct-mode source priority

In direct mode:

1. use precomputed overview artifacts if they exist
2. use processed parquet / local extracts
3. use demo fixtures only if no real source exists

26.2 Direct-mode allowed retrieval strategies

• bbox + haversine radius filtering
• direct DuckDB parquet queries
• ZIP-level aggregate lookup for EMS / Fire
• cached precomputed JSON where appropriate

26.3 Direct-mode honesty rule

If direct mode uses rough proxies:

• say so in data_gaps
• do not imply precision that does not exist

---

27. Remote SSH Testing Plan

We should explicitly support backend-only smoke testing on the official machine.

27.1 Minimum tests to run now

Test A: detail JSON shape

• real point
• radius 500m
• one ranking order

Expect:

• valid JSON
• non-empty current_state
• non-empty priority_actions

Test B: ranking sensitivity

Same point, same radius, two different ranking orders.

Expect:

• different weights
• meaningfully different top-ranked actions

Test C: overview fallback

No precompute ready.

Expect:

• valid overview response
• overview_ready = false

Test D: overview minimal success

At least one overview artifact exists.

Expect:

• non-empty cells
• correct category_id

27.2 Current feasibility status

The SSH tests already prove direct mode is viable enough to start backend implementation and smoke testing immediately.

---

28. NVIDIA / CUDA Compatibility

The teammate-owned NVIDIA ecosystem work should be easy to plug in, not blocked by our backend design.

28.1 What our backend should naturally support

• cuDF / RAPIDS for overview precompute and large aggregations
• DuckDB/parquet for detail realtime queries
• local model endpoint abstraction
• batch watchlist runs

28.2 Recommended split

• cuDF / RAPIDS
  • overview precompute
  • heavy groupbys
  • baseline generation
  • batch pre-aggregation
• DuckDB
  • realtime point/radius filtering
  • lightweight local queries
• LLM runtime
  • short brief generation only

28.3 Why this is enough

We do not need to own benchmark work ourselves. We only need to ensure our logic can benefit from those choices.

---

29. Benchmark Story

Teammates are already running benchmark work, but we should design the backend so their numbers matter.

The remote benchmark path already showed strong cuDF wins on real data for:

• load
• string filter
• sort

Therefore:

• overview precompute should be the first place to exploit GPU aggregation wins
• detail mode should remain simpler and latency-friendly

These benchmark numbers should appear in final slides, but the backend architecture must already reflect them.

---

30. Apex Lessons

The local apex folder is useful for architecture thinking, not as a core product dependency.

30.1 What is worth borrowing

• API-first organization
• shared core, multiple consumers
• runtime/provider abstraction
• a repeatable operator-facing security gate

30.2 How it maps here

In Urban Dossier:

• overview
• detail
• watchlist

should all consume the same core analysis capability.

And:

• SkillDataProvider
• DirectQueryDataProvider

should be swappable runtime providers.

30.3 What not to do

Do not make Apex or pentest tooling the main product story. It should remain a quality gate, but a real one that can block unsafe merges.

---

31. Module Layout

src/
  api.py
  config.py
  categories.py
  providers/
    base.py
    skill_provider.py
    direct_provider.py
  overview.py
  detail.py
  metrics.py
  trend_engine.py
  pattern_detector.py
  priority_engine.py
  evidence.py
  scoring.py
  report.py
  watchlist.py

31.1 Responsibilities

• providers/base.py
  • provider interface
• providers/skill_provider.py
  • Skill-backed source reads
• providers/direct_provider.py
  • legacy hardcoded retrieval refactor
• overview.py
  • overview orchestration + fallback
• detail.py
  • realtime detail orchestration
• trend_engine.py
  • trend computations
• pattern_detector.py
  • multi-signal patterns
• priority_engine.py
  • ranking logic
• evidence.py
  • evidence table + verification
• report.py
  • brief generation
• watchlist.py
  • batch wrapper around detail

---

32. Implementation Phases

Phase 1

• add provider abstraction
• implement DirectQueryDataProvider
• set URBAN_DOSSIER_DATA_MODE=direct
• implement /api/analyze-point minimal path

Phase 2

• implement /api/overview fallback
• add first overall and one category precompute layer
• expose data_mode in responses

Phase 3

• wire trend engine
• wire pattern detection
• wire priority engine
• wire evidence and fallback brief

Phase 4

• add watchlist batch wrapper
• add SkillDataProvider skeleton
• keep method signatures stable

Phase 5

• switch default to auto when Skill is ready
• refine coverage/manifest integration

Phase 6

• harden the prize path for Pensar/Apex
• lock the award path to restricted CORS + token-protected heavy endpoints
• enforce request caps and timeout budgets
• run the standing Pensar/security gate before each demo or merge

---

33. Open Questions For Teammates

These do not block us from writing backend skeletons, but they do affect final integration:

1. final first-pass category list
2. exact frontend expectation for overview rendering payload
3. radius options to expose in UI
4. first standardized Skill/data outputs that will really ship
5. final local model endpoint choice on the official machine

None of these should block direct-mode backend testing.

---

34. Pensar Prize Interpretation

The rules include a bounty for:

• Least Likely to get Hacked
• requirement: run Pensar on the build and fix the issues it surfaces

This means the backend should be designed so the prize path is:

• real
• repeatable
• local
• measurable
• identical to the real demo behavior

It does not mean:

• writing a fake-safe build just for the scan
• temporarily removing risky endpoints only for judging
• claiming security from architecture slides alone

34.1 The correct security target

The correct target for this prize is:

• a local hackathon demo backend
• running on the same machine family and same codebase as the demo
• with the same real endpoints the frontend consumes
• with bounded request cost
• with no obvious cross-origin abuse path
• with no unprotected batch-amplification path

34.2 Current risks already observed in our own testing

Before hardening, our own backend testing already demonstrated these real risks:

• wildcard CORS on heavy endpoints
• no token/auth on heavy endpoints
• watchlist resource amplification
• very large legal request windows for detail mode
• missing category whitelist for overview selection

Those must be treated as real findings, not hypothetical concerns.

---

35. Pensar-Hardened Backend Requirements

To make the backend realistically defensible for the prize path, v3.6.1 requires:

35.1 Endpoint exposure rules

• GET /api/health may remain open
• GET /api/categories and GET /api/coverage may remain open only if they do not expose filesystem paths, secrets, or privileged internals
• POST /api/overview, POST /api/analyze-point, and POST /api/watchlist/run must require a local demo token
• watchlist must be treated as a privileged/internal endpoint even in demo mode

35.2 Request-bound rules

• DetailRequest.radius_m should be capped to a safe range; recommended max 1000 or 1500
• DetailRequest.time_window_days should be capped to a safe range; recommended max 730
• WatchlistRequest.seeds must have a hard max; recommended 10
• watchlist seeds should be deduplicated before analysis
• any request exceeding budget should return a structured error instead of being silently truncated

35.3 Category and path safety rules

• category_id must be restricted to overall or known map-driving categories
• no user input may influence arbitrary file paths
• overview file names must be derived from whitelisted category ids only
• provider mode must remain a backend/server decision, not a user-controlled request parameter

35.4 Model safety rules

• local model base URLs must be restricted to local/allowlisted addresses
• the report path must tolerate model failure cleanly
• the fallback brief must remain a first-class output path
• prompt construction should prefer structured data and bounded summaries

35.5 Prize-path runtime mode

For the actual security-award path, the recommended mode is:

• URBAN_DOSSIER_DATA_MODE=direct
• restricted CORS
• token-protected heavy endpoints
• Skill support compiled in but not enabled unless it has passed the same gate

This is not cheating because it is the real runtime path we currently ship and test.

---

36. Pensar Test Protocol

Pensar/Apex should be treated as a standing gate, not a one-time demo rehearsal.

36.1 Minimum local test setup

Before running Pensar:

1. start the real backend service
2. use the same build and same config that the frontend uses for the demo
3. ensure heavy endpoints are exposed exactly as they will be shown
4. ensure the direct-mode data path is populated with real or realistic local data

Operational note:

• launch Apex with pensar
• run the security review against the live local backend, not a mock or reduced endpoint surface

36.2 Minimum manual abuse tests we must pass

Before or alongside Pensar, we should manually verify:

1. a hostile browser origin cannot call heavy endpoints
2. watchlist rejects over-limit seed counts
3. extreme detail requests are rejected or bounded
4. unknown category_id values are rejected cleanly
5. model outage still yields a safe deterministic response

36.3 Minimum Pensar review scope

Pensar should be used against:

• the running backend as a blackbox target
• the local backend codebase as a whitebox review target

The review should explicitly check for:

• authentication gaps
• resource exhaustion
• prompt/data injection opportunities
• unsafe path handling
• overly broad CORS
• missing timeout/limit enforcement

36.4 Pass/fail rule

We only claim the prize path is ready if:

• Pensar findings are reviewed
• actionable findings are fixed in code
• the fixes are re-tested on the same real endpoints
• no special "scan-only" build is introduced

---

37. No-Cheating Enforcement

To keep this prize attempt credible, the following are forbidden:

• branching the code just for Pensar while demoing a different build
• disabling watchlist or detail only during the scan
• returning pre-canned safe data to Pensar while serving live logic to the frontend
• adding Pensar-specific user-agent or source checks
• claiming Skill mode is secure without testing Skill mode

37.1 Allowed simplifications

These are allowed because they reflect real deployment choices, not fake-safe tricks:

• running only direct mode in the award path if Skill is not ready
• using a local demo token for protected endpoints
• limiting seed count, radius, and time window to a safe demo budget
• falling back from LLM output to deterministic markdown if the local model is unavailable

---

38. Change Management Gate

Every backend change after v3.6.1 must pass a standing security gate.

38.1 Triggering changes

The gate must run whenever we change:

• API request models
• provider implementations
• overview file-loading logic
• detail query logic
• watchlist behavior
• model prompt construction
• auth/CORS/config behavior

38.2 Required gate checklist

For each qualifying change:

1. run backend smoke tests
2. run manual abuse tests
3. run Pensar/Apex review
4. fix actionable issues
5. re-run smoke tests and Pensar
6. record what changed and what was fixed

38.3 Minimum artifact we should keep

For each security-relevant change, keep:

• commit hash
• runtime mode used
• endpoints tested
• Pensar finding summary
• fixes applied
• retest result

This creates an honest record that the backend keeps earning the prize claim over time.

---

39. Final Direction

Urban Dossier v3.6.1 should be understood as:

• a complete backend architecture, not just a concept note
• one that merges:
  • v3 deterministic analysis and code skeletons
  • v3.5.x overview/detail split
  • v3.5.3 provider abstraction and direct-query fallback
  • a real Pensar/Apex hardening gate

The correct final interpretation is:

• overview is fixed and pre-rendered
  • overall + one map per single category
• detail is personalized and realtime
  • click + radius + drag order -> local ranked priorities
• watchlist is batch reuse of detail
  • fixed platform ordering
• Skill is preferred when available
• direct query is the current test path and permanent fallback
• the security-award path uses the same real backend, not a special fake-safe variant

That is the most complete, practical, and review-ready version of the backend engineering plan.