Skip to content

blocksense-v3.5.3-engineering.md

Latest commit

 

History

History
933 lines (634 loc) · 21.3 KB

blocksense-v3.5.3-engineering.md

File metadata and controls

933 lines (634 loc) · 21.3 KB
Raw

Urban Dossier NYC v3.5.3 - Backend Engineering Document

Target reader: backend engineer / Codex / Claude Code / team manager

Date: 2026-04-11 Hackathon: NVIDIA Spark Hack NYC 2026 Hardware: Acer Veriton GN100 / DGX Spark / NVIDIA GB10 Grace Blackwell / 128GB unified memory Runtime target: 100% local, offline-capable Scope owned by us: backend analysis engine, overview service, detail-time ranking, API contract, evidence, brief generation, backend-only smoke tests Not our scope: frontend UI implementation, final dataset selection, final Skill implementation owned by teammates

0. Purpose

This v3.5.3 document is the practical execution version after one more team constraint became clear:

  • the Skill side is still not settled
  • backend still needs to be testable now
  • we do not want to throw away the Skill interface that may arrive later

So v3.5.3 formalizes a dual-path backend:

  1. Preferred path
    • use the standardized Skill/data interface when it becomes available
  2. Current test path
    • use hardcoded direct data retrieval in backend code
    • query local parquet / CSV / processed extracts directly

The backend must behave the same at the API level in both modes.

That means:

  • the frontend should not care whether data came from Skill or direct query
  • the ranking / evidence / report logic should not care either
  • only the data-provider layer changes

This makes v3.5.3 the most practical handoff version:

  • we can test backend immediately
  • we preserve the future Skill integration path
  • we reduce dependency risk on another workstream

1. What Changed From v3.5.2

v3.5.2 already got the product split right:

  • overview = overall + single-category fixed layers
  • detail = clicked-point + radius + drag ranking + real-time recompute
  • watchlist = batch reuse of detail logic

v3.5.3 adds one new implementation rule:

Skill is no longer a runtime dependency for testing. Skill becomes an optional provider behind a stable backend interface.

So the architecture is now:

  • same product behavior as v3.5.2
  • different runtime fallback strategy

2. Product Definition

Urban Dossier remains:

  • a fully local civic analysis backend
  • with a pre-rendered overview layer
  • and a real-time detail layer

But v3.5.3 adds a more realistic delivery rule:

  • if Skill is ready, use Skill outputs
  • if Skill is not ready, backend directly queries the available local data assets

The product promise to frontend stays the same:

  1. overview buttons switch between fixed layers
  2. clicking a point enters detail mode
  3. drag order only affects local real-time priority ranking

3. Correct Frontend Interpretation

The latest frontend requirement still stands exactly as in v3.5.2.

3.1 Overview state

Overview should only support:

  • one overall map
  • one single-category map per button

Overview does not depend on user drag ranking.

3.2 Detail state

After the user clicks a map point:

  • the frontend can expose drag ordering
  • the backend receives priority_order
  • the backend recomputes local priorities in real time

3.3 Watchlist state

Watchlist is not user-personalized.

It should reuse the detail engine with:

  • fixed default platform ordering
  • fixed area seeds or map cells

4. Why v3.5.3 Exists

This version exists to solve a very practical delivery problem.

If we require Skill before backend testing, we block ourselves on another workstream.

If we remove Skill entirely, we create migration cost later.

So the right compromise is:

  • preserve Skill-facing contracts
  • implement direct-query fallback immediately
  • make direct-query the currently enabled test mode

This is a better engineering move than waiting.

5. Core Architecture

                     +------------------------------+
                     |  API Layer                   |
                     | overview / detail / watchlist|
                     +--------------+---------------+
                                    |
                                    v
                     +------------------------------+
                     | Shared Analysis Core         |
                     | trends / patterns / priority |
                     | evidence / report            |
                     +--------------+---------------+
                                    |
                     +--------------+---------------+
                     | Data Provider Abstraction    |
                     +-------+----------------------+
                             |
              +--------------+--------------+
              |                             |
              v                             v
   +------------------------+   +--------------------------+
   | SkillDataProvider      |   | DirectQueryDataProvider  |
   | preferred future path  |   | current test default     |
   +------------------------+   +--------------------------+

The big idea is simple:

  • all product logic is above the provider layer
  • all source-specific mess stays below it

6. Runtime Modes

6.1 Supported data modes

Add an explicit backend config:

URBAN_DOSSIER_DATA_MODE = auto | skill | direct

Recommended meaning:

  • direct
    • use hardcoded backend retrieval only
    • current default for testing
  • skill
    • require Skill outputs
    • fail clearly if unavailable
  • auto
    • use Skill if ready
    • otherwise fall back to direct query

6.2 Current test default

For now:

URBAN_DOSSIER_DATA_MODE=direct

This is the correct current choice because Skill delivery is not stable yet.

6.3 Future production/default mode

Once Skill is confirmed working on the official machine, recommended default becomes:

URBAN_DOSSIER_DATA_MODE=auto

That gives us:

  • no rewrite of backend API
  • graceful use of Skill
  • safe fallback if Skill breaks on-site

7. Data Provider Contract

The provider contract is the key new addition in v3.5.3.

Everything above this layer should consume a stable interface.

7.1 Required provider methods

Recommended interface:

class DataProvider:
    def get_overview_layer(self, view_mode: str, category_id: str | None, viewport: dict | None, zoom: int | None) -> dict: ...
    def get_point_signals(self, latitude: float, longitude: float, radius_m: int, time_window_days: int) -> dict: ...
    def get_local_timeseries(self, latitude: float, longitude: float, radius_m: int, time_window_days: int) -> dict: ...
    def get_baselines(self) -> dict: ...
    def get_context_items(self, latitude: float, longitude: float, radius_m: int) -> dict: ...
    def get_coverage(self) -> dict: ...

This is intentionally backend-facing, not Skill-facing.

7.2 Why this matters

If we keep this interface stable:

  • the frontend API does not change
  • the detail engine does not change
  • only the provider implementation changes

That is exactly what we want.

8. Provider Implementations

8.1 SkillDataProvider

This is the preferred future path.

It should read from Skill-prepared outputs such as:

  • standardized signal tables
  • precomputed overview layers
  • area timeseries
  • baselines
  • manifest files

This path should assume cleaner inputs and less ad hoc filtering.

8.2 DirectQueryDataProvider

This is the currently enabled test path.

It should use hardcoded backend retrieval logic against:

  • processed parquet files
  • precomputed JSON artifacts
  • local CSVs when needed

It should be built from the older code path already present in the local backend repo.

This is not "throwaway hack code." It is the operational fallback path.

9. What “Direct Query” Means In Practice

This section is important because the fallback should not be vague.

9.1 Reuse the old metrics-style retrieval path

The existing local backend already proves direct retrieval can work:

  • bounding-box + radius filtering
  • direct DuckDB parquet queries
  • ZIP-level aggregate lookup for EMS / Fire
  • evidence creation from matched rows

v3.5.3 should explicitly preserve that approach as the fallback test backend.

9.2 Direct query sources

Direct query mode can read from:

  • processed parquet
  • cache/precomputed JSON
  • cache/report markdown
  • demo fixtures

Priority order for direct mode:

  1. precomputed overview artifacts if they exist
  2. processed parquet / local extracts
  3. demo fixtures only when real data is unavailable

9.3 Current remote-machine feasibility

We already confirmed over SSH that direct local queries are feasible against real data for:

  • collisions
  • rodent
  • 311
  • public toilets

That means direct mode is not theoretical. It is the current practical route for backend-only smoke tests.

10. Overview Layer In v3.5.3

Overview logic stays the same as v3.5.2, but the source of data now depends on provider mode.

10.1 Overview should still only expose:

  • overall
  • one map per single category

10.2 In direct mode

Overview should try:

  1. pre-rendered overview parquet / JSON if present
  2. lightweight direct aggregate generation if feasible
  3. fallback response with overview_ready = false

10.3 In skill mode

Overview should read:

  • Skill-prepared overview layers
  • Skill-generated manifest

10.4 Overview fallback contract

Fallback remains:

{
  "schema_version": "v3.5.3",
  "mode": "overview",
  "view_mode": "overall",
  "category_id": null,
  "cells": [],
  "coverage": {
    "overview_ready": false,
    "missing_categories": ["overall", "safety", "traffic", "facilities"]
  },
  "ui_message": "Overview not yet available. Click a point for realtime detail analysis."
}

11. Detail Layer In v3.5.3

Detail remains the real product core.

The logic is unchanged from v3.5.2:

  1. accept clicked point + radius + priority order
  2. query local signals
  3. compute trends
  4. detect patterns
  5. apply ranking weights
  6. emit priority_actions, why_now, current_state, detail_items, evidence

The only new rule is:

  • data retrieval comes from the active provider

That means detail mode is where we can test the product now even before Skill is ready.

12. User Preference Model

No change from v3.5.2.

Preference only matters in detail mode.

12.1 Weight conversion

Recommended first-release formula:

weight(rank) = decay ^ (rank - 1)
decay = 0.72

Approximate weights:

  • rank 1 -> 1.00
  • rank 2 -> 0.72
  • rank 3 -> 0.52
  • rank 4 -> 0.37
  • rank 5 -> 0.27

12.2 What it affects

Weights should influence:

  • final priority ranking
  • report emphasis
  • tie-breaking across categories

Weights should not influence:

  • raw retrieval counts
  • evidence integrity
  • overview map choice

13. Shared Core Analysis

The analysis core should remain source-agnostic.

That means the same core functions should work with either provider:

  • trend engine
  • pattern detector
  • priority engine
  • evidence builder
  • report generation

This is the whole point of introducing the provider abstraction.

14. Trend Engine

Trend logic remains as previously defined.

Each signal should compute:

  • recent_delta
  • seasonal_delta
  • baseline_gap
  • persistence

14.1 Reference implementation

def compute_recent_delta(values_by_period):
    recent = values_by_period.get("last_30d", 0)
    previous = values_by_period.get("prev_30d", 0)
    if previous == 0:
        return {"change_pct": None, "label": "no_baseline"}
    pct = (recent - previous) / previous * 100
    return {
        "change_pct": round(pct, 1),
        "label": "rising" if pct > 10 else "falling" if pct < -10 else "stable",
    }


def compute_seasonal_delta(values_by_period):
    current = values_by_period.get("last_90d", 0)
    same_last_year = values_by_period.get("same_90d_last_year", 0)
    if same_last_year == 0:
        return {"change_pct": None, "label": "no_baseline"}
    pct = (current - same_last_year) / same_last_year * 100
    return {
        "change_pct": round(pct, 1),
        "label": "rising" if pct > 15 else "falling" if pct < -15 else "stable",
    }

14.2 v3.5.3-specific note

In direct mode, timeseries may initially be rougher because they are assembled directly from local tables.

That is acceptable for testing as long as:

  • schema is stable
  • output shape is stable
  • evidence and caveats remain honest

15. Pattern Detection

Pattern detection remains rule-based.

Recommended first rules:

  • sanitation complaints rising + rodent rising
  • collision pressure rising + EMS response worsening
  • housing violations unresolved + AEP/building risk present

This layer should not care whether source data came from Skill or direct query.

16. Priority Engine

Priority logic also remains the same.

16.1 Core formula

priority_score =
    category_weight
  * severity
  * momentum
  * confidence
  * actionability

16.2 Reference implementation

def compute_priority_actions(current_state, trends, baselines, priority_weights, signal_to_category):
    candidates = []
    for signal_name, trend in trends.items():
        config = PRIORITY_SIGNALS.get(signal_name)
        if not config:
            continue

        category_id = signal_to_category.get(signal_name, "other")
        category_weight = priority_weights.get(category_id, 0.30)

        severity = compute_severity(current_state, signal_name, baselines)
        momentum = compute_momentum(trend)
        confidence = compute_confidence(trend, current_state, signal_name)
        actionability = config.get("actionability", 0.5)

        priority_score = category_weight * severity * momentum * confidence * actionability

        if priority_score > 0.05:
            candidates.append({
                "signal": signal_name,
                "category_id": category_id,
                "action": config["action_template"],
                "priority_score": round(priority_score, 3),
                "evidence_ids": config["evidence_ids"],
            })

    candidates.sort(key=lambda x: x["priority_score"], reverse=True)
    for i, c in enumerate(candidates, start=1):
        c["rank"] = i
    return candidates[:5]

17. Evidence And Verification

Evidence remains essential.

In direct mode this is even more important, because direct retrieval can be rougher than future Skill-curated outputs.

So v3.5.3 should enforce:

  • all surfaced issues need evidence ids
  • data gaps must be explicit
  • fallback mode must never present rough retrieval as certainty

18. API Surface

The frontend API must not change depending on provider mode.

18.1 POST /api/overview

Request:

{
  "view_mode": "overall",
  "category_id": null,
  "viewport": {
    "north": 40.92,
    "south": 40.49,
    "east": -73.68,
    "west": -74.27
  },
  "zoom": 11,
  "render_mode": "h3_cells"
}

Response:

{
  "schema_version": "v3.5.3",
  "mode": "overview",
  "view_mode": "overall",
  "category_id": null,
  "layer_mode": "h3_r8",
  "cells": [],
  "coverage": {
    "overview_ready": true,
    "available_categories": ["overall", "safety", "traffic", "facilities"],
    "missing_categories": []
  },
  "data_mode": "direct"
}

18.2 POST /api/analyze-point

Request:

{
  "latitude": 40.7579,
  "longitude": -73.9999,
  "radius_m": 500,
  "priority_order": ["safety", "traffic", "facilities"],
  "time_window_days": 365
}

Response:

{
  "schema_version": "v3.5.3",
  "mode": "detail",
  "data_mode": "direct",
  "target": {},
  "priority_profile": {
    "order": ["safety", "traffic", "facilities"],
    "weights": {
      "safety": 1.0,
      "traffic": 0.72,
      "facilities": 0.52
    }
  },
  "priority_actions": [],
  "why_now": [],
  "current_state": {},
  "detail_items": {
    "map_points": [],
    "nearby_facilities": [],
    "building_flags": [],
    "recent_incidents": []
  },
  "trends": {},
  "patterns": [],
  "evidence_table": [],
  "data_gaps": [],
  "scores": {},
  "report_summary": "",
  "report_markdown": ""
}

The extra data_mode field is useful during testing and debugging.

18.3 GET /api/categories

No change.

18.4 GET /api/coverage

Should now additionally report:

  • active data mode
  • whether Skill provider is available
  • whether direct provider is available

18.5 POST /api/watchlist/run

No user preference order. Use fixed platform ordering and whichever provider is active.

19. Standardized Data Contract

The Skill-facing contract remains in place.

That means we still want the future Skill/data side to provide:

  1. point_events
  2. area_timeseries
  3. building_signals
  4. context_places
  5. baselines
  6. manifest

19.1 Why keep the contract now

Even though we are not currently depending on Skill, we keep the contract because:

  • it preserves the future integration path
  • it keeps backend expectations explicit
  • it avoids a rewrite when Skill is finally ready

19.2 Direct mode compatibility rule

Direct mode is allowed to build provider outputs from whatever local raw/processed assets exist, as long as it adapts them into the same internal shapes expected above.

20. Direct Query Test Strategy

This is the most important practical section for v3.5.3.

20.1 What should be test-enabled now

We should immediately support backend-only testing for:

  • one overview fallback endpoint
  • one overview success path if layer exists
  • one detail clicked-point analysis path
  • one ranking sensitivity test

20.2 Minimum real data set for testing

Direct mode can already test against a small real subset such as:

  • collisions
  • rodent
  • 311
  • public toilets
  • EMS ZIP aggregate

This is enough to validate:

  • request shape
  • ranking logic
  • evidence flow
  • report fallback flow

20.3 Smoke tests we should actually run

Test A: detail JSON shape

One real point, one radius, one ranking order.

Expected:

  • valid response
  • non-empty current_state
  • non-empty priority_actions

Test B: ranking sensitivity

Same point, same radius, two ranking orders.

Expected:

  • different priority_profile.weights
  • meaningfully different top action ordering

Test C: overview fallback

No precomputed layer available.

Expected:

  • overview_ready = false
  • frontend-safe message

Test D: overview success

At least one category layer exists.

Expected:

  • non-empty cells
  • correct category_id

21. Remote SSH Feasibility

The remote machine already proved direct mode is viable.

We confirmed over SSH that local queries can run against real remote data for:

  • collisions
  • rodent
  • 311
  • public toilets

So v3.5.3 should treat:

  • direct mode as the current execution path
  • skill mode as the preferred future upgrade path

This is exactly the right order for delivery.

22. NVIDIA / CUDA Compatibility

Teammates are handling the NVIDIA ecosystem optimization path, so our backend should remain compatible rather than prescriptive.

v3.5.3 should naturally support:

  • overview precompute on cuDF/RAPIDS
  • detail queries on DuckDB/parquet
  • batch watchlist runs
  • local model endpoint abstraction

The provider split makes this easier:

  • direct provider can use current DuckDB/parquet testing immediately
  • skill provider can later consume GPU-accelerated curated outputs

23. Apex Lessons

The local apex folder still matters for architecture thinking.

The most useful lessons are:

  • API-first organization
  • shared core, multiple consumers
  • runtime/provider abstraction

That lines up exactly with v3.5.3:

  • overview, detail, and watchlist share one core
  • Skill and direct query are swappable providers

We still should not make Apex the product story itself.

24. Recommended Module Layout

src/
  api.py
  config.py
  categories.py
  providers/
    base.py
    skill_provider.py
    direct_provider.py
  overview.py
  detail.py
  metrics.py
  trend_engine.py
  pattern_detector.py
  priority_engine.py
  evidence.py
  scoring.py
  report.py
  watchlist.py

24.1 Important note

The old direct-query code from the previous backend should not be deleted.

It should be refactored into providers/direct_provider.py.

That is the main implementation move in v3.5.3.

25. Implementation Order

Phase 1

  • add provider abstraction
  • implement DirectQueryDataProvider
  • set URBAN_DOSSIER_DATA_MODE=direct
  • wire /api/analyze-point through provider layer

Phase 2

  • implement overview fallback API
  • add one or two direct-mode overview layers
  • expose data_mode in responses

Phase 3

  • add SkillDataProvider skeleton
  • keep method signatures stable
  • keep it disabled until Skill outputs are confirmed

Phase 4

  • switch default to auto when Skill is actually usable

26. Open Questions We Still Need From Teammates

  1. final first-pass category list
  2. exact frontend expectation for overview rendering
  3. detail radius options exposed in UI
  4. first standardized Skill/data outputs that will actually ship
  5. final local model endpoint choice on the official machine

But none of these should block direct-mode backend testing.

27. Final Direction

The correct v3.5.3 direction is:

  • keep the v3.5.2 product behavior
    • overview = overall + single-category fixed layers
    • detail = click + radius + drag order -> live local priorities
  • but change runtime dependency strategy
    • current test mode = direct hardcoded backend retrieval
    • future preferred mode = Skill-backed provider
    • stable backend interface for both

This is the most practical version for the team right now because it lets the backend move immediately without breaking the future Skill integration path.