fix: detect stale pipeline jobs by progress

[Marquis03]

Fixes #189.

Problem and reproduction

The post-market pipeline was marked as failed after a fixed 20-minute wall-clock timeout even while it was still reporting progress.

Two observed adjustment-factor sync runs were cancelled after 1,282 and 1,336 seconds. Their latest progress messages were 41/139 and 50/139 batches respectively, demonstrating that the jobs were slow but not stalled. This prevented large initial synchronizations from completing and delayed later stages such as ETF data preparation.

Root cause

JobStore.reap_stale() measured elapsed time from started_at. The normal 1,200-second timeout therefore applied to total job runtime rather than inactivity.

The stale path also released the heavy-run lock even though the worker thread could not be forcibly stopped. Retrying could allow the previous worker and a new job to write the same data concurrently.

Solution

• Add last_progress_at to active jobs.
• Initialize it when a job starts and refresh it on each progress callback.
• Detect stale jobs from the time since the latest progress instead of total runtime.
• Fall back to started_at for active jobs created without the new field.
• Make the progress check and transition to failed atomic under the job-store lock.
• Keep the heavy-run lock until the worker exits naturally, preventing overlapping writers.
• Update pipeline API comments to describe inactivity-based cleanup.
• Add regression tests for long-running active jobs, genuinely stale jobs, legacy active jobs, and run-slot ownership.

Compatibility

The new job field is additive. Existing job data without last_progress_at continues to use started_at, and no API field is removed or renamed. Data-provider, Parquet, financial-data, and frontend contracts are unchanged.

If a worker is permanently stuck, the job is marked failed but the execution slot remains unavailable until the process is restarted. This is an intentional fail-closed behavior to prevent concurrent writes.

Performance

Each progress callback performs one UTC timestamp update. Stale checks remain O(1), and no data-path scans or network calls are added.

Validation

• python -m pytest tests/test_pipeline_and_monitor_fixes.py -q — 12 passed.
• Targeted Ruff check for the changed code passed; existing unrelated warnings in the touched legacy files were excluded.
• git diff --check passed.
• Manual data-sync retest did not reproduce the premature cancellation.

UI evidence

No UI behavior or layout changed.

Risk and rollback

The remaining risk is a provider call that blocks longer than its job inactivity threshold without emitting progress. Such a worker will fail closed and require a backend restart before another heavy data task can begin.

The change can be rolled back by reverting this commit; no data migration is required.