Adaptive Background Sync for Commons Metadata

[ayushshukla1807]

GSoC 2026 Blueprint — Phase 3: Adaptive Background Sync for Commons Metadata

Status: Active architecture blueprint, cited in my GSoC 2026 proposal (Phase 3, Weeks 9–10). Tracks the async import engine design.

Problem

The current labs.py get_files() implementation executes a synchronous MySQL query against the Commons replica database. For categories with large file counts (>5,000 images), this query:

1. Times out — the Toolforge MySQL replica enforces a query timeout, returning 0 results for large categories
2. Blocks the WSGI thread — no other requests can be served during the query
3. Ignores redirects — partially fixed in PR fix: follow category redirects during file import #428, but the redirect chain is still resolved in a single query

Proposed Architecture: Chunked Async Import

sequenceDiagram
    participant Coord as Coordinator
    participant API as POST /import
    participant Worker as Background Worker
    participant Commons as Commons Replica (MySQL)
    participant Cache as Import Cache (Redis/SQLite)

    Coord->>API: import_method=category, category=WLM_2024
    API-->>Coord: 202 Accepted {job_id: "abc123"}
    API->>Worker: enqueue(job_id, category)

    loop Chunked fetch (500 files/page)
        Worker->>Commons: SELECT ... LIMIT 500 OFFSET N
        Commons-->>Worker: Batch of file metadata
        Worker->>Cache: store batch
        Worker->>API: progress update (N/total)
    end

    Worker->>API: job complete
    Coord->>API: GET /import/abc123/status
    API-->>Coord: {status: "complete", files_imported: 8432}

Loading

Why This Matters for GSoC

Large Wiki Loves competitions (WLM, WLE) routinely have 10,000–50,000 submissions. The current synchronous import is the single biggest operational blocker for using Montage at full competition scale. This is the highest-impact infrastructure improvement in the proposal.

Implementation Plan

Component	Technology	Notes
Job queue	Python threading.Thread or Celery	Start simple, Celery if needed
Progress tracking	SQLite job table	Lightweight, no new deps
Chunked query	LIMIT/OFFSET pagination	500 files per chunk
Redirect resolution	Existing PR #428 logic	Reuse and extend

[ayushshukla1807]

I am closing this PR to reduce repository noise. The core fixes relevant to my GSoC Proposal are being manually consolidated into PR #454 and PR #415 to make it substantially easier for the maintainers to review my code. The larger concepts discussed here will be implemented incrementally and manually if my proposal is accepted.

[ayushshukla1807]

I have stripped the AI formatting from the description and reopened this PR so I can manually improve its code over the coming days, fulfilling my promise.

[ayushshukla1807]

Closing this conceptual proposal. Consolidating my Open Source footprint to prioritize high-value, locally verified bug fixes for the current review window.