[bug]: Plane (self-hosted, commercial) • silo + backend • Import • Jobs never leave `TRANSFORMING` state

[nanx64]

Summary

On self-hosted Plane (commercial/enterprise), an import never completes: the job is stuck at
status TRANSFORMING forever, even though the data is fully imported. The state machine that finalizes
a job is also what dequeues the next one, so one stuck import blocks all subsequent imports.

Even though the issue was spotted during a Linear import process, the research suggests the import code path is source-agnostic, so GitHub/Jira/etc. imports are affected the same way.

Environment

• Deployment: self-hosted (Docker / Swarm), commercial/enterprise images.
• Affected versions: v3.0.0 and v3.1.4 (current stable) confirmed. The defect is present in
  every image inspected: v3.0.0, v3.0.1, v3.1.0, v3.1.4, v3.1.5-rc1, v3.2.0-rc3, preview.
• Components involved: silo (importer), backend (api + celery worker), Postgres.

Steps to reproduce

1. Configure a Linear import with a valid PAT against a workspace.
2. Run the import.
3. Watch the job in the UI (or import_jobs / import_reports in Postgres).

Expected

Job progresses INITIATED → PULLING → TRANSFORMING → PUSHING → FINISHED; the UI shows the import as
complete; the next queued import starts.

Actual

• Issues and modules are created in the target project (the load succeeds).
• import_reports shows completed_batch_count == total_batch_count (e.g. 3/3) with issue counts
  populated (e.g. imported=123, errored=45, total=168).
• import_jobs.status stays TRANSFORMING indefinitely; the job never finalizes.
• Because finalization never fires, the next queued import never starts - the importer queue stalls.

silo logs confirm the data phase completed, then only UI polling remains:

[LINEAR][<job>] Finished pushing data to batch 🚀 ------------------- [....]
Message processed in 3000ms
GET /api/jobs?source=LINEAR&workspaceId=... 200   # (repeats; no further work)

Root cause

silo hard-gates the TRANSFORMED → PUSHING transition on a report field the backend never exposes.

silo - apps/silo/dist/start.mjs, BaseDataMigrator.update():

case "TRANSFORMED":
  if (report.transformed_batch_count != null && report.total_batch_count != null) {
    if (report.transformed_batch_count + 1 === report.total_batch_count) {
      // -> status "PUSHING"
    } else {
      // -> increment transformed_batch_count
    }
  }

backend - the ImportReport model (plane/ee/models/job.py) defines only total_batch_count,
completed_batch_count, total_issue_count, imported_issue_count, errored_issue_count. There is
no transformed_batch_count field or migration, and ImportReportCountIncrementAPIView
(plane/silo/views/importer_report.py) never increments it. The API serializer
ImportReportAPISerializer (plane/silo/serializers/job.py) is a ModelSerializer with
fields = "__all__", so it cannot surface a field that doesn't exist on the model.

Therefore the API response has no transformed_batch_count key, so in silo
report.transformed_batch_count is undefined, undefined != null is false, the whole
TRANSFORMED block is skipped, the job never reaches PUSHING, and the code that marks the job
finished (and dequeues the next job) is never reached.

Issues still land because the backend's Celery consumer creates them independently of this status
field, which is why the project fills up while the job appears "stuck".

Impact

• Every import ends in a permanent TRANSFORMING state (reads as hung/broken to users).
• The importer queue stalls: no further import can start until the stuck job is cleared manually.

Workaround (operational, no code change)

After the batches complete, force the job to FINISHED, guarded so it only touches jobs whose data
load is genuinely done:

UPDATE import_jobs
SET status = 'FINISHED', updated_at = now()
WHERE status = 'TRANSFORMING'
  AND report_id IN (
    SELECT id FROM import_reports
    WHERE completed_batch_count >= total_batch_count
      AND total_batch_count > 0
  );

Must be repeated for every new import until the defect is fixed.

Suggested fix

Make the backend provide what silo consumes - on ImportReport:

1. Add transformed_batch_count = models.IntegerField(default=0) + migration.
2. Increment it in ImportReportCountIncrementAPIView alongside the existing counters.

(ImportReportAPISerializer needs no change, given fields = "__all__".) With the field present and
incremented, the shipped silo state machine advances to PUSHING / FINISHED normally.

Alternative: change silo to not depend on transformed_batch_count (e.g. gate on
completed_batch_count / total_batch_count, which the backend already provides).

Fields

Steps to reproduce

<already-stated-above>

Environment

Production

Browser

Latest Chromium

Edition

Commercial

Version

v3.0.0, v3.0.1, v3.1.0, v3.1.4, v3.1.5-rc1, v3.2.0-rc3, preview

[makeplane]

Synced with Plane Workspace 🔄

[GIT-339] [bug]: Plane (self-hosted, commercial) > silo + backend > Linear Import > Jobs never leave TRANSFORMING state