feat: durable out-of-process job scheduling for file_processors (#6160)

## Summary

Inline providers run in the server's event loop today, so a single large
`file_processors.process_file` call blocks the whole server, and the
work is an
in-process closure that dies on restart.

This PR introduces **one substrate** that delivers both **process
isolation** and
**durable scheduling**: a job is an addressable record in a SQL-backed
queue that a
separate worker process leases by id. That simultaneously gives
non-blocking
execution (separate process, separate GIL) and survives restarts. The
engine is
generic; only a thin per-API adapter is provider-specific.

Scope is intentionally **file_processors only**. Migrating `batches` and
promoting
other inline APIs are deferred follow-ups.

## Design

New package `src/ogx/core/jobs/`:

- **`queue.py`** — `JobQueue` over `SqlStore` (the DB is the IPC
channel). Atomic
guarded-UPDATE leasing claims a job exclusively; `fail()` requeues until
`max_attempts`; `reclaim_stale()` returns expired leases to `scheduled`
on startup
  (the durability/restart property).
- **`worker.py`** — `WorkerPool` spawns N processes (spawn context, own
GIL). Each
rebuilds the real provider impl and its dependencies from a
`ProviderDescriptor`,
then loops: lease → run method → complete/fail, heartbeating the lease.
- **`proxy.py`** — generic `JobBackedProxy` base (enqueue, blocking
run-and-wait,
  get, cancel, list) plus a `WORKER_PROXY_FACTORIES` registry.
- **`file_processor_proxy.py`** — the only file-processor-specific code:
maps the
`FileProcessors` protocol onto the base, stages direct uploads into the
Files API
so payloads carry only a `file_id` (never bytes), and self-registers via
  `register_worker_proxy(Api.file_processors, ...)`.
- **`dispatch.py` / `runtime.py` / `bootstrap.py`** — per-(api, method)
payload
(de)serialization, and process-global runtime construction wired into
the stack
  lifecycle.

Wiring:

- `InlineProviderSpec` gains `execution_mode` (`"inline" | "worker"`),
default
`"inline"` so **no other API's topology changes**. The resolver returns
the
  registered worker proxy instead of importing the impl in-process when
  `execution_mode == "worker"`.
- All four `file_processors` specs (auto, pypdf, markitdown, docling)
are set to
worker mode — the problem is file size and blocking in general, not CPU
heaviness.

### Adding worker mode for another API later

Purely additive, no engine/resolver/worker changes: set `execution_mode:
worker` on
the spec, add a `dispatch.py` entry, write a ~15-line proxy adapter,
call
`register_worker_proxy(...)`.

## API changes

`file_processors` is `/v1alpha`, so no minimum deprecation window
applies.

- `POST /file-processors/process` is marked **deprecated** — it still
works and keeps
  its signature, but now runs out-of-process via the job engine.
- New async endpoints: `POST /file-processors/jobs` (submit),
  `GET /file-processors/jobs`, `GET /file-processors/jobs/{job_id}`,
`POST /file-processors/jobs/{job_id}/cancel`, with `ProcessFileJob` and
  `ListProcessFileJobsResponse` models. OpenAPI specs regenerated.

## Test plan

Unit coverage for every layer of the substrate:

- `test_queue.py` — lease/complete/fail/retry/cancel/reclaim,
lease-expiry recovery
- `test_worker.py` — worker dispatch (complete and failure paths)
- `test_build_impl.py` — descriptor → real provider reconstruction (with
a dependency)
- `test_proxy.py` — proxy round-trip incl. the deprecated blocking
surface
- `test_resolver_worker_mode.py` — the resolver seam: a worker spec
mounts the proxy
  and registers a reconstructable descriptor that captures dependencies

```bash
uv run --no-sync pytest tests/unit/core/jobs/ -q
```

```
25 passed
```

No regressions in the surrounding suites:

```bash
uv run --no-sync pytest tests/unit/core tests/unit/providers/files -q
```

```
347 passed (+ jobs)
```

## Out of scope (follow-ups)

- Streaming the upload to storage so large files never transit server
memory.
- Migrating `batches` onto this substrate.
- Off-box workers over HTTP (the queue boundary already makes this
reachable).
- Promoting other inline APIs to `execution_mode: worker`.
- End-to-end integration tests (real spawned-worker round-trip,
durability across
  restart, non-blocking-latency proof).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Charlie Doern <cdoern@redhat.com>
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
Co-authored-by: Francisco Javier Arceo <farceo@redhat.com>

Author: cdoern

client-sdks/stainless/openapi.yml

@@ -3717,13 +3717,158 @@ paths:
       tags:
       - File Processors
       summary: Process a file into chunks ready for vector database storage.
-      description: Process a file into chunks ready for vector database storage. Supports direct upload via multipart form or processing files already uploaded to file storage via file_id. Exactly one of file or file_id must be provided.
+      description: Deprecated. This endpoint blocks until processing completes, which can hold a connection open for large files. Use POST /file-processors/jobs to submit the file asynchronously and poll GET /file-processors/jobs/{job_id} for the result. Supports direct upload via multipart form or processing files already uploaded to file storage via file_id. Exactly one of file or file_id must be provided.
       operationId: process_file_v1alpha_file_processors_process_post
       requestBody:
         content:
           multipart/form-data:
             schema:
               $ref: '#/components/schemas/Body_process_file_v1alpha_file_processors_process_post'
+      deprecated: true
+  /v1alpha/file-processors/jobs:
+    get:
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ListProcessFileJobsResponse'
+        '400':
+          $ref: '#/components/responses/BadRequest400'
+          description: Bad Request
+        '429':
+          $ref: '#/components/responses/TooManyRequests429'
+          description: Too Many Requests
+        '500':
+          $ref: '#/components/responses/InternalServerError500'
+          description: Internal Server Error
+        default:
+          $ref: '#/components/responses/DefaultError'
+          description: Default Response
+      tags:
+      - File Processors
+      summary: List file-processing jobs.
+      description: List file-processing jobs, most recent first.
+      operationId: list_process_file_jobs_v1alpha_file_processors_jobs_get
+      parameters:
+      - name: after
+        in: query
+        required: false
+        schema:
+          anyOf:
+          - type: string
+          - type: 'null'
+          description: Return jobs after this job ID.
+          title: After
+        description: Return jobs after this job ID.
+      - name: limit
+        in: query
+        required: false
+        schema:
+          type: integer
+          maximum: 100
+          minimum: 1
+          description: Maximum jobs to return.
+          default: 100
+          title: Limit
+        description: Maximum jobs to return.
+    post:
+      responses:
+        '200':
+          description: The created file-processing job.
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ProcessFileJob'
+        '400':
+          $ref: '#/components/responses/BadRequest400'
+          description: Bad Request
+        '429':
+          $ref: '#/components/responses/TooManyRequests429'
+          description: Too Many Requests
+        '500':
+          $ref: '#/components/responses/InternalServerError500'
+          description: Internal Server Error
+        default:
+          $ref: '#/components/responses/DefaultError'
+          description: Default Response
+      tags:
+      - File Processors
+      summary: Submit a file for asynchronous processing.
+      description: Submit a file for processing and return immediately with a job handle. The work runs out-of-process so the server is not blocked. Poll GET /file-processors/jobs/{job_id} until the status is terminal. Supports direct upload via multipart form or a previously uploaded file_id.
+      operationId: create_process_file_job_v1alpha_file_processors_jobs_post
+      requestBody:
+        content:
+          multipart/form-data:
+            schema:
+              $ref: '#/components/schemas/Body_create_process_file_job_v1alpha_file_processors_jobs_post'
+  /v1alpha/file-processors/jobs/{job_id}:
+    get:
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ProcessFileJob'
+        '400':
+          $ref: '#/components/responses/BadRequest400'
+          description: Bad Request
+        '429':
+          $ref: '#/components/responses/TooManyRequests429'
+          description: Too Many Requests
+        '500':
+          $ref: '#/components/responses/InternalServerError500'
+          description: Internal Server Error
+        default:
+          $ref: '#/components/responses/DefaultError'
+          description: Default Response
+      tags:
+      - File Processors
+      summary: Retrieve a file-processing job.
+      description: Retrieve the current state of a file-processing job, including its result once completed.
+      operationId: retrieve_process_file_job_v1alpha_file_processors_jobs__job_id__get
+      parameters:
+      - name: job_id
+        in: path
+        required: true
+        schema:
+          type: string
+          title: Job Id
+  /v1alpha/file-processors/jobs/{job_id}/cancel:
+    post:
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ProcessFileJob'
+        '400':
+          $ref: '#/components/responses/BadRequest400'
+          description: Bad Request
+        '429':
+          $ref: '#/components/responses/TooManyRequests429'
+          description: Too Many Requests
+        '500':
+          $ref: '#/components/responses/InternalServerError500'
+          description: Internal Server Error
+        default:
+          $ref: '#/components/responses/DefaultError'
+          description: Default Response
+      tags:
+      - File Processors
+      summary: Cancel a file-processing job.
+      description: Cancel a scheduled or in-progress file-processing job.
+      operationId: cancel_process_file_job_v1alpha_file_processors_jobs__job_id__cancel_post
+      parameters:
+      - name: job_id
+        in: path
+        required: true
+        schema:
+          type: string
+          title: Job Id
   /v1/messages:
     post:
       responses:
@@ -11547,6 +11692,32 @@ components:
         Represents token usage details including input tokens, output tokens, a
         breakdown of output tokens, and the total tokens used. Only populated on
         batches created after September 7, 2025.
+    Body_create_process_file_job_v1alpha_file_processors_jobs_post:
+      properties:
+        file:
+          anyOf:
+          - type: string
+            contentMediaType: application/octet-stream
+          - type: 'null'
+          description: The File object to be uploaded and processed. Mutually exclusive with file_id.
+        file_id:
+          anyOf:
+          - type: string
+          - type: 'null'
+          description: ID of file already uploaded to file storage. Mutually exclusive with file.
+        options:
+          anyOf:
+          - additionalProperties: true
+            type: object
+          - type: 'null'
+          description: Optional processing options. Provider-specific parameters.
+        chunking_strategy:
+          anyOf:
+          - type: string
+          - type: 'null'
+          description: Optional chunking strategy for splitting content into chunks. Must be valid JSON string.
+      type: object
+      title: Body_create_process_file_job_v1alpha_file_processors_jobs_post
     Body_create_skill_v1alpha_skills_post:
       properties:
         file:
@@ -12728,6 +12899,16 @@ components:
       - cached_tokens
       title: InputTokensDetails
       description: A detailed breakdown of the input tokens.
+    JobStatus:
+      type: string
+      enum:
+      - completed
+      - in_progress
+      - failed
+      - scheduled
+      - cancelled
+      title: JobStatus
+      description: Status of a job execution.
     ListConnectorsResponse:
       properties:
         data:
@@ -12768,6 +12949,24 @@ components:
       - has_more
       title: ListMessageBatchesResponse
       description: Response from GET /v1/messages/batches.
+    ListProcessFileJobsResponse:
+      properties:
+        data:
+          items:
+            $ref: '#/components/schemas/ProcessFileJob'
+          type: array
+          title: Data
+          description: The list of file-processing jobs.
+        has_more:
+          type: boolean
+          title: Has More
+          description: Whether more jobs are available after this page.
+          default: false
+      type: object
+      required:
+      - data
+      title: ListProcessFileJobsResponse
+      description: Response model listing file-processing jobs.
     ListSkillVersionsResponse:
       properties:
         object:
@@ -13634,6 +13833,44 @@ components:
       - reasoning_tokens
       title: OutputTokensDetails
       description: A detailed breakdown of the output tokens.
+    ProcessFileJob:
+      properties:
+        job_id:
+          type: string
+          title: Job Id
+          description: Unique identifier for the job.
+        status:
+          $ref: '#/components/schemas/JobStatus'
+          description: Current execution status of the job.
+        created_at:
+          type: integer
+          title: Created At
+          description: Unix timestamp (seconds) for when the job was created.
+        result:
+          anyOf:
+          - $ref: '#/components/schemas/ProcessFileResponse'
+            title: ProcessFileResponse
+          - type: 'null'
+          description: The processed file result. Present only once the job has completed successfully.
+          title: ProcessFileResponse
+        error:
+          anyOf:
+          - type: string
+          - type: 'null'
+          description: Error message. Present only if the job failed.
+      type: object
+      required:
+      - job_id
+      - status
+      - created_at
+      title: ProcessFileJob
+      description: |-
+        An asynchronous file-processing job.
+
+        Returned when a file is submitted for processing via the job-based API. Poll
+        by job_id until ``status`` is terminal (completed/failed/cancelled). When
+        completed, ``result`` holds the processed chunks; when failed, ``error``
+        explains why.
     ProcessFileResponse:
       properties:
         chunks:
@@ -15048,6 +15285,18 @@ components:
             The container image to use for this implementation. If one is provided, pip_packages will be ignored.
             If a provider depends on other providers, the dependencies MUST NOT specify a container image.
           nullable: true
+        execution_mode:
+          default: inline
+          description: |2
+
+            How this provider's work is executed. 'inline' runs the implementation in the server
+            process (default, unchanged behavior). 'worker' runs the implementation in a separate
+            worker process via the job queue, so long or large operations do not block the server.
+          enum:
+          - inline
+          - worker
+          title: Execution Mode
+          type: string
         description:
           anyOf:
           - type: string
@@ -15787,16 +16036,6 @@ components:
       - batch_id
       title: CancelBatchRequest
       type: object
-    JobStatus:
-      description: Status of a job execution.
-      enum:
-      - completed
-      - in_progress
-      - failed
-      - scheduled
-      - cancelled
-      title: JobStatus
-      type: string
     Job:
       description: A job execution instance with status tracking.
       properties:

docs/docs/providers/file_processors/inline_docling.mdx

@@ -11,8 +11,8 @@ description: |
   - **Layout preservation** — tables, lists, and nested structures are converted to Markdown
   - **Multi-format support** — PDF, DOCX, PPTX, HTML, and images
   - **Better RAG quality** — structured chunks with heading metadata produce more relevant retrieval results
-  - **VLM-based processing** — optionally route Vision Language Model inference through the stack's model-serving
-    infrastructure for richer document understanding (layout analysis, OCR via vision models)
+  - **VLM configuration validation** — VLM processing requires worker-side inference routing, which is not yet
+    supported; configuring `vlm_model` currently produces a clear startup error
 
   ## Usage
 
@@ -33,23 +33,13 @@ description: |
       config: {}
   ```
 
-  ### Enabling VLM Processing
+  ### VLM Processing
 
-  To enable VLM-based document processing, set `vlm_model` to a vision model registered with the
-  stack's inference API. The VLM pipeline routes inference through the stack's model-serving
-  infrastructure — no separate GPU resources are needed for document processing.
+  VLM-based document processing is not currently available while this provider runs in worker
+  mode because auto-routed inference cannot yet be reconstructed in the worker process. A
+  configuration with `vlm_model` fails startup rather than silently running a different pipeline.
 
-  ```yaml
-  file_processors:
-    - provider_id: docling
-      provider_type: inline::docling
-      config:
-        vlm_model: granite-docling-258M
-        vlm_preset: granite_docling
-  ```
-
-  When `vlm_model` is not set or no inference provider is available, the processor gracefully
-  degrades to the standard non-VLM pipeline.
+  Leave `vlm_model` unset to use the standard non-VLM pipeline.
 
   ## Installation
 
@@ -80,8 +70,8 @@ preserves semantic boundaries. It supports PDF, DOCX, PPTX, HTML, and images.
 - **Layout preservation** — tables, lists, and nested structures are converted to Markdown
 - **Multi-format support** — PDF, DOCX, PPTX, HTML, and images
 - **Better RAG quality** — structured chunks with heading metadata produce more relevant retrieval results
-- **VLM-based processing** — optionally route Vision Language Model inference through the stack's model-serving
-  infrastructure for richer document understanding (layout analysis, OCR via vision models)
+- **VLM configuration validation** — VLM processing requires worker-side inference routing, which is not yet
+  supported; configuring `vlm_model` currently produces a clear startup error
 
 ## Usage
 
@@ -102,23 +92,13 @@ file_processors:
     config: {}
 ```
 
-### Enabling VLM Processing
+### VLM Processing
 
-To enable VLM-based document processing, set `vlm_model` to a vision model registered with the
-stack's inference API. The VLM pipeline routes inference through the stack's model-serving
-infrastructure — no separate GPU resources are needed for document processing.
-
-```yaml
-file_processors:
-  - provider_id: docling
-    provider_type: inline::docling
-    config:
-      vlm_model: granite-docling-258M
-      vlm_preset: granite_docling
-```
+VLM-based document processing is not currently available while this provider runs in worker
+mode because auto-routed inference cannot yet be reconstructed in the worker process. A
+configuration with `vlm_model` fails startup rather than silently running a different pipeline.
 
-When `vlm_model` is not set or no inference provider is available, the processor gracefully
-degrades to the standard non-VLM pipeline.
+Leave `vlm_model` unset to use the standard non-VLM pipeline.
 
 ## Installation