Plan and implement full TAPAS-xq processing pipeline

[candyhazlett]

Description

Right now the TAPAS-xq integration only handles the first step — storing the TEI file via `ProcessTeiFileJob`. The full pipeline will eventually need to cover:

1. Store TEI file to TAPAS-xq
2. Get MODS metadata back from TAPAS-xq
3. Run XSL transformations per view package to produce HTML fragments for the TAPAS reader

Each of these is its own job with its own error handling and status. This issue is for planning and building out the rest of that pipeline.

Related issues

Prerequisites

• Rename processing_status to a more specific name #43 — Rename `processing_status` to a more specific name (should happen before or alongside this work)
• Handle 422 from tapas-xq as InvalidTeiError with descriptive message #51 — Handle 422 from tapas-xq as `InvalidTeiError`
• Client#post and #delete don't rescue 404 from tapas-xq #52 — `Client#post` and `#delete` don't rescue 404
• Refactor TapasXq::Client — audit @ usage and extract shared error handlers #41 — Refactor `TapasXq::Client`

Sub-tasks tracked separately

• Add TapasXq::DeletionService to wrap DELETE endpoints #53 — Add `TapasXq::DeletionService`
• Add DeleteFromTapasXqJob and DeleteProjectFromTapasXqJob #54 — Add `DeleteFromTapasXqJob` and `DeleteProjectFromTapasXqJob`
• Add after_destroy callbacks on CoreFile and Project to clean up tapas-xq #55 — Add `after_destroy` callbacks on `CoreFile` and `Project`
• Add SyncTapasXqMetadataJob and after_update callbacks to keep tapas-xq in sync #56 — Add `SyncTapasXqMetadataJob` and `after_update` callbacks
• Add TapasXq::ViewPackagesService and sync ViewPackage model from tapas-xq registry #57 — Add `TapasXq::ViewPackagesService` and sync `ViewPackage` model
• Add TapasXq::DeriveReaderService to generate reading interface XHTML #58 — Add `TapasXq::DeriveReaderService` to generate reading interface XHTML

Related storage work

• Move MODS and TFE XML out of SQL storage #44 — Move MODS and TFE XML out of SQL storage (also covers `RetrievalService` wiring decision)

Steps

1. Resolve prerequisites (Rename processing_status to a more specific name #43, Handle 422 from tapas-xq as InvalidTeiError with descriptive message #51, Client#post and #delete don't rescue 404 from tapas-xq #52, Refactor TapasXq::Client — audit @ usage and extract shared error handlers #41) before or alongside this work
2. Design the job chain — likely jobs that trigger the next step on success
3. Wire `TapasXq::RetrievalService` into the application flow — the service already exists with `retrieve_tei`, `retrieve_mods`, and `retrieve_tfe` methods; the work is deciding where each is called (see Move MODS and TFE XML out of SQL storage #44 for details)
4. Implement the XSL transformation step via `TapasXq::DeriveReaderService` (Add TapasXq::DeriveReaderService to generate reading interface XHTML #58), blocked on view packages sync (Add TapasXq::ViewPackagesService and sync ViewPackage model from tapas-xq registry #57)
5. Update status tracking to handle multiple pipeline stages
6. Implement `POST /tapas-xq/:project-id/:doc-id/tei` (TEI-only store, no MODS/TFE regeneration) — low priority; only needed if a use case emerges for re-storing TEI without regenerating supplemental files
7. Write specs for each new job and service

Acceptance criteria

• TEI store, MODS retrieval, and XSL transformation each have their own job
• Status tracking reflects per-stage state
• Pipeline works end-to-end in development with TAPAS-xq running
• Specs cover happy path and error cases for each stage

[candyhazlett]

Related issues and clarifications

The sub-tasks of this pipeline are now tracked individually. Links below for coordination:

Error handling (prerequisite)

• Handle 422 from tapas-xq as InvalidTeiError with descriptive message #51 — Handle 422 from tapas-xq as InvalidTeiError with descriptive message
• Client#post and #delete don't rescue 404 from tapas-xq #52 — Client#post and #delete don't rescue 404

Client refactor (prerequisite)

• Refactor TapasXq::Client — audit @ usage and extract shared error handlers #41 — Refactor TapasXq::Client — audit instance variable usage and extract shared error handlers

Pipeline additions

• Add after_destroy callbacks on CoreFile and Project to clean up tapas-xq #55 — Add after_destroy callbacks on CoreFile and Project to clean up tapas-xq
• Add DeleteFromTapasXqJob and DeleteProjectFromTapasXqJob #54 — Add DeleteFromTapasXqJob and DeleteProjectFromTapasXqJob
• Add SyncTapasXqMetadataJob and after_update callbacks to keep tapas-xq in sync #56 — Add SyncTapasXqMetadataJob and after_update callbacks
• Add TapasXq::ViewPackagesService and sync ViewPackage model from tapas-xq registry #57 — Add TapasXq::ViewPackagesService and sync ViewPackage model
• Add TapasXq::DeriveReaderService to generate reading interface XHTML #58 — Add TapasXq::DeriveReaderService to generate reading interface XHTML

Related model/storage work

• Rename processing_status to a more specific name #43 — Rename processing_status to a more specific name (should happen before or alongside this work, as noted in the issue description)
• Move MODS and TFE XML out of SQL storage #44 — Move MODS and TFE XML out of SQL storage (see comment on Move MODS and TFE XML out of SQL storage #44 re: RetrievalService dependency)

---

Note on step 3 ("Implement MODS retrieval"): TapasXq::RetrievalService already exists and wraps all three GET endpoints (retrieve_tei, retrieve_mods, retrieve_tfe). The work here is deciding where each method gets called in the application flow and wiring it in — not building the service from scratch. See the comment on #44 for the full decision breakdown, since the answer depends on how MODS/TFE storage is resolved.