[Feature]: Skip or blacklist previously failed documents during bucket ingestion

[lucaseduoli]

Feature Area

• Ingestion (document processing, upload, Docling)
• Backend/API (Python/FastAPI)

Problem Description

Currently, when executing bucket ingestion through the UI, documents that previously failed to ingest (for instance, corrupt .docx or unsupported/broken files) are re-attempted every single time.

While OpenRAG is smart enough to skip already-ingested files by checking the active knowledge collection, it does not keep a record of failed files. As we continuously add new documents to the bucket, the system ends up wasting resources and time retrying all previously failed files, which significantly slows down the ingestion of new documents.

Proposed Solution

We propose introducing a mechanism to track failed document ingestions and exclude them from subsequent runs:

1. Persistent Failure Registry: Store metadata (e.g., file path, S3/COS URI, file hash, and last error message) of documents that failed to process.
2. Exclude on Ingestion: Update the ingestion pipeline to skip documents that are present in the failure registry.
3. UI Integration:
   • Provide a way in the UI (e.g., in the Knowledge settings or ingestion status page) to view failed documents and their corresponding error messages.
   • Add a "Retry" or "Clear Failure State" action to manually force a retry once a file has been corrected or updated in the bucket.
4. Retry Budget (Optional): Support a maximum retry limit (e.g., attempt to process a failed file up to 3 times before permanently blacklisting it).

Use Case

As a developer/administrator managing knowledge sources with external COS buckets,
I want OpenRAG to remember and skip documents that have consistently failed ingestion,
so that the ingestion pipeline is not bottlenecked and can quickly process new files added to the bucket.

Alternatives Considered

• Manual Bucket Cleanup: Manually identifying and deleting corrupt or unparseable files from the source bucket. However, this is not scalable or feasible when the bucket is fed by automated upstream systems or external users.
• Manual Exclude Rules: Manually configuring exclusion regexes in the UI. This is error-prone and does not dynamically catch unexpected corrupt files.

Priority

• Would improve my workflow