No chuncks are getting generated from pdf file

[090Gamer]

With the MCP setup LLM gets access to the Morphik and performs ingest

list-allowed-directories
search-files
ingest-file-from-path
retrieve-chunks
retrieve-chunks

Then it gets stuck on the retrieve chunks with no progress.

Log file:

2025-06-05 16:19:39,992 - core.workers.ingestion_worker - INFO - File download took 0.00s for 2.80MB
2025-06-05 16:19:41,054 - core.workers.ingestion_worker - INFO - Document retrieval took 1.01s
2025-06-05 16:19:41,078 - core.workers.ingestion_worker - INFO - Initial document update took 0.02s
2025-06-05 16:19:41,085 - core.workers.ingestion_worker - WARNING - No text chunks extracted after parsing. Will attempt to continue and rely on image-based chunks if available.
2025-06-05 16:19:41,086 - core.workers.ingestion_worker - INFO - Text chunking took 0.00s to create 0 chunks
2025-06-05 16:19:41,086 - core.workers.ingestion_worker - ERROR - Error processing ingestion job for file abc.pdf: No content chunks (text or image) could be extracted from the document
2025-06-05 16:19:41,115 - core.workers.ingestion_worker - INFO - Updated document ab3e6c97-df0a-412d-aa4d-57470b4a4593 status to failed

Terminal output:

morphik-1 | INFO: 172.16.9.1:41444 - "POST /retrieve/chunks HTTP/1.1" 200 OK
morphik-1 | INFO: 172.16.9.1:33314 - "POST /ingest/file HTTP/1.1" 200 OK
worker-1 | 16:19:39: 0.15s → e756ce20119547928c41c036e9bc72d6:process_ingestion_job(auth_dict={'entity_type': 'developer', 'entity_id': 'dev_user', 'app_id': None,…)
worker-1 | 16:19:41: 1.13s ← e756ce20119547928c41c036e9bc72d6:process_ingestion_job ● {'status': 'failed', 'filename': 'abc.pdf', 'error': 'No content chunks (text…
morphik-1 | INFO: 172.16.9.1:33314 - "POST /retrieve/chunks HTTP/1.1" 200 OK
postgres-1 | 2025-06-05 16:21:00.990 UTC [57] LOG: checkpoint starting: time
postgres-1 | 2025-06-05 16:21:03.563 UTC [57] LOG: checkpoint complete: wrote 26 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=2.506 s, sync=0.058 s, total=2.573 s; sync files=19, longest=0.008 s, average=0.004 s; distance=23 kB, estimate=1451 kB
morphik-1 | INFO: 172.16.9.1:33314 - "POST /retrieve/chunks HTTP/1.1" 200 OK
morphik-1 | INFO: 172.16.9.1:33314 - "POST /retrieve/docs HTTP/1.1" 200 OK
worker-1 | 16:21:58: recording health: Jun-05 16:21:58 j_complete=3 j_failed=0 j_retried=0 j_ongoing=0 queued=0

Using openAI embeddings and openAI model, though 0 usage of the model so far. Only embeddings are getting triggered, but with zero cost

[090Gamer]

Update. With .txt documents something is happening, but still no data is acessible by LLM:

2025-06-08 19:09:43,685 - core.workers.ingestion_worker - INFO - === Ingestion Performance Summary ===
2025-06-08 19:09:43,685 - core.workers.ingestion_worker - INFO - Total processing time: 5.98s
2025-06-08 19:09:43,685 - core.workers.ingestion_worker - INFO - - parse_file: 2.76s (46.2%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - split_into_chunks: 1.50s (25.0%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - retrieve_document: 1.01s (16.9%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - generate_embeddings: 0.51s (8.6%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - store_chunks_and_update_doc: 0.04s (0.7%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - update_document_parsed: 0.01s (0.2%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - deserialize_auth: 0.00s (0.0%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - download_file: 0.00s (0.0%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - create_chunk_objects: 0.00s (0.0%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - apply_post_parsing_rules: 0.00s (0.0%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - colpali_create_chunks: 0.00s (0.0%)
2025-06-08 19:09:43,686 - core.workers.ingestion_worker - INFO - - colpali_generate_embeddings: 0.00s (0.0%)

But with pdfs fail to get chuncks.

The reply to get-document function:

[
{
"type": "text",
"text": "Document with ID 40f3b771-561e-4aae-aceb-113ceb97246d not found"
}
]

In the console:

INFO: 172.16.9.1:60272 - "POST /documents/40f3b771-561e-4aae-aceb-113ceb97246d HTTP/1.1" 405 Method Not Allowed

Although, if I access the link via browser I get the following:

{
"external_id": "40f3b771-561e-4aae-aceb-113ceb97246d",
"owner": {
"id": "dev_user",
"type": "developer"
},
"content_type": "application/pdf",
"filename": "full_text.pdf",
"metadata": {
"external_id": "40f3b771-561e-4aae-aceb-113ceb97246d"
},
"storage_info": {
"key": "ingest_uploads/741cd0d8-5860-49a8-b599-d45797552e77/full_text.pdf",
"bucket": "storage"
},
"storage_files": [
{
"bucket": "storage",
"key": "ingest_uploads/741cd0d8-5860-49a8-b599-d45797552e77/full_text.pdf",
"version": 1,
"filename": "full_text.pdf",
"content_type": "application/pdf",
"timestamp": "2025-06-08T19:12:08.943151Z"
}
],
"system_metadata": {
"error": "No content chunks (text or image) could be extracted from the document",
"status": "failed",
"content": "",
"created_at": "2025-06-08T19:12:08.913481+00:00",
"updated_at": "2025-06-08T19:12:13.297462+00:00",
"folder_name": null
},
"additional_metadata": {
},
"access_control": {
"admins": [
"dev_user"
],
"readers": [
"dev_user"
],
"user_id": [
"dev_user"
],
"writers": [
"dev_user"
],
"app_access": []
},
"chunk_ids": []
}

[ArnavAgrawal03]

Looking into it right now!

[ArnavAgrawal03]

have you verified the allowed directories?

[090Gamer]

@ArnavAgrawal03 Which directories should be allowed?
I fixed my install by adding a network of mcp-client to the morphik-core components. Now get summery of txt docs, but pdfs are still give no chuncks.

I use two docker containers (stacks) to run morphic-core and mcp client.
I bind a volumes from my system towards:

• morphik core: app/storage , app/logs and app/morpik.toml
• MCP client: app/storage and use the flag of allowed dir with app/storage

Overall the list-allowed-dir and ingest-file-from-path and delete-file methods work with mcp client. But some methods are not allowed as you can see from the terminal output. No problems with methods now.

As a sidenote, I only managed to run the client with the URI of my server on which I am running the docker with port number opened for morphik-core, with localhost or 0.0.0.0 with proper port there was an error on ingestion.

I can get a summery of .txt file now, but not of .pdf file (no chuncks are generated for some reason)

The summery is super slow, though I use openAI models for everything
text file with 4 words:

morphik-1 | 2025-06-09 07:12:47 - core.api - INFO - === Retrieve Chunks: 'test.txt...' Performance Summary ===
morphik-1 | 2025-06-09 07:12:47 - core.api - INFO - Total time: 43.66s
morphik-1 | 2025-06-09 07:12:47 - core.api - INFO - - retrieve_embeddings_and_auth: 43.61s (99.9%)

[090Gamer]

@ArnavAgrawal03 In terminal logs when ingesting pdf I also see :

2025-06-10 18:10:16 - core.services.document_service - INFO - No authorized documents found

I can clearly see the pdf in mounted directory (under ingested_uploads), the same way as for .txt files

[ArnavAgrawal03]

Debugging right now - thanks for letting me know

[ArnavAgrawal03]

This should be resolved now./