Skip to content

Commit 4f8f39f

Browse files
sahana-sreeramsahana-sreeram
committed
feat(file_processors): add remote::unstructured-api provider
Add Unstructured.io as a remote file processor provider supporting 65+ file formats. Implementation follows the same pattern as remote::docling-serve provider with unit tests providing coverage Signed-off-by: Sahana Sreeram <sahanasreeram01@gmail.com>
1 parent 34186e4 commit 4f8f39f

8 files changed

Lines changed: 806 additions & 0 deletions

File tree

‎docs/docs/providers/file_processors/remote_unstructured-api.mdx‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
description: |
3+
[Unstructured.io](https://unstructured.io) is a multi-format document parser that supports 65+ file types
4+
including emails (EML/MSG), legacy documents, presentations, spreadsheets, and more. This provider uses
5+
the Unstructured.io SaaS API for cloud-based document processing with advanced table and image detection.
6+
7+
## Supported Formats
8+
9+
- **Documents**: PDF, DOC, DOCX, PPTX, XLSX, ODT, RTF, EPUB
10+
- **Email**: EML, MSG (unique capability)
11+
- **Web**: HTML, Markdown, XML, JSON
12+
- **Images**: PNG, JPG, TIFF (with OCR)
13+
- **Text**: TXT, CSV
14+
- **65+ formats total** — see [Unstructured format support](https://docs.unstructured.io/pipelines/supported-file-types)
15+
16+
## Features
17+
18+
- **Multi-format support** — 65+ file types including email formats (EML/MSG)
19+
- **Cloud-based processing** — no local dependencies or system requirements
20+
- **Table detection** — extracts tables with structure preservation
21+
- **Image detection** — identifies and extracts image elements
22+
- **SOC2/HIPAA/GDPR certified** — suitable for regulated industries
23+
24+
## Usage
25+
26+
Get an API key from [Unstructured.io](https://unstructured.io) (free tier available), then start OGX:
27+
28+
```bash
29+
UNSTRUCTURED_API_KEY=your-api-key ogx stack run \
30+
--providers "file_processors=remote::unstructured-api,files=inline::localfs,vector_io=inline::faiss,inference=inline::sentence-transformers,inference=remote::ollama" \
31+
--port 8321
32+
```
33+
34+
Or add it to a custom `run.yaml`:
35+
36+
```yaml
37+
file_processors:
38+
- provider_id: unstructured
39+
provider_type: remote::unstructured-api
40+
config:
41+
api_key: ${env.UNSTRUCTURED_API_KEY}
42+
```
43+
44+
## When to Use
45+
46+
- **Diverse formats**: Need to process emails, legacy documents, or 10+ different file types
47+
- **Managed service**: Want zero setup and no system dependencies
48+
- **Compliance**: Require SOC2/HIPAA/GDPR certified processing
49+
- **Email RAG**: Building customer support or communication archive applications
50+
51+
For faster processing with fewer formats, use `inline::docling` instead.
52+
53+
## Performance
54+
55+
- Processing speed: ~1-2 seconds per page
56+
- Best for: Documents <100 pages
57+
- Cost: ~$0.01 per page (verify current pricing with Unstructured.io)
58+
59+
## Documentation
60+
61+
See [Unstructured.io documentation](https://docs.unstructured.io) for API details and format support.
62+
sidebar_label: Remote - Unstructured-Api
63+
title: remote::unstructured-api
64+
---
65+
66+
# remote::unstructured-api
67+
68+
## Description
69+
70+
71+
[Unstructured.io](https://unstructured.io) is a multi-format document parser that supports 65+ file types
72+
including emails (EML/MSG), legacy documents, presentations, spreadsheets, and more. This provider uses
73+
the Unstructured.io SaaS API for cloud-based document processing with advanced table and image detection.
74+
75+
## Supported Formats
76+
77+
- **Documents**: PDF, DOC, DOCX, PPTX, XLSX, ODT, RTF, EPUB
78+
- **Email**: EML, MSG (unique capability)
79+
- **Web**: HTML, Markdown, XML, JSON
80+
- **Images**: PNG, JPG, TIFF (with OCR)
81+
- **Text**: TXT, CSV
82+
- **65+ formats total** — see [Unstructured format support](https://docs.unstructured.io/pipelines/supported-file-types)
83+
84+
## Features
85+
86+
- **Multi-format support** — 65+ file types including email formats (EML/MSG)
87+
- **Cloud-based processing** — no local dependencies or system requirements
88+
- **Table detection** — extracts tables with structure preservation
89+
- **Image detection** — identifies and extracts image elements
90+
- **SOC2/HIPAA/GDPR certified** — suitable for regulated industries
91+
92+
## Usage
93+
94+
Get an API key from [Unstructured.io](https://unstructured.io) (free tier available), then start OGX:
95+
96+
```bash
97+
UNSTRUCTURED_API_KEY=your-api-key ogx stack run \
98+
--providers "file_processors=remote::unstructured-api,files=inline::localfs,vector_io=inline::faiss,inference=inline::sentence-transformers,inference=remote::ollama" \
99+
--port 8321
100+
```
101+
102+
Or add it to a custom `run.yaml`:
103+
104+
```yaml
105+
file_processors:
106+
- provider_id: unstructured
107+
provider_type: remote::unstructured-api
108+
config:
109+
api_key: ${env.UNSTRUCTURED_API_KEY}
110+
```
111+
112+
## When to Use
113+
114+
- **Diverse formats**: Need to process emails, legacy documents, or 10+ different file types
115+
- **Managed service**: Want zero setup and no system dependencies
116+
- **Compliance**: Require SOC2/HIPAA/GDPR certified processing
117+
- **Email RAG**: Building customer support or communication archive applications
118+
119+
For faster processing with fewer formats, use `inline::docling` instead.
120+
121+
## Performance
122+
123+
- Processing speed: ~1-2 seconds per page
124+
- Best for: Documents <100 pages
125+
- Cost: ~$0.01 per page (verify current pricing with Unstructured.io)
126+
127+
## Documentation
128+
129+
See [Unstructured.io documentation](https://docs.unstructured.io) for API details and format support.
130+
131+
132+
## Configuration
133+
134+
| Field | Type | Required | Default | Description |
135+
|-------|------|----------|---------|-------------|
136+
| `api_key` | `SecretStr` | No | | API key for authenticating with Unstructured.io SaaS API (get one from https://unstructured.io) |
137+
| `default_chunk_size_tokens` | `int` | No | 800 | Default chunk size in tokens when chunking_strategy type is 'auto' |
138+
139+
## Sample Configuration
140+
141+
```yaml
142+
api_key: ${env.UNSTRUCTURED_API_KEY}
143+
```

‎pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,7 @@ starter = [
120120
"torch",
121121
"tqdm",
122122
"tree_sitter",
123+
"unstructured-client>=0.25.0,<0.44.0",
123124
"weaviate-client>=4.16.5",
124125
]
125126

@@ -205,6 +206,7 @@ unit = [
205206
"ollama",
206207
"sqlite-vec",
207208
"together",
209+
"unstructured-client>=0.25.0,<0.44.0",
208210
]
209211
# These are the core dependencies required for running integration tests. They are shared across all
210212
# providers. If a provider requires additional dependencies, please add them to your environment
@@ -531,6 +533,7 @@ module = [
531533
"google.genai.*",
532534
"docling.*",
533535
"docling_core.*",
536+
"unstructured_client.*",
534537
]
535538
ignore_missing_imports = true
536539

‎src/ogx/providers/registry/file_processors.py‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,76 @@ def available_providers() -> list[ProviderSpec]:
191191
## Documentation
192192
193193
See [Docling Serve's documentation](https://github.com/docling-project/docling-serve/blob/main/docs/README.md) for more details on setup and configuration.
194+
""",
195+
),
196+
RemoteProviderSpec(
197+
api=Api.file_processors,
198+
provider_type="remote::unstructured-api",
199+
adapter_type="unstructured-api",
200+
pip_packages=["unstructured-client>=0.44.0"],
201+
module="ogx.providers.remote.file_processor.unstructured_api",
202+
config_class="ogx.providers.remote.file_processor.unstructured_api.UnstructuredApiFileProcessorConfig",
203+
api_dependencies=[Api.files],
204+
description="""
205+
[Unstructured.io](https://unstructured.io) is a multi-format document parser that supports 65+ file types
206+
including emails (EML/MSG), legacy documents, presentations, spreadsheets, and more. This provider uses
207+
the Unstructured.io SaaS API for cloud-based document processing with advanced table and image detection.
208+
209+
## Supported Formats
210+
211+
- **Documents**: PDF, DOC, DOCX, PPTX, XLSX, ODT, RTF, EPUB
212+
- **Email**: EML, MSG (unique capability)
213+
- **Web**: HTML, Markdown, XML, JSON
214+
- **Images**: PNG, JPG, TIFF (with OCR)
215+
- **Text**: TXT, CSV
216+
- **65+ formats total** — see [Unstructured format support](https://docs.unstructured.io/pipelines/supported-file-types)
217+
218+
## Features
219+
220+
- **Multi-format support** — 65+ file types including email formats (EML/MSG)
221+
- **Cloud-based processing** — no local dependencies or system requirements
222+
- **Table detection** — extracts tables with structure preservation
223+
- **Image detection** — identifies and extracts image elements
224+
- **SOC2/HIPAA/GDPR certified** — suitable for regulated industries
225+
226+
## Usage
227+
228+
Get an API key from [Unstructured.io](https://unstructured.io) (free tier available), then start OGX:
229+
230+
```bash
231+
UNSTRUCTURED_API_KEY=your-api-key ogx stack run \\
232+
--providers "file_processors=remote::unstructured-api,files=inline::localfs,vector_io=inline::faiss,inference=inline::sentence-transformers,inference=remote::ollama" \\
233+
--port 8321
234+
```
235+
236+
Or add it to a custom `run.yaml`:
237+
238+
```yaml
239+
file_processors:
240+
- provider_id: unstructured
241+
provider_type: remote::unstructured-api
242+
config:
243+
api_key: ${env.UNSTRUCTURED_API_KEY}
244+
```
245+
246+
## When to Use
247+
248+
- **Diverse formats**: Need to process emails, legacy documents, or 10+ different file types
249+
- **Managed service**: Want zero setup and no system dependencies
250+
- **Compliance**: Require SOC2/HIPAA/GDPR certified processing
251+
- **Email RAG**: Building customer support or communication archive applications
252+
253+
For faster processing with fewer formats, use `inline::docling` instead.
254+
255+
## Performance
256+
257+
- Processing speed: ~1-2 seconds per page
258+
- Best for: Documents <100 pages
259+
- Cost: ~$0.01 per page (verify current pricing with Unstructured.io)
260+
261+
## Documentation
262+
263+
See [Unstructured.io documentation](https://docs.unstructured.io) for API details and format support.
194264
""",
195265
),
196266
]

‎src/ogx/providers/remote/file_processor/unstructured_api/__init__.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Copyright (c) The OGX Contributors.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the terms described in the LICENSE file in
5+
# the root directory of this source tree.
6+
7+
from typing import Any
8+
9+
from ogx_api import Api
10+
11+
from .config import UnstructuredApiFileProcessorConfig
12+
13+
14+
async def get_adapter_impl(config: UnstructuredApiFileProcessorConfig, deps: dict[Api, Any]):
15+
from .unstructured_api import UnstructuredApiFileProcessor
16+
17+
assert isinstance(config, UnstructuredApiFileProcessorConfig), f"Unexpected config type: {type(config)}"
18+
19+
files_api = deps.get(Api.files)
20+
if files_api is None:
21+
raise ValueError(
22+
"Failed to find required dependency: files API is required for unstructured-api file processor"
23+
)
24+
25+
impl = UnstructuredApiFileProcessor(config, files_api)
26+
return impl
27+
28+
29+
__all__ = ["UnstructuredApiFileProcessorConfig", "get_adapter_impl"]

‎src/ogx/providers/remote/file_processor/unstructured_api/config.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Copyright (c) The OGX Contributors.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the terms described in the LICENSE file in
5+
# the root directory of this source tree.
6+
7+
from typing import Any
8+
9+
from pydantic import BaseModel, Field, SecretStr
10+
11+
from ogx_api.vector_io import VectorStoreChunkingStrategyStaticConfig
12+
13+
14+
class UnstructuredApiFileProcessorConfig(BaseModel):
15+
"""Configuration for Unstructured.io API file processor."""
16+
17+
api_key: SecretStr = Field(
18+
description="API key for authenticating with Unstructured.io SaaS API (get one from https://unstructured.io)"
19+
)
20+
default_chunk_size_tokens: int = Field(
21+
default=VectorStoreChunkingStrategyStaticConfig.model_fields["max_chunk_size_tokens"].default,
22+
ge=100,
23+
le=4096,
24+
description="Default chunk size in tokens when chunking_strategy type is 'auto'",
25+
)
26+
27+
@classmethod
28+
def sample_run_config(cls, **kwargs: Any) -> dict[str, Any]:
29+
return {
30+
"api_key": "${env.UNSTRUCTURED_API_KEY}",
31+
}

0 commit comments

Comments
 (0)