Skip to content

Commit b041781

Browse files
authored
Added dynamic batching with file-type optimization and Removed batch config runtime arguments from ingestion API (#305)
* Remove batch config runtime arguments from ingestion API Signed-off-by: Swapnil Masurekar <smasurekar@nvidia.com> * feat: add dynamic batching with file-type optimization Signed-off-by: Swapnil Masurekar <smasurekar@nvidia.com> --------- Signed-off-by: Swapnil Masurekar <smasurekar@nvidia.com>
1 parent 7ef0517 commit b041781

11 files changed

Lines changed: 724 additions & 49 deletions

File tree

deploy/compose/docker-compose-ingestor-server.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,7 @@ services:
147147
# NV-Ingest Batch Mode Configurations
148148
NV_INGEST_FILES_PER_BATCH: ${NV_INGEST_FILES_PER_BATCH:-16}
149149
NV_INGEST_CONCURRENT_BATCHES: ${NV_INGEST_CONCURRENT_BATCHES:-4}
150+
ENABLE_NV_INGEST_DYNAMIC_BATCHING: ${ENABLE_NV_INGEST_DYNAMIC_BATCHING:-True}
150151

151152
# Tracing
152153
APP_TRACING_ENABLED: ${APP_TRACING_ENABLED:-"False"}

deploy/helm/nvidia-blueprint-rag/values.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -415,6 +415,7 @@ ingestor-server:
415415
# === NV-Ingest Batch Mode Configurations ===
416416
NV_INGEST_FILES_PER_BATCH: "16"
417417
NV_INGEST_CONCURRENT_BATCHES: "4"
418+
ENABLE_NV_INGEST_DYNAMIC_BATCHING: "True"
418419

419420
# === Tracing ===
420421
APP_TRACING_ENABLED: "False"

docs/accuracy_perf.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@ These variables require trial and error tuning for optimal performance.
6161

6262
| Name | Default | Description | Advantages | Disadvantages |
6363
|----------------------|------------|---------------------|----------------------|--------------------------|
64-
| `NV_INGEST_CONCURRENT_BATCHES` | 4 | Controls the number of parallel batch processing streams. Also available as `concurrent_batches` runtime argument in API requests to override this default. | - You can increase this for systems with high memory capacity. <br/> | - Higher values require more system memory. <br/> - Requires careful tuning based on available system resources. <br/> |
65-
| `NV_INGEST_FILES_PER_BATCH` | 16 | Controls how many files are processed in a single batch during ingestion. Also available as `files_per_batch` runtime argument in API requests to override this default. | - Adjust this to helps optimize memory usage and processing efficiency. <br/> | - Setting this too high can cause memory pressure. <br/> - Setting this too low can reduce throughput. <br/> |
64+
| `NV_INGEST_CONCURRENT_BATCHES` | 4 | Controls the number of parallel batch processing streams. | - You can increase this for systems with high memory capacity. <br/> | - Higher values require more system memory. <br/> - Requires careful tuning based on available system resources. <br/> |
65+
| `NV_INGEST_FILES_PER_BATCH` | 16 | Controls how many files are processed in a single batch during ingestion. | - Adjust this to helps optimize memory usage and processing efficiency. <br/> | - Setting this too high can cause memory pressure. <br/> - Setting this too low can reduce throughput. <br/> |
6666
| `APP_NVINGEST_ENABLE_PDF_SPLIT_PROCESSING` | `false` | Set to `true` to enable parallel processing of PDF pages by splitting them into chunks. Most useful when ingesting a small number of large PDFs. Also available as `enable_pdf_split_processing` runtime argument in API requests to override this default. | - Reduces ingestion time for multi-page PDFs. <br/> | - Requires tuning `APP_NVINGEST_PAGES_PER_CHUNK` for optimal performance. <br/> - May increase memory usage. <br/> |
6767
| `APP_NVINGEST_PAGES_PER_CHUNK` | 16 | Controls how many pages are grouped per chunk when `APP_NVINGEST_ENABLE_PDF_SPLIT_PROCESSING` is enabled. Smaller chunks = more parallelism but more overhead. Also available as `pdf_split_processing_options.pages_per_chunk` runtime argument in API requests to override this default. | - Smaller values increase parallelism. <br/> - Larger values reduce overhead. <br/> | - Requires tuning based on PDF sizes and available resources. <br/> |
6868

docs/api_reference/openapi_schema_ingestor_server.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -824,7 +824,7 @@
824824
"title": "Data",
825825
"description": "JSON data in string format containing metadata about the documents which needs to be uploaded.",
826826
"examples": [
827-
"{\"collection_name\": \"multimodal_data\", \"blocking\": false, \"split_options\": {\"chunk_size\": 512, \"chunk_overlap\": 150}, \"custom_metadata\": [], \"generate_summary\": false, \"documents_catalog_metadata\": [], \"summary_options\": null, \"enable_pdf_split_processing\": false, \"pdf_split_processing_options\": {\"pages_per_chunk\": 16}, \"enable_parallel_batch_mode\": true, \"concurrent_batches\": 4, \"files_per_batch\": 16}"
827+
"{\"collection_name\": \"multimodal_data\", \"blocking\": false, \"split_options\": {\"chunk_size\": 512, \"chunk_overlap\": 150}, \"custom_metadata\": [], \"generate_summary\": false, \"documents_catalog_metadata\": [], \"summary_options\": null, \"enable_pdf_split_processing\": false, \"pdf_split_processing_options\": {\"pages_per_chunk\": 16}}"
828828
]
829829
}
830830
},
@@ -850,7 +850,7 @@
850850
"title": "Data",
851851
"description": "JSON data in string format containing metadata about the documents which needs to be uploaded.",
852852
"examples": [
853-
"{\"collection_name\": \"multimodal_data\", \"blocking\": false, \"split_options\": {\"chunk_size\": 512, \"chunk_overlap\": 150}, \"custom_metadata\": [], \"generate_summary\": false, \"documents_catalog_metadata\": [], \"summary_options\": null, \"enable_pdf_split_processing\": false, \"pdf_split_processing_options\": {\"pages_per_chunk\": 16}, \"enable_parallel_batch_mode\": true, \"concurrent_batches\": 4, \"files_per_batch\": 16}"
853+
"{\"collection_name\": \"multimodal_data\", \"blocking\": false, \"split_options\": {\"chunk_size\": 512, \"chunk_overlap\": 150}, \"custom_metadata\": [], \"generate_summary\": false, \"documents_catalog_metadata\": [], \"summary_options\": null, \"enable_pdf_split_processing\": false, \"pdf_split_processing_options\": {\"pages_per_chunk\": 16}}"
854854
]
855855
}
856856
},

src/nvidia_rag/ingestor_server/ingestion_state_manager.py

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,6 @@ def __init__(
3434
documents_catalog_metadata: list[dict[str, Any]] | None = None,
3535
enable_pdf_split_processing: bool = False,
3636
pdf_split_processing_options: dict[str, Any] | None = None,
37-
enable_parallel_batch_mode: bool = True,
3837
concurrent_batches: int = 4,
3938
files_per_batch: int = 16,
4039
):
@@ -63,8 +62,6 @@ def __init__(
6362
self.enable_pdf_split_processing = enable_pdf_split_processing
6463
self.pdf_split_processing_options = pdf_split_processing_options
6564

66-
# Batch processing configuration
67-
self.enable_parallel_batch_mode = enable_parallel_batch_mode
6865
self.concurrent_batches = concurrent_batches
6966
self.files_per_batch = files_per_batch
7067

src/nvidia_rag/ingestor_server/main.py

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@
6161
)
6262
from nvidia_rag.ingestor_server.task_handler import INGESTION_TASK_HANDLER
6363
from nvidia_rag.rag_server.main import APIError
64+
from nvidia_rag.utils.batch_utils import calculate_dynamic_batch_parameters
6465
from nvidia_rag.utils.common import (
6566
create_catalog_metadata,
6667
create_document_metadata,
@@ -268,9 +269,6 @@ async def upload_documents(
268269
vdb_auth_token: str = "",
269270
enable_pdf_split_processing: bool = False,
270271
pdf_split_processing_options: dict[str, Any] | None = None,
271-
enable_parallel_batch_mode: bool = True,
272-
concurrent_batches: int = 4,
273-
files_per_batch: int = 16,
274272
) -> dict[str, Any]:
275273
"""Upload documents to the vector store.
276274
@@ -289,14 +287,19 @@ async def upload_documents(
289287
if vdb_endpoint is None:
290288
vdb_endpoint = self.config.vector_store.url
291289

290+
# Calculate dynamic batch parameters
291+
files_per_batch, concurrent_batches = calculate_dynamic_batch_parameters(
292+
filepaths=filepaths,
293+
config=self.config,
294+
)
295+
292296
state_manager = IngestionStateManager(
293297
filepaths=filepaths,
294298
collection_name=collection_name,
295299
custom_metadata=custom_metadata,
296300
documents_catalog_metadata=documents_catalog_metadata,
297301
enable_pdf_split_processing=enable_pdf_split_processing,
298302
pdf_split_processing_options=pdf_split_processing_options,
299-
enable_parallel_batch_mode=enable_parallel_batch_mode,
300303
concurrent_batches=concurrent_batches,
301304
files_per_batch=files_per_batch,
302305
)
@@ -858,9 +861,6 @@ async def update_documents(
858861
vdb_auth_token: str = "",
859862
enable_pdf_split_processing: bool = False,
860863
pdf_split_processing_options: dict[str, Any] | None = None,
861-
enable_parallel_batch_mode: bool = True,
862-
concurrent_batches: int = 4,
863-
files_per_batch: int = 16,
864864
) -> dict[str, Any]:
865865
"""Upload a document to the vector store. If the document already exists, it will be replaced.
866866
@@ -934,9 +934,6 @@ async def update_documents(
934934
vdb_auth_token=vdb_auth_token,
935935
enable_pdf_split_processing=enable_pdf_split_processing,
936936
pdf_split_processing_options=pdf_split_processing_options,
937-
enable_parallel_batch_mode=enable_parallel_batch_mode,
938-
concurrent_batches=concurrent_batches,
939-
files_per_batch=files_per_batch,
940937
)
941938
return response
942939

@@ -2087,7 +2084,7 @@ async def __perform_shallow_extraction_workflow(
20872084
num_batches,
20882085
)
20892086

2090-
if not state_manager.enable_parallel_batch_mode:
2087+
if not self.config.nv_ingest.enable_parallel_batch_mode:
20912088
# Sequential batch processing
20922089
total_failed = 0
20932090
for i in range(
@@ -2234,7 +2231,7 @@ async def __run_nvingest_batched_ingestion(
22342231
)
22352232

22362233
# Process batches sequentially
2237-
if not state_manager.enable_parallel_batch_mode:
2234+
if not self.config.nv_ingest.enable_parallel_batch_mode:
22382235
logger.info("Processing batches sequentially")
22392236
all_results = []
22402237
all_failures = []

src/nvidia_rag/ingestor_server/server.py

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -363,21 +363,6 @@ def validate_summary_configuration(self) -> "DocumentUploadRequest":
363363
description="Options for PDF split processing.",
364364
)
365365

366-
enable_parallel_batch_mode: bool = Field(
367-
default=CONFIG.nv_ingest.enable_parallel_batch_mode,
368-
description="Enable parallel batch processing.",
369-
)
370-
371-
concurrent_batches: int = Field(
372-
default=CONFIG.nv_ingest.concurrent_batches,
373-
description="Number of batches to process concurrently.",
374-
)
375-
376-
files_per_batch: int = Field(
377-
default=CONFIG.nv_ingest.files_per_batch,
378-
description="Number of files to process in each batch.",
379-
)
380-
381366
# Reserved for future use
382367
# embedding_model: str = Field(
383368
# os.getenv("APP_EMBEDDINGS_MODELNAME", ""),
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
"""
16+
Batch processing utilities for the ingestor server.
17+
18+
This module provides utilities for calculating optimal batch parameters
19+
based on file characteristics and system resources.
20+
"""
21+
22+
import logging
23+
from pathlib import Path
24+
25+
from nvidia_rag.utils.configuration import NvidiaRAGConfig
26+
27+
logger = logging.getLogger(__name__)
28+
29+
30+
# Text-like file extensions that process quickly
31+
# These are mapped to TXT or HTML in EXTENSION_TO_DOCUMENT_TYPE
32+
TEXT_LIKE_EXTENSIONS = frozenset({
33+
"txt",
34+
"md",
35+
"json",
36+
"sh",
37+
"html",
38+
})
39+
40+
# Optimal batch parameters for text-like files
41+
# Text files process quickly, so we use larger batches with sequential processing
42+
TEXT_FILE_BATCH_SIZE = 200
43+
TEXT_FILE_CONCURRENT_BATCHES = 1
44+
45+
# Threshold percentage to determine if workload is text-heavy
46+
TEXT_FILE_PERCENTAGE_THRESHOLD = 50.0
47+
48+
49+
def calculate_dynamic_batch_parameters(
50+
filepaths: list[str],
51+
config: NvidiaRAGConfig,
52+
) -> tuple[int, int]:
53+
"""
54+
Calculate optimal batch parameters dynamically based on file characteristics.
55+
56+
Analyzes file extensions to determine optimal batching strategy:
57+
- Text-like files (html, json, md, sh, txt): Faster processing, larger batches with less concurrency
58+
Uses files_per_batch={TEXT_FILE_BATCH_SIZE}, concurrent_batches={TEXT_FILE_CONCURRENT_BATCHES}
59+
- Complex files (pdf, images, docx, pptx, media): Smaller batches with higher concurrency
60+
Uses default configured values
61+
62+
The function can be enhanced in the future with additional factors:
63+
- File sizes and complexity (number of pages, resolution, etc.)
64+
- Available system resources (memory, CPU)
65+
- Historical performance metrics
66+
- Target latency and throughput requirements
67+
68+
Args:
69+
filepaths: List of file paths to be processed
70+
config: NvidiaRAGConfig instance containing batch configuration
71+
72+
Returns:
73+
Tuple of (files_per_batch, concurrent_batches)
74+
- files_per_batch: Number of files to include in each batch
75+
- concurrent_batches: Number of batches to process concurrently
76+
"""
77+
# Check if dynamic batching is enabled
78+
if not config.nv_ingest.enable_dynamic_batching:
79+
# Return default configured values without analysis
80+
logger.info(
81+
f"Dynamic batching is disabled. Using default configuration parameters for files processing: "
82+
f"files_per_batch={config.nv_ingest.files_per_batch}, concurrent_batches={config.nv_ingest.concurrent_batches}"
83+
)
84+
return config.nv_ingest.files_per_batch, config.nv_ingest.concurrent_batches
85+
86+
# Analyze file extensions
87+
if not filepaths:
88+
logger.warning("Empty filepaths list provided to dynamic batch calculator")
89+
return config.nv_ingest.files_per_batch, config.nv_ingest.concurrent_batches
90+
91+
# Extract extensions and count text-like files
92+
text_file_count = 0
93+
total_files = len(filepaths)
94+
extension_counts = {}
95+
96+
for filepath in filepaths:
97+
# Extract extension (lowercase, without dot)
98+
ext = Path(filepath).suffix.lstrip(".").lower()
99+
100+
# Track extension distribution
101+
extension_counts[ext] = extension_counts.get(ext, 0) + 1
102+
103+
# Check if it's a text-like file
104+
if ext in TEXT_LIKE_EXTENSIONS:
105+
text_file_count += 1
106+
107+
# Calculate percentage of text-like files
108+
text_file_percentage = (text_file_count / total_files) * 100 if total_files > 0 else 0
109+
110+
# Log file distribution for debugging
111+
logger.debug(
112+
f"File distribution analysis: Total={total_files}, "
113+
f"Text-like={text_file_count} ({text_file_percentage:.1f}%), "
114+
f"Extensions={dict(extension_counts)}"
115+
)
116+
117+
# Decision logic: If majority (>TEXT_FILE_PERCENTAGE_THRESHOLD%) are text-like files, optimize for them
118+
if text_file_percentage > TEXT_FILE_PERCENTAGE_THRESHOLD:
119+
files_per_batch = TEXT_FILE_BATCH_SIZE
120+
concurrent_batches = TEXT_FILE_CONCURRENT_BATCHES
121+
logger.info(
122+
f"Dynamic batching: Detected {text_file_percentage:.1f}% text-like files. "
123+
f"Using optimized parameters for text processing: "
124+
f"files_per_batch={files_per_batch}, concurrent_batches={concurrent_batches}"
125+
)
126+
else:
127+
# Use default configuration for other files (PDFs, images, media, etc.)
128+
files_per_batch = config.nv_ingest.files_per_batch
129+
concurrent_batches = config.nv_ingest.concurrent_batches
130+
logger.info(
131+
f"Dynamic batching: Detected {text_file_percentage:.1f}% text-like files. "
132+
f"Using default configuration parameters for files processing: "
133+
f"files_per_batch={files_per_batch}, concurrent_batches={concurrent_batches}"
134+
)
135+
136+
return files_per_batch, concurrent_batches

src/nvidia_rag/utils/configuration.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -382,6 +382,11 @@ def validate_chunk_settings(self) -> "NvIngestConfig":
382382
env="NV_INGEST_CONCURRENT_BATCHES",
383383
description="Number of batches to process concurrently",
384384
)
385+
enable_dynamic_batching: bool = Field(
386+
default=False,
387+
env="ENABLE_NV_INGEST_DYNAMIC_BATCHING",
388+
description="Enable dynamic calculation of batch parameters based on file characteristics",
389+
)
385390
enable_pdf_split_processing: bool = Field(
386391
default=False,
387392
env="APP_NVINGEST_ENABLE_PDF_SPLIT_PROCESSING",

tests/unit/test_ingestor_server/test_ingestor_library.py

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -702,9 +702,6 @@ async def test_csv_deletion_timing_sequential_batches(self, ingestor):
702702
filepaths=filepaths,
703703
collection_name="test_collection",
704704
custom_metadata=[],
705-
enable_parallel_batch_mode=False,
706-
concurrent_batches=1,
707-
files_per_batch=1,
708705
)
709706

710707
await ingestor._NvidiaRAGIngestor__run_nvingest_batched_ingestion(
@@ -754,9 +751,6 @@ async def test_csv_deletion_timing_parallel_batches(self, ingestor):
754751
filepaths=filepaths,
755752
collection_name="test_collection",
756753
custom_metadata=[],
757-
enable_parallel_batch_mode=True,
758-
concurrent_batches=2,
759-
files_per_batch=1,
760754
)
761755

762756
await ingestor._NvidiaRAGIngestor__run_nvingest_batched_ingestion(
@@ -793,9 +787,6 @@ async def test_csv_deletion_single_batch_mode(self, ingestor):
793787
filepaths=filepaths,
794788
collection_name="test_collection",
795789
custom_metadata=[],
796-
enable_parallel_batch_mode=False,
797-
concurrent_batches=1,
798-
files_per_batch=1,
799790
)
800791

801792
await ingestor._NvidiaRAGIngestor__run_nvingest_batched_ingestion(
@@ -838,9 +829,6 @@ async def test_csv_deletion_with_no_csv_file(self, ingestor):
838829
filepaths=filepaths,
839830
collection_name="test_collection",
840831
custom_metadata=[],
841-
enable_parallel_batch_mode=False,
842-
concurrent_batches=1,
843-
files_per_batch=1,
844832
)
845833

846834
await ingestor._NvidiaRAGIngestor__run_nvingest_batched_ingestion(
@@ -885,9 +873,6 @@ async def test_csv_deletion_with_missing_csv_file(self, ingestor):
885873
filepaths=filepaths,
886874
collection_name="test_collection",
887875
custom_metadata=[],
888-
enable_parallel_batch_mode=False,
889-
concurrent_batches=1,
890-
files_per_batch=1,
891876
)
892877

893878
with pytest.raises(FileNotFoundError, match="File not found"):

0 commit comments

Comments
 (0)