graph LR
Data_Ingress_Caching["Data Ingress & Caching"]
Dataset_Management["Dataset Management"]
Data_Augmentation_Engine["Data Augmentation Engine"]
Data_Transformation_Utilities["Data Transformation & Utilities"]
Data_Loader_Builder["Data Loader Builder"]
Data_Exploration_Visualization["Data Exploration & Visualization"]
Data_Ingress_Caching -- "loads and caches data for" --> Dataset_Management
Dataset_Management -- "relies on" --> Data_Ingress_Caching
Dataset_Management -- "integrates" --> Data_Augmentation_Engine
Data_Augmentation_Engine -- "provides transformation logic to" --> Dataset_Management
Dataset_Management -- "utilizes" --> Data_Transformation_Utilities
Data_Transformation_Utilities -- "offers helper functions consumed by" --> Dataset_Management
Data_Loader_Builder -- "leverages objects from" --> Dataset_Management
Dataset_Management -- "provides objects to" --> Data_Loader_Builder
Data_Transformation_Utilities -- "might use" --> Data_Ingress_Caching
Data_Ingress_Caching -- "provides low-level file access to" --> Data_Transformation_Utilities
Data_Exploration_Visualization -- "queries and analyzes data managed by" --> Dataset_Management
Dataset_Management -- "exposes information for" --> Data_Exploration_Visualization
Data_Exploration_Visualization -- "might use" --> Data_Transformation_Utilities
Data_Transformation_Utilities -- "provides capabilities to" --> Data_Exploration_Visualization
The Dataset Management & Preprocessing subsystem is crucial for preparing data for the DocLayout-YOLO model. It encompasses functionalities from raw data loading to advanced augmentation and format conversion, ensuring data is in an optimal state for model consumption.
Manages the low-level loading of images and data from various sources (including specialized inputs like video streams) and handles efficient caching to RAM or disk. It acts as the foundational layer for data access.
Related Classes/Methods:
Provides the primary abstraction for datasets, overseeing operations like image integrity verification, caching and loading dataset labels, and managing initial data transformations. It represents the core dataset object.
Related Classes/Methods:
Implements a suite of data augmentation techniques (e.g., mosaic, affine transformations, random flips, HSV adjustments) to enrich the dataset, prevent overfitting, and improve model generalization during training.
Related Classes/Methods:
Offers general utility functions for data verification (images, labels), handles various annotation format conversions (e.g., COCO to YOLO, DOTA to YOLO OBB, polygons to masks), and provides specialized dataset splitting functionalities (e.g., DOTA dataset windowing).
Related Classes/Methods:
Responsible for constructing efficient data loaders (e.g., PyTorch DataLoaders) that feed preprocessed and augmented data to the model during training and inference, optimizing batching and parallel processing.
Related Classes/Methods:
Facilitates interactive data exploration through functionalities like embedding creation, similarity searches, and SQL queries on the dataset, complemented by a graphical user interface for visualization and user interaction.
Related Classes/Methods: