graph LR
scripts_data_collector_base["scripts.data_collector.base"]
scripts_dump_bin["scripts.dump_bin"]
qlib_data_data["qlib.data.data"]
qlib_data_dataset_handler["qlib.data.dataset.handler"]
qlib_data_dataset_processor["qlib.data.dataset.processor"]
scripts_dump_bin -- "consumes raw data produced by" --> scripts_data_collector_base
qlib_data_data -- "initiates the process of dataset creation by interacting with" --> qlib_data_dataset_handler
qlib_data_dataset_handler -- "applies various data transformations and feature engineering steps to the datasets by utilizing the functionalities provided by" --> qlib_data_dataset_processor
The Qlib data subsystem is structured around a clear data pipeline, beginning with raw data acquisition and culminating in prepared datasets for machine learning. The scripts.data_collector.base component is responsible for the initial collection and saving of raw financial data. This raw data is then consumed by scripts.dump_bin, which transforms it into Qlib's optimized binary format for efficient storage and retrieval. Further down the pipeline, qlib.data.data serves as the central access point for all data operations, orchestrating the loading of various data elements and initiating the dataset creation process. This process is managed by qlib.data.dataset.handler, which fetches and prepares data by applying a series of transformations and feature engineering steps provided by qlib.data.dataset.processor. This modular design ensures a robust and efficient flow from raw data to analysis-ready datasets.
Orchestrates the collection of raw financial data from external sources, manages instrument lists, normalizes datetime information, and handles the initial saving of collected data. It serves as the base for various specific data collection implementations.
Related Classes/Methods:
Responsible for ingesting raw data (often from data collectors) and converting it into Qlib's optimized binary format. It handles the reading and writing of calendars, instruments, and features, ensuring data is stored efficiently for retrieval.
Related Classes/Methods:
Acts as the primary high-level interface for accessing and managing data within Qlib. It handles loading calendars, instruments, features, and orchestrates the creation of datasets, abstracting the underlying storage and caching mechanisms. This component is central to providing a unified data view to other parts of the system.
Related Classes/Methods:
Manages the creation and processing of datasets. It orchestrates the fetching of data and the application of various data processors to prepare the data for machine learning tasks, ensuring data is in the correct format and structure for model consumption.
Related Classes/Methods:
Provides a suite of data preprocessing and feature engineering functionalities. These processors perform transformations such as normalization, denoising, and handling of missing or infinite values on the datasets, which is crucial for preparing high-quality input for AI/ML models.
Related Classes/Methods: