graph LR
Trainer["Trainer"]
Dataset_Manager["Dataset Manager"]
Model_Architecture["Model Architecture"]
Evaluator["Evaluator"]
Model_Persister["Model Persister"]
Corpus_Transformer["Corpus Transformer"]
Corpus_Splitter["Corpus Splitter"]
Vocabulary_Builder["Vocabulary Builder"]
Trainer -- "utilizes" --> Model_Architecture
Trainer -- "consumes data from" --> Dataset_Manager
Trainer -- "invokes" --> Evaluator
Trainer -- "uses" --> Model_Persister
Corpus_Splitter -- "provides data to" --> Dataset_Manager
Vocabulary_Builder -- "provides vocabulary to" --> Dataset_Manager
Dataset_Manager -- "provides data to" --> Trainer
Corpus_Transformer -- "provides data to" --> Corpus_Splitter
The khaiii training subsystem orchestrates the entire machine learning model training lifecycle. It begins with raw corpus data being processed by the Corpus Transformer and then divided into training, validation, and test sets by the Corpus Splitter. Concurrently, the Vocabulary Builder constructs the necessary vocabulary from the corpus. The Dataset Manager then prepares this processed data and vocabulary for consumption by the Trainer. The Trainer is the central component, responsible for managing the training epochs, interacting with the Model Architecture for forward and backward passes, and periodically invoking the Evaluator to assess model performance. Finally, the Model Persister handles saving the trained model and its configuration. This structured flow ensures efficient data preparation, robust model training, and reliable model persistence.
The core orchestrator of the machine learning training process. It manages training epochs, handles model state (saving/restoring checkpoints), and coordinates the flow of data and model updates.
Related Classes/Methods:
Responsible for loading, preprocessing, and providing training and evaluation datasets in a format suitable for the model's input. It handles batching and data iteration.
Related Classes/Methods:
Defines the neural network structure, including various layers (e.g., embedding, convolutional, hidden layers) that constitute the machine learning model. It encapsulates the forward pass logic.
Related Classes/Methods:
Assesses the performance of the trained model by calculating various metrics (e.g., word accuracy, morph accuracy) on validation or test datasets.
Related Classes/Methods:
Manages the serialization and deserialization of the trained model, including its configuration, learned weights, and any other relevant metadata, enabling the model to be saved and loaded.
Related Classes/Methods:
Preprocesses raw corpus data, performing initial transformations such as tokenization, normalization, or format conversion, making it suitable for further processing.
Related Classes/Methods:
Divides the transformed corpus into distinct subsets for training, validation, and testing, ensuring proper evaluation and preventing data leakage.
Related Classes/Methods:
Creates the vocabulary (e.g., character, morpheme, or word mappings to indices) from the corpus, which is essential for converting textual data into numerical representations for the model.
Related Classes/Methods: