graph LR
LoRA_Fine_tuning_Orchestrator["LoRA Fine-tuning Orchestrator"]
Training_Loop_Core["Training Loop Core"]
Data_Loading_Preparation["Data Loading & Preparation"]
LoRA_Layer_Implementation["LoRA Layer Implementation"]
DoRA_Implementation["DoRA Implementation"]
Quantization_Algorithms["Quantization Algorithms"]
Loss_Functions["Loss Functions"]
LoRA_Fine_tuning_Orchestrator -- "utilizes" --> Data_Loading_Preparation
LoRA_Fine_tuning_Orchestrator -- "delegates to" --> Training_Loop_Core
Training_Loop_Core -- "relies on" --> Loss_Functions
Training_Loop_Core -- "integrates" --> LoRA_Layer_Implementation
Training_Loop_Core -- "integrates" --> DoRA_Implementation
DoRA_Implementation -- "supports" --> Quantization_Algorithms
The mlx-lm project's fine-tuning subsystem is orchestrated by the LoRA Fine-tuning Orchestrator, which initiates and manages the entire process. It utilizes the Data Loading & Preparation component to prepare datasets and delegates to the Training Loop Core for the iterative optimization process. The Training Loop Core is central to the training, relying on Loss Functions for gradient computation and integrating both LoRA Layer Implementation and DoRA Implementation to apply low-rank adaptation techniques. Notably, DoRA Implementation is designed to support Quantization Algorithms, allowing it to operate on and potentially re-quantize models that have undergone quantization, thereby enabling efficient fine-tuning of compressed models. This modular design allows for flexible integration of various adaptation and optimization techniques.
Acts as the high-level entry point and orchestrator for the LoRA fine-tuning workflow. It manages the end-to-end process, including initiating training and evaluation.
Related Classes/Methods:
Implements the fundamental training loop, handling iterative optimization, forward/backward passes, and model updates.
Related Classes/Methods:
Provides utilities for loading, preprocessing, and preparing datasets from various sources (e.g., local files, Hugging Face Hub) for the fine-tuning process.
Related Classes/Methods:
Contains the specific architectural implementations of LoRA layers, enabling low-rank adaptation within models.
Related Classes/Methods:
Provides the core logic for applying DoRA (Weight-Decomposed Low-Rank Adaptation) to models, including converting base layers and fusing weights. It is designed to be compatible with both unquantized and quantized linear layers.
Related Classes/Methods:
This acts as a collective component encompassing various quantization algorithms (dwq, dynamic_quant, awq, gptq). Each sub-module within mlx_lm.quant is responsible for implementing its specific quantization technique to reduce model size and optimize for inference.
Related Classes/Methods:
Provides functions for computing gradients during optimization.
Related Classes/Methods: