Data_Preprocessing_Utilities.md

graph LR
    Input_Orchestration_Service["Input Orchestration Service"]
    Image_Preprocessing_Module["Image Preprocessing Module"]
    Audio_Preprocessing_Module["Audio Preprocessing Module"]
    Prompt_Formatting_Service["Prompt Formatting Service"]
    Tokenizer_Management_Service["Tokenizer Management Service"]
    Video_Vision_Processing_Module["Video Vision Processing Module"]
    Video_Frame_Retrieval_Module["Video Frame Retrieval Module"]
    Shared_Resource_Loader["Shared Resource Loader"]
    Input_Orchestration_Service -- "delegates to" --> Image_Preprocessing_Module
    Input_Orchestration_Service -- "delegates to" --> Audio_Preprocessing_Module
    Prompt_Formatting_Service -- "works in conjunction with" --> Tokenizer_Management_Service
    Video_Vision_Processing_Module -- "depends on" --> Video_Frame_Retrieval_Module
    Input_Orchestration_Service -- "invokes" --> Shared_Resource_Loader
    Tokenizer_Management_Service -- "invokes" --> Shared_Resource_Loader

Loading

Details

The Data Preprocessing & Utilities subsystem is a critical shared service within the mlx-vlm project, responsible for transforming diverse raw input modalities (images, audio, text, video) into a standardized, VLM-consumable format. This includes specialized operations like prompt formatting and tokenizer interactions.

Input Orchestration Service

Acts as the primary entry point for input preparation, coordinating and delegating specific preprocessing tasks based on the input modality. It ensures that all diverse inputs are channeled through the appropriate processing pipelines.

Related Classes/Methods:

• mlx_vlm.utils.prepare_inputs:722-829

Image Preprocessing Module

Specializes in handling image inputs, performing necessary transformations such as loading, resizing, and normalization to prepare images for the VLM.

Related Classes/Methods:

• mlx_vlm.utils.process_image:615-620

Audio Preprocessing Module

Manages the preprocessing of audio inputs, including loading audio files and resampling them to meet the VLM's required format and specifications.

Related Classes/Methods:

• mlx_vlm.utils.load_audio:631-653

Prompt Formatting Service

Responsible for structuring and formatting chat messages and other textual inputs into coherent prompts, integrating various modalities as needed, making them ready for tokenization.

Related Classes/Methods:

• mlx_vlm.prompt_utils.apply_chat_template:378-474

Tokenizer Management Service

Handles the loading, initialization, and application of the VLM's tokenizer, which is essential for converting formatted text into numerical tokens that the VLM can process.

Related Classes/Methods:

• mlx_vlm.tokenizer_utils.load_tokenizer:314-344

Video Vision Processing Module

Coordinates the processing of visual information specifically derived from video inputs, preparing individual video frames for subsequent VLM analysis.

Related Classes/Methods:

• mlx_vlm.video_generate.process_vision_info:319-347

Video Frame Retrieval Module

Focuses on the retrieval and initial extraction of individual frames from a given video source, serving as a foundational step for video processing.

Related Classes/Methods:

• mlx_vlm.video_generate.fetch_video:233-298

Shared Resource Loader

Provides a general utility for loading common resources such as models, processors, and configuration files that are required by various preprocessing steps across different modalities.

Related Classes/Methods:

• mlx_vlm.utils.load:259-306

FAQ