graph LR
Text_Preprocessing["Text Preprocessing"]
Tokenizer["Tokenizer"]
Text_Preprocessing -- "relies on" --> Tokenizer
click Text_Preprocessing href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/langextract/Text_Preprocessing.md" "Details"
The Text Preprocessing subsystem is responsible for preparing raw text content for consumption by Large Language Models (LLMs). Its primary boundary encompasses functionalities related to text division, sizing, and tokenization.
Text Preprocessing [Expand]
Prepares raw text content for LLM consumption by dividing large input documents into smaller, manageable chunks to comply with LLM context window limitations. It relies on tokenization for accurate sizing of these chunks.
Related Classes/Methods:
Text Preprocessing
Handles the process of converting raw text into tokens and provides functionality for counting these tokens. This is crucial for accurately determining the size of text segments before they are fed into LLMs.
Related Classes/Methods:
Tokenizer