Skip to content

on_boarding.md

Latest commit

 

History

History
110 lines (64 loc) · 6.77 KB

on_boarding.md

File metadata and controls

110 lines (64 loc) · 6.77 KB
Raw
graph LR
    Data_I_O["Data I/O"]
    Text_Preprocessing["Text Preprocessing"]
    Schema_Definition["Schema Definition"]
    Prompt_Generation["Prompt Generation"]
    LLM_Integration_Layer["LLM Integration Layer"]
    Extraction_Resolution_Alignment["Extraction Resolution & Alignment"]
    Extraction_Orchestration["Extraction Orchestration"]
    Visualization["Visualization"]
    Data_I_O -- "provides raw documents to" --> Text_Preprocessing
    Text_Preprocessing -- "supplies text chunks to" --> Prompt_Generation
    Schema_Definition -- "guides prompt construction within" --> Prompt_Generation
    Prompt_Generation -- "sends formatted prompts to" --> LLM_Integration_Layer
    LLM_Integration_Layer -- "returns raw LLM output to" --> Extraction_Resolution_Alignment
    Schema_Definition -- "guides parsing and validation within" --> Extraction_Resolution_Alignment
    Extraction_Resolution_Alignment -- "provides structured and aligned extractions to" --> Extraction_Orchestration
    Extraction_Orchestration -- "requests and utilizes LLM instances from" --> LLM_Integration_Layer
    Extraction_Orchestration -- "delivers final annotated documents to" --> Data_I_O
    Extraction_Orchestration -- "sends annotated documents to" --> Visualization
    click Text_Preprocessing href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/langextract/Text_Preprocessing.md" "Details"
    click Prompt_Generation href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/langextract/Prompt_Generation.md" "Details"
    click LLM_Integration_Layer href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/langextract/LLM_Integration_Layer.md" "Details"
    click Extraction_Resolution_Alignment href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/langextract/Extraction_Resolution_Alignment.md" "Details"
    click Extraction_Orchestration href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/langextract/Extraction_Orchestration.md" "Details"
Loading

CodeBoardingDemoContact

Details

The langextract architecture is structured as a flexible, LLM-agnostic pipeline for information extraction. It orchestrates a clear data flow, starting with Data I/O for document ingestion, followed by Text Preprocessing to prepare content. Schema Definition provides the blueprint for extraction, guiding Prompt Generation and Extraction Resolution & Alignment. The LLM Integration Layer serves as a pluggable interface for various LLMs, facilitating the core inference step. Extraction Orchestration acts as the central control, managing the end-to-end process from chunking to final structured output. Results are then either persisted via Data I/O or presented through Visualization. This modular design ensures clear component boundaries, enabling easy extension and adaptation to new LLMs or extraction tasks, making it ideal for visual representation as a sequential data flow diagram with feedback loops.

Data I/O

Manages the loading of raw input documents and the saving of processed, annotated data.

Related Classes/Methods:

  • langextract/io.py

Text Preprocessing [Expand]

Prepares raw text content for LLM consumption through tokenization, segmentation, and chunking.

Related Classes/Methods:

  • langextract/chunking.py

Schema Definition

Defines the expected structure of the extracted information, guiding both prompt generation and post-processing.

Related Classes/Methods:

  • langextract/schema.py

Prompt Generation [Expand]

Constructs structured prompts for LLMs based on preprocessed text chunks and the defined extraction schema.

Related Classes/Methods:

  • langextract/prompting.py

LLM Integration Layer [Expand]

Provides a unified, abstract interface for interacting with various LLM providers, handling model instantiation and inference calls.

Related Classes/Methods:

  • langextract/factory.py
  • langextract/inference.py

Extraction Resolution & Alignment [Expand]

Parses raw LLM text output into structured data and aligns extracted spans back to their original positions.

Related Classes/Methods:

  • langextract/resolver.py

Extraction Orchestration [Expand]

Coordinates the entire extraction pipeline, managing data flow, LLM invocations, and multi-pass extractions.

Related Classes/Methods:

  • langextract/__init__.py
  • langextract/annotation.py

Visualization

Renders the final annotated documents into a human-readable and interactive format for review and presentation.

Related Classes/Methods:

  • langextract/visualization.py

FAQ