Skip to content

Home

Jump to bottom
Karunya Chavan edited this page May 4, 2026 · 1 revision

1. Project Overview

Semantixel is a lightweight, web-based platform that abandons rigid, lexically-bound file management in favor of intelligent, associative image search. Powered by OpenAI's CLIP, HuggingFace Transformers, and Doctr OCR, it enables users to query image datasets using natural language descriptions, visual similarity matching, and exact text detection.

The system is highly optimized for personal knowledge bases, research datasets, developer screenshot archives, and complex media collections where traditional metadata keyword search yields diminishing returns. By projecting visual features and linguistic semantics into a shared high-dimensional space, Semantixel mathematically bridges the semantic gap between human intent and unstructured binary storage.

Key Features

  • Zero-Shot Natural Language Retrieval: Retrieve images and extracted video frames using descriptive text captions via CLIP embeddings.
  • Cross-Modal Similarity Search: Discover visually related images from a reference image or media identifier.
  • OCR-Backed Information Extraction: Automatically detect and index literal text embedded within documents and screenshots.
  • Unified Cross-Boundary Search: Integrate Google Drive via OAuth to seamlessly search cloud-hosted assets alongside local directories.
  • Graph-Based Exploration: Interactively browse the similarity relationships between indexed items within the web interface.

2. Architecture

Semantixel operates on a deeply decoupled architecture, separating the web-based User Interface from the intensive mathematical operations of vector embedding and information retrieval.

System Architecture Diagram

classDiagram
    class WebUI {
        <<Frontend>>
        +HTML/CSS Interface
        +Search Dashboard
        +Graph Explorer
    }
    
    class FlaskServer {
        <<Backend API>>
        +Serve Search Results()
        +Serve Media Content()
    }

    class EmbeddingModels {
        <<Processing Engine>>
        +CLIP Vision Encoder
        +CLIP Text Encoder
        +Doctr OCR
    }
    
    class IngestionPipeline {
        <<Processing Engine>>
        +Process Images()
        +Extract Video Frames()
    }

    class VectorDatabase {
        <<Storage Layer>>
        +ChromaDB Dense Vectors
    }
    
    class SparseIndex {
        <<Storage Layer>>
        +BM25 Lexical Hashes
    }

    WebUI --> FlaskServer : HTTP REST Requests
    FlaskServer --> EmbeddingModels : Triggers Encoding/Search
    FlaskServer --> IngestionPipeline : Triggers Indexing
    IngestionPipeline --> EmbeddingModels : Routes Data for Projection
    EmbeddingModels --> VectorDatabase : Read/Write Latent Vectors
    EmbeddingModels --> SparseIndex : Read/Write OCR Hashes
Loading

Architectural Reasoning

The system implements a hybrid retrieval strategy. While dense vector embeddings (CLIP/ChromaDB) excel at conceptual mapping, they can overlook domain-specific identifiers. By running the Doctr OCR pipeline in parallel and hashing the extracted text into a BM25 sparse index, Semantixel ensures mathematical retrieval parity across both abstract semantic concepts and exact literal queries.

3. Setup & Installation

Constructing a system capable of real-time semantic processing requires a carefully managed environment.

Prerequisites

  • Python: Version 3.11 is strictly required.
  • Hardware: A CUDA-capable GPU is highly recommended to accelerate tensor manipulation and deep learning inference during the indexing phase.
  • Environment Manager: Conda or a standard Python environment manager.

Local Setup Steps

Execute the following commands to initialize the local environment and index your directories:

# 1. Create and activate the environment
conda create -n semantixel python=3.11 -y
conda activate semantixel

# 2. Install dependencies
pip install -r requirements.txt

# 3. Configure the application (Edit config.yaml parameters)

# 4. Run a full local directory scan to index media
python main.py --scan

# 5. Start the web server
python wsgi.py

# Alternatively, run the default combined flow
python main.py

4. Configuration

System behavior and runtime state are dynamically managed via the config.yaml file, ensuring deep neural parameters and filesystem boundaries can be altered without modifying the core Python scripts.

Core Configuration Parameters

  • include_directories: Defines the strict absolute paths of local directories to scan.
  • exclude_directories: Specifies sub-directories to deliberately bypass.
  • batch_size: Controls the matrix throughput for the embedding models.
  • clip: Specifies the CLIP provider and model checkpoint settings.
  • text_embed: Dictates the text embedding provider settings.
  • ocr_provider: Selects the OCR backend engine (e.g., Doctr).
  • google_drive: Stores the integration flags for connected cloud sources.

5. Core Modules/Components

The repository is modularized to ensure high cohesion and separation of concerns.

Codebase Structure

  • semantixel/: The core processing backend containing embedding providers, APIs, and source integrations.
  • settings/: Manages the desktop configuration UI.
  • UI/: Houses the web interface assets and Flow Launcher integrations.
  • db/: The persistence layer where the localized ChromaDB and BM25 artifacts are dynamically created at runtime.
  • docs/: Technical design notes and documentation.

Module Relationships Diagram

graph TD
    A[main.py / wsgi.py] --> B(semantixel/ Backend Core)
    B --> C{db/ Persistence}
    B --> D[settings/ Desktop Config]
    B --> E[UI/ Web Interface]
    
    C --> F[(ChromaDB)]
    C --> G[(BM25 Index)]
    
    B -. Reads .-> H[config.yaml]
Loading

6. API/Endpoints & Request Flow

The backend utilizes a Flask API to serve both search computations and visual media directly to the frontend interface.

Request/Data Flow Sequence Diagram

This sequence illustrates the zero-shot cross-modal retrieval workflow when a user submits a natural language query.

sequenceDiagram
    participant User
    participant WebUI as Web UI
    participant FlaskAPI as Flask API
    participant CLIP as CLIP Text Encoder
    participant DB as ChromaDB / BM25
    
    User->>WebUI: Enters textual query (e.g., terminal failure)
    WebUI->>FlaskAPI: GET /search?q=query
    FlaskAPI->>CLIP: Forward text string
    CLIP-->>FlaskAPI: Return high-dimensional Query Vector
    FlaskAPI->>DB: Execute K-NN Cosine Similarity Search
    DB-->>FlaskAPI: Return top N Media Identifiers
    FlaskAPI->>WebUI: Serve JSON Results & Media payloads
    WebUI-->>User: Render visual image grid
Loading

7. Database/Data Flow (Ingestion)

Semantixel abstracts physical storage boundaries by indexing files using "source-aware media identifiers" rather than brittle, localized absolute file paths.

Ingestion Data Flow

  1. Discovery: The system scans configured local directories and connected sources.
  2. Extraction: Files are loaded; video assets undergo temporal frame extraction.
  3. Projection: Image matrices are passed through the CLIP vision-encoder to generate dense vectors. Concurrently, Doctr OCR extracts literal text.
  4. Persistence: Dense vectors and source-aware identifiers are stored in ChromaDB, while OCR text is hashed into the BM25 index.

Database Schema Relationships

erDiagram
    SOURCE_AWARE_IDENTIFIER ||--o{ CHROMADB_VECTOR : maps_to
    SOURCE_AWARE_IDENTIFIER ||--o{ BM25_DOCUMENT : hashes_to

    SOURCE_AWARE_IDENTIFIER {
        string URI
        string Source_Type
        string MIME_Type
    }
    CHROMADB_VECTOR {
        array Embeddings
        json Metadata
    }
    BM25_DOCUMENT {
        string OCR_Text
    }
Loading

8. Authentication & Security

Semantixel guarantees data sovereignty and operational security by strictly sandboxing operations and handling OAuth flows entirely locally.

Local Sandboxing

Local media access is mathematically restricted to the paths explicitly defined in the include_directories configuration; external URL handling is strictly validated before any image-query ingestion can occur.

Google Drive OAuth Workflow

To blend cloud storage with local search, the system integrates a secure Google Cloud OAuth web application flow.

  1. Create a Google Cloud OAuth client of type Web application.
  2. Configure the redirect URI exactly as: http://localhost:23107/integrations/google_drive/auth/callback.
  3. Download the client secret JSON and update config.yaml.
  4. Run python main.py --scan to fetch, embed, and synchronize Drive assets alongside local files.

Security Note: The OAuth integration stores a local token file for subsequent access. OAuth client secrets and token files must remain local and are correctly ignored by Git.

9. Deployment

As a personal knowledge base and retrieval tool, Semantixel is designed for local, offline-capable deployment.

Deployment Flow Diagram

flowchart LR
    A[Clone Repository] --> B[Setup Conda Environment]
    B --> C[Configure config.yaml]
    C --> D[Run main.py --scan]
    D --> E[Launch Flask via wsgi.py]
    E --> F((Localhost:23107))
    
    subgraph External Dependencies
        G[Google Cloud Console OAuth]
    end
    
    C -.-> G
    G -.-> E
Loading

10. Troubleshooting

  • Google Drive Sync Failures: Ensure the Google Cloud Redirect URI perfectly matches http://localhost:23107/integrations/google_drive/auth/callback. Verify the client secret JSON is correctly referenced.
  • Poor Search Results: If lexical text queries are failing on screenshots, ensure the ocr_provider is actively enabled in config.yaml to run the Doctr extraction phase.
  • Missing Media: Verify that the missing file types fall under supported formats and exist inside paths explicitly listed under include_directories.

11. FAQ

Q: Does Semantixel require an active internet connection? A: No. Core capabilities execute entirely offline. An internet connection is only required to integrate Google Drive.

Q: What file types are supported? A: Standard images, documents containing visual text (via OCR), and video assets (by extracting temporal keyframes).

Q: Does it assume a local file path for all search results? A: No. Indexed items are successfully resolved through "source-aware media identifiers," allowing local and cloud files to be served natively side-by-side.

Clone this wiki locally