Loyca.ai 🐦

Atestiguo que la Lloica tiene el pecho manchado por obra de una buena acción. Y en premio de ella, ..., desde hoy en adelante tendrá sobre su noble pecho un escudo escarlata. — Marta Brunet

	Loyca.ai

The long-tailed meadowlark (Leistes loyca) is a passerine bird of southern South America

A desktop AI assistant with contextual awareness and adaptive behavior

Loyca.ai is a desktop application that "sees" the user's screen. It is designed to provide intelligent assistance by leveraging advanced AI capabilities and intelligent screen analysis. It has been built with Tauri 2.0, using SvelteKit as frontend.

The idea was originated from thinking about Clippy — that classic, intrusive office assistant. Loyca.ai takes the concept and flips it: instead of randomly popping up, it quietly observes your screen, figures out what you need, and decides when to help. Smart, contextual, and actually useful.

The project is focused on local inference and privacy. It supports any OpenAI-API compatible endpoint, such as LM-Studio or Ollama. If total privacy is not a concern, then you can even use OpenRouter. All the data is stored locally using SQLite with rusqlite and sqlite-vec for vector storage.

Configuration Window that you can see once the application starts

Installation

The application has been successfully tested in the following environments:

• ✅ Windows 11 64-bit
• ✅ macOS 14 Sonoma 64-bit
• ✅ Archlinux - GNOME* 48 (Wayland** and X11)

* For GNOME, the Windows-Call extension is required in order to get the focused window title, and process information.

** Wayland is a very strict protocol and the avatar won't be displayed always in the same position.

You can visit Releases to download the latest version, or follow Development to build it from source.

To enable the assistant, you must first set the model to use. We recommend to use qwen/qwen2.5-vl-7b as vision model and openai/gpt-oss-20b as chat model. However, you can just use one model if you want (it must have vision capabilities). Also, note that for using tools the model must accept them.

What can Loyca.ai do?

Well, the main goal is to provide intelligent suggestions based on user context and screen analysis. For that, Loyca.ai uses a Vision Language Model (VLM) to understand the actual focused application window. After analyzing the screen for a while, a request to determine the user's intent and state is sent. This information, combined with other features, is used as input for a Contextual Bandit, which will decide whether to provide assistance or not. The following diagram illustrates the main logic of Loyca.ai:

However, that is not the only aspect of Loyca.ai's functionality. You can use it directly as an AI chatbot, supporting even MCP servers.

As MCP Server

If you don't want to use it as an AI chatbot or assistant, you can still start a MCP server to use a set of tools:

• get-user-context: Extract the user's context from their recent activity. This include:
  • Latest apps used (prioritized by focus time)
  • Actual focused app
  • Latest user's intent and state
• semantic-search-screenshots: Perform semantic search on screenshot descriptions to find relevant visual content. For each screenshot found, it returns a description, category and id.
• ocr_screenshot: Perform OCR on a screenshot using its ID. Currently, it uses the ocrs crates for that (still thinking about using the VLM instead).

You can see the implementation in src-tauri/src/llm/mcp_server.rs.

Technical Details

🖼️ Screenshot Analysis

The analysis is done by providing few-shot examples of screenshots and their descriptions, along with a category and keywords. You can see the implementation in src-tauri/src/llm/image_analysis.rs.

The categories are :

• code editor: Writing code, IDE usage
• terminal: Command line, shell operations
• document editor: Word processing, writing documents
• spreadsheets: Excel, Google Sheets, data tables
• database tools: SQL editors, database management (actual tools, not reading about databases)
• email app: Email clients, composing/reading emails
• chat/messaging: Slack, Discord, instant messaging
• video conferencing: Zoom, Teams, video calls
• file manager: File explorers, directory browsing
• music streaming: Spotify, Apple Music, audio streaming
• video streaming: YouTube, Netflix, video content
• social media: Twitter, Facebook, Instagram, LinkedIn, Reddit
• online shopping: E-commerce, shopping websites
• research/browsing: Web browsing, reading articles, StackOverflow, documentation
• game: Gaming applications
• other: Anything that doesn't fit the above categories

The system has some logic to decide whether or not a new image analysis must be performed. It first checks if the similarity between the current image and the last analyzed image is below a certain threshold using a RMS score. If the similarity is extremely high, it is likely the user is away from their computer so it will not perform a new analysis. The same if the similarity is high but not enough time has passed since the last analysis. For the image comparison, Loyca.ai uses image-compare crate.

🎯 User Intention Analysis

Similar to the screenshot analysis, few-shot examples are provided. The LLM will answer with a user's intention based on the current context, along with a user's state and keywords. You can see the implementation in src-tauri/src/llm/user_intention.rs.

The possible user's states are:

• flowing: User is in a deep, productive flow state, working smoothly and with high focus on a cohesive set of work-related or skill-development tasks. They are making clear progress.
• struggling: User shows signs of difficulty, confusion, or distraction based on their window content and behavior (e.g., frequent searching for errors, rereading the same document).
• idle: User has minimal meaningful activity or is in passive consumption mode with very low engagement.
• focused: User is concentrated on a single task or a cohesive set of related tasks. This state can apply to both work and complex leisure activities, but is distinct from flowing as it describes concentration, not necessarily productive output.
• learning: User is actively consuming educational content, documentation, or tutorials with the clear goal of acquiring new knowledge or skills.
• communicating: User is primarily engaged in communication activities (email, chat, video calls).
• entertaining: User is primarily engaged in leisure or recreational activities. This includes both passive consumption (music, video) and active engagement (social media, gaming) that is not related to work or skill development.

The system can decide to skip the request if the Jaccard similarity between the latest keywords and the keywords in the user's current context is below a certain threshold.

After getting the response, Loyca.ai triggers the contextual bandit system to adapt its behaviour.

🧠 Contextual Bandit System

A Contextual Bandit is a simplified version of a reinforcement learning algorithm that can be used for online decision-making. Basically:

1. The agent observes some context
2. It chooses one action (arm) from a set of possible actions based on the context.
3. It receives a reward for that action.

The contextual bandit system is implemented using the huggingface/candle project, particularly the NeuralUCB-Diagonal algorithm from "Neural Contextual Bandits with UCB-based Exploration" (D Zhou et al. 2019).

For our use case, the contextual bandit works as follow:

1. User's context is provided to the agent
2. Agent chooses if user required (arm=0) or not (arm=1) assistant
3. The agent receives a reward* and its trained

• If arm=0, the avatar won't display any alert.
• If arm=1, the avatar will display an alert for 20 seconds. Depending on the user's action, which can be to accept, reject, or ignore, the agent will receive a different reward.

* The rewards also depends on the user's state of the current context, you can check the BanditContext.compute_reward method from src-tauri/src/contextual_bandit/context.rs

The implementation of the NeuralUCB-Diagonal algorithm can be found in src-tauri/src/contextual_bandit/neural_ucb_diag.rs.

Development

Prerequisites

• Node.js v18+ and pnpm
• Rust toolchain (latest stable)
• SQLite

Installation

1. Clone and setup

   git clone https://github.com/vokturz/loyca-ai.git
   cd loyca-ai
   pnpm install

2. Development

   pnpm tauri dev

3. Production build

   pnpm tauri build

📁 Project Structure

loyca-ai/
├── src/                          # SvelteKit 5 frontend
│   ├── routes/                   # Page routing
│   │   ├── +page.svelte          # Main configuration interface
│   │   └── avatar/               # Floating avatar interface
│   └── lib/                      # Shared components and utilities
├── src-tauri/                    # Rust backend
│   ├── src/
│   │   ├── contextual_bandit/   # Contextual Bandit system
│   │   ├── embedding/           # Text similarity service
│   │   ├── llm/                 # AI model integration
│   │   ├── sql/                 # Database operations
│   │   ├── background_tasks.rs  # Image and User analysis in background
│   │   ├── constants.rs         # Constants for the application
│   │   ├── helpers.rs           # Helper functions for the application
│   │   ├── lib.rs               # Entry point of the Tauri application
│   │   ├── logging.rs           # Logging utilities
│   │   ├── main.rs              # Main file
│   │   ├── ocr.rs               # OCR service
│   │   └── window_manager.rs    # OS window detection
│   └── tauri.conf.json          # Tauri configuration
└── static/                      # Static assets and icons

To Do

👨‍💻 Frontend

• Implement the Chat Session component and support going back to older conversations
• Implement the Avatar Customization component, so users can personalize their experience.
• Add zoom in/out functionality for the text in the avatar window.

🖥️ Backend

• Handle all possible MCP tool responses in LLM Client (actually it supports only text responses, and is not using the output schema).
• Create a way to standardize a app title. For example, "Releases - Vokturz/loyca-ai" and "loyca-ai/README.md - Vokturz/loyca-ai" should be both part of the same app.
• Add a memory system, so the assistant can remember user interactions and preferences.
• Implement the Avatar Customization component
• Create custom errors for each module