You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Breeze Automatic is a sophisticated server designed to power advanced conversational AI experiences, built around a **Pipecat-based Voice Agent** for robust, real-time voice assistants.
3
+
Clairvoyance is a powerful, multi-agent conversational AI platform designed to support sophisticated, real-time voice and data interactions. It is built on a modular architecture featuring a FastAPI server that manages and orchestrates multiple specialized voice agents.
4
4
5
-
## 1. Core Component: Pipecat Voice Agent
5
+
## 1. Core Architecture
6
6
7
-
The application's core is a standalone voice agent built on the Pipecat framework. It's launched as a subprocess by the main FastAPI server and handles the end-to-end voice conversation flow, including:
8
-
* Speech-to-Text (STT)
9
-
* Language Model (LLM) interaction with dynamic tool use
10
-
* Text-to-Speech (TTS)
7
+
The platform is built around a few key components:
8
+
9
+
***FastAPI Server:** The central application that exposes API endpoints, manages agent lifecycles, and handles incoming requests.
10
+
***Voice Agents:** Specialized, independent agents responsible for handling different conversational workflows. Each agent is built using a robust framework to manage real-time communication.
11
+
***Automatic Agent:** A Pipecat-based agent designed for dynamic data retrieval and analytics conversations. It can operate in `live` mode with real-time data or `test` mode with dummy data.
12
+
***Breeze Buddy Agent:** An agent focused on telephony and workflow-driven conversations, such as order confirmations. It integrates with multiple telephony providers like Twilio and Exotel.
13
+
***Database Integration:** The application uses a database to store configuration, track calls, and manage other persistent data.
14
+
***Docker Support:** The project includes a `Dockerfile` for easy containerization and deployment.
11
15
12
16
## 2. Key Features
13
17
14
-
***Dual-Mode Operation:**Can run in `live` mode with real-time data fetching or `test` mode using dummy data.
15
-
***Dynamic Tool Loading:** The voice agent dynamically loads tools based on the operating mode and provided credentials (e.g., Juspay and Breeze tools are only loaded in `live` mode with valid tokens).
16
-
***Multi-Provider Analytics:**Integrates with both **Juspay** and **Breeze** APIs to fetch a wide range of analytics data, including sales, orders, marketing, and checkout metrics.
17
-
***Personalized Prompts:**The agent's system prompt can be personalized with the user's name for a more engaging experience.
18
-
***Environment-Driven Configuration:** All sensitive keysand settings are managed via environment variables.
19
-
***Modular & Scalable Architecture:** The project is structured for clarity, maintainability, and easy extension with new tools or providers.
18
+
***Multi-Agent Support:**Designed to run multiple, distinct voice agents (`Automatic`, `Breeze Buddy`) within a single platform.
19
+
***Telephony Integration:** The `Breeze Buddy`agent connects with external telephony providers (Twilio, Exotel) to manage real voice calls.
20
+
***Dynamic Tool Loading:**The `Automatic` agent dynamically loads tools based on the operating mode and credentials, allowing it to interact with services like Juspay and Breeze for analytics.
21
+
***Workflow-Driven Conversations:**Agents can follow predefined workflows, such as the order confirmation process in `Breeze Buddy`.
22
+
***Environment-Driven Configuration:** All sensitive keys, API endpoints, and settings are managed via a `.env` file.
23
+
***Modular & Scalable:** The project is structured for maintainability and easy extension with new agents, tools, or services.
20
24
21
25
## 3. Project Structure
22
26
23
-
The project is organized into the main FastAPI server (`app/`) and the Pipecat voice agent (`app/agents/voice/automatic/`).
27
+
The project is organized into a main FastAPI application (`app/`) with a clear separation of concerns for agents, API routing, database management, and core services.
│ │ └── breeze_buddy/ # Telephony and workflow agent
37
+
│ ├── api/
38
+
│ │ └── routers/ # FastAPI routers for different endpoints
39
+
│ ├── core/
40
+
│ │ └── config.py # Configuration and environment variable management
41
+
│ ├── database/
42
+
│ │ ├── accessor/ # Database access logic
43
+
│ │ └── queries/ # SQL queries
44
+
│ ├── scripts/
45
+
│ │ └── create_tables.py # Script to initialize database tables
46
+
│ └── services/
47
+
│ └── langfuse/ # Integration with Langfuse for tracing
48
+
├── Dockerfile # Docker configuration for containerization
49
+
├── requirements.txt # Python dependencies
44
50
└── run.py # Script to run the server
45
51
```
46
52
@@ -49,44 +55,44 @@ The project is organized into the main FastAPI server (`app/`) and the Pipecat v
49
55
### Prerequisites
50
56
51
57
* Python 3.8+
52
-
* Access to Azure OpenAI and Daily.co APIs with valid keys.
58
+
* Database (e.g., PostgreSQL)
59
+
* Access to required third-party APIs (e.g., Azure OpenAI, Daily.co, Twilio/Exotel) with valid keys.
53
60
54
61
### Installation Steps
55
62
56
63
1.**Clone the repository.**
57
-
2.**Create and activate a virtual environment.**
64
+
2.**Create and activate a virtual environment:**
65
+
```bash
66
+
python -m venv venv
67
+
source venv/bin/activate
68
+
```
58
69
3. **Install dependencies:**
59
70
```bash
60
71
pip install -r requirements.txt
61
72
```
62
73
4. **Set up Environment Variables:**
63
-
Create a `.env` file in the project root with the following variables:
64
-
*`DAILY_API_KEY`: **Required**.
65
-
*`AZURE_OPENAI_API_KEY`: **Required**.
66
-
*`AZURE_OPENAI_ENDPOINT`: **Required**.
67
-
*`GOOGLE_CREDENTIALS_JSON`: **Required**. Path to your Google Cloud credentials JSON file.
68
-
*`GEMINI_API_KEY`: **Required**for the Gemini Live Proxy.
74
+
Create a `.env` file in the project root by copying `.env.example` and filling inthe required values for the database, API keys, and other configurations.
75
+
5. **Initialize the Database:**
76
+
Run the script to create the necessary tables in your database.
77
+
```bash
78
+
python -m app.scripts.create_tables
79
+
```
69
80
70
81
## 5. Running the Server
71
82
72
-
Execute the `run.py` script:
83
+
Execute the `run.py` script to start the FastAPI server:
73
84
```bash
74
85
python run.py
75
86
```
76
87
The server will start on `http://0.0.0.0:8000` by default.
77
88
78
-
## 6. How It Works: The Voice Agent Flow
79
-
80
-
1. A client sends a POST request to the `/agent/voice/automatic` endpoint on the FastAPI server.
81
-
2. The payload includes the `mode` (`live` or `test`) and various tokens/IDs (`eulerToken`, `breezeToken`, `shopId`, etc.).
82
-
3. The server creates a new Daily.co video room for the voice session.
83
-
4. It then launches the Pipecat voice agent as a **new subprocess**, passing the mode, tokens, and shop details as command-line arguments.
84
-
5. Inside the agent's `__init__.py`, the `initialize_tools` function is called.
85
-
6. This function checks the `mode` and the presence of tokens to decide which toolsets to load:
86
-
* **System tools** are always loaded.
87
-
* In `test` mode, **dummy tools** are loaded.
88
-
* In `live` mode, if tokens are present, the corresponding **real-time Juspay and Breeze tools** are loaded.
89
-
7. The agent's system prompt is personalized with the user's name if provided.
90
-
8. The agent connects to the Daily room and begins the conversation, now equipped with the appropriate set of tools for the session.
91
-
92
-
This architecture allows for clean separation of concerns and enables the creation of highly contextual and capable voice assistants.
89
+
## 6. How It Works
90
+
91
+
1. The FastAPI server starts and initializes the API routers.
92
+
2. When a request is made to an agent-specific endpoint (e.g., `/breeze-buddy/make-call`), the corresponding router handles it.
93
+
3. The router logic invokes the appropriate agent manager or service (e.g., `CallsManager`for`Breeze Buddy`).
94
+
4. The agent manager orchestrates the workflow, which may involve:
95
+
* Interacting with a database to fetch configuration.
96
+
* Making calls to external services (e.g., starting a call via Twilio).
97
+
* Launching an agent as a subprocess to handle the real-time conversation.
98
+
5. The voice agent connects to the communication service (like Daily.co or a direct telephony stream) and manages the STT -> LLM -> TTS pipeline, using its specialized tools to complete its task.
0 commit comments