Merge pull request #17 from vxcontrol/14-enhancement-openrouter-configuration

Add ctester utility for testing LLM agents

Author: asdek

.vscode/launch.json

@@ -45,5 +45,23 @@
             "url": "http://localhost:8000",
             "webRoot": "${workspaceFolder}/frontend/src"
         },
+        {
+            "type": "go",
+            "request": "launch",
+            "name": "Launch Agents Tests",
+            "program": "${workspaceFolder}/backend/cmd/ctester/",
+            "envFile": "${workspaceFolder}/.env",
+            "cwd": "${workspaceFolder}",
+            "env": {},
+            "args": [
+                // "--config", "${workspaceFolder}/examples/configs/openrouter.provider.yml",
+                "--config", "${workspaceFolder}/examples/configs/deepinfra.provider.yml",
+                // "--report", "${workspaceFolder}/examples/tests/openrouter-report.md",
+                "--report", "${workspaceFolder}/examples/tests/deepinfra-report.md",
+                "--agents", "all",
+                "--tests", "all",
+                "--verbose"
+            ]
+        },
     ]
 }

Dockerfile

@@ -61,6 +61,9 @@ RUN --mount=type=cache,target=/go/pkg/mod \
 # Build backend
 RUN go build -trimpath -o /pentagi ./cmd/pentagi
 
+# Build ctester utility
+RUN go build -trimpath -o /ctester ./cmd/ctester
+
 # STEP 3: Build the final image
 FROM alpine:3.21
 
@@ -83,11 +86,17 @@ RUN mkdir -p \
     /opt/pentagi/ssl \
     /opt/pentagi/fe \
     /opt/pentagi/logs \
-    /opt/pentagi/data
+    /opt/pentagi/data \
+    /opt/pentagi/conf
 
 COPY --from=be-build /pentagi /opt/pentagi/bin/pentagi
+COPY --from=be-build /ctester /opt/pentagi/bin/ctester
 COPY --from=fe-build /frontend/dist /opt/pentagi/fe
 
+# Copy provider configuration files
+COPY examples/configs/openrouter.provider.yml /opt/pentagi/conf/
+COPY examples/configs/deepinfra.provider.yml /opt/pentagi/conf/
+
 COPY LICENSE /opt/pentagi/LICENSE
 COPY NOTICE /opt/pentagi/NOTICE
 COPY EULA.md /opt/pentagi/EULA

README.md

@@ -11,6 +11,7 @@
 - [Quick Start](#-quick-start)
 - [Advanced Setup](#-advanced-setup)
 - [Development](#-development)
+- [Testing LLM Agents](#-testing-llm-agents)
 - [Building](#%EF%B8%8F-building)
 - [Credits](#-credits)
 - [License](#-license)
@@ -658,6 +659,147 @@ Run the command(s) in `frontend` folder:
 
 Open your browser and visit the web app URL.
 
+## 🧪 Testing LLM Agents
+
+PentAGI includes a powerful utility called `ctester` for testing and validating LLM agent capabilities. This tool helps ensure your LLM provider configurations work correctly with different agent types, allowing you to optimize model selection for each specific agent role.
+
+The utility features parallel testing of multiple agents, detailed reporting, and flexible configuration options.
+
+### Key Features
+
+- **Parallel Testing**: Tests multiple agents simultaneously for faster results
+- **Comprehensive Test Suite**: Evaluates basic completion, JSON responses, function calling, and more
+- **Detailed Reporting**: Generates markdown reports with success rates and performance metrics
+- **Flexible Configuration**: Test specific agents or test groups as needed
+
+### Usage Scenarios
+
+#### For Developers (with local Go environment)
+
+If you've cloned the repository and have Go installed:
+
+```bash
+# Default configuration with .env file
+cd backend
+go run cmd/ctester/*.go -verbose
+
+# Custom provider configuration
+go run cmd/ctester/*.go -config ../examples/configs/openrouter.provider.yml -verbose
+
+# Generate a report file
+go run cmd/ctester/*.go -config ../examples/configs/deepinfra.provider.yml -report ../test-report.md
+
+# Test specific agent types only 
+go run cmd/ctester/*.go -agents simple,simple_json,agent -verbose
+
+# Test specific test groups only
+go run cmd/ctester/*.go -tests "Simple Completion,System User Prompts" -verbose
+```
+
+#### For Users (using Docker image)
+
+If you prefer to use the pre-built Docker image without setting up a development environment:
+
+```bash
+# Using Docker to test with default environment
+docker run --rm -v $(pwd)/.env:/opt/pentagi/.env vxcontrol/pentagi /opt/pentagi/bin/ctester -verbose
+
+# Test with your custom provider configuration
+docker run --rm \
+  -v $(pwd)/.env:/opt/pentagi/.env \
+  -v $(pwd)/my-config.yml:/opt/pentagi/config.yml \
+  vxcontrol/pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/config.yml -verbose
+
+# Generate a detailed report
+docker run --rm \
+  -v $(pwd)/.env:/opt/pentagi/.env \
+  -v $(pwd):/opt/pentagi/output \
+  vxcontrol/pentagi /opt/pentagi/bin/ctester -report /opt/pentagi/output/report.md
+```
+
+#### Using Pre-configured Providers
+
+The Docker image comes with pre-configured provider files for OpenRouter or DeepInfra:
+
+```bash
+# Test with OpenRouter configuration
+docker run --rm \
+  -v $(pwd)/.env:/opt/pentagi/.env \
+  vxcontrol/pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/openrouter.provider.yml
+
+# Test with DeepInfra configuration
+docker run --rm \
+  -v $(pwd)/.env:/opt/pentagi/.env \
+  vxcontrol/pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/deepinfra.provider.yml
+```
+
+To use these configurations, your `.env` file only needs to contain:
+
+```
+LLM_SERVER_URL=https://openrouter.ai/api/v1      # or https://api.deepinfra.com/v1/openai
+LLM_SERVER_KEY=your_api_key
+LLM_SERVER_MODEL=                                # Leave empty, as models are specified in the config
+LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/openrouter.provider.yml  # or deepinfra.provider.yml
+```
+
+#### Running Tests in a Production Environment
+
+If you already have a running PentAGI container and want to test the current configuration:
+
+```bash
+# Run ctester in an existing container using current environment variables
+docker exec -it pentagi /opt/pentagi/bin/ctester -verbose
+
+# Generate a report file inside the container
+docker exec -it pentagi /opt/pentagi/bin/ctester -report /opt/pentagi/data/agent-test-report.md
+
+# Access the report from the host
+docker cp pentagi:/opt/pentagi/data/agent-test-report.md ./
+```
+
+### Command-line Options
+
+The utility accepts several options:
+
+- `-env <path>` - Path to environment file (default: `.env`)
+- `-config <path>` - Path to custom provider config (default: from `LLM_SERVER_CONFIG_PATH` env variable)
+- `-report <path>` - Path to write the report file (optional)
+- `-agents <list>` - Comma-separated list of agent types to test (default: `all`)
+- `-tests <list>` - Comma-separated list of test groups to run (default: `all`)
+- `-verbose` - Enable verbose output with detailed test results for each agent
+
+### Example Provider Configuration
+
+Provider configuration defines which models to use for different agent types:
+
+```yaml
+simple:
+  model: "provider/model-name"
+  temperature: 0.7
+  top_p: 0.95
+  n: 1
+  max_tokens: 4000
+
+simple_json:
+  model: "provider/model-name"
+  temperature: 0.7
+  top_p: 1.0
+  n: 1
+  max_tokens: 4000
+  json: true
+
+# ... other agent types ...
+```
+
+### Optimization Workflow
+
+1. **Create a baseline**: Run tests with default configuration
+2. **Experiment**: Try different models for each agent type
+3. **Compare results**: Look for the best success rate and performance
+4. **Deploy optimal configuration**: Use in production with your optimized setup
+
+This tool helps ensure your AI agents are using the most effective models for their specific tasks, improving reliability while optimizing costs.
+
 ## 🏗️ Building
 
 ### Building Docker Image