Embedding Providers Implementation

[asdek]

Description of the Change

This PR implements comprehensive support for embedding providers and introduces an etester utility for testing and managing embeddings. The implementation adds constructors for Cybertron, Huggingface, Bedrock, and VoyageAI embedding providers, updates the embedding configuration in various files, and provides a new command-line tool for embedding testing and management.

Problem

The project lacked proper support for various embedding providers and tools for testing and debugging embedding functionality. This made it difficult to work with different embedding models and to effectively test and manage embeddings in the system.

Solution

The solution consists of two main components:

1. Implementation of missing embedding provider constructors in the embedding.go file with support for OpenAI, Ollama, Mistral, Jina, Hugging Face, GoogleAI, VoyageAI
2. Creation of an etester utility with commands for testing, displaying statistics, flushing documents, and reindexing embeddings

Type of Change

• 🐛 Bug fix (non-breaking change which fixes an issue)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔧 Configuration change
• 🧪 Test update
• 🛡️ Security update

Areas Affected

• Core Services (Frontend UI/Backend API)
• AI Agents (Researcher/Developer/Executor)
• Security Tools Integration
• Memory System (Vector Store/Knowledge Base)
• Monitoring Stack (Grafana/OpenTelemetry)
• Analytics Platform (Langfuse)
• External Integrations (LLM/Search APIs)
• Documentation
• Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: latest
Docker Version: 28.0.1
Host OS: Linux/macOS
LLM Provider: [all]
Enabled Features: [Embedding Provider]

Test Steps

1. Build the etester utility with go build -o etester cmd/etester/main.go
2. Run ./etester test to verify embedding provider connectivity
3. Test embedding functionality with various providers (Cybertron, Huggingface, Bedrock, VoyageAI)
4. Use ./etester info to check embedding statistics
5. Verify management commands with ./etester flush and ./etester reindex

Test Results

The embedding providers correctly initialize and connect to the respective services. The etester utility successfully performs all operations including testing, statistics gathering, flushing, and reindexing.

Security Considerations

The implementation adds support for external embedding providers which require API keys and credentials. These are handled securely through environment variables and configuration files. No plaintext credentials are stored in the code.

Performance Impact

The new embedding providers offer various performance characteristics depending on the model and service used. Performance testing shows acceptable response times for embedding generation. The etester utility includes verbose output options that can help diagnose performance issues.

Documentation Updates

• README.md updates
• API documentation updates
• Configuration documentation updates
• GraphQL schema updates
• Other: .env.example updates with embedding configuration options

Deployment Notes

To use the new embedding providers, users need to:

1. Set appropriate environment variables for the desired embedding provider
2. Update configuration files to specify the provider and model
3. For Docker deployments, the updated Dockerfile includes the etester utility

New environment variables:

• EMBEDDING_PROVIDER: Type of embedding provider to use
• EMBEDDING_MODEL: Specific model for the selected provider
• EMBEDDING_API_KEY: API key for the embedding provider
• Various provider-specific configuration options

Checklist

Code Quality

• My code follows the project's coding standards
• I have added/updated necessary documentation
• I have added tests to cover my changes
• All new and existing tests pass
• I have run go fmt and go vet (for Go code)
• I have run npm run lint (for TypeScript/JavaScript code)

Security

• I have considered security implications
• Changes maintain or improve the security model
• Sensitive information has been properly handled

Compatibility

• Changes are backward compatible
• Breaking changes are clearly marked and documented
• Dependencies are properly updated

Documentation

• Documentation is clear and complete
• Comments are added for non-obvious code
• API changes are documented

Additional Notes

The etester utility provides a flexible framework for testing and managing embeddings that can be extended with additional functionality in the future. The implementation of multiple embedding providers allows users to choose the most appropriate model for their specific use case and performance requirements.