Add Function Tester (ftester) utility and integrate DuckDuckGo and Perplexity search engines

[asdek]

Description of the Change

Problem

The current PentAGI system lacks tools for direct testing and debugging of individual functions and AI agent behaviors. Additionally, the search capabilities need to be expanded to include more modern providers and the LLM provider selection needs enhancement.

Solution

This PR introduces ftester, a versatile utility for debugging and testing individual PentAGI functions. It also adds support for DuckDuckGo and Perplexity search engines, updates provider configurations, and improves documentation.

Key improvements:

• New ftester utility for targeted testing of system functions and AI agent behaviors
• Added support for DuckDuckGo and Perplexity search engines
• Enhanced API configurations and provider handling
• Improved error handling and logging
• Comprehensive documentation updates

Type of Change

• 🐛 Bug fix (non-breaking change which fixes an issue)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔧 Configuration change
• 🧪 Test update
• 🛡️ Security update

Areas Affected

• Core Services (Frontend UI/Backend API)
• AI Agents (Researcher/Developer/Executor)
• Security Tools Integration
• Memory System (Vector Store/Knowledge Base)
• Monitoring Stack (Grafana/OpenTelemetry)
• Analytics Platform (Langfuse)
• External Integrations (LLM/Search APIs)
• Documentation
• Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: latest
Docker Version: 24.0.5+
Host OS: Ubuntu 22.04/macOS Sonoma/Windows 11
LLM Provider: OpenAI, Anthropic, DeepSeek, OpenRouter
Enabled Features: [Langfuse, Grafana, OpenTelemetry]

Test Steps

1. Install PentAGI with the updated docker-compose.yml
2. Verify new search engines work by creating a flow that uses DuckDuckGo and Perplexity
3. Test the new ftester utility:

   # Test in mock mode because flow id is equal 0
   docker exec -it pentagi /opt/pentagi/bin/ftester terminal -command "ls -la" -cwd "/work" -message "List files"
   
   # Test with a real flow
   docker exec -it pentagi /opt/pentagi/bin/ftester -flow 1 describe

4. Verify Langfuse captures ftester function calls and interactions

Test Results

The tests successfully verified:

• DuckDuckGo and Perplexity search engines work correctly
• ftester utility can operate in both mock and real modes
• All function calls are properly logged in Langfuse and Observability stack
• Updated provider configurations work with various LLM providers

Security Considerations

This PR doesn't modify the security model. The ftester utility adheres to the same container isolation principles as the main application. All external API calls use the same security mechanisms already in place.

Performance Impact

Performance testing shows:

• DuckDuckGo searches are typically ~20% faster than Google searches
• Perplexity searches provide more contextually relevant results
• ftester has minimal overhead when running in production containers
• LLM agent testing shows good performance with the updated provider configurations

Documentation Updates

• README.md updates
• API documentation updates
• Configuration documentation updates
• GraphQL schema updates
• Other: Added comprehensive documentation for ftester utility

Deployment Notes

This PR adds new environment variables that should be added to your .env file:

PERPLEXITY_API_KEY=your_perplexity_key
PERPLEXITY_MODEL=sonar-pro
PERPLEXITY_CONTEXT_SIZE=medium

For users who want to use the new search engines, these variables should be set. The deployment process is the same as before, but with access to new functionality.

Checklist

Code Quality

• My code follows the project's coding standards
• I have added/updated necessary documentation
• I have added tests to cover my changes
• All new and existing tests pass
• I have run go fmt and go vet (for Go code)
• I have run npm run lint (for TypeScript/JavaScript code)

Security

• I have considered security implications
• Changes maintain or improve the security model
• Sensitive information has been properly handled

Compatibility

• Changes are backward compatible
• Breaking changes are clearly marked and documented
• Dependencies are properly updated

Documentation

• Documentation is clear and complete
• Comments are added for non-obvious code
• API changes are documented

Additional Notes

The ftester utility is an extremely powerful tool for development and debugging. It allows developers and security professionals to:

1. Directly test individual components without running the entire system
2. Debug AI agent behaviors in specific flow contexts
3. Verify environment variables and external service connectivity
4. Develop and refine new prompts and agent behaviors with quick feedback

This enables much faster development cycles and more targeted debugging, especially when working with complex AI interactions.

The new search engines also significantly expand PentAGI's research capabilities, providing more diverse sources of information and reducing dependency on any single provider.